Optical connectivity a good choice in the data center

by Doug Coleman

Optical connectivity can maximize a data center’s overall operations efficiency. Optical cable with laser-optimized 50-µm OM3 fiber provides bandwidth capabilities that support up to 10-Gbit/sec transmission for existing applications as well as future-data-rate applications of 16 Gbits/sec to 100 Gbits/sec and beyond. The transmission performance, data-rate scalability, pathway-and-space utilization, electronics port density, power and cooling efficiencies, and ease of installation and testing are factors that make optical connectivity a serious contender to 10GBase-T and copper connectivity in the data center.

IEEE 802.3ae released the optical 10-Gbit Ethernet standard in 2002, with the 10GBase-SR physical media dependent (PMD) for short-range links up to 300 meters emerging as the dominant and best-suited 10-Gbit connectivity solution in the data center using OM3 fiber. The 10GBase-T standard, approved in 2006, provides guidance for 10-Gbit transmission on 4-pair twisted-pair copper cable within a 500-MHz bandwidth. But the outlook for transmission faster than 10 Gbits/sec on copper links is doubtful due to the required distances in the data center as well as the LAN.

10-Gig on twisted pair

Category 6 UTP copper cable has been widely deployed in data centers, but it has limitations associated with supporting 10-Gbit/sec transmission. The cable reach is rated in the range of 37 to 55 meters, but many industry experts believe that anything longer than 37 meters may be subject to alien crosstalk because of theextended operating frequency.

Category 6 performance issues prompted development of Category 6A, which is intended to support 10-Gbit/sec operation up to 100meters. Category 6A cable, however, typically increases the outside diameter of previous-generation Category 6 copper cable by about 40% in an attempt to mitigate internal and external noise impairments. Such a cable design can be more challenging to route through network pathways and requires more complex testing and field-termination methods than a single, slender fiber-optic cable.

When copper connectivity is used, most 10-Gig copper systems will likely require removal of legacy Category 6 UTP cable and installation of Category 6A . To date, the Category 6A detailed cable standards from the Telecommunications Industry Association (TIA; www.tiaonline.org) and the International Electrotechnical Commission (IEC; www.iec.ch) have not yet been completed. So, all Category 6A networks installed before the standards are completed are effectively proprietary solutions withpotential interoperability issues. Minimal 10-Gig interoperability has been demonstrated among different Category 6A and hardware manufacturers.

Shielded Category 6 and 7 copper cables are also being considered in place of UTP copper cables for 10-Gig operation. UTP cables have been the de facto copper cables used in North America. In comparison, shieldedcopper cables have rarely been deployed due to special installation and termination practices. Thecable’s stiffness, weight, and large bend radii, as well as grounding and bonding issues, make it complex to install. In addition, some in the industry believe they present Power over Ethernet (PoE) concerns related to heat dissipation.

The extended operating frequency range of 10-Gbit copper cable requires higher power consumption (10 to 15 watts) of the 10-Gbit interfaces due to increased insertion loss and the need to overcome internal and external crosstalk issues. Electronic digital signal processing (DSP) techniques are required to correct internal noise impairments, which contribute significantly to the inherent time delay while recovering the transmitted data packets-otherwise known as latency.

Data centers demand low-latency physical (PHY) performance. One industry expert estimated that a one-millisecond advantage in trading applications can be worth $100 million a year to a major brokerage firm. 10-Gig optical PHY has 1,000 times better latency performance than 10-Gig copper. 10-Gig optical has typical PHY latency in the nanosecond range, whereas 10-Gig copper has PHY latency measurable in microseconds. DSP algorithms cannot be used to mitigate alien crosstalk or other external sources of electromagnetic interference due to their random statistical nature. The UTP cable relies entirely on the (larger) cable design and physical configuration of the cable in pathways and spaces to address the external alien crosstalk noise ingress.

Pathway and space use

The larger diameter of Category 6A cable can also significantly compromise efficient use of pathways and spaces, such as wire baskets, trays, conduits, and racks. Two Category 6A UTP cables consume the same area equivalent of a single 216-fiber ribbon cable. The high fiber density, combined with the small diameter of optical cable, maximizes the raised-floor pathway and space efficiencies for routing and cooling, and offers superior pathway use when routed in aerial cable trays.

The 216-fiber ribbon cable (top) can accommodate 108 duplex connections, while the bundle of Category 6A cables can accommodate 48 such connections.

Because of the larger diameter and abundance of copper cable, a greater amount of jacketing and insulating materials can cause additional fuel load in the pathways and increase the volume of hazardous waste going into landfills from construction debris. Copper cables’ congestion in pathways also increases the potential for damage to electronics through overheating by restricting the much-needed airflow to regulate temperature. In addition, air-cooling/damming effects can interfere with ventilation systems’ ability to remove dust and dirty.

A typical plenum Category 6A cable weighs 46 pounds per 1,000 feet. The accumulated weight of Category 6A cabling alone to serve a 108-circuit, 200-foot length 10-Gbit/sec installation will be approximately 1,000 pounds-compared with 40 pounds for the same length of a 216-fiber optical cable. The 5x greater weight of Category 6A cables will require additional hardware costs to support the load, and may contribute to cable strain-relief issues in hardware, as well as compression issues in trays and conduits.

Concerning stress, Category 6A installers must ensure they do not exceed the 25-pound tensile rating and avoid untwisting of wires, while a single-fiber 2.0-mm jumper boasts a 50-pound tensile rating and a multi-fiber cable can have a tensile strength of 600 pounds.

In addition, care must be taken not to violate the copper cable’s 4x outside-diameter bend radius. Violating the bend radius distorts the cable’s physical geometric properties, which degrades transmission performance. Optical cables have an equivalent or better bend performance because of their small diameter, which easily facilitates installation without compromising transmission performance.

Installation and field testing

A labor premium over traditional 1000Base-T systems is anticipated for the installation and testing of 10GBase-T systems. 10-Gig copper can be much more complex to field-test than previous-generation copper systems, and requires sophisticated test equipment. Testing is performed on each cable across the 1- to 500-MHz frequency spectrum to determine conformance for insertion loss, return loss, pair-to-pair near-end crosstalk (NEXT), power-sum NEXT, pair-to-pair equal-level far-end crosstalk (ELFEXT), power-sum ELFEXT, propagation delay, length, delay skew, and wire map. Complex and often time-consuming alien crosstalk measurements must also be made.

Some Category 6A UTP cable manufacturers, and others, are suggesting random alien crosstalk field testing to demonstrate total cable conformance. Considering the financial impact of data center downtime, contractors, network designers, and end users are not expected to accept random testing. Industry experts have stated as much as 3.75 hours are required to perform 100% total testing on a bundle of 24 copper cables. Conversely, 10-Gbit/sec field testing for optical cables simply requires the standard end-to-end link-loss measurement.

It’s expected that 10-Gbit/sec copper port density will be four to eight ports per card due to power requirements of 10 to 15 watts per port, as well as being limited by heat dissipation and crosstalk issues. In high-density applications, this power usage is significant, not only in consumption, but also in the generation of heat, which requires cooling to protect transmission equipment from rising temperatures.

The American Society of Heating, Refrigeration, and Air Conditioning Engineers (ASHRAE; www.ashrae.org) guidelines state that for every kilowatt of power, an equal thermal unit of cooling is required. More power consumption requires greater cooling and humidity control, which increases operational costs of uninterruptible power supply (UPS) systems and backup power supplies.

The 216-fiber ribbon cable (top) can accommodate 108 duplex connections, while the bundle of Category 6A cables can accommodate 48 such connections.

Industry experts have stated that every one-cent increase in energy cost per kilowatt-hour can amount to $3 million per year in energy and cooling costs in a large data center. Also, for every dollar spent on powering servers and equipment in a data center, firms should expect to spend another dollar to as much as $2 to $3 to cool the hardware. Major Tier 1 switch manufacturers are not expected to offer 10GBase-T commercial products until mid-2008, as silicon-chip manufacturers continue to address issues with power consumption.

Considering the fiber advantages

Accordingly, 10-Gbit/sec optical components provide clear advantages over copper twisted pair:

• 10-Gbit X2 transceivers support up to 16 ports per line card. Maximum power dissipation is 4 watts per port;
• 10-Gbit XFP optical transceivers support up to 24 to 36 ports per line card. Maximum power dissipation is 2.5 watts per port;
• Emerging 10-Gbit SFP+ optical transceivers will support up to 48 ports per line card. Maximum power dissipation will be 1 watt per port. The SFP+ transceiver will offer significantly lower cost compared to the X2 and XFP transceivers;
• A 10-Gbit copper system will require more switches and line cards to match the bandwidth capability of 10-Gbit optical.

10GBase-SR server adapter cards typically use less than 9 watts of power to transmit up to 300 meters, while recently announced 10GBase-T cards use 25 watts to reach up to 30 meters. The optical adapter is easily powered from the server PCI-Express slot without an external power feed.

Ethernet adapters were announced in late January to support 10GBase-T. They consume just less than 25 watts to support a maximum transmission distance of 30 meters. Greater service distances would require an additional electrical feed to power the copper adapter card, since the PCI-Express slot can only provide up to 25 watts. As with the 10-Gbit copper switches, the 10-Gbit copper server adapter card’s high power-consumption and cooling needs result in higher operational costs. To date, copper server and card adapter server manufacturers have demonstrated minimal interoperability with 10GBase-T switch vendors. Switch-vendor support is at least a year away due to power challenges.

In addition to the characteristics already detailed, optical systems can claim five attributes-extended distance capability, power and cooling needs, port density, pathway and space use, and a migration path to higher speeds-that put it at an advantage over twisted-pair copper alternatives.

Extended distance capability. Laser-optimized 50-µm multimode OM3 fiber provides 10-Gbit serial transmission to a maximum distance of 300 to 550 meters. Fiber can offer network designers more flexibility in their planning, and the advantage of being able to use new and reconfigurable architectures in the data center. OM3 50-µm fiber supports data rates beyond 10-Gbit and is to be included in the IEEE 100-Gbit/sec Project Authorization Request (PAR) to operate at a minimum 100-meter distance. In the best-case scenario, Category 6A, Category 6F/UTP, and Category 7 copper are limited to 100 meters and are restricted to four connections in the 10GBase-T channel.

Power and cooling. 10-Gig optical switch electronics and server adapter cards require less power to operate compared to 10-Gig copper. 10-Gbit SFP+ optical transceivers will consume a maximum of 1.0 watt per port, compared to 10 to 15 watts per port for a 10GBase-T copper switch. 10GBase-SR server adapter cards typically use less than 9 watts to transmit up to 300 meters, while recently announced 10GBase-T cards use 25 watts to transmit up to 30 meters. Significant silicon-chip development will likely be required to reduce the power consumption of 10-Gbit copper interfaces. Although industry expectation is that silicon-chip power consumption may be reduced, the high insertion loss of copper cables at the extended frequency range needed to support 10-Gbit Ethernet and electronic DSP noise-reduction circuitry means that energy consumption will inevitably be higher than that of low-loss fiber interconnects. The silicon-chip size-reduction work is not expected to be completed in the near future as major performance issues, such as electrostatic discharge (which impacts reliability), must be resolved. The power consumption of10-Gbit copper electronics and adapter cards will drive the need for excessive cooling, meaning greater energy use and running costs for data centers and possibly larger uninterruptible power supplies. Industry experts have stated that 10GBase-T over Category 6 or 7 twisted-pair cables can extend up to 100 meters, but power requirements hinder its cost-effectiveness.

Port density. Fiber provides a higher 10-Gbit port density per electronic line card and patch panel when compared to copper. Many manufacturers’ 10-Gbit UTP copper patch-panel densities are reduced by up to 50% to mitigate crosstalk impairments by increased spacing between connector ports. Fiber can accommodate higher port densities, up to 1,728 fibers, in a 4U housing.

Pathway and space congestion. High fiber density, combined with the smaller diameter of optical cable, maximizes the raised-floor pathway and space use for routing and cooling. Optical cables also offer superior pathway usage when routed in aerial cable trays. The larger Category 6A outer diameter impacts conduit size and fill rate, as well as cable management due to the increased bend radius. Common commercial cable-pulling lubricants have been shown to affect copper- cable attenuation at the high operating frequencies of 10-Gbit Ethernet. Bundled copper cable also inhibits removal of abandoned cable and presents serious alien crosstalk issues in raised-floor and aerial pathways.

Migration to higher speeds. Laser-optimized 50-µm multimode OM3 fiber provides a migration path for supporting higher data rates such as 16-Gbit and 32-Gbit Fibre Channel, and 100-Gbit Ethernet. Copper twisted-pair cable cannot support these speeds at market-required distances. Industry experts have stated that instead of a wholesale transition to 10-Gbit Ethernet over copper, the networking industry may well segue into optical fiber, as the next speed is 100-Gbit Ethernet and copper will not work over required distances at that data rate.

Planning ahead

Should you ever require cabling infrastructure to support 10-Gbit speeds in the future, now is the time to consider optical connectivity in the data center. Optical cable with laser-optimized 50-µm OM3 fiber provides bandwidth capabilities that support legacy and future-data-rate applications without the costs and downtime of retiring a copper cable plant and recabling. Optical-fiber connectivity can provide optimized transmission performance, extended distance capability, best use of pathways and spaces, ease of installation and testing, reduction in power consumption, cooling and electronic costs, and the highest 10-Gbit electronic and patch-panel densities.

Twisted-pair copper cabling has been struggling to keep up with bandwidth demands virtually since it entered the data center market. There have been six generations of copper cable in the past 20 years, each one designed to address the next wave of bandwidth demand and each one becoming more complex to design around and install.

DOUG COLEMAN is manager of technology and standards with Corning Cable Systems (www.corningcablesystems.com).