On the road to 100-Gbits/sec transmission

Development of the next generation of Ethernet is well underway.

by Andrew Oliviero

As anyone who is carefully watching can tell, the data communications industry is moving to 100-Gbits/sec transmission speeds. The questions become:

• What are the applications and key network points driving the need for 100-Gbits/sec in public networks and private enterprises?
• Who are the most likely early adopters of this next-generation technology?

This article presents answers to these questions, and explains why temporary solutions (such as link aggregation) are not ideal to fully address these overloaded networks.

Among transceiver options being explored is OM3 fiber using low-cost 850-nm parallel optics arrays. As a full-duplex link, with 12 fibers running each direction, this solution would use a total of 24 fibers for the complete link.

We’ll also discuss the current status of the standards process, and what still needs to happen before a standard is written. The final section addresses the transceiver technologies and options being considered to meet 100-Gbits/sec speeds for OM3 multimode and singlemode fiber, and provides some assumptions on likely cost differences between the two.

Drivers for 100-Gbit transmission

As high-speed broadband services offered by fiber-to-the-x (FTTx)-focused telecommunications carriers and cable television companies are becoming more available, consumers are taking advantage of the many novel applications offered to them. Content providers are pushing the bandwidth requirements by developing more new applications and services, so that video-on-demand, HDTV, IPTV, Internet gaming, MySpace, YouTube, and digital-photo transfers, which could only be envisioned in the past, are now a reality.

In short, we are seeing a push by content/service providers and a pull by the consumer. The lesson: Increase the size of the access pipelines and demand will come.

These events have led to continuous and rapid growth of the network and Internet traffic, which has placed an incredibly high demand on the existing infrastructure. Network carriers, service providers, and Internet exchanges are feeling this load on their networks and are seeking higher-speed solutions in a hurry.

In the private sector, there is also a drive for higher network speeds for LANs and storage area networks (SANs). This demand comes from high-bandwidth applications, such as video-based streaming and downloading, videoconferencing, and Voice over IP.

Data-center servers, too, will continue to experience a rise in traffic and bandwidth demand, as more information is being generated and stored today than ever before. With recent government data warehousing legislation and recommendations for the medical and financial industries, along with redundancy to protect against catastrophic loss, data centers and SANs are expected to see further upgrading to higher networking speeds. In fact, storage standards, such as Fibre Channel and Infiniband, have already developed roadmaps for speeds up to 100 Gbits/sec and beyond.

Another key driver for higher networking speeds is the high-performance computing (HPC) market. Supercomputers and HPC networks now under development will require a minimum of 100-Gbits/sec transmission speeds for short links ranging from only a few inches to hundreds of meters. In some cases, these will be used to link major supercomputer clusters between research-and-development departments of universities and medical facilities.

Link aggregation (LAG), an IEEE 802.3ad standard, is being deployed to address this increased demand with current 10-Gbits/sec server and networking equipment; however, many believe that LAG is just a temporary fix. It can be complex to use, making traffic engineering and management much more challenging. What’s more, capacity expansions and troubleshooting of multiple physical links become much more difficult.

LAG’s limitations create inefficient distribution of large flows and, ultimately, uneven distribution of traffic. All in all, many within IEEE feel that better solutions are required to address this demand directly.

Where will we see 100-Gig?

Before discussing the standards under development with an eye toward 100 Gbits/sec, let’s review more closely the early adopters and key network points that will use these next-generation speeds.

Not surprisingly, the early adopters will be carrier networks (e.g., Verizon, AT&T), triple-play service providers (e.g., network carriers and cable TV companies), Internet exchange carriers (e.g., Yahoo!) and specific enterprise users with extremely high throughput speeds.

Early deployment of next-generation high speeds will occur in key high-bandwidth switching, routing, and aggregation interconnect points for:

• Service-provider backbones supporting the metro, core, and access parts of their networks;
• Internet exchanges;
• Interconnection links in data center and storage servers of corporate enterprise networks; and
• Interconnects for high-performance supercomputing networks in medical and R&D enterprises.

Deployment within LAN riser backbones (interconnecting LAN workgroup switches to core switches or campus LAN backbones) is not expected for quite some time. Most importantly, these next-generation speeds are not intended for interconnecting desktop computers to LAN workgroup switches, which have historically been the main driver for network equipment and switch port demand.

As a result, unlike the high volumes of 10/100/ 1000-Mbits/sec Ethernet port sales over the years, initial volumes for 100-Gbits/sec Ethernet ports are anticipated to be more modest. But this does not imply a reduction in the need or value of 100-Gbit Ethernet to address the applications previously discussed, because 100-Gbits/sec transmission provides a solution for applications that have been demonstrated to need bandwidth beyond existing capabilities.

High Speed Study Group takes action

IEEE 802.3 formed the High Speed Study Group (HSSG) in late 2006 to investigate the need for a next Ethernet speed, and to offer objectives as part of a project authorization request (PAR) should it decide to recommend the creation of a task force to write a standard. The HSSG is an internationally represented group of component, switch, and cabling manufacturers, as well as end users representing private and public networks. Two ad-hoc committees, the Fiber Optic Ad Hoc and Reach Ad Hoc, support the group’s efforts.

In their evaluation of next Ethernet speed proposals, the HSSG followed the five-criteria validation process established by the IEEE:

1. Broad market potential;
2. Compatibility;
3. Distinct identity;
4. Technical feasibility;
5. Economic feasibility.

A considerable number of presentations have been made within the HSSG and the ad-hoc committees to validate the five criteria. During the November 2006 IEEE 802.3 plenary, the HSSG voted to support 100 Gbits/sec as the next Ethernet speed.

The following specific objectives have been accepted since that meeting:

• Support full-duplex operation only;
• Preserve the 802.3/Ethernet frame format at the MAC client service interface;
• Preserve minimum and maximum frame size of current 802.3 standard;
• Support a speed of 100 Gbits/sec at the MAC/PLS service interface;
• Support at least 10 kilometers on singlemode fiber (metropolitan and enterprise networks);
• Support at least 40 kilometers on singlemode fiber (long haul);
• Support at least 100 meters on OM3 multimode fiber;
• Support at least 10 meters on copper;
• Support a bit error rate better than or equal to 10 to 12 at the MAC/PLS service interface.

The HSSG’s next step is to finalize support, document, and submit the PAR to IEEE to initiate writing the standard. When accepted by the IEEE 802.3 committee, the HSSG will be concluded and all efforts will move to specifying the technical details of exactly how to meet the objectives.

The 40-Gbits/sec debate

But this has not occurred yet. During the process, there have been many proponents of a 40-Gbit/sec speed requirement to be included in the PAR (in addition to the 100-Gbits/sec objective) to support the server and data center/SAN markets. There have been many debates over the last year as to the economic feasibility and broad market potential for this intermediate speed, and whether this would slow down the development of the much-needed 100-Gbits/sec standard.

The High Speed Study Group and Fiber Ad Hoc committee are evaluating support of singlemode fiber using CWDM optics in a two-fiber duplex link, where multiple wavelengths would operate over a single fiber in each direction.

Strong cases, however, have been made in support of 40-Gbits/sec and the HSSG is now working on a method of satisfying both the 40- and 100-Gbits/sec advocates in a way that does not hinder progress toward a final PAR. (See “IEEE Ethernet High Speed closes in on initial approval,” page 17.)

The group’s next step is to submit the PAR and obtain approval. After the PAR is accepted, the IEEE will begin writing the next-generation Ethernet standard. The current target is to initiate work this year and publish it in 2010.

Based on the fiber-cabling objectives agreed upon in the HSSG, transceivers will be developed to support singlemode fiber and OM3 multimode fiber (also known as 850-nm laser-optimized 50-µm multimode fiber). Since standard 62.5-µm (OM1) and 50-µm fiber (OM2) will not be supported at 100-Gbits/sec, OM1 and OM2 are no longer recommended for new data center and storage area installations, or HPC environments, where futureproofing to higher speeds is important.

Transceiver and optical-fiber options

The Fiber Optic Ad Hoc committee is also evaluating the transceiver options. It is proposing the use of existing transceiver technologies, such as parallel optical interfaces (sometimes referred to as space-division multiplexing) and coarse wavelength division multiplexing (CWDM), using transceivers with speeds of 10 to 50 Gbits/sec. The soon-to-be-published TIA TSB-172 serves as an excellent tutorial on the details of these transmission technologies.

For OM3 multimode fiber, the HSSG and Fiber Ad Hoc are evaluating the use of low-cost 850-nm parallel optics transceiver arrays, or a combination of parallel optic arrays and CWDM. The former is the leading candidate. With this approach, twelve 10-Gbit/sec 850-nm optical transmitters and receivers are packaged in an array and attached to OM3 fibers using 12-fiber MPO array connectors. The data is divided equally among the available channels.

For example, 12 OM3 fibers, each operating at 10 Gbits/sec at 850 nm, can be aggregated into a 100-Gbits/sec system (12 fiber x 10 Gbits/sec parallel array). The type of encoding being proposed would limit the channel to 100 Gbits/sec instead of 120 Gbits/sec. Because this is a full-duplex link with 12 fibers running in each direction, a total of 24 fibers would be used for a complete link.

This strategy can also be used to support 40-Gbits/sec speeds over OM3 fiber. In this case, four or six OM3 fibers, each operating at 10 Gbits/sec at 850 nm, can be aggregated to 40 Gbits/sec. A total of 12 fibers would be used in this link, as opposed to 24 fibers in a 100-Gbits/sec link. In general, the parallel solution is relatively simple and low-cost, since it uses the same circuits multiple times.

To reduce the cost of the electronics and for the OM3 option, transceiver manufacturers are proposing to loosen the encircled flux and/or spectral width specifications of existing 10GBase-SR transceivers. As a result, the transmittable distance over OM3 fiber would be reduced from 300 meters to as low as 100 meters, depending on the degree of change, despite OM3 fiber’s very high bandwidth. In this case, OM3 fiber’s bandwidth is not the limitation; instead, the desire to reduce the cost of these 12 transceiver arrays is becoming the driver.

Balancing act

Because these future speeds are intended for data center environments, however, 100 to 150 meters should be sufficient. During the standards-development efforts, transceiver and fiber manufacturers will establish the proper balance of specifications to minimize cost and maximize transmittable distance.

The HSSG and Fiber Ad Hoc are evaluating the support of singlemode fiber using CWDM optics in a two-fiber duplex link. In this case, multiple wavelengths would be operating over a single fiber in each direction. An example of this technique is the 10GBase-LX4 transceiver. For 100-Gbits/sec systems, the following are being considered in a 20-nm spacing range around 1310 nm:

• 10 wavelengths x 10 Gbits/sec;
• 5 wavelengths x 20 Gbits/sec;
• 4 wavelengths x 25 Gbits/sec; and
• 2 wavelengths x 50 Gbits/sec.

At this point, the 4 x 25-Gbits/sec transceiver is a leading candidate. Installing low- or zero-water-peak singlemode fiber (ITU G.652D-compliant) provides the most flexibility to deploy any of the proposed singlemode fiber solutions.

Why not use singlemode fiber with a single laser (serial transmission) operating at 100-Gbits/sec? Such a laser simply is not commercially available today, and probably will not be for a long time. It will be challenging to develop and produce such a laser cost-effectively. Therefore, despite singlemode fiber’s exceptionally high bandwidth, achieving higher speeds on singlemode will require optics using multiple lasers to drive multiple wavelengths.

Several presentations have been made in the HSSG estimating the cost differences between future multimode and singlemode 100-Gbits/sec systems.

Cost factors considered

The advantage for OM3 mutimode fiber systems involves the readily available, even lower-cost 850-nm vertical-cavity surface-emitting laser (VCSEL) transceiver. 850-nm transceivers have continued to favor multimode systems for 1- and 10-Gbits/sec systems. The existing manufacturing platform and market volumes for 10GBase-SR ports provide economically favorable conditions for the development of 12-VCSEL arrays.

But because multiple OM3 fibers must be used in the parallel technique, these systems will be more sensitive to the length of the cabling in the channel than CWDM transmission over singlemode. That means the relative cost benefit of parallel systems has diminishing benefits as the channel length increases.

The singlemode CWDM systems take advantage of low-cost singlemode cable, but at the expense of higher complexity in the transmitter and receiver than with the parallel optical technique. In other words, the same transceiver- and connector-alignment challenges that can drive up the cost of 1310-nm components when used with singlemode fiber are magnified even further as the number of wavelengths is increased. Plus, these transceivers are not available, and extra R&D will be required to bring these to market.

Since optical port costs typically make up the largest percentage of total system cost, the cost advantages held by 850-nm-based systems are projected to hold true at these higher speeds. In general, OM3 multimode fiber will continue to be the most cost-effective choice for short-reach applications at higher speeds. Zero-water-peak singlemode fiber is best used for long distances.

Next generation on the horizon

There is very strong industry support for 100-Gbits/sec and possibly 40-Gbits/sec transmission speeds in public and private networks to support triple-play services, significant amounts of video-based applications, data-center storage increases, and high-performance computing. The IEEE group is addressing these needs and will soon commence writing the next-generation Ethernet standard.

OM3 multimode fiber is poised to support short-reach solutions cost-effectively, whereas singlemode fibers will continue their place in outside plant, long-reach solutions.

ANDREW OLIVIERO is senior product manager with OFS (www.ofsoptics.com).