A January 2020 white paper by Arista Networks (NYSE: ANET) answers a host of technical questions regarding emerging trends in 400G transceivers and cabling for data centers.
Highlights of the document include the following:
What are the benefits of moving to 400G technology?
"400G platforms allow data centers and high-performance computing environments to address growing needs for higher bandwidth at lower cost and power per gigabit. Key benefits include:
• Increase switching bandwidth by a factor of 4. Migrating from 100G to 400G systems increases the bandwidth per RU from 3.2-3.6T to 12.8-14.4T / RU.
• Enable higher density 100G ports using optical or copper breakouts. A 32 port 1RU 400G system enables 128 100GE ports / RU. This enables a single Top of Rack (TOR) leaf switch to connect to multiple racks of servers or Network Interface Cards (NICs).
• Reduce the number of optical fiber links, connectors and patch panels by a factor of 4 when compared to 100G platforms for the same aggregate bandwidth.
• Enable 2-4X lower cost and power / bit, reducing capex and opex."
What form-factors are used for 400G optics and cables?
"There are two form-factors defined for 400G optical modules and cables:
• The OSFP: The OSFP stands for “Octal Small Form-factor Pluggable”. It is described as an “Octal” module because the electrical interface of an OSFP connector consists of 8 electrical lanes, running at 50Gb/s each, for a total of bandwidth of 400Gb/s.
• The QSFP-DD: The QSFP-DD stands for “Quad Small Form-factor Pluggable (QSFP) – Double Density (DD)”. The electrical interface of a QSFP-DD connector also has 8 electrical lanes, running at 50Gb/s each, for a total bandwidth of 400Gb/s. The QSFP-DD form factor is similar to the QSFP form factor, except a second row of electrical contacts has been added to increase the number of high-speed electrical lanes from 4 (in a QSFP) to 8 (in a QSFP-DD)."
Can I plug an OSFP module into a QSFP-DD port, or a QSFP-DD module into an OSFP port?
"No. The OSFP and the QSFP-DD are two physically distinct form factors. If you have an OSFP system, then OSFP optics and cables must be used. If you have a QSFP-DD system, then QSFP-DD optics and cables must be used."
Can I have an OSFP on one end of a 400G link, and a QSFP-DD on the other end?
"Yes. The OSFP and QSFP-DD describe the physical form factor of the module. As long as the Ethernet media types are the same (i.e. both ends of the link are 400G-DR4, or 400G-FR4 etc.), OSFP and QSFP-DD modules will interop with each other."
What is the difference between a ‘400G’ and ‘200G’ breakout DAC?
"Arista offers copper breakout cables in 400G and 200G versions. Both versions include breakouts from an OSFP / QSFP-DD port to 2x QSFP, 4x QSFP, and 8x SFP ports. The key differences between the 400G and 200G versions are:
1. The 400G cables are designed to carry 8x 50Gb/s PAM-4 electrical lanes, while the 200G cables are designed to carry 8x 25Gb/s NRZ electrical lanes (see a later section of this document for a detailed explanation of PAM-4 and NRZ signaling). The high-speed 50G PAM-4 signals place more stringent signal integrity requirements on the DACs, and the 400G cables are specified to support these signals. The 400G cables will also support lower speed 25Gb/s NRZ signals, but there may be interop issues when using 400G cables in legacy systems designed for 25Gb/s NRZ cables (see explanation below).
2. Each connector on the DAC cable (i.e. the OSFP / QSFP-DD and QSFP / SFP connectors) contains an EEPROM which is programmed to indicate the signal format and max data rate supported by the cable. The EEPROMs in the 400G cables indicate support for up to 50G PAM-4 data rates, while the EEPROMs in the 200G cables indicate support up to only 25G NRZ data rates.
Breakout DACs are often used to connect Arista switches to other non-Arista equipment such as server NICs (Network Interface Cards). The NIC software will usually read the DAC connector’s EEPROM to determine what type of device has been plugged in.
The software installed on many legacy NICs may not recognize the newer signal speeds and modulation formats, and hence may not successfully recognize cables programmed for 50Gb/s PAM-4 signals. The 200G cables can be used when connecting to legacy equipment that will only recognize DACs programmed for 25Gb/s data rates."
Read the full white paper (PDF).
