A government laboratory compares fiber and copper--and chooses copper for its network.
Dean Collins, John Antonishek, and Alan Mink / NIST
The National Institute of Standards and Technology (nist), part of the U.S. Department of Commerce, is upgrading the cabling of existing buildings and installing cabling in a new laboratory in a building under construction. The goal of the upgrade is to provide nist with as modern a data-communications network as possible within cost constraints. nist is a scientific research laboratory that explores the physical sciences and information technology. Typical calculations include simulations and analysis of physical systems and visualization of their results. Network communication is required to support e-mail, Internet access, cluster and distributed computing, client/server computing, and access to the central supercomputing facility.
The nist campus in Gaithersburg, MD, covers about 578 acres, with 31 buildings serving a staff of more than 3000. The buildings are connected with a fiber-optic Fiber Distributed Data Interface backbone. The cabling within the buildings is mostly Category 3 copper cabling, but some sites use Category 5 and fiber-optic cabling. Most of the horizontal runs are less than 100 meters in length.
The recabling of existing buildings and the cabling of the new laboratory involve tradeoffs between copper and fiber-optic cable and the associated switches. Singlemode fiber costs about 5% more than multimode, but associated network interface cards (nics) and switches can cost significantly more. Although singlemode fiber can support higher bit rates over longer distances than multimode, it is not necessary for the desktop environment. Thus, we eliminated singlemode from our considerations and focused on just multimode fiber. We looked at the various tradeoffs from two different viewpoints:
- the cost of installing and maintaining the cabling
- computer-imposed limitations on the use of the cabling.
Our major cabling options included Category 5, Enhanced Category 5, and Category 6 copper cable, as well as fiber-optic cable. Copper cabling, at the rated speeds, is limited to 100-meter runs. In a typical nist office building, this constraint does not present any undue hardship. With fiber, it`s possible to service more, if not all, of the building from one central telecommunications closet (TC), but in practice this arrangement becomes cumbersome because of the amount of space necessary to support all of the cable and the limited amount of space available in a cable tray.
Network active devices for fiber-optic cabling are about four to six times more expensive per port than for copper cabling. The density of fiber-optic ports is also much lower and requires additional equipment racks and modules. Since the fiber-optic equipment supports a lower port density, more patch-panel and equipment space is necessary--almost four times the equipment space and one-and-a-half times more patch-panel space. Since fiber requires more racks to support the same number of ports, it also requires additional air conditioning. Furthermore, fiber patch cords used in the TCs and offices are five times more expensive than their copper counterparts.
The installation costs for fiber (including pulling, terminating, and testing) are twice that of copper cabling. Since maintenance agreements are a percentage of the initial cost, the same factors for component costs and installation costs are relevant. More of our in-house staff already are trained to pull, terminate, and test copper cabling than fiber-optic cabling, thereby reducing the need for external support to maintain a copper-cabling infrastructure.
Installation and components
The cost difference for cabling one typical nist laboratory--considering only the installation and component cost--is approximately $1.2 million. If this difference is applied across the entire campus, we estimate that the additional cost for using fiber optics would be approximately $14.5 million.
Based on our evaluation, we reached the following conclusions: For recabling the existing buildings we chose Category 5 copper, which provided cost benefits and sufficient capacity--up to 155 megabits per second--to support most network users. This decision was based on our having only a five- to 10-year horizon before all the existing buildings are scheduled to be renovated. At that time, cabling requirements and options will be reevaluated. For the cabling of the new laboratory, we chose Category 6 because it is the most advanced copper solution currently available; while there is a minimum cost differential for the material, there is no additional installation cost. It is only 10% more expensive than Category 5 and 5% more expensive than Enhanced Category 5. Furthermore, it is completely backward-compatible with existing Category 3 and Category 5 installations.
User considerations
From another perspective, the computer user doesn`t care about electrons or photons. The user is interested in performance, which includes effective communication latency and bandwidth; interoperability between protocols; availability of network components; ease of use; and the cost of the computer itself.
Simply put, what measurable benefit will the computer user obtain for the extra cost of fiber optics?
For the user, there is no advantage from either copper or fiber. The transmission medium--whether copper or fiber--is transparent to the user, even to the system device-driver programmer, because the nics hide that level of detail. The device-driver software in the operating system (OS) deals with a logically delimited data packet, frame, or segment. Therefore, there is no difference in ease of use.
Neither fiber nor copper has an impact on communications protocols or their interoperability. All the protocol layers, except the physical layers, are handled by the processor--or the processor side of the nic--while the physical layer is transparent to the processor and handled by the media (fiber or copper) side of the nic (see "How protocols affect communication speed," page 24).
Latency is of interest to the end-user. In distributed computing, latency can be described as how long it takes for a message to travel from one computer to another. But it turns out that network latency--the delay introduced while a packet of data is stored, pro-cessed, and then forwarded--is not a determining factor in select-ing either copper or fiber. Network latency depends on the effective communications speed and the length of the packet being sent.
For example, communication latency for a 200-megahertz Pentium running on a Fast Ethernet local area network (lan) can be broken down as shown in the graph on page 19. The data packet in this example has a minimum length of 4 bytes and a maximum length of 1453 bytes. Depending on the packet size used, the communication latency can range from approximately 84 to 248 microseconds. The protocol stack terms are dependent on processor speed and could also be reduced by advanced protocols such as Operating System Bypass. The nic direct-memory-access transfer terms are only affected by memory access speeds. The only term dependent on bandwidth is network latency, which is a function of network speed and packet size. If a gigabit connection were used, the latency associated with the maximum packet length would decrease from approximately 122 to 12 microseconds. However, the total communication latency would still be dominated by the non-bandwidth-related terms (protocol stacks and direct-memory-access transfer).
Looking at cost and availability of nics and their associated device drivers, we see that Fast Ethernet is a de facto standard and thus inexpensive. A device driver is software that is part of the OS and acts as an interface between a specific device and the OS. On the device side, this software handles all the details of a single device, such as an nic, a disk, or a keyboard. This software is so specific that different models and versions of a device from the same manufacturer may require a different device driver.
On the OS side, this piece of software hides all those device-specific details and presents a uniform interface to the OS for that class of device. An atm/oc-3 155-Mbit/sec nic is available for most major computer input/output (I/O) buses and is moderately priced, but device drivers are not available for all operating systems. The faster atm/oc-12 622-Mbit/sec nic is very sparsely available and so are the necessary device drivers; it is also expensive. The Gigabit Ethernet nic is just starting to become available. These early offerings are expensive, but will undoubtedly come down in cost. Very fast atm connections (above OC-12) are not currently aimed at the desktop, and nics are not available.
From the standpoint of the computer user, the choices depend on the user`s own criteria. If price, performance, and interoperability are the most important factors, Fast Ethernet is the choice. It is low-cost, uses copper, has good performance, high availability, and is the de facto standard, thus providing interoperability.
If local performance is the only important factor, Gigabit Ethernet and atm/oc-12 are good choices. They are fast networks, use fiber (or copper for Gigabit Ethernet), are moderate to high in cost, and are selectively available. However, achieving high performance requires nonstandard protocols.
If global interoperability and performance are the most important factors, use either Fast Ethernet or atm/oc-3. These both have good performance, use either copper or fiber, have moderate to low costs, high to reasonable availability, support standard Internet protocols for interoperability, and have long-distance capability.
For fiber-optic technology to become attractive for the particular scenario we have looked at, some of the following must occur:
Components and nics must become more readily available and cost less.
Device drivers must be readily available and supported so devices can easily be interfaced to different computers and operating systems.
Fiber must have a smaller footprint to allow high densities on patch panels and network devices.
Fiber must become easier and less expensive to terminate.
Users` bandwidth needs must require fiber.
Standard protocols faster than tcp/ip must be developed and commonly used.
At this time, for our nist application, copper is still the clear winner. On cabling considerations alone, copper provides nist definite cost benefits while supporting sufficient bandwidth in our five- to 10-year horizon. nist computer users cannot use the longer run length and higher bandwidth of fiber, thus nist users would see no performance difference between copper and fiber. Fiber-optic technology is changing rapidly, however, as is copper technology.
This article is based on a presentation made at Kessler Marketing Intelligence Corp.`s 21st Annual Newport Conference on Fiberoptics Markets (Newport, RI) in October 1998.
Latency breakdown for a 200-MHz Pentium computer on a Fast Ethernet local area network shows that the medium of the network--copper or fiber--has little effect on communication latency. This latency is most affected by the communication-protocol software and electronics as well as the length of the data packet.
How protocols affect communication speed
Currently all communication processing is done electronically in parallel (8, 16, or 32 bits) for two reasons:
The data is meaningful only in groups and, by dealing with a parallel group, you have more time to process the data than at the bit rate.
Optical processing of protocols is not yet feasible, but research is progressing in this area.
Interoperability of protocols is handled by the protocol stack, which allows only well-known protocols to be used together. For example, routers convert between a number of well-known protocols such as Ethernet and Fiber Distributed Data Interface. Nonstandard protocols, including many applications programming interfaces (apis), cannot interoperate. Well-known standard protocols also provide stability and reliability, mostly through massive exposure and fine-tuning over an extended period. Performance, on the other hand, is achieved via nonstandard apis or Operating System Bypass protocols, although an api called Virtual Interface Architecture is being proposed by an industrial group that includes Intel.
How does all this affect communication speeds? The graph compares memory access speeds (cpu-to-memory) with network speeds (network I/O-to-memory). The Y axis shows common network speeds. The X axis shows memory access speeds. Two representative points on the X axis are personal computers based on a 200- and 400-megahertz Pentium microprocessor. The memory access (read and write) speed is derived by doubling the measured memory copy (read-followed-by-write) speed.
Network communication requires user data to be encapsulated and transferred from memory (usually specified by its virtual address) to a nic plugged into the I/O bus. The actual transfer is accomplished via direct memory access by the nic. The nic "packages" the data and transmits it onto the communications medium, possibly traversing through a number of switches, which may buffer and process some or all of the message before passing it along. When it reaches the destination nic, the process is reversed, placing the data into memory (a user-specified buffer).
The dashed diagonal line in the figure represents the upper limit of communication speed, equal to memory access speed. This upper limit may be a desirable goal, but not practically achievable.
All current microprocessors running the tcp/ip protocol can use the maximum bandwidth of the current common commodity networks such as Fast Ethernet and atm/oc-3. Current processors can execute the instructions for the common tcp/ip protocols faster (between 150 and 600 megabits per second) than the bandwidth of these networks can transmit them. Our performance measurements have shown that although atm/oc-3 has a 50% higher bandwidth than Fast Ethernet, using tcp/ip over atm/oc-3 only obtains about 30% higher communications throughput than tcp/ip over Fast Ethernet. Furthermore, on a 200-MHz computer, tcp/ip would limit communications throughput to less than 200 Mbits/sec. Thus, providing a fiber-based gigabit-per-second network interface for such a machine would result in only marginal communications throughput compared to an OC-3 155-Mbit/sec link.
But to achieve gigabit-per-second communication requires OS Bypass protocols. These significantly reduce the intervention of the processor to execute the communication protocol by moving most of the previous software instructions to hardware. This technique is limited by the speed of the I/O bus. In many computers today, that is the pci (peripheral component interconnect) bus. To achieve OS Bypass protocols requires the communication be done directly to the user buffers, bypassing the OS. This approach requires the user buffers be locked in physical memory and that the nic be programmable to maintain the list of user buffer physical addresses and be able to multiplex different message streams from different sources. To achieve even higher-speed communication requires faster networks and switches connected to faster I/O buses, or even direct attachment to the memory bus, in addition to bypassing the OS.
Dean R. Collins, is division chief; John K. Antonishek is group leader, Network and Telecommunications Systems; and Alan Mink is project leader, Scalable Parallel Systems and Applications Group, of the High Performance Systems & Services Div. of the National Institute of Standards and Technology (Gaithersburg, MD). Collins can be contacted at [email protected].