Top-of-rack architecture remains popular in data centers

By Patrick McLaughlin

In data center architectures, switch-to-server connections traditionally are the lowest-speed connections. But that fact should not be interpreted to mean they are the least important. From the standpoint of speed, switch-to-switch connections serve as the backhaul or backbone, and as such typically are multiple times the speed of switch-to-server connections. Yet those switch-to-server connections are commonly referred to as “edge” connections, and are vital because servers are, for practical purposes, clients’ interface with the data center.

The speeds and architectures of switch-to-server connections are several, and common setups include top-of-rack, end-of-row, and middle-of-row. Siemon’s Valerie Maguire and Betsy Conroy recently authored a paper titled “The copper vs. fiber war is over and everybody won. Where do we go from here?” The paper covers a number of topics on cabling media and applications. Among the topics are data center environments and architectures. The authors state, “The ability of twisted-pair copper cabling to support speeds of 10 Gbits/sec makes it the preferred choice for today’s data center switch-to-server connections. With cabling channel lengths supported up to 100 meters and transceiver costs well below that of optical fiber, Category 6A and higher copper cabling is currently well suited to support a variety of architectures for switch-to-server connections, including top of rack, middle of row, and end of row scenarios.”

The paper explains that in today’s most-demanding environments, switch-to-server connections are pushing beyond 10-Gbit/sec speeds; it details the 25-Gbit/sec and 40-Gbit/sec Base-T specification efforts taking place within the IEEE and refers to Category 8 as the twisted-pair medium ideally suited to support these speeds.

While 10G-plus speeds are elbowing their way into the switch-to-server environment, top-of-rack appears to be firmly cemented as the architecture by which many of these connections are made. A little more than a year ago BSRIA (www.bsria.co.uk) published results of a global data center survey, in which it inquired about network-connectivity speeds and architectures-along with other areas-in different data center environments.

The data on top, from 2014, shows 3 percent of switch-to-server connections exceeded 10-Gbit/sec speeds. BSRIA projected that by 2016 that number would grow to 24 percent.

Among the high-level conclusions drawn from the survey is that the top-of-rack (ToR) architecture is more popular in colocation facilities than it is in enterprise data centers. “The uptake of ToR is higher for the colocation segment, at 61 percent, than for the enterprise segment, with 42 percent opting for ToR,” BSRIA reported. A third of enterprise data center connections are made using centralized switching, while 15 percent are made using end-of-row and another 5 percent using middle-of-row switching architectures.

Often discussed hand-in-hand with top-of-rack/end-of-row is the use of point-to-point and structured cabling. BSRIA’s survey illustrated a correlation between the two. “The use of non-structured cabling point-to-point links was significantly higher for the colocation segment, which also has a higher use of ToR architecture,” BSRIA pointed out. In total the survey yielded even results, with 50 percent using point-to-point and the other 50 percent using structured cabling. But looking at the enterprise data center segment in particular, the correlation is strong. ToR switching is in place for 42 percent of enterprise data center connections, and point-to-point cabling is in use for 46 percent. Of note, BSRIA’s inquiry about point-to-point versus structured cabling specifically pertained to 10G links.

Point-to-point connections and structured cabling were in a dead heat when BSRIA examined 10G links in enterprise and colocation data centers. Further detail shows that point-to-point is more prevalent in colocation facilities than in enterprise data centers.

Other data in BSRIA’s survey back up the assertion by Siemon’s Maguire and Conroy that switch-to-server links are outgrowing 10G. “The study highlighted quite significant levels of 1G and 10G both in switch-to-switch and switch-to-server links and expected progression to 40G and 100G planned for 2016,” BSRIA noted. In switch-to-server links, speeds up to 10G combined to account for 92 percent of connections, but BSRIA expected the number to drop to 73 percent by 2016.

In other recent analysis, BSRIA discussed trends in network and cabling using for large data centers (which it defines as having more than 500 servers and more than 20,000 square feet of floor space), medium data centers (100 to 500 servers and 2,000 to 19,999 square feet), and small data centers (25 to 100 servers and 500 to 1,999 square feet). Large data centers have a high uptake of ToR architecture, but also implement EoR, MoR and centralized switching, BSRIA said. It added that in these environments, Category 8 cabling has the potential to replace point-to-point direct attachments.

Medium data centers also have a high uptake of ToR, and their fiber use is increasing. And small data centers/computer rooms mainly use copper cabling. Many small and medium data centers use singlemode fiber from the entrance to the main distribution area to support telecommunications services, BSRIA added.

The dynamics of connection speeds, architectures and cabling media are some of the factors that make data center administration a continual challenge and balancing act. Faster speeds often drive change, but efficient administration and the ease of anticipated future upgrades also play a role.

Patrick McLaughlin is our chief editor.