WHITE PAPER
In the Data Center - Where and What Am I Testing?
Download PDF
Overview
The data center is the heart of every enterprise network that enables the transmission, access and storage of all information. Here, cabling connects enterprise local area networks (LANs) to switches, servers, storage area networks (SANs), and other active equipment that supports all applications, transactions and communication. It also is where the LAN connects to service provider networks that provide access to the Internet and other networks outside of the facility.
As the amount of information and applications continues to grow, data centers small and large are expanding their capacity to house increasing quantities of active equipment and more links than ever before while also needing to enable high-bandwidth, low-latency data transmission to and from equipment. Regardless of the size and type of the data center, the switching topology and the applications, the underlying cabling infrastructure that creates all the links needed to connect data center equipment follows the same basic design principles established by industry standards.
The ANSI/TIA-942-A Infrastructure Standard for Data Centers references the TIA-568 series of the cabling standards but contains additional information appropriate for data centers. It outlines the specific functional areas of the data center, providing minimum recommendations for pathways and spaces, backbone and horizontal cable media distances, redundancy, cable management and environmental considerations. Similar to TIA-942-A, other data center standards like ISO/IEC 24764 Information technology - Generic Cabling Systems for Data Centres and ANSI/BICSI 002-2014 Data Center Design and Implementation Best Practices also outline various functional areas of the data center that define equipment placement while allowing for scalability and reliability.
Depending in which functional area of the data center you are testing, there are different applications, cabling and connectivity that you will encounter. Understanding the functional areas of the data center and what you will likely need to test in each can help prepare you for data center testing.
On This Page
Starting in the Entrance Room (ER)
The ER (also sometimes referred to as the entrance facility) is considered the service entrance and contains demarcation points to the service provider’s network. It may also contain demarcation points for backbone cabling to other buildings such as in a campus environment. The ER houses service provider equipment and may be located inside or outside of the data center, depending on the service provider requirements. In large hyperscale or colocation data centers, there may be multiple ERs that provide access to multiple service providers.
The ER is where outside plant cable transitions to the data center’s backbone cabling. While some high-count OSP copper cable for voice service may come into the EF, the primary type of cabling found here is singlemode fiber running 40 and 100 Gig applications such as 40GBASE-LR4, 100GBASE-ER4 or 100GBASE-LR4. There may also be some incoming multimode backbone fiber from nearby campus buildings running 10, 40 or 100 Gig applications such as 10GBASE-SR, 40GBASE-SR4 or 100GBASE-SR4.
To achieve the transition from OSP fiber to premise fiber in the ER, fusion splicing is the most common method of termination with fiber panels housing duplex singlemode fiber connections for connection to service provider equipment. Sometimes incoming backbone indoor/outdoor fiber from nearby buildings will pass through the ER and directly into the main distribution area.
So what will you be testing in the ER? In most cases, the service provider demarcation is installed and tested by the service provider, but there may be singlemode or multimode backbone fiber connections here to other buildings that will need to be tested. Due to longer outside plant cable runs between buildings, difficulty in physically inspecting the fiber and the common use of splicing, Tier 2 testing with an Optical Time Domain Reflectometer (OTDR) such as Fluke Networks’ OptiFiber® Pro can be ideal in these spaces. This will help locate splices and determine specifically what is causing a high loss measurement when troubleshooting.
An optical loss test is also commonly used for certifying the campus fiber backbone to ensure that insertion loss meets the requirements of the applications. In addition, one often finds multi-fiber (i.e., 12-fiber) MPO connectivity in the ER, which is required to support higher bandwidth multimode applications such as 40 and 100GBASE-SR4. With MPO connectivity, using a tester with MPO testing capabilities is much faster and more accurate than using duplex tester that requires the use of MPO to LC fan-out cords, a 3-cord reference method and about 15 different steps to test each of the six pairs of duplex fibers.
Moving On To the Main Distribution Area (MDA)
As the central point of distribution, the MDA houses core switches and routers for connecting to the LAN, SANs and other areas of the data center. This area may serve multiple horizontal distribution areas (HDAs) or equipment distribution areas (EDAs) in the data center, as well as telecommunications rooms (TRs) located throughout a facility. The MDA typically includes high-density backbone multimode or singlemode fiber cross-connects with multimode being the most common for connecting to other areas of the data center due to its lower transmission equipment cost and shorter distances. In addition to duplex fiber connectivity, multi-fiber MPO connectivity is becoming more prevalent in this space here to support higher bandwidth applications such as 40 and 100GBASE-SR4.
Every data center has at least one MDA, and while it is typically located within the data center, larger colocation facilities may place the MDA in a separate secure location. In smaller data centers, the MDA may also include a horizontal cross-connect for connecting to equipment in EDAs that are served directly from the MDA.
So what will you be testing in the MDA? Here you will primarily be testing singlemode or multimode fiber with duplex LC/SC connectivity or multi-fiber MPO connectivity. The other ends of these links are located in the intermediate distribution area, horizontal distribution area or equipment distribution area. The primary performance parameter you need to test for is insertion loss as it is critical to supporting the application, especially in 40 and 100GBASE-SR4 applications that have stringent insertion loss requirements. An optical loss test set will provide the most accurate loss measurements.
The Optional Intermediate Distribution Area (IDA)
The IDA is an optional area primarily used in large data centers, such as those distributed across multiple floors or multiple rooms. Referred to as an intermediate distributor (ID) in the ISO/IEC 24764 standard, IDAs may include intermediate cross-connects and are designed to enable data center growth or provide segmentation for specific applications. Each room or floor may have one or more IDAs, which serves one or more horizontal distribution areas (HDAs) and equipment distribution areas (EDAs) within the data center, and one or more TRs located outside of the data center space.
Aggregation switches and fiber cross-connects are typically found in the IDA, so the types of connections here are similar to what would be found in the MDA – fiber enclosures containing either multimode or singlemode and either duplex fiber connectors or MPOs. In the IDA, you will be testing the other end of the fiber links from the MDA, and again, insertion loss is the key parameter and an optical loss test set will provide the most accurate results.
Somewhere in the Horizontal Distribution Area (HDA)
While a very small data center where the MDA can directly support all equipment via a horizontal cross-connect might not contain an HDA, most data centers will contain at least one HDA to serve as the distribution point for the EDA. However, in a top-of-rack (ToR) scenario where access switches reside in each cabinet within the EDA and connect directly to switches in the IDA or MDA, the HDA is eliminated.
Akin to the TR in the LAN where backbone cabling transitions to horizontal cabling, the HDA houses aggregation switches, access switches, SAN switches and Keyboard/Video/Mouse (KVM) switches to serve the equipment (i.e., servers) located in the EDA. A large data center will typically have several HDAs to serve multiple EDAs of the data center. HDAs can be centralized in their own separate area or located at the end-of-row (EoR) or middle-of-row (MoR) location for the equipment they serve, which means you’ll be testing an HDA and EDA all within a single row.
Fiber backbone cabling from the MDA (or IDA) terminates at the HDA to provide the fiber uplinks to larger core switches. The HDA will therefore contain similar fiber connectivity as to what is found in the MDA. While the fiber from the MDA may utilize MPO trunk cabling, typically the connections at the front of the fiber panels in this space will house duplex LC/SC connectivity since the switches here are lower speed.
The HDA also houses horizontal cross-connects and features Category 6 or higher copper connections for connecting switches to the EDA via applications like 1000BASE-T or 10GBASE-T. So the HDA is therefore one of those data center spaces where you will likely find yourself testing both copper and fiber cabling—fiber uplinks to the MDA (or IDA) and copper links to the EDA. This is also the area of the data center where Category 8 cabling will be found to support upcoming 25GBASE-T and 40GBASE-T applications. Ideally, you want a copper certification tester for this space that can test everything from Category 5e to Category 8 such as Fluke Networks’ DSX CableAnalyzer™ Series tester since you could encounter various performance levels. For even more flexibility, OTDR and loss modules can be added to the DSX Series to support fiber testing.
Perhaps in a Zone Distribution Area (ZDA)
The optional ZDA, which is not commonly used in most enterprise data centers, is essentially like a consolidation point within the horizontal cabling between the HDA and the EDA and contains no active equipment. It is not recommended to contain a cross-connect, but may contain an interconnect for terminating horizontal cabling from the EDA. The ZDA can be convenient where it is not feasible to install patch panels within the HDA for connecting to the EDA, and it can enable easier reconfigurations in very large data centers. Just like with a consolidation point, it is important to test the permanent copper links between the HDA and the ZDA separately and then test the entire link from the HDA to the EDA with the ZDA included to rule out any problems with the termination at the ZDA.
Ending in the Equipment Distribution Area (EDA)
EDAs are spaces allocated for end equipment, such as servers and storage equipment mounted in cabinets or racks. Due to the increasing amount of applications and server virtualization, space within the EDA is typically at a premium. Here, horizontal cables from HDAs (or ZDAs) such as the aforementioned Category 6 or higher copper cabling terminates at patch panels in the cabinet or rack that correspond to the equipment they serve. The application most commonly used for these access switch-to-server connections is 10GBASE-T. This is another area of the data center where Category 8 cabling will be found to support upcoming 25GBASE-T and 40GBASE-T applications.
The EDA also allows for point-to-point cabling between equipment, such as in a ToR scenario where access switches reside in the EDA typically in each cabinet and connect directly to the MDA or IDA, eliminating the HDA. In a ToR application, SFP+ or SFP28 twinax direct attach cables (DACs) are often used for connecting directly from ToR access switches to servers in the same cabinet, while higher speed QSFP+ and QSFP28 DACs are typically used for connections from SAN switches to storage equipment. Testing SFP/QSFPs modules involves verifying that power is properly being delivered.
Best Practices Hold True
Any time you're dealing with fiber, it's essential to make sure to practice good hygiene, and that means endface inspection and cleaning. Not only can contamination on the fiber cause performance problems, it can also transfer from the patch cord to the equipment port and even damage transceivers. Before connecting a cord - even a new one out of the bag - make sure it's clean by inspecting it. If it's dirty, clean it and then inspect it again to make sure you've removed the contamination.
Many data centers have taken to using Velcro(R) strips to attach easy-to-use fiber click cleaners to every rack to encourage good fiber hygiene.
Regardless of where you are testing in the data center, best testing practices still need to be followed. First of all, when it comes to testing multimode fiber links, encircled flux (EF)-compliant testing is required by industry standards for 40 and 100 gigabit links. In addition, as bend insensitive multimode fiber (BIMMF) grows in popularity, the use of the EF method becomes even more critical because a mandrel is not able to produce a tight enough bend to remove the higher order modes that BIMMF confines to the fiber core. It is also important to use non-BIMMF test cords to prevent overly optimistic results.
Another key best practice for testing multimode fiber includes using the 1-cord reference method. While a 2-cord reference may seem easier, referencing both test cords results in optimistic results and may provide negative loss results. Only the 1-cord reference includes the loss of the connections at both ends of the channel for the highest accuracy. Furthermore, many cabling vendors reject results carried out with the 2-cord reference, which could prevent acquiring a warranty.
If you’re conducting Tier 2 testing of fiber in the ER of MDA using an OTDR, measuring the loss of specific events like connectors and splices, as well as overall link loss, depends on the direction from where the measurement is made. That is why birdirectional testing that averages the results from measurements made at both ends of the link is needed. While bidirectional testing may have once been time consuming, there are OTDRs available today that simplify the process by using a loop at the remote end that allows for testing from one end and automatically reporting the bidirectional average for that link.
Copper testing in the data center also has some key considerations. Alien Crosstalk is the key performance parameter that can impact the ability of 10GBASE-T to operate, and since 10GBASE-T is the typical application used for switch-to-server links in the data center, testing for Alien Crosstalk is often required to warranty a system. This is often done using a sample size, or whatever the requirements are of your cabling vendor. When specifying a sample size for Alien Crosstalk testing, it is wise to test an equal number of short, medium and long disturbed links.
Another copper testing consideration is shield integrity. Shielded copper cabling is seen more in the data center than any other space, and Category 8 cabling is a shielded cable. Shielded cabling offers far superior noise immunity compared to unshielded, and if installed correctly, you should see hardly any Alien Crosstalk on a shielded cabling system. But if it’s not installed correctly, even shielded cabling can fail. In a data center application that uses shielded cabling where one grounded patch panel is connected to another grounded patch panel, an open shield on a cable can result in failed Alien Crosstalk testing. While most testers look for simple continuity between the shield on the main unit and the shield on the remote unit, that DC signal will look for any way it can to get to the remote unit—including through the common building ground to which the patch panels and racks are connected. That means that the tester will show a connected shield even when it isn’t. Thankfully this can be avoided with a tester with the ability to test for shield integrity.
