Jonathan A. Zdziarski email@example.com
August 5, 2003
|"These facilities are one of only a few places where an individual is capable of introducing heavy, unchecked equipment, leaving it in or near a large public concentration of business, and is able to communicate remotely with the equipment from virtually anywhere in the world. "|
I’ve spent a significant portion of the past ten years of my
professional career working for corporations with large co-location
facilities. Co-location facilities
provide a cost effective data center solution for many companies, both small
and large, enabling remote hosting of equipment in a climate controlled
environment usually with several redundant high-speed connections to the
Internet. These facilities are
responsible for a significant percentage of electronic business performed in
In having the privilege of working with a number of these
facilities, I’ve also had the opportunity to witness the vulnerabilities that
could give themselves over to terrorist activities. Prior to
Many co-location facilities are strategically placed in areas where a significant amount of business is occurring…major peering points, large corporate concentrations, and many general terrorist targets. Some facilities are within immediate proximity to targets such as the New York Stock Exchange and the public and private networks that are responsible for the Internet as well as military and public service networks. A single target among many, if taken out, could seriously cripple the Internet let alone the number of critical private networks sharing the same fiber. Due to the placement of such facilities, they are unfortunately an ideal target for terrorists to take advantage of. These facilities are one of only a few places where an individual is capable of introducing heavy, unchecked equipment, leaving it in or near a large public concentration of business, and is able to communicate remotely with the equipment from virtually anywhere in the world.
Before delving into the different vulnerabilities these facilities are faced with, it is necessary to get a general overview of what the average facility looks like.
A co-location facility is for the most part a large data center located either in its own building or sharing an office building where slices of space (usually anywhere from 1/5 of a rack to multiple racks) are carved out and leased to an individual or corporation. The leasing party (e.g. the customer) is responsible for moving their equipment into the facility, setting it up on the racks, and connecting it to whatever network is provided by the facility (if any). Most facilities are comprised of the following components:
Most facilities are unmanned, and remotely managed from a NOC, or network operations center. This NOC could be in the room next door, on a different floor, or in a nearby building. The NOC is usually responsible for the electronic state of the entire company including their network, external customers (leased lines), internal customers (co-located customers), peering sessions, and etcetera. There is generally no group focused specifically on the co-location facility’s process and procedures, although there are occasionally one or two overworked individuals responsible for the cabling and such.
There are generally two groups of individuals clear to
access these facilities at any given time: customers who have procured space,
and telco/fire/electircal maintenance technicians from vendors, the city, and other such individuals.
From a customer perspective, the standard procedure to procure rack space in a co-location facility is to sign and pay on a contract through the company’s sales representative. The customer’s network access will then be provisioned and rack space assigned. Access control for customers is generally more formal than maintenance: the customer will most likely be given access cards, a key, or other means to access the facility at which point it will be their responsibility to install their equipment and bring it online.
From a maintenance perspective, the individual wishing to access the facility will generally be asked to provide an identification card and/or information about the service call. Since the maintenance may be for a specific customer, there is generally no maintenance authorization log. An individual performing maintenance on the facility may be from one of many local carriers with hardware in the facility, an electrician configuring a rack for a customer, fire marshall (who frequently inspect facilities), or even a "tech" from a customer sent out to unplug a system. Guest cards are usually given to the technician upon arrival, or the technician is escorted into the facility and left to work.
Since a majority of these co-location facilities are unmanned (perhaps with the exception of a security guard), it is the customer or maintenance technician’s responsibility to conduct themselves in a professional manner by not stealing or sabotaging the equipment of another customer or vendor. Should an individual visit the facility during a lull time (the evening, for example), they will be virtually alone. The individual will usually bring the necessary equipment in on a cart, swipe their card, possibly sign in, and be cleared for access to the facility.
While there are usually some deterrents in place, such as cages for larger customers, security cameras, and possibly an on-site security guard, these facilities are generally not monitored with much attention or scrutiny. Generally offenses of this nature are only detected after the fact. The responsibility of the security guard is to insure that unauthorized individuals are not allowed into the facility, that nobody walks in with a baseball bat and starts smashing equipment, etcetera. They are not by any means equipped with the ability to determine if a chassis is armed with explosives – or even if the equipment a customer is working on belongs to them. Even to engineers who work in a co-location facility on a daily basis, most of the work customers in the facility perform are considered to be “their technical stuff” and rarely ever noticed let alone challenged.
Authorized guests of the facility will frequently have several large pieces of heavy equipment stored in the facility. These can include enterprise-class servers or mainframes, network equipment, telco gear, large batteries, and possibly even their own hardware closets. It is not uncommon to see computer systems five or six feet high with large locking doors for disk storage or processor blades nor is it uncommon to see a group of deep-cycle batteries with bare terminals connected to telco equipment. Other equipment involved usually includes copper and fiber optic cabling, miscellaneous small devices, and possibly even the customer’s own remote access cameras.
Now that we’ve taken a basic look at the design and procedure of co-location facilities, it is time to discuss the vulnerabilities of these facilities. As I said before, these facilities are one of only a few places where an individual, with some money in hand, is able to bring heavy equipment in virtually unchecked, be able to leave it in a place where it will remain untouched, and remotely communicate with that equipment from anywhere in the world. This makes co-location facilities an ideal target for terrorists to use as a means of introducing explosive devices (even possibly a small nuclear device) into a large business or residential area, and detonate the explosives remotely having enough time to leave the country if so desired. Since co-location facilities are frequently located in buildings where there are concentrations of telecommunications networks tied together, even a small explosion capable of taking out only the building could seriously cripple commerce.
Co-location facilities being a corporate business, it is the goal of a vast majority of facilities to give business to any individual or corporation that can afford the services. There are generally no requirements to purchase rack space other than meeting the financial obligation and agreeing to the contract. The corporate strategy behind these facilities also compensates for the fact that many new small businesses will need these facilities’ services.
For this and many other reasons, a vast majority of co-location facilities are not very familiar with all of their customers. Certainly no background checks are performed on individuals or businesses seeking to lease rack space in the facility. In many cases a business license is also not necessary as an individual is just as valuable a customer as a business.
The first step a terrorist group would likely perform in a scenario such as this would be to find a company with a co-location facility either:
The sales representatives for the facility are usually more than happy to give an individual a guided tour through their facility and will generally like to brag about big name customers in the facility as well. This walk through the facility, while helping to close the deal, also gives a potential attacker the following information:
Once the attacker has this information, they can make an even better judgment about whether or not the facility is a suitable point of attack. Should the attacker desire to become a customer, a contract will be filled out. The customer will then agree to a set of terms and conditions, provide billing information, and hand over a check. Once this is complete, the provisioning process begins and the rack space is assigned. In many cases, this entire process can take less than 24 hours.
Once the attacker/customer has been given access to the co-location facility, equipment may be introduced into the facility and bolted to the racks. In all my experience I have never found a single provider that allows their employees to inspect the customers’ hardware or require a copy of the keys to such hardware remain on the premises. The attacker can easily smuggle in whatever equipment they desire without detection. This can include explosive devices, small nuclear devices, electromagnetic devices, etcetera.
Due to the large and heavy nature of most enterprise class computer equipment, plenty of free space is already available in any large computer chassis to accommodate such devices as well as keep them cool and stable. The Sun E450 is an ideal example of such a chassis, as a significant amount of free space is already available and would require little or no modification to hide such a device. We used to call them the beer coolers for their size and cooling capabilities. There is also the Cisco 12000 series routers, large "stackable" type computer systems, and plenty of other options as well.
Using the serial cable or other interfaces, these devices can be connected to a computer which is then connected to the Internet via the network connection supplied by the facility. Some of the possible ways such weapons could be smuggled in include:
Once the attacker has brought in their equipment, it can sit in plain daylight inside a computer chassis, or even underneath the raised floor where it will be out of sight. The employees of most facilities are not expected to nor do they “meddle” in their customers’ business. I’ve found in many facilities, one could easily steal another company’s hardware while being watched through a camera by the NOC dismissed with the thought, “They must be with Company ABC. I’m not going to challenge them and get in trouble”.
Aside from timers and RF detonators, there is IP. As the co-location business involves Internet connectivity, very high speed connections are usually provided to the customers. Connecting this to the computer that has been brought in provides a remote means of detonation from anywhere in the world over any layer of encryption the attacker finds necessary. Should a small nuclear device have been smuggled in, detonating the equipment from inside the building could easily take out the city block.
A more complex approach might include scouring a news site for mentions of particular key words such as “Bin Laden Captured”. Once the system has been connected to the Internet, it can be trained to detonate on any event ranging from the system date to the score at the last Yankee's game.
Should the attacker be more interested in damaging equipment from another business located in the facility, smaller explosives will most likely be used. Something as simple as water itself can damage and bring down a large computer network. Unfortunately, cages do not keep water out, and many of the different designs of overhead conduits make it even easier to irrigate every machine in the room.
Now that some of the very basic vulnerabilities have been discussed, let’s take a look at some of the ways security can be improved in these co-location facilities. These are just a few suggestions out of many that would both serve as a deterrent and help detect such violations.
This method of gaining access to a facility hinges completely
on the lack of proper procedures and authorization checks at a company. The
"ignorance factor" is taken advantage of to authenticate phone calls, uniforms,
and paperwork. The steps to fixing the hole in the maintenance arena include:
An undetected attack in this arena hinges on the ability for the individual to become a customer. The biggest problem with the co-location business is that there is not a very solid relationship between the provider and the customer. When an individual rents an apartment, their credit and references are checked, the individual is interviewed, and based on the character of the individual, it is determined whether or not they are allowed to rent. Unfortunately due to the technology industry failing and in the name of privacy and professionalism, renting rack space in a co-location facility requires no such check in most cases. The only reason most sales representatives even know what their customers do is from casual conversation and not background checks. Once the baton passes to provisioning or network operations, nobody knows anybody. The customer is now a number, and so no questions are asked.
Facilities located in areas with a high concentration of people or businesses should make every attempt to know who their customers are and where they’ve come from. Having background checks part of the standard contract, visiting their place of business, and learning as much about the customer are all ways to insure that you’re not doing business with the wrong group. This will force attackers to build long term relationships with their targets making it all the more possible for them to expose a suspicion.
New customers should provide more specific information about their business. Are they a web hosting company? An online trading company? What do they intend to use the hardware in the rack space for? Sure they won’t say “to blow up Wall Street” but if they don’t have an answer, that’s suspicious. Does the customer have a website? How did they pay…cash, personal check, company check, credit card or wire transfer?
Some information can be discerned from common interaction with the customer. If the company has been around for several years, has offices, and a staff then it is most likely a legitimate business. On the other hand, if the company is a new virtual company with a residential address, there is significant reason to raise an eyebrow. Co-location facilities already have the liberty to discriminate based on many other criteria such as whether or not the company spams, broadcasts pornography, etcetera. Insuring that the customer has a justifiable business is certainly a responsible approach to new customers.
Some basic policies regarding the hardware that is stored in the facility can help give the provider some additional means of inspecting equipment and insuring new customers don’t attempt to smuggle explosive devices. Having a clause in a contract that requires a copy of all system keys to be stored on the premises and a requirement stating the provider “reserves the right to inspect any hardware brought into the facility for malfunction or malicious use” gives the provider the ability to perform periodic checks of all hardware for not only this purpose, but to insure there are no fire risks, loose power cables, or any other facility hazard. Such a policy is easy to justify.
Explosive detection tools are available as well to scan new hardware. Even K-9 officers can be trained to detect up to 11 distinct odors of explosives. An occasional pass-through of the facility by a trained K-9 officer can provide a non-intrusive way to check out new customers without the need to even touch their hardware. Other actions such as checking underneath the raised floor for any devices, closely monitoring a customer’s actions and behavior, and etcetera are all less intrusive ways to keep an eye out for suspicious activity..
Paying attention to the kind of hardware the customer brings in is another good way to identify suspicious activity. Is the user a dialup Internet provider? Why do they not have any dialup access equipment such as modem banks? Are they a web hosting company? Why do they have only one large server instead of several small ones? Are they a Linux shop? Why do all of their terminals have little window icons on them? As small as they are, these inconsistencies can pile up and help to identify a customer who is not really who they say they are (even if you're just looking for spammers).
Finally, implementing a network provisioning period that allows the customer to install their hardware before their network connectivity or POTS lines become available will give the provider plenty of time to perform any such inspections before they are given the ability to communicate with their hardware from anywhere in the world. Granted this would not stop a bomb on a timer, however an individual who would seek to launch this type of attack would most likely do so for the purpose of doing it remotely.
Restricting building and rooftop access
If the facility is located in the same building as a targeted corporation, or if there is a possibility of a different style of attack (for example, a releae of biological toxin from the roof top), the individual may attempt to use this opportunity to attack in other ways. By having 24 hour access to the co-location facility, the attacker is also given 24 hour access to the entire building (in many cases). Having a building security guard monitor the status of individuals in the building will help detect if a customer who is supposed to be on the 10th floor is snooping around on the 4th floor. Policies for introducing new equipment outside of normal business hours can be implemented requiring permission from the facility provider. Additional steps similar to these can and should be taken to secure the building outside of business hours.
Many facilities providers allow their customers to network to the roof of the building where antennas or satellite gear are mounted. Restricting this type of access to where the customer must be accompanied by staff during normal business hours will help prevent any “soft target” attacks on the building.
Hiring an outside firm to test the effectiveness of the policies a company has in place for knowing/choosing their customers and managing the facility will help expose any loose individuals in the company. Hire a firm to create a fake company with some suspicious ties and attempt to become a customer. Do they succeed? What about equipment policy violations? Can they effectively pose as a maintenance technician and gain access? A terrorist isn't likely to be walking in with guns blazing and take over the building, but rather play a game of invisibility and misdirection, taking advantage of the very nature of of "good-willed" individuals...these external tests can help identify these weak areas.
In summary, there may be no easy cookie cutter plan for improving the security of co-location facilities; however it is of the utmost importance to take the necessary steps to protect these facilities from a terrorist attack. Detecting the vulnerabilities in your individual corporation is the first step. Once they are exposed, finding an effective way to fix them will help make the facility a secure place of commerce rather than the next target of attack
The bottom line is: how difficult is it for anybody out there to sneak their way into your facility as a tech or a customer, and sneak in a dangerous device? If the answer is “not very difficult” then you have some vulnerability in your policies that could potentially expose you to an attack.