How IT Leaders Can Best Plan for Disaster

As recovery efforts in the wake of Hurricane Sandy continue, it's time to think about how your IT organization will prepare for the next disaster.

By Thor Olavsrud
Tue, November 06, 2012

Hurricane Sandy left devastation in its wake, first pounding the Caribbean and then pummeling the Mid-Atlantic and Northeast regions of the U.S. People and businesses are now struggling to get back on their feet. As a CIO or IT leader, you need to ask yourself: Will your organization be prepared for the next disaster?

Sandy left at least 106 dead in the U.S. and left millions without power. In New York City, many homes and businesses have suffered severe damage, the public transit system remains snarled, utilities are struggling to reconnect services and gasoline is scarce. Damage from the storm is projected to be as much as $50 billion.

Residents—some of whom are now homeless or without power, heat or running water—are trying to find a way forward. Many businesses, too, were not prepared for the magnitude of the storm and are now struggling to get back on their feet. Disruptions in communications, power and transit are conspiring to make it a challenging prospect at best. But with the right planning, says Nicolas Dubus, IT director for Florida-based eTailer CableOrganizer.com, you can overcome these challenges with little or no business disruption when disasters strike in the future.

Dubus learned this lesson the hard way: In 2005, Hurricane Wilma hit Florida like a hammer, flooding much of the southern portion of the state. Wilma was responsible for $20.6 billion of damage in the U.S. ($24.5 billion in 2012 dollars). CableOrganizer.com, with headquarters in Fort Lauderdale, was without power for four days.

Communication Is a Key Issue to Disaster Recovery

"More than power, communication was the issue," Dubus says. "We didn't have Internet. Cell phones weren't working. Regular phone lines weren't working. You couldn't communicate with your vendors."

Wilma was a wakeup call for CableOrganizer.com, a largely IT system-driven business that sells electrical, telecom/datacom/networking, home theater, cable and wire management products. After that experience, it got serious about disaster preparedness.

"Each year, prior to the month of June when hurricane season commences, we review our disaster-prevention procedures to ensure readiness in the event that a tropical storm or a hurricane heads our way," he explains. "But rather than a simple checklist and seasonal preparatory efforts, we review and maintain disaster prevention and recovery plans for all levels of the operation year-round to best assure we're truly prepared."

"Whether you have a small company or a big company, when there's a disaster like Sandy, it's very important to have a plan on paper and review it frequently," he adds. "Now we have a plan and it's documented. We know who's going to do what things and when. We know who's supposed to come here to the building, who's supposed to take care of different things in the building. We even have a blog that is private to the company where the owner is able to post updates and status to the Web."

First and foremost, Dubus says, no matter how high-tech your business is, old school is the way to go when it comes to disaster preparedness.

"One of the very first things I would say is to create a list of contact information for IT and the whole company," he says. "For instance, how to contact your manager and the other people you need to contact. It should be on paper. You want to have all of the information for your phone system provider, web server hosting, Internet contact—all the contacts you can imagine. Have it on paper and keep it updated."

It's About People and Planning

For a large, distributed enterprise, paper may be less of an immediate concern, but being able to reach out to your people should still be your first priority, says Daniel Newton, senior vice president of operations at Datapipe.

Datapipe is a provider of managed hosting services and data center infrastructure for IT services and cloud computing. Datapipe's headquarters are in Jersey City, N.J. on the other side of the Hudson River from New York City. It maintains data centers in Somerset, N.J., San Jose, Calif and the U.K. and China. Even with two data centers in New Jersey, Datapipe suffered no outages or failures as a result of the storm.

"We focused on our people first and foremost," Newton says. "We had to ensure that we had the availability of our people and we had to make sure they were OK."

As soon as Newton's team was sure there was a reasonable probability of the storm hitting New Jersey, they made plans to move a number of people from the New Jersey office to an office in Austin, Texas for the week. The team also booked hotel rooms near the New Jersey facilities so they could walk to work if transit was impossible. Newton says the company even stocked up on supplies in the facility in case employees had to spend several days there. Team members in other locations around the world were also notified that their shifts might have to be flexible so they could pick up any slack.

"We did have people displaced," Newton says. "We did have people that didn't have power and we did have people that could not easily get into our critical infrastructure to help support our clients."

Even with those issues, Newton says Datapipe was able to take a roll call of all its employees in the wake of the storm and make sure everyone was OK.

Protect Your Facilities

Next you need to think about your facilities. Dubus suggests multiple Internet connections so you can try to keep those connections up and running if possible. CableOrganizer.com has five connections, including a satellite dish on the roof of its building that connects to HughesNet. It also maintains several analog phone lines and phones that do not require power to function. Dubus notes that he's also given key personnel satellite phones to use if no other communication networks are available.

Datapipe, on the other hand, relies on softphones to keep their communications flowing.

"We would have to lose telcos across the entire East Coast and probably telcos on the West Coast as well for them to go down," Newton says.

As for power, having a gasoline-powered generator in your building may be a short-term fix, but it's probably not the best one unless you also store plenty of fuel onsite.

"The problem with that is when there's no power, it can be very hard to get gas," he says. "I go to the gas station and the line is 10 hours long. What we did is we purchased solar panels and set them on the roof. We also have a propane-powered generator and we have a 1,000 gallon propane tank on the premises. We can keep that generator running for several weeks if necessary."

Datapipe had diesel generators on hand. It never lost power from its utility, but Newton notes that power did fluctuate at times, so his team ran the generators for two days to keep the fluctuations from affecting their equipment.

Ahead of the storm, Newton's team made sure all of the fuel for all of the generators was replenished and able to run them for a minimum of three days. The company also had a full fuel truck onsite with guaranteed redelivery.

"The focus for us is on disaster preparedness and business continuity so you don't actually have to use your disaster recovery," Newton says. "I think that's one of the reasons we've come through this unscathed."

More Disaster Preparedness Tips

Don't wait for the next hurricane to be on the radar to start your planning process. Here are some additional tips for your disaster preparedness plan:

Maintain your surge protectors and UPS units. Don't let simple things trip you up when it comes to keeping your computers and servers powered. Routinely test and maintain the UPS surge protectors for all your computers and server banks. Replace the UPS units for desktops every two years and for servers every three to five years.

Back up your data and store copies off-site. At the very least, store tape backups in a protected, remote location. Better yet, co-locate as well. And don't simply expect your backups to work. Test your backups regularly. CableOrganizer.com runs tests on small files every week to ensure the integrity of the backup files. Twice a month, it runs a complete restore of all backup data.

Maintain off-site, disaster-proof data centers. Keep your Web servers in various locations to ensure your business's connection to the outside world remains uninterrupted. CableOrganizer.com house some servers in a server room at its headquarters in Fort Lauderdale, but it has also strategically divided Web servers among several additional data centers in Connecticut, Texas and Florida. It maintains a cage with the Host.net data center in Boca Raton, which houses critical servers, the company's phone system, domain controller, SQL databases and application and file servers.

Add backup customer support options. You have to be able to maintain contact with your customers. In addition to its main phone system and backup analog lines, CableOrganizer.com subscribes to a disaster recovery plan with GlobalResponse, a Margate, Fla.-based firm whose building is equipped to face a disaster, and which is connected to AT&T through an OC-12 Sonnet SMARTring. CableOrganizer.com also maintains a backup customer support group with GlobalResponse that can take phone calls on its behalf.

Thor Olavsrud covers IT Security, Big Data, Open Source, Microsoft Tools and Servers for CIO.com. Follow Thor on Twitter @ThorOlavsrud. Follow everything from CIO.com on Twitter @CIOonline and on Facebook. Email Thor at tolavsrud@cio.com