WIRELESS ’93 CTIA CONVENTION
DALLAS, TEXAS. MARCH 4, 1993
Disasters, whether natural or man-made, bring out the best and the worst in people and organizations.
The worst is seen when the disasters are either man-made, e.g. war, sabotage (mining a dam in Bosnia) or can occur after a natural disaster such as looting or exploitation of people caught in the disaster.
But today, we are here to look at how disasters can also bring out the best in people. Our speakers will be dealing with a seldom-recognized benefit of the new wireless telephone service now available across our Continent.
It will be a story of not only how the cellular service served in the public during and after natural disasters, but of the people in the cellular companies who went above and beyond to provide help. These are stories of which everyone in the cellular industry can be very proud.
BACKGROUND TO DISASTER RECOVERY
But, first, let me provide some background to these stories by looking at the whole field of disaster recovery in the wireless communications industry.
Plans for recovery from all kinds of occurrences are part of the normal design of any well-engineering cellular company. There is little sense in relying on cellular service to help others in the event of a natural disaster if the system itself is the first victim.
Just like any military operation, plans must be in place for every eventuality. Staff must be trained to act according to this plan. There must be regular tests to ensure that the plans work.
This is a huge topic and I am certainly not going to go into the details. However, a quick overview will alert you for the kinds of things you should listen for as our speakers tell you of their personal experiences. At least, I will outline some of the categories of planning necessities for disaster recovery.
A perfect wireless network would have so much redundancy, it would never be out of service. Clearly a company providing this would never make any money and so compromises are needed.
The usual criterion is Maximum Allowable Downtime (MAD). This is a judgmental factor of risk management. It obviously has dollar-and-cent implications involving lost revenue versus the capital cost of additional redundancy. This is also where insurance may play a role.
Of far more importance to today’s discussion, however, is how this translates into network design incorporating all the usual techniques of battery back-up in all cellsites, diesel back-up in addition at the MSO, redundant backhaul facilities and so on.
But regardless of the redundancy of the physical plant, portions of that plant can go out of service in a natural disaster. Towers do collapse in a hurricane. Therefore, even more planning is needed for such things as mobile sites that can be moved to a location on short notice.
These are often referred to as Cells on Wheels or COWs. There have even been proposals for a Switch on Wheels (would that be referred to as a SOW?). However, switches are not normally as big a problem and the rerouting to other switches in the network is another approach, although this takes considerable planning. It would only be in the event of a major problem, e.g. a fire at the switch site where a SOW would become essential and then, given the cost, it would likely have to be on an Air Force One to be practical for delivery anywhere in the country.
As we know, most switches have considerable redundancy already and it is our experience that software is far more likely to provide temporary problems than hardware. That is why we all put so much time and effort into rapid restart procedures.
However, there are many other considerations beyond just good network design,
• the recovery plan must include clear responsibilities for the recovery teams. There needs to be a central Command Centre already designated in each area. Bear in mind that networks may be national but disasters are local. Decentralization is essential;
• notification needs to be formalized. Are the home phone numbers of your staff on a computer which may also be out of commission? Who notifies senior management of the problem? Who gets in touch with the authorities?
• a supply of emergency phones is necessary. Where are these? Who has access through security? To whom are these distributed, in what priority?
• the personal safety of staff is a consideration. How far do you risk your staff in the event of disaster?
• public relations becomes very important. How are people to know what services are available?
In Cantel’s Disaster Recovery Network Handbook, we listed 86 possible threats as part of our vulnerability analysis. Given the list of potential problems, it is a testimony to the industry that we have had so few serious service outages. It is even more impressive that we have been able to react so well to external disasters.
It is now my pleasure to introduce three members of the industry who will tell you of their company’s reaction to very difficult situations.
You have heard some amazing stories of disaster preparedness and heart-warming response. If there is one theme running through all of these presentations, it would be:
• the necessity of having a well thought-through disaster preparedness plan;
• having top executive commitment to ensure that everyone in the company believes in the plan and will react with enthusiasm and dedication when the occasion requires it.