 |
|
-
How Did It Affect Your Business?
The blackout of August 14, 2003 affected 50 million people in
Canada and the US -- many businesses were shut down completely,
finishing the first day in the dark while people stranded in
elevators awaited rescue and rush hour traffic coped with an
abrupt loss of traffic control. In Ontario, some were without
power for days, and in at least one area, the resumption of
power was accompanied by a transformer fire that brought back
the blackout. Because of the slow restart of electrical
services, businesses and residents were asked to minimize
electrical consumption to avoid consumption-reducing rotating
blackouts. As this is being written, normalcy is slowly
returning. And with it, hope that perhaps the tightly
coupled inter-dependencies that characterize the modern world
can be tweaked somehow so that this does not happen again. Let
us hope that this will be true some day. But over the last few
years, this has been just one example of an external event that
has abruptly interrupted "normal" activities. Unlike a
deteriorating weather situation, this event was sudden and
unexpected. The opportunity exists, while events are fresh in
our minds, to reflect on the blackout, its impact, and what we
did to recover -- with the objective being to evaluate our
responses to learn from the experience and be better prepared
for the next time.
-
Things to Consider
One possible approach to reviewing the experience would be to
perform a day-by-day review of the event and its consequences:
write down the problems, mitigation responses and impacts. Try
to apply 20/20 hindsight -- what worked well, what didn't, what
should be done differently "the next time". Some questions are
very industry-specific, for example:
Retail -- did the cash registers stop abruptly?
Was there any information loss? Was any business conducted with
the normal systems down? Has it been possible to verify the
perpetual inventory now that "normalcy" has been restored?
Manufacturing -- what happened to materials in
process at the time of the failure? Were any processing
electronics damaged by the power loss or restart? Any impact on
in-plant materials tracking?
Financial Services -- how were transactions in
process at the time of the power failure handled? Did this leave
any uncovered positions? Any customer impact? Were any trading
reports impacted?
As the timeline is revisited, here are a few things that may be
helpful to include in the discussions:
-
Communications
One common problem within the GTA(Toronto) was the unreliability
of the cellular phone network. Phones would work erratically or
not at all. In general, land lines were reliable, but access to
them was sometimes crippled by failure of locally powered
electronic phones. If contacting vendors, local authorities or
communicating within the management team were activities that
were performed during the blackout and recovery, how much was
reliant on affected technology? Was it necessary to provide
customer information and media releases? Do essential staff
carry a management contact list with every key number listed?
And would a simple telephone and printed staff call list have
been helpful?
-
Emergency Management Plan
The blackout was an emergency that affected many people
simultaneously. Did an emergency plan exist and was it followed
during the event? If so, what were its short-falls? Did it include
provisions for wide-area disasters? Or was there the assumption
that impact would be confined? The recent blackout and ice storm of
1998 are good examples of events that affect a wide area. Don't
have a plan? Then this may be a good time to start -- even a
simple outline is better than nothing.
-
Emergency Backup Power
Some businesses attempted to carry on during the initial period of
the blackout as though nothing was wrong, just an interruption in
the usual services. But their work areas were dark and potentially
dangerous. A small investment in emergency lighting would have
made things much easier for the first few hours. Similarly, most
computers lost current work when they shutdown abruptly, the
latest updates not having been flushed to disk. This can
cause file corruption in addition to data loss. An investment in
local battery backup would have bought time to perform an orderly
shutdown and capture final figures if necessary. At this office,
for example, all systems run from a single UPS with a 1 hour
capacity; if the device signals the server that power has gone away, an an
orderly shutdown automatically propagates. Investments in battery
backup and emergency lighting can make a big difference in the
initial period. For longer operation, a local backup generator is
required and in a wide area disaster, few generators are
available.
-
Reserve Supplies
In some areas, the drinking water systems failed or lost
protective pressure. Food and fuel are other resources that
suddenly became in short supply. Were these local issues of
concern? If so, were adequate local supplies available, or should
one goal be to establish or increase supplies?
-
Saved Computer Information
Two general classes of information should be reviewed:
system/application backups and critical information reports. If
the blackout caused any equipment damage or data loss, were the
regular backups adequate to recover business processes? Was the
information readily available and usable? If not, are there
changes to the backup process and media handling that should be
considered -- soon? And during the blackout or recovery, was there
any critical information that was needed but was trapped inside an
inaccessible computer? If so, are there any practical processes
that can be put in place to produce external reports and refresh
them periodically?
-
Technical Contingency Plan
If computers are an essential part of the business process, was
there an emergency recovery plan in place? Did it work? Was the
business able to utilize the recovered services? How successful
was the information recovery? What parts of the process need to
be changed in the future? If a plan did not exist, would it have
helped?
-
Business Insurance
Now that the recovery is under way, was any business insurance
adequate for the experienced interruption? Was it possible to
contact the appropriate parties when they were needed? Did the
provisions of the policy cover the loss? If not, are there any
changes needed to the coverage (or provider)?
-
Energy Management Practices
The provincial government asked that firms and citizens practice
extreme energy conservation to reduce the strain on the recovering
electrical system during the days following the blackout. If there
was an attempt to comply with this request, how successful was it?
How did this affect the overall business functioning? Are any of
the changes desirable to be continued -- even after the electrical
grid is restored to normalcy? Did this provide any insight into
how energy is used locally? Did the exercise suggest any changes
for the future?
-
Sources of Information
As with any post-incident review, supplemental information is
always helpful when formulating action plans to improve response
for next time. In the areas of emergency management, contingency
planning and energy conservation there are many resources
available on the Internet. The links section of the Technology
Strategists website provides references to a number of
organizations and information sources.
Technology Strategists is a Toronto, Ontario-based management
consulting firm established in 1995, providing business
contingency planning, application and system troubleshooting,
infrastructure development and operational process services.
|
|
