Most people can recall where they were on September 11, 2001, at the moments the planes struck the World Trade Center twin towers. The disturbing memory is etched on my mind, first receiving the call, then later coming to terms with the grim reality as it was relayed around the world by the media. As a so-called business continuity expert, the sensation of loss was amplified by past experience and an insight of what was to come.
For two decades, both before and after the London bombings of the 1990s, a handful of specialists including myself have preached, lived and breathed disaster recovery (DR) in its many guises and now, despite the billions of dollars spent to secure the way we do business and proof it against disruption, we are confronted by this apocalyptic scenario.
So what does 9/11 really mean for business? Has the world changed? Will normality be resumed in a year, a decade, or is this the beginning of a new era, a Vietnam or a Cold War? This is a chilling and uncertain possibility.
In this article I have attempted to crystallise what I believe are the future implications of September 11 for global markets and world trade generally.
Evolution
Technically, we are told, the term 'DR' (disaster recovery) was superseded in the early 1990s by 'contingency planning', later by 'business continuity' and recently by 'operational risk management'. To be sure, the discipline is maturing, but it's important to understand what it means, where the advances came from and their relative value.
From my perspective there are two key differences between then and now. First, business continuity in its purest form means that from the outside, despite inner chaos, there is negligible perception of anything untoward. This implies that reputation will be preserved, business revenues undiminished, growth continuous and competitive edge maintained. This amounts to perfection, the holy grail of business continuity; the reality is rarely so aesthetically or financially pleasing, although some very impressive organisations all but achieved this following 9/11.
Second, DR traditionally was the preserve of technology, the replacement of business-critical equipment (often a mainframe environment) in an acceptably short time, leaving business to sort itself out within the criteria imposed by the technologists. In this respect the business continuity school scores significant points, underscored in red by the experiences of 9/11. Business continuity uses business need to set the priorities and timeframes that must be achieved by IT and all other infrastructure providers -- not the other way around. It puts the boot, very logically, on the other foot. (Note that, for the purposes of this article and for the sake of familiarity, I will use the DR acronym and assume that it now encompasses those aspects of business continuity that have been added since the phrase was coined.)
From a world trade perspective, DR's role is akin to that of insurance and could even be conceived as a form of marketing. A firm's ability to demonstrate resilience gives its counterparties, its suppliers and its clients the confidence they need to commit to large-scale business. Indeed, a select few world-class companies require evidence of DR provision before taking on a new supplier.
But for the majority, enthusiasm and demand for DR from executives remains fickle. Despite the advances described above and the shocks delivered by Sarin gas, the IRA, Y2K, and a dozen other 'lessons', DR rarely rises above 'grudge purchase' status, and never more so than in times of recession.
We must hope and believe that the memory of 9/11 will have a different, enduring effect, although the evidence I have gathered in some organisations already suggests that the contrary may be true.
A changing climate
In the introduction I alluded to a change in the political climate, brought about by terrorism, its parameters redefined on September 11 in terms of scale, simplicity, motivation, targeting and human misery. Al Qaeda has ably demonstrated that globalisation is not restricted to peaceable organisations and that terrorism is the political weapon of choice for some globally active factions.
It is a sobering thought that the first ever conference on biochemical terrorism took place in February 2002. Equally worrying is the number of apparently independant and relatively minor incidents that seem to be occurring.
The business of global terrorism affects all legitimate trade. It gives it new impetus and urgency and a set of negated assumptions: urgent operational challenges that must be met. It also begs many unanswered questions, not least 'could this happen again?'
We have to assume that it could, particularly where the democratic, multi-ethnic, multi-ideological nature of first world societies leaves them exposed and open to disruption from within. Activism, anti-capitalism, anti-progress, race- and religion-related unrest feature regularly in the media and typically focus on causing intensive localised disruption to a target organisation.
These realisations have caused our DR focus to change once again and already organisations are adopting new procedures to manage the new risks as they perceive them.
We have already begun to react as we seek to protect ourselves, and the pressure will increase as customers, auditors, regulators and markets demand that we become measurably resilient. Our immediate actions will be largely in the form of 'sharp end' preventive tactics with emphasis on viability and realism. Our longer term response may include cultural change, reducing travel, self-insuring as premiums soar and perhaps, as in Johannesburg, we will abandon the central business district as the preferred place of work, creating 'doughnut cities' with a no-go core.
Geography and real estate
September 11 saw a significant change in attitude toward real estate. Leaders have assumed (correctly in my view) that CBDs, symbolic buildings and those shared by high-profile organisations have a greater likelihood of being targeted. So the prestige and convenience associated with premier locations must now be offset against the additional risks borne by the firms that occupy them. Nor has this factor been lost on employees: a survey carried out by globalcontinuity.com shortly after the tragedy indicated a strong reaction against working in high-rise offices, where 43% said they would strongly resist such an offer.
Sceptics now acknowledge that wide-area disasters will occur and organisations within a substantial radius of an impact can be affected both directly and indirectly (yet still I occasionally find difficulty in persuading organisations to plan for this). These effects can range from physical damage through to denial of access for prolonged periods and the loss of vital utilities. Each can be manifested in multiple forms and with multiple compounding side effects. The number of detailed scenarios is endless and we cannot hope to plan for all of them; a new approach is required where plans are conceived with an underlying emphasis on flexibility and interpretation.
Some unfortunate organisations lost both their primary and back-up sites as a result of 9/11, their worst-case provision compromised by a desire for operational convenience. A second survey supported this, showing a surprising number of organisations with recovery sites either within the CBD, within 2 km of the primary site or both; of these, many were already considering a move further afield. Experience has long shown that out-of-town or remote recovery locations reduce the chance of incidental concurrent outage and that multi-site distributed organisations tend to be more resilient than single-site businesses.
Location, evacuation capacity, prominence, the businesses carried on by co-tenants and the location of alternative operating centres are all now important factors in deciding where to locate your business.
A fair share
Some organisations opt to use specialist DR service providers rather than bear the cost of maintaining their own 'warm site'. These DR providers generate revenue by selling the opportunity to use their workspace, equipment, technicians and other assets to multiple customers. They argue that the chance of concurrent demand or 'invocation', to use an industry term, is vanishingly small and, in the majority of cases (localised interruptions, computer-specific failures and so on), this formula works particularly well.
However, unless great care is taken by providers in managing risk ratios, syndication does not work well for wide-area incidents. On September 11, many businesses found their contracted recovery venues already occupied by others who shared their syndication on a first come, first served basis. In one case, a firm called to invoke its provider some eight minutes after the event and was informed that they were the eleventh in line. Such firms either received a small percentage of their anticipated service level or were diverted to more remote centres not catered for in their plans. (Others were magnificently served by their DR providers and none to my knowledge were simply left out in the cold.)
In my opinion, syndication breaks down because beyond a notional break-even number of participants, each additional subscription represents clear profit and encourages high syndication rates, sometimes on specific units of equipment. In my experience, it is almost unheard of for hot site providers to disclose the exact number, identity, location or contractual terms relating to other syndicants. DR suppliers also generally charge invocation fees to occupants of their facilities, charges that will be recouped in many cases through insurance claims.
Perhaps the new risk climate will encourage DR firms to offer more transparency on their risk management processes and provide stronger guarantees to syndicants.
A safe supply
Despite DR providers' undoubted pre-occupation with caring for the many firms they continue to support in and around New York, it would be wrong to suggest that calm has been restored to the DR market. Its principal players have had the time to catch their breath and the market space remains active, turbulent and changing.
Polarisation is occurring, with a small number of industrial behemoths steadily reeling-in the weaker or smaller players. A fierce battle was waged recently as third-placed SunGard (DR revenues of around USD410m last year according to Gartner Dataquest) and fourth-placed HP (USD135m) fought over second-placed but faltering Comdisco (USD480m). SunGard's eventual successful acquisition means it is now placed almost on a par with IBM, which accounts for around 40% of the worldwide business continuity services market. Opinion on the outcome of this change is divided, with some viewing it as anti-competitive, reducing choice in an already cramped market. Others see it as a beneficial consolidation, increasing the depth of resource available to them should disaster strike.
As a by-product of this trend, the number of specialist DR suppliers is also dwindling. Earlier in the year Sema group was acquired by Schlumberger, originally an oilfield services provider, now repositioned as a global technology services company. IBM, HP, GE, SunGard, DEC and many other mainstream technology providers now offer disaster recovery services as just one part of a diverse technology portfolio. Relatively few now deliver DR as their mainstream focus. Perhaps the wider market has become too uncertain to support so concentrated an offering, or perhaps the pace of technological advance and level of investment required to build a viable DR enterprise presents too great a challenge to niche providers.
One scenario, then, would be that DR attains the status of a 'value added service' available to be purchased from a systems or services provider. Given this, I envisage few niche entrants to the DR market, although other major players may join the fray, attracted by DR's lure of short-term growth. Within the new commercial formula we should expect the 'high street retailer effect' to prevail, encouraging us to buy a three- or five-year disaster recovery 'insurance policy' whenever a major IT installation is commissioned. Long-term flexible commitments like this are profitable, increasing the stability and forecast net worth of providers.
A third significant trend, a blurring of the distinctions between the high availability and disaster recovery markets, is also evident. Many of the organisations mentioned in this article already offer distributed high-availability services. Interestingly, if clients opt for multi-site resilient operations then conventional DR contracts will become increasingly obsolete.
In the longer term we can anticipate that, in keeping with multi-site policies, organisations and particularly financial institutions will build in resilience by investing in distributed high-availability solutions. An increasing number, although still a minority, will find they no longer need a DR provider and will bring the entire capability in-house.
Perhaps we will come to regard September 11 as the cataclysmic event that reshaped the DR industry, killing off the dinosaurs and evolving tough new breeds capable of ensuring our businesses survive these harsh conditions.
People
One of the lasting messages conveyed by the reports of colleagues, the press and those directly involved in recovery following September 11, is the astonishing fragility of the conventional business infrastructure. In many cases we rely on the web of interconnection and interaction of multiple assets, so much so that an event beyond a set piece scenario results in temporary chaos and delay beyond tolerable limits.
Backups, recovery centres, instruction manuals, plans and all the DR measures imaginable cannot of themselves resolve the chaotic disorder arising from major disruption and it is people, not systems, who dominate the recovery process. They alone are capable of injecting the necessary order, offering the resourcefulness, flexibility and motivation required to rebuild and re-tune so complex a system.
Reports from firms who successfully recovered from the World Trade Center catastrophe unfailingly highlight the crucial part played by staff. One organisation reported making over one million client calls, providing 250,000 free meals and operating 40 command centres. They stated that "employees and vendors will always rise to the occasion and can accomplish 'miracles' to recover the firm".
Paradoxically, in most organisations few staff willingly participate in DR-related activities, partly because they receive little recognition for it and partly because it is seen as a defensive, low-yield activity, lacking in kudos. This perception may stem from the overly prescriptive or analytic methods historically employed, the failure by executives to sell the need to line management, and the pass-or-fail test regimes that have traditionally been imposed. A number of organisations, including my own, are now working in ways that correct this condition.
Need to communicate
No matter how willing the workforce, it cannot function if it cannot communicate. Yet in the hours immediately following the attack, almost all fixed line and mobile networks in the vicinity of the World Trade Center became inoperable. This was due to the destruction of critical supplier infrastructure, the saturation of the system by the sheer volume of calls being made and the seizure of reserve bandwidth by the emergency services. The resulting telecomm blackout meant that businesses found it difficult to organise recovery and were unable to contact or account for many of their employees. Any recovery processes reliant on data transfer via public or private networks were also disrupted.
This incredible telecoms blackout has encouraged many executives and infrastructure managers to review their dependency on existing networks and look for ways to create resilience and redundancy for these systems. This is borne out by Gartner Dataquest's findings that, following September 11, 50% of United States businesses expect to realign their internal budgets to allow for increased spending on telecom resilience and services.
The powerful message that businesses cannot rely on the telecom service when it needs it most makes innovation seem inevitable. A McKinsey report found that firms were "overly vulnerable to 'choke points', telephone switches and other hubs through which key information flows". The report recommended that businesses should develop "an alternative communications system for emergencies" and even suggested that "the financial services industry should consider developing a secure network for use in emergency situations that does not rely on the main telecommunications network or on the mobility of participants".
The telecomm failure had an immediate effect on the Internet industry, with the initial shock and subsequent slow recovery affecting many web-dependent companies. Those directly affected by the failures saw their websites out of service for days, delivering a severe blow to e-commerce related businesses in the area. Other companies suffered temporary service problems as the surge in Internet usage around the world overloaded the otherwise unaffected Internet infrastructure.
In the weeks immediately following 9/11 there was evidence that firms whose business model relied on the Internet for e-commerce sales, or for other mission-critical activities, such as communicating with staff and customers, had also begun to investigate and implement resilience solutions, such as load balancing, DNS rerouting and the adoption of multi-site operations.
The importance of telecoms is set to increase still further, spurred by the popular disinclination to travel. Improvements in streaming media, video-conferencing and remote-meeting technology seem set to encourage this trend, still further increasing our dependence on technology. In the same way that work expands to fill the time available, so it seems that telecoms bandwidth will continue to be soaked up just as fast as it becomes affordable.
Driven by competitive necessity, telecom companies must now find new ways to respond, differentiating and perhaps offering high-priced contingency bandwidth and systematically eliminate these choke points.
Conclusion
The need for business continuity and/or disaster recovery has been hammered home by the tragedy of September 11, painfully focusing our attention on the vulnerability of western culture and business in general. Yet it seems unlikely that the terrorist threat will be nullified in the foreseeable future.
In most jurisdictions, a firm's principals are held legally responsible for safeguarding its stakeholders' interests, a remit that demands the effective and prudent management of operational risk. The persistent threat profile described in this article insists that executives act to prepare and protect the organisations in their charge.
To satisfy this legal, commonsense and competitive necessity, we should aim for a multi-faceted response whose key points are to:
- immediately fill any apparent gaps in corporate defences -- increase front-desk security, train executives to handle crises, teach postroom staff to deal with powder-filled envelopes -- but not in isolation;
- insist that utility companies (including telecoms) provide diverse routed capacity for use in an emergency;
- site recovery locations or processing centres, or both, out-of-town, maintaining a 'thin blue line' style of presence;
- research advanced technology solutions that allow data and processing to be rapidly replicated between sites, ideally in real-time;
- devise a planned response that is flexible, capable of handling diverse scenarios and delivering near-continuous business;
- train people and rehearse the planned response until individuals are confident and can use their initiative, adapting and responding with minimal instruction.
Together, these measures will increase organisational resilience, building a solid prevention-and-cure capability. Rejecting them could be viewed as somewhat short-sighted.
More information about JR Consulting Partners Ltd can be found at www.jrcpl.com.
