Planning for Disaster Recovery: What Could Possibly Go Wrong?

Where disaster recovery is concerned, having the right tools and procedures in place is paramount, of course. Our Data Protection Services are your powerful defenses to recover from data corruption, our Server Recovery Solutions enable you to recover systems quickly, and our Consulting & Strategy capabilities provide expert guidance along the way, but first you need a comprehensive business disaster recovery plan. Before finalizing your plan, you need to ask yourself some questions. What should be the basics of your DR plan? What steps should the plan include? Ask yourself what problems would test the strength and thoroughness of your plan.

What could go wrong?

Aside from an invasion by Martians, what could cause a disaster for your company? We tend to think more in terms of weather events, , but there are plenty of other causes, such as hacks, IT failures, employee sabotage; and physical plant problems like power outages, plumbing crises, and roof leaks. Think through the possibilities and create a plan around the most plausible scenarios.

How well do you know your vendors?

Be sure that you have contact information for all your vendors and that you stay in touch with them. When disaster strikes, you’ll be glad you did.

What about SLAs?

In planning your disaster recovery procedures, don’t overlook your technology partners.

How detailed should your communication plan be?

In previous posts, we have stated the importance of a well-designed communication plan that keeps the whole organization informed.

• Provide your vendors and customers with timely information posted on your website and delivered through email. Let them know what to expect and don’t be defensive about any responsibility you bear for the problem. Remember, the coverup is usually worse than the transgression.
• To the extent possible, have statements drafted in advance so that the need to explain something doesn’t catch you unprepared.
• Public statements should include information on what to expect going forward.
• Let the business press know the score.
• Update your messages whenever there is anything new to report.
• Give your people clear written instructions on what to say and not say to inquirers. Not everyone should be able to speak for the company and everyone who is should have the same story.
• Decide who needs to learn what from whom, between coworkers, vendors and customers. When disaster hits, concerned parties need to know who to call and what information to convey.

Who does what?

You will want to designate a core team to handle the most important aspects of the recovery, but everyone in the company should be part of the process. There is such a thing as assigning tasks in too much detail, so that people don’t take the time to learn their roles, but it’s important that everyone in the organization plays a part in the recovery plan. Ideally, all team members should have clearly written descriptions of their duties. Since not everyone is at his or her desk all day, and vacations and other absences are constant, be sure that key responsibilities can be handled by backup personnel.

How much downtime can you stand?

Companies’ ability to tolerate freeze-ups varies greatly, so you need to examine this issue in depth.

• Inventory your systems and applications and become familiar with product information available from your vendors.
• Evaluate each critical system and application to see how long you can keep operating if it is disabled. (For more on downtime, check out our downtime calculator.)
• Look for acceptable workarounds when critical systems go down.
• Identify the systems that, while they may not be the most important, are best able to help you through a period of downtime.
• Include in your plan the order in which you want your systems to be restored. Divide them into two or more levels of time sensitivity, separated in terms of importance versus urgency. Which ones do you need to be operable as rapidly as possible and which ones, though important, can remain inactive for a while without causing too much damage?

Who’s taking notes?

Someone in your organization should be responsible for writing down all your plans, precisely and thoroughly, including a complete description of the recovery team and each person’s assignments. Not only is it important that someone be given specific responsibility for transcribing the plan, it should be someone with the writing skills needed to make the material comprehensible, complete and unambiguous.

Which way to the storm cellar?

All concerned parties should know how to continue doing their jobs during the crisis, including changes in location. Since a disaster may involve damage to your physical location and danger to your people, you should have an alternate workplace available, somewhere people can continue their work with as little stress and disturbance as possible. Everyone should know where to go and how to set up workspaces when they get there. If working remotely is in the mix, teleworkers should know how to become productive quickly and smoothly.

How do you know your plans will work?

Without regular, in-depth testing, you don’t. Test often and thoroughly, because a perfect plan that’s out of date is no plan at all. Testing once a quarter is not overkill.

• As you go through your testing procedures, be especially aware of the little things: minor technical weaknesses that are insignificant when everything is going well but which become catastrophes when disasters arise.
• Pay attention to the human element as well. For example, every time someone leaves the company, moves to a new position internally, acquires new skills or takes on new responsibilities, that person’s role in the recovery plan will change.
• One key to good testing procedures is a simulation, making the test as close to real as possible. Staging live drills will keep everyone mindful of possible disasters and sharpen performance.
• Don’t think of test failures as bad, and don’t brush them off. A test failure can be fixed and can prevent serious damage in the future.

Finally, when anything in the plan changes, even the slightest detail, don’t forget to make sure it is reflected in the written record. It’s not easy to make sure everyone knows to notify the official scribe when a change takes place, but failure to take that small action could impair recovery efforts.