Unsurprisingly, PGX is involved in a lot of database emergency responses. Every emergency is different, but there are some basic rules that everyone should follow for any emergency reponse.
–
- Wind your watch.
No one has a watch that winds anymore, but the point is: take a deep breath. Give yourself a minute, or two, or five to gather data. A too-fast response is the main way a problem becomes a disaster.
- No one on the call who doesn’t have something to do.
When you have the incident call, keep the group down to people who can actually (a) supply information, or (b) do mitigations. You want as small a group as possible.
- Have a final decision-maker.
Five peer engineers sitting around debating what to do is not a response team: it’s a group therapy session. Have someone on the call who can make a final decision as to what action to take at each step.
- Analyze, don’t argue.
If the data points away from a hypothesis as to what is happening, accept that fact, but remember that it is about probabilities, not facts. “It can’t be that” closes off a line of inquiry when it can, in fact, be that.
- Point and call.
Before making any change, the person making the change explains it out loud to someone else. This is the technological equivalent of pointing and calling, and it works wonders.
- Appoint someone to deal with senior management.
There are some managers who have the idea the reason the problem hasn’t been fixed is that the engineers haven’t been yelled at by someone with a lofty-enough title yet. Have some brave soul be the communications path to and from management so that the engineers can actually fix it.
- No post-mortem until after the corpse is cold.
Don’t try to do a post-mortem while the emergency is on-going. Collect data, and wait a week or so before writing it up. You’ll have better data, more detachment, and during the emergency, you have much better things to do.
–
Emergencies are when you have to be the most organized. Drill on this. You won’t regret it.
“Train driver pointing” photograph by Alan Levine via Wikimedia Commons, licensed under CC BY 2.0. Original can be found at Flickr.
