Nobody likes a crisis; they are difficult, troubling and sometimes dangerous. For most of us in web operations, the chances are slim that a crisis will be truly life-threatening, but when millions of dollars are on the line it can feel like a pressure cooker and have a negative impact on lifestyle, relationships and mental health. Most companies go to extreme lengths to avoid crises, and even when one does occur, the typical response is to first deal with it, and subsequently pretend it never happened. As if the memories are too painful to discuss, we avoid the topic all together; talking to your customers about it is quite risky. It is probably best not to mention it at all; you should just move on. Unfortunately, for most organizations, reacting like this means missing a grand opportunity to make your company better.
Like any organization, we are always on the lookout for new talent. Of course, you want people who are "smart and get things done", but beyond that, I have found one particular personality trait to be critical to long term success at OmniTI; the ability to stay calm in a crisis. While I tend to think OmniTI does well in avoiding them, we do have a tendency to attract customers with a lot on the line and, apparently, with a critical mass of customers, so we are no strangers to crises. While it is pretty clear to me that composure under stress is a fundamental requirement in high-stakes jobs (like large-scale web operations), I think it is generally helpful in any situation. Contingency planning can only get you so far, and when your packets are spilling all over the floor you need to keep a clear head about you to make sure you can assess and remediate as if you've done so since kindergarten. If you can't remain calm, the situation can deteriorate quickly. Turning to the blame game before solving the problem at hand is a sure sign of such deterioration. You must fight that urge. If you can't, your team can't be as open with communications as you need them to be, and your recovery time will suffer. Be upfront about how you want your teams to respond, ideally before problems arise. There are real crises in the world where people die at the hands of companies; walking though one of these exercises can be humbling and enlightening; James Lukaszewski takes us through a "Death by Burger" scenario in his Seven Dimensions of Crisis Communication Management, and outlines positive and negative ways that a company can respond to such an incident.
That said, resolving a crisis should not only be about solving the problem at hand. When calamities occur, it's important to recognize that your company has an opportunity for introspection. What is it about your processes that led to the crisis you've just survived? Do your process and tool chains do everything they need to do? Don't just determine if they work, but do they do the job in they way you would like the job to be done. Seldom will you arrive at good answers to these questions through the normal course of business. Even if you think failure is human (perhaps especially when you think so), it's important to understand what processes failed or what information was unavailable that led to this human error. That information is crucial because, in most cases, the people on your team are acting in a manner they think is safe and appropriate -- and in the best interest of the company. The knee-jerk reaction in these cases is often a summary dismissal, but that will often leave you with the core issue unaddressed: they thought their actions were acceptable. If you fail to gain an understanding of the underlying causes, this bleak episode is likely to become a rerun; either with a new employee, or perhaps with an existing team member who also doesn't understand where the appropriate lines need to be drawn.
One thing I believe is very helpful is to look at how others handle these things. The Internet is new, Web Operations even newer; but crisis management and postmortem analysis are not. Quite often I see people lay blame at either the wrong people or processes in times of (and even after) failure. Ideally you should not be trying to lay blame at all, but instead figure out where improvements are needed. Many people mistakenly assume that crises are born out of mistakes; often they are not. As businesses grow over time, it's easy for plans that were once appropriate to become inadequate. You need to look at your systems holistically. For folks in Web Operations, a healthy understanding of why complex systems fail can help you gain a better perspective.
If you are running a team, you owe it to yourself to turn crises into dialogue. If your customers were affected, be honest with them about where things went wrong, and why what you did was the appropriate thing or how you plan to adjust course going forward. Be careful not to overreact; the goal should not be to add more process, but rather to improve process. Your next crisis is your next great opportunity to learn more about your organization and to strengthen it for the future. Don't miss it!
