§ Seeds / 2014 / Summer

Summer 2014 an OmniTI publication



The Amazing Dr. Black

When my son Alex was about four years old, he started getting sick about once a month. We would take him to the doctor, and the doctor would say he had an infection, give us antibiotics and send us on our way. It only took a few instances of this before we noticed the pattern and became concerned. We voiced our concerns to the doctor, but the pattern did not change. So we started seeing a different doctor each time. We believed that eventually we would find a doctor who would help him.

As you can imagine, seeing doctor after doctor was very difficult for us, and during that time, I came to a horrifying realization: doctors are just people. Just like everyone else, the level of performance between any two of them can vary wildly. Of course, there can be many reasons for poor performance, but in my experience the more common ones are:

Not Knowing

Years ago, a team I managed had an engineer, who was in charge of a sizable Ruby on Rails application, leave and I was left without anyone with that certain set of skills. I went to one of our team members and said “Hey - I know you don’t know much about Ruby or Rails, but please support this app as best you can.” So going into it, I knew, and he knew that he didn’t know. And, when you know you don’t know, that’s a better place to be than when you don’t know you don’t know.


Because we knew we didn’t know, when acute or chronic problems presented themselves I was able to tell the project manager, “We are doing the best we can with an engineer who has little experience with this technology.” This isn't the type of thing you want to say, but at least it allows you to set everyones expectations correctly.

As we were working through our son's illness, I think some of the doctors we took Alex to see genuinely did not know why my son was stuck in this cycle of illness, but that didn't stop them from prescribing something that gave us an expectation that *this* would be the solution we were looking for.

Not Caring

Fast forward a few years, and another engineer on my team was trying to solve a problem with one of our web applications. He had been restarting the app routinely, trying to deal with some intermittent issues. Then came the automated script that ran once-a-day to restart the app. Then once-an-hour. I finally set aside some time and had a look at the problem myself.

As it turned out, the restart command being issued wasn’t even restarting the application. I felt compelled to ask, “Did you even check to see if the application was being restarted?” As an engineer, if I don’t verify that my fix is at the very least doing what I intended, can it really be said that I care? As a doctor, and really in any profession, I have to say that I think the same rules apply. While I can’t say for certain that any of the doctors we saw didn’t care enough to give my son the very best treatment, I did get that feeling more than once.

My Hands Are Tied

I see this all the time. We are not finished with the product, but it has to be working in production like YESTERDAY, PEOPLE! We produce code for businesses. That code makes them money, or saves them money, or keeps track of their money. In any case: money. If we have to tack on a piece of code with some duct tape and a few nails we had laying around at the last minute, sometimes we do it. We hope to find the right balance, because we are constantly trying to do just that. But, as Leonardo da Vinci is credited with saying, “Art is never finished, only abandoned.”

The point is, as software engineers, sometimes we have to decide between implementing the quick solution or the right solution. There are a finite amount of resources over time that can be applied to solve a problem; that’s just reality. However, there is a balance to be sought. And, when you are the one being affected by limitations, being aware of them doesn’t do much to ease the pain.

Of course the same thing applies to medical care. If you consider the state of medicine in the U.S. over the last several decades, whatever your views are, I think we can agree there is an imbalance which leads to insufficient resources to adequately care for people. I remember a specific conversation I had with one of Alex’s doctors at the HMO; I left with the feeling that rules and costs were inhibiting her from doing more in-depth treatment that could have helped my son.


Of course, even if you have people with the right knowledge, who care about the problem, and have the resources to work on it, communication is still CRUCIAL. I have seen emotional and procedural escalations occur because of a lack of communication more than any other single aspect of a customer relationship. As a manager, one thing I counseled on repeatedly is that you must answer communications in a timely manner. I know it is more fulfilling to send an email saying, “I completed this task, and it’s in production and working great!” rather than some variation on why it’s not done yet, but you need to keep the customer informed.

No communication leaves the customer not knowing anything about the state of their issue [1]. Is it fixed or finished? Is my issue even being worked on? Did the engineer assigned to it quit and move to Djibouti under an assumed name to escape my wrath? I can honestly think of very few instances where communication of bad news caused a freakout. I can remember many instances where no communication did.

Taking my son through the medical process, we rarely received communication in detail, especially in response to our questions and concerns. We were left in the dark far too often, which was not just unfortunate, but it left us feeling totally alone.

You Are NOT on Your Own

One thing I was taught at a customer service job many years ago was to never let the customer feel like they are on their own. The most basic example being, “I’m going to transfer you to department N. If you get cut off or they are not able to help you, here is a number where you can reach me directly. Please call me back and I will continue to help.” This is comforting to the customer, and they will remember your company in a positive light. I have tried to incorporate this strategy into every aspect of my dealings with customers, both internal and external. “You are not alone,” is one of the best feelings you can give another human being.

The Worst News

Once, an internal customer (a group from within our same company) sent us this email:

“It’s been three months now since the promised delivery date for this product has passed and we still don’t have it. We need to know now if you can provide this to us or not.” Although we had kept them up-to-date with frequent communications and regular meetings, our lateness had finally exceeded their patience.

Unfortunately, this particular software project turned out to be much more involved than we had initially thought. After talking through the issues with our lead developer, I sent this reply:

“We are very sorry, but after encountering many unforeseen complications in the existing data structure, we no longer believe we will be able to deliver this product.”

Wow. That was not a fun email to send. Was there blowback? Surprisingly, none at all. In fact, I don’t think we ever heard from them again. It left a sick feeling in the pit of my stomach for a while, because we wanted to “do good” but this time, we failed. And yet, I think there is a strength, help, and hope in this reply. “We can’t help you,” stated in no uncertain terms, frees you to try another path. It also frees us to stop sweating something we are losing ground on every day. I never heard “I can’t help you” from any of the doctors we saw, but can you see where that might have helped us?

Trying Different Things

It’s been said that the definition of insanity is doing the same thing over and over, and expecting different results. If that’s true, I have seen a lot of insanity.

Earlier I had mentioned a case where we had an issue that we tried to solve through restarting a web application. As it happened, the way this problem manifested was that the application would just stop accepting connections, and the customer's browser would just sit and wait. Our support engineer would restart the application manually, and in the interest of fostering good communication, would inform the customer that the problem was fixed. This was fine the first few times, but after:
“I restarted it.”
“I restarted it.”
“I restarted it.”
. . .understandably, they started to become upset. When they asked the support engineer, he would say, “Well, I can’t tell what’s wrong with it after I restart it.” This was a bit of a catch–22. The customer needed it working again right away, but they wanted the chronic problem fixed too.

I finally said: “The next time the app stops responding, don’t restart it. We need to examine it while it’s broken.” And I communicated this intention to the customer, both beforehand and during the following outage.

Do you want to know what the root cause was? So do I. We never figured it out. But because we took the time to fully analyze the broken app, we could set up some solid monitoring that could detect when the problem occurred and instantly restart the application. Once that was in place, we found that the application rarely restarted, and when it did, it was within a minute of the app failing to respond. Outages became insignificant to the customer, and they were satisfied.

Substitute antibiotics for “I restarted it” and you have our medical dilemma exactly. You “fixed” it, Doctor, but have you really fixed it? No. You kicked the can down the road, and we’ll have to deal with the problem again soon.

Dr. Black

In our quest to find a doctor who could fix our son, my wife and I finally found Dr. Black. When we talked to him the first time about Alex’s recurring illness, he listened. I mean he really listened. And I think we left with antibiotics, because Alex still needed them, but we also left with something new: Flonase (for nasal congestion, where the infection would always start.)

And guess what? That didn’t fix his problem. He was still sick a few weeks later, BUT. . .

A Cut Above the Rest

What made this doctor exceptional was that every single time we saw him, his approach was, “That didn’t work? Well, let’s try something else.” It was such a relief to have someone working on finding a real solution. He may not have known what the problem was, but he let us know that he was as determined to find a solution as we were, and that he would work with us until we did. And a few tries later, he found it. It was Rhinocort, a different medicine in the same family as the first medicine we tried. He pointed out that some people have success with one and not the other.

Since this was preventative medicine, we could give it to my son whenever he started to get congested and not have to give him a course of antibiotics every month. Eventually we were able to stop altogether and he has been very healthy for more than a decade now.

I will never forget The Amazing Dr. Black, and I will always try to emulate his conscientious work ethic.

1.   OK, sometimes they can tell it’s still not working, but most customers will be unsatisfied with this amount of information.