Thirty years ago, as a new business continuity coordinator, I discovered that preventing bad things from happening was not always possible. Not only was it not possible from a financial perspective, it wasn’t possible because there were vendors upon whom I depended. This was in an era where vendor management simply did not exist. Most of the vendors didn’t care about safety, security, or staying operational the way I did. It was deeply troubling, to the point where I was unable to accept the risks created by the vendors with which my company contracted. The C-suite was willing to assume some of that risk in favor of not damaging our vendor and partner relationships. I would get a nice pat on the head and be told “it would never happen, don’t worry about it.”
The patronization only further aggravated my outlook, that is, until events began to occur that caused a loss of revenue through disrupted operations. The dam in the revenue stream received immediate attention by the C suite and I was told to fix it, with a warning not to let it happen again. Exasperated but gratified for permission to correct an intolerable amount of risk, I began to set controls in place that could help mitigate our exposure to our vendors risk appetites.
In one of the first events that was “never going to happen,” the building manager (vendor 1) for one of our remote offices hired a sub-contractor (vendor 2) to replumb the floor above our office suite. That was all fine. The building manager made sure the water to the floor was shut off and the sub-contractor had the floor plans where the building utilities were marked clearly. He also did a walk-through of the systems on the floor at the beginning of the project. The sub-contractor had drained the pipes filling them with air before they began cutting. The work was being done after hours to prevent disruption to the tenants on the other floors due to noise. This was all great, so what could go wrong?
As they were cutting and pulling the old pipes out, there was a problem. One of the pipes they cut still had water in it. Not only did they not expect any water, they were unprepared for this type of water. As it gushed out, the smell of it made it impossible to stay where they were. A significant amount of water began pooling on the floor. The sub-contractor’s employee ran to find the shut-off valve but found it was already off. He called the building manager who had to come and see what the matter really was. Since the work was being done after hours, that took time because he was not on premise. All the while the water kept pouring in and the smell worsened.
They finally figured out it was the sprinkler system, but the maintenance engineer had the key and he was not on premise! Hours after the cut, they were able to unlock the sprinkler room and shut off the water. Decades-old water had flooded the floor. It also meant the skyrise was not habitable because of the compromised fire safety equipment. But rather than notify the tenants, they decided to wait.
Somehow it escaped the building manager’s thought that water follows gravity. Below them was our suite which had been affected. The black water had filled up the filing cabinets with original signature documents, beneficiary designation cards, and one-of-a-kind contracts. The dark water flooded our floor with 4 inches of putrid liquid that smelled worse than the sewer. It was on the keyboards and laptops and computers, on the phones and notepads and calendars and the artwork. It soaked the SVP’s desk and drawers and family photos. It soaked everything and sat unattended all night long until our office manager stepped off the elevator for work at 7 a.m. the next day.
The smell drove him back off the floor for a few minutes to get fresh air. He found a way to cover his face and walked back onto the floor and opened the office door with his key. Black water ran out all over the lobby on that floor. He saw the destruction, moved back off the floor, and called me.
We of course implemented our recovery plans and filed an insurance claim. But here is what we learned the hard way.
- The building vendor had no emergency plan.
- Our building lease did not provide any financial recourse for when mismanagement of the facility shut down our operations.
- The sub-contractor was not thoroughly vetted by the building manager. The work had been assigned to a new employee of the subcontractor with less experience than the owner of the company. The building manager could have avoided engaging them had they done their due diligence before hiring them or at least ensured there was better supervision of the project.
- Documents that are freeze-dried still stink – horribly. We spent a fortune the next year on surgical gloves. We ended up asking clients to sign new forms and contracts, so our staff did not have to endure the smell of the old ones.
- And as with all relocation events, the disrupted work environment affects daily productivity, not once but twice. Once moving out and once moving back in after it’s cleaned up or a new facility is found.
Over the years, the vendor outages or their decisions and actions affecting company operations have occurred far more frequently than fires or floods. It came in second place to power loss as the root cause of events to which I’ve responded. Any way I count, those events directly affecting places where I managed continuity and operational risk, the vendors or their subcontractors were often culprits or contributors.
Unlike 30 years ago, today we talk a lot about vendor and supply chain management. It’s terrific and needed. However, it’s easy just to consider key suppliers of resources and overlook the day-to-day vendors who are in and out of your building. In the last few years I know of a couple local situations where subcontractors to a building vendor caused outages. One led to a fire behind the walls of a newer facility, resulted in relocation, and a disruption lasting more than six months. The other was a roof leak caused by a subcontractor who walked on a roof where they had been asked not to walk. It broke the membrane and leaked water through the ceiling into the data center, which required a temporary shutdown of equipment. Both events impacted the revenue streams of those companies.
Most of us do a decent job of controlling cyber and financial risks. We work hard to protect our data. Yes, the focus today on operational risk is better, yet there is still room for most of us to improve. Expanding the scope of your vendor risk management program and making sure it covers everyone who has anything to do with your physical operations will help. Part of that is talking to your vendors, asking the questions needed to determine how they select and vet their employees and subcontractors is a good next step. Another suggestion is to watch them when there is opportunity. Be curious and ask them questions. Not like a lawyer, but as one who is genuinely interested in their processes. This will accomplish two things for you. First, you might find out something which the vendor owner won’t tell you. Second, your questions may cause them to be more considerate of what they are doing, which may in turn prevent a mistake which has the potential to create a negative impact on your operations.
Outages, even small ones, drain not just revenue but also your employees’ available time to work on those revenue-producing activities. Managing vendors is just a part of a great risk management program. We’ve come a long way in the last 30 years, but from my perspective we will never be done making improvements. There is always more we can do to minimize our risk exposures.