When a vendor hosting one of our core systems went offline due to a ransomware attack, it brought into focus that our eggs were in a basket that we were not holding. Suddenly we were at the mercy of their IT department trying to restore our systems and recover our data.

As you can imagine, that was not a comfortable place to be.

Sixty-five days with a critical system offline, with nothing more to do than pass along vague updates and missed milestones from my vendor to my increasingly anxious stakeholders, taught me some valuable lessons about assuming that vendors are prepared and capable of handling a cyber-attack.

By making a few changes to the way we hire vendors, even when all of our eggs are in one basket, we can better ensure the safety of organizational systems and data.

Vet Your Vendors

This one might seem obvious, but there are many cases in which we inherit established systems, an existing system is bought by a new vendor, or a new version changes the delivery method or technology stack behind an application. Vetting our vendors can be a time-consuming and painful process, but it is the crucial first step.

As more and more vendors are moving their software to subscription models, we are increasingly dependent on vendors to protect our core systems and critical data. True SAAS solutions are designed and built from the ground up to be delivered through online mediums with redundancy and Disaster Recovery (DR) plans built into the infrastructure, especially when operating on one of the major providers such as AWS or Azure.

Hosted solutions, by contrast, are often moved to servers on premises at the developer’s offices and you are simply given remote access to run the software via a Web App, RDP portal, or Citrix connection. Beware of vendors simply hosting their solutions and selling them as a subscription. Their desire to convert to a more viable long-term pricing model often comes as an afterthought and lacks the same level of native protections that true SAAS solutions typically offer. High costs, lack of expertise, and short-sightedness often plague these environments and result in a lack of redundancies, security, performance, and DR capabilities.

“Vendor vetting is not a one-and-done task.”

No matter where your vendor falls on this spectrum, it’s imperative to do a deep dive with vendors to understand their security, DR plans, and their capabilities. Ideally, vendors will be prepared for these conversations, often with customer-facing security documentation, testing cycles, results, or even recent tabletop exercise results to share with you.

To ensure that you are getting the best quality service, vendors should live up to the standards you would expect if you were hosting the system in-house. Those that fall short need to be replaced whenever possible. If the system is too, specialized, expensive, or unique to replace, work with them to better understand your expectations and put measures in place to protect your company. This kind of guidance will not only help protect you against disaster but can foster a rapport with your vendor and establish mutually beneficial goals that strengthen the relationship.

Vendor vetting is not a one-and-done task. Annual security reviews should be scheduled with vendors of core systems, and internal policies and procedures should be updated accordingly with any changes identified by the reviews. A close partnership with vendors throughout your relationship will ensure your security expectations are understood, met, and carried out.

Contracts With Teeth

Regardless of how prepared your vendors are, you will still be watching from the sidelines if something goes wrong. At that point, it is too late to start wondering what obligations they must participate in and any financial or reputational damage your company will incur as a result.

The fact is software contracts are written by the vendor and will, more often than not, have verbiage in them that protects them from any financial repercussions due to outages. You may be surprised how often the contract could be interpreted to mean you would even pay the entire subscription rate regardless of how often or how long the system is offline. IT leadership and your legal team should thoroughly review this part of the contract.

When possible, adjust the contract language to include protections for your company that extend beyond the bare minimum to cover expenses, such as consulting fees, insurance deductibles, late fees, employee overtime, and even reputational damages incurred due to the unavailability of the system. Never accept a contract that does not at least indicate that you will only pay for services based on their availability and uptime.

These kinds of contractual negotiations can be difficult as vendors are often hesitant to put their money where their mouth is when it comes to uptime guarantees. At the least, sending a redlined contract back to them, with teeth in it, shows them how serious you are about protecting your company and serves as leverage in negotiating their obligation to protect your company.  

Take Back What Eggs You Can

Since using SAAS or hosted solutions will require you to put a large amount of trust in your vendor, it is imperative to ensure that safety nets within your control are in place. Work with your vendors to provide backups of your data, configurations, and even code when possible.

Use your own infrastructure (on-premises or cloud) to take snapshots of your environment. By maintaining your own backups, you are taking back the ability to participate materially in the event of an outage, standing up your own instance of the system, or at least maintaining access to and the integrity of your data.

Vendors will rarely have an objection to this arrangement, but if they do, you should push back with the contractual changes. “If you are not going to commit to damages contractually, then you need to commit to providing us daily backups of the server on our own infrastructure.“

Get creative about how you work with the vendor to ensure you can put yourself in the game instead of just sitting on that sideline.

Penthouse to Basement

Cybersecurity protects you–until it doesn’t. Disaster recovery plans are great–until they fail. Backups are comforting–until they are corrupted. Thinking through every variable and every scenario that can manifest from a cyber attack is difficult, and invariably, something will catch you off guard. It is our job as IT leaders to walk through each version of an event.

Most of our time is spent in the penthouse, protected by our cybersecurity defenses that we have expended time, energy, and money to establish. It’s comfortable here, surrounded by our comfy firewalls and 24/7 monitoring. But, as anyone who’s been around the block a few times will tell you, it’s not a matter of if, but when you will be impacted by a cybersecurity event.

As a security event cascades through your various lines of defense, you may eventually find yourself in the basement…what happens then?

With each level of protection you put in place, you should always conclude with the question “And what if that fails?” This leads you to plan the next contingency. Each contingency moves you down a floor until you reach the basement… “What if we can never recover this system?”

You, the subject matter experts, and stakeholders in the system itself should create this plan. This is an uncomfortable question to ask your stakeholders, and the number of times I heard the reply “Well I will just quit,” tells me just how important it is to have contingency plans in place.

Have the stakeholders think through what they would need if they had to do this process manually. What resources would we need? What outside help could we leverage? What are the most critical deliverables? What alternative systems are out there, and how quickly could we transition?

Having alternative workflows in place that can keep the business moving forward will eliminate the need to be reactionary if the worst-case scenario comes to pass. By vetting the various lines of defense you are creating with the stakeholders you are ensuring that you have thought through everything and demonstrate the efforts you and your team are taking to ensure they are protected.

“Put all your eggs in one basket… and WATCH THAT BASKET”

Mark Twain

Ultimately, the impact of a cyber incident on your company will come down to how prepared you are. We cannot afford to be lulled into a sense of false security by farming out our data and systems to third parties. Because, when you are standing over a bunch of broken eggs, it is not going to matter who dropped the basket.