Monday, December 14, 2020

Who Watches the Watchers?

On December 13, 2020, the Cybersecurity and Infrastructure Security Agency posted an Emergency Directive that describes actions federal agencies should take as a result of a supply chain attack against certain SolarWinds products, namely those built upon the Orion platform. The short take is that any organization using SolarWinds Orion solutions should immediately power these systems off, unless they have the means by which forensic images can be captured while the systems are still online.

SolarWinds develops monitoring tools for all manner of IT components: on-prem physical and virtual servers, storage, cloud instances, web sites, and event logs. They're widely deployed in public, private, and military settings. And they're wildly popular with the tech influencers on Twitter; I've been a SolarWinds MVP for longer than I can remember.

Attacks on monitoring systems raise an interesting question: how do you monitor a monitor? Most systems monitoring solutions are deployed as the end of a chain: you build an infrastructure, you build the systems, applications, networks, and then you build a monitor to keep an eye on them all. Consideration is rarely given to whether the monitor is doing its job with integrity: have you ever deployed a monitor to monitor the monitor? Likely no, as throughout the industry we've accepted that a monitor is the last in a set of pieces to any IT puzzle.

The advisory from this weekend challenges that assumption, and once again reveals the risk we all face when operating IT infrastructure at any scale. No component or system can be trusted without a method for verification in place. And actions of these systems' administrators should be closely monitored for atypical behavior.

It's unclear at this point whether the recent patch from SolarWinds addresses the supply chain issue. My recommendation is to wait for CISA to provide further guidance before you return your SolarWinds Orion servers to the networked world.