Monday, December 14, 2020

Who Watches the Watchers?

On December 13, 2020, the Cybersecurity and Infrastructure Security Agency posted an Emergency Directive that describes actions federal agencies should take as a result of a supply chain attack against certain SolarWinds products, namely those built upon the Orion platform. The short take is that any organization using SolarWinds Orion solutions should immediately power these systems off, unless they have the means by which forensic images can be captured while the systems are still online.

SolarWinds develops monitoring tools for all manner of IT components: on-prem physical and virtual servers, storage, cloud instances, web sites, and event logs. They're widely deployed in public, private, and military settings. And they're wildly popular with the tech influencers on Twitter; I've been a SolarWinds MVP for longer than I can remember.

Attacks on monitoring systems raise an interesting question: how do you monitor a monitor? Most systems monitoring solutions are deployed as the end of a chain: you build an infrastructure, you build the systems, applications, networks, and then you build a monitor to keep an eye on them all. Consideration is rarely given to whether the monitor is doing its job with integrity: have you ever deployed a monitor to monitor the monitor? Likely no, as throughout the industry we've accepted that a monitor is the last in a set of pieces to any IT puzzle.

The advisory from this weekend challenges that assumption, and once again reveals the risk we all face when operating IT infrastructure at any scale. No component or system can be trusted without a method for verification in place. And actions of these systems' administrators should be closely monitored for atypical behavior.

It's unclear at this point whether the recent patch from SolarWinds addresses the supply chain issue. My recommendation is to wait for CISA to provide further guidance before you return your SolarWinds Orion servers to the networked world.

Wednesday, November 11, 2020

Hybrid Cloud for Everyone (#CFD9)

A panel of delegates and a select group of technology companies convened at Cloud Field Day 9 last week to discuss innovations in the cloud services market. I'm fortunate to have attended as a delegate. In this capacity, I enjoyed a front row seat for presentations on the latest in cloud technology.

Still locked in a staring contest with cloud? You're not alone.
While every presentation showcased the capabilities that made each solution unique, a theme quickly emerged during the event: the promise of a Kubernetes-based hybrid cloud world is still just beyond the reach of most enterprises and businesses. Outside of the hyperscaler and Fortune 500 crowd, IT leaders and technology professionals have yet to find practical ways to efficiently and effectively leverage the variety of public cloud services that are available today.

Two common cloud migration strategies (lift-and-shift and refactor) have run their course. What options are left to the late adopters? Is it too late to move forward with a hybrid cloud strategy, especially if you haven't even figured out a single cloud strategy?

To hear from VMware, Red Hat, Pure Storage, StorPool, NetApp, AWS, and Scality, the answer is a hard no (where no is a good thing).

If you haven't adopted a cloud strategy of any variety yet, you're not alone. The implications of a shift from traditional on-premises workloads where the server is the base unit of abstraction are non-trivial, and impact more than just your operations. You've heard of containers and docker and kubernetes, but these are not minor projects for an organization to tackle on their own. Even using managed solutions like Google Kubernetes Engine and Azure Kubernetes Service are serious undertakings that require a highly specialized skillset (enter the SRE) to properly build and manage. But good SREs are in high demand, are hard to find, and should be freed from operations to spend their time making your applications faster and better.

In particular, VMware recognizes that their customers inhabit the entire spectrum of cloud adoption:

  • On-prem / private cloud
  • Cloud curious
  • Hybrid cloud
  • Cloud first
VMware's challenge as a company is to meet the needs of their customers, regardless of each customer's progress in their cloud journey. To this effect, the company is using its suite of vRealize applications, particularly the vRealize Operations Manager tool, to assist in the planning stages of a cloud project. And they've embraced a growing trend within the industry: data center extension. VMConAWS is a very attractive solution for on-prem shops that have a major investment in vSphere but want to safely expand to a cloud environment.

I'll have more on VMware's cloud solutions, along with overviews of each of the presenters from Cloud Field Day 9, published soon. In the meantime, you can view all of the presentations here.

Friday, July 31, 2020

visiting the country from the city

Roll your troubles on down the hill
It's been a long while and it'll be longer still
Til you pay off your debts
And sign your last bill
Then we'll roll you on down
Where the water moves still

Tuesday, May 19, 2020

Today is Tomorrow

It's the seventh of May, and it's still the same today that it's been since early March. Sunrises and sunsets, digits increment on iOS home screens across the country, and we're still stuck on repeat 1, like the time we discovered "Jungle Boogie" after watching Pulp Fiction and nearly wore out the polycarbonate compact discs that we stored music on thirty years ago. The weather's getting nicer, and playing with my daughter in the sunshine feels pretty good. Maybe working from home all of the time is the change I needed.

Two decades ago a friend recommended that I read Tom Robbin's excellent book Fierce Invalids Home from Hot Climates. Aside from the antihero Switters and the three female characters who exert incredible force on his story arc, one character stands out in these days of quarantine: End of Time.

From the novel:
(Fer-de-lance had concluded that the shaman’s name could be more accurately translated to mean End of Future, or more explicitly yet, Today Is Tomorrow. Accent on the verb. Today Is Tomorrow.)
Today is Tomorrow, because I started writing this on May first, then changed the first line to the seventh (after changing it first to the fourth), and now it's May the eighteenth. But it's still the same today it's been since early March. And it's this same day we'll live for the foreseeable future.

I push my daughter on the yellow swing in the front yard, and she lists the places she and I will visit "after the virus." In no particular order:

  • Cross Street Market
  • The pizza place
  • The chocolate shop
  • The coffee shop
  • The bakery
  • The playground
  • The grocery store for pizza and sushi and big chocolate chip cookies
  • Another pizza shop
  • Yet another pizza shop
  • Starbucks
  • Rita's

Her list makes me smile. It's a list of all the places she and I visited in the normal life of running between appointments, sprinting to the city for afternoon classes and having time to kill in Federal Hill. Running down the sidewalks. Never passing up an opportunity to pick up a few raspberry truffles and milk chocolate pretzels. She and I laughing in the market.

The time is passing, I think.

I lent my copy of Fierce Invalids to a good friend who offered his copy of A Visit from the Goon Squad in return. It feels like only yesterday when I made that trade. It was seven years ago.

Seven years, or a few months, maybe it was yesterday. It's all whimsy. We're all my grandfather now.

Thursday, March 5, 2020

Email Engineering and Duress-Driven Design

Two observations for you. If you're busy, you can just go ahead and agree with me now and move along.

Engineering by Email is Evil

Have you ever been part of an email thread at work that starts out with an innocent question like, "what's CPU usage on cluster alphabet?" but then devolves into a highly technical planning session with multi-paragraph diatribes about design decisions and flimsy, off-the-cuff justifications? It usually spirals into chaos soon thereafter as disagreement abounds. And decisions are made based on who replied to the thread (which is saved by everyone as a "get out of jail free" card).

This is engineering by email. And it is pure evil.

I've developed an informal method for determining when engineering by email is occurring: I look at the height of the scrollbar relative to the height of the window. If the scrollbar's height is 1/4 or less of the window's height: run. And let's not even talk about how horizontal scrolling is widely understood to be a bad UX idea.

Engineering is a process, just like design, digestion, and the Xiphoid. Ok, maybe not just like those things. But it's a process nonetheless. It requires a deep understanding of your requirements, constraints, assumptions, and other influences. It can't simply be typed into existence in response to an email. And email is certainly not a suitable repository for such documentation.

“But no one uses email anymore! We use slack! We use Teams! We used Cisco Webex Teams! We use HipChat!”

Oh shut up already. Everyone uses email for work. No temporal walled garden app will kill email. But I’ll entertain that thought a moment and tell you that if you use any of these messaging platforms as your primary source of inter- and intra-office communications, then the same applies.

In many ways, we’ve lost the true meaning of engineering. It’s a process, not a product. A science, not a suggestion. Engineering isn’t a one-line message that says, “Maybe we can use the DR site since it has excess capacity?” That’s an idea, part of a brainstorming effort. It can influence engineering, but on its own this is NOT engineering.

Duress-Driven Design

You can't build something out of fear. Your motivation can't be to not get fired or publicly shamed. These conditions lead to duress-driven design. And like engineering by email, it is also pure evil.

Your project timeline needs to reflect the enormity of the problem you're solving. While we all like to take our turn as the superhero that saves the day by delivering an amazing solution in a highly compressed time frame, we’ll only burn out all the faster for it. And bypassing development, or feverishly condensingit, is never a good idea.

What's the Point?

The point is that if you work in an enterprise environment where the two items above are not only tolerated but heralded as "working hard," you should get the fuck out as soon as possible.

Wednesday, January 22, 2020


It's the warmest New Year's Day I can remember, but as I recently shared with my wife, I can't remember things like weather from one year to the next anymore. I can't remember if I ever remembered such a thing as fleeting and uninteresting as the weather. Sometimes I record the temperature and the sunniness of a given day in one corner of the too infrequent entries in my beekeeping journal, if I remember. But sometimes, I don't.

New Year's Day is a good day for optimists, and I continue to be one. You imagine a whole year laid out before you. You schedule successes and celebrations. You anticipate the positive results of challenges that you don't yet see coming. You hope that this warm winter weather is a seasonal fluke, and not the harbinger of irreversible climate change. Optimists aren't naive, we're just hopeful.

Even pessimists find a reason to be joyous on January 1st. If nothing else, it's a milestone at which you can stop and turn around, look at the road behind you, and be glad it's behind you. If nothing else.

Optimism isn't foolish head-in-the-clouds dimwittedness, though my cantankerous coworkers would certainly disagree. It's the result of a constant accounting of experience, of failures and defeats, of missed opportunities and unexpected wins. This optimist recounts these ups and downs on a near hourly basis, likely the result of undiagnosed ADHD and a persistent feeling of having left a pot on the burner last night.

My bees did not survive the winter. I've lost two hives in two seasons. But I'll try again this spring. Each failure instructs. Maybe the hives swarmed, and the frozen bees in the deeps were just the remnants of the colony. Or maybe I starved them because I don't yet understand how to sustain bees through the single-digit colds of the mid-Atlantic. My beekeeping gloves are turning a mesmerizing shade of amber, and my hive tool is starting to look like an old tool. I like how old tools look. I look at my hands, and they look like old tools, too.

I ran two half marathons in two weeks last fall: one on asphalt, the other on earth. Trail running is fine, but it's not for me and my declining vision. Depth perception is important when navigating roots and rocks, and I still haven't had my glasses repaired since I fell face-first in the single track around Loch Raven. My mind is twenty but my body is twice that and some.

Being an optimist means compiling a list of things that you'll likely fail at over the next twelve months, and doing them anyway. So raise a glass of the mead you'd like to brew in the fall. We drink, dear friends, to future failures.