Thursday, December 8, 2016

Ellicott City

I'm from Baltimore.

No, really. I mean, I don't have much of that ridiculous, sweet Baltimore twang when I speak. You'd have to really listen to me say give-away words like mojito and ocean to pick it up. Shit, I even pronounce the T in Baltimore; even Dominic West blurts out Bawlmer in a more authentic manner than I ever could. Colloquialisms like downy ocean... forget it. And I've never even considered referring to anyone as hon.

But I'm from Baltimore. At least, I was born there.

Until I turned 20, I lived in Ellicott City. It's a tiny suburb of Baltimore, one that drivers recognize primarily by the exit sign on I-95. You probably haven't given it any thought, really. And you probably think it's pronounced Elli-COT City. And you're wrong.

It's Elli-KIT.

Until the historic rain of Saturday, July 30th, Elliott City (more accurately, historic Elliott City, or Old Elliott City as we call it) wore all the adjectives you'd expect: sleepy, quaint, quirky, quiet. A comforting collection of cafes and consignment, all rolling down a curving hill towards the Patapsco river. Coffee and tea, boutiques, paintings and pottery, bottled beer, antique furniture. All of the shoppes you'd expect from a historic town that's older than our country.


But in the span of four hours, under the relentless advance of over 6 1/2" of rain, Elliott City sustained unimaginable damage. Roads, sidewalks, and subterranean structures were flooded and washed away. At the bottom of the hill, where Main Street intersects with Maryland Avenue, a location you may know because of the clock but I know because of the Phoenix, the falling rain accumulated into angry eddies the color of coffee with cream and carried cars into the river. And most tragically, two people lost their lives, overcome by the worst deluge the city had seen in over forty years.

On Sunday morning, we woke up to a new Old Elliott City.

Metaphor

If you've read enough American literature, you're probably already thinking that floods wash away the old to make room for the new. That floods are cleansing in nature, and are natural, and are part of the unforgiving circle of life. Floods are metaphors for the need to, on occasion, purge a place or a mind of its cobwebs and hangups.

But how can anyone argue that here, in the case of the small town that, four months after these rains, is still only partially reopened? There's no need for renewal in a district that prides itself on the historic. Old Ellicott City cannot be made new by external forces; its continued existence is in spite of these efforts. What makes Ellicott City forever young is its embrace of history. It's not irony. It's austere truth.

Hometown

I say I'm from Baltimore, but I've never lived there. And though I spent countless nights wandering Fells Point in search of oxblood docs after hitting Funk's Democratic Coffee Spot, the Bohemian Cafe, and the Daily Grind (all of which are ghosts of Baltimore now, except the Daily Grind, which grew up and left the chicken paintings behind), I don't hold any valid claim to growing up in the city. I always felt bad about listing Baltimore as my location on Twitter, until I deleted my Twitter account, that is.

I'm from Baltimore, unless you're from Baltimore. Then I'm from a million miles away.

I say I'm from Ellicott City, which is more true in that it's where my grandparents lived, and where my parents still live, in an untouched enclave surrounded by the "new" neighborhood that's been there for 30 years now, and where I lived for so long. But Ellicott City outside of Main Street has changed so rapidly and drastically, that now when I pass through or visit, it's like trying to remember a dream. Bits are familiar, yes there's where Donut World used to be, wasn't that Crab Shanty before? Commercial churn altered the landscape of route 40, and now I navigate by intersections, not storefronts.

And Old Ellicott City, which four months later has done the impossible and reopened many of its deluged shoppes, dried out its cafes, and allows pedestrians to wander all the way to the bottom of the hill to see the clock once again in its place. Old Ellicott City is different now. Washed away and rebuilt, perfectly if you don't know any better.

But stand at the bottom of the hill at night. Put your back to Cacao Lane and face south. You'll see lights on in Bean Hollow, which is still the Riverside to me, and contractors busily repairing the best coffee shop you've never visited. And if you can bear it, look left, where the Phoenix sits on the corner of Main and Maryland. I tried this three weeks ago, but couldn't bring myself to do it. It's like pausing "It's a Wonderful Life" just as George Bailey is on the bridge, ready to jump, before the snow starts to fall again. You can feel the emotion sneaking up, so you run.

I'm learning that home isn't a place, it's a feeling. Home is chatting with your forever ago friend in the basement of the rathskeller where Batman still serves up brews. Home is texting your dad and sister about trips to Walt Disney World from twenty five years ago. Home is telling your kids stories about when you were their age, riding store-brand BMX bikes through Turf Valley just to get to Boone's Lane. Home is buying a painting from your high school art teacher when you're forty years old and just watched your old stomping grounds get washed away, and holding on tight to a perfectly captured moment of Main Street in the light, Early Sunday Morning, at the top of the hill, before the rains, before the churn and turmoil and destruction pulled you twenty years into the past and left you there to find your way back on your own.

Early Sunday Morning

Tuesday, November 1, 2016

Sunset

Ranks of perpendicular lines cascade ever downward from the upper left corner of the monitor. Anemic arrows droop from the end of one line to the start of a new line below. Some lines are parallel; others serial. Angry words like analysis and requirements and discussion and testing precede some lines, while tepid words like development and deployment and documentation and design precede the rest. And all the while time marches to the east, in projected pursuit of a single, ultimate moment: your baby's death, neatly documented on a Gantt chart.

Maybe your baby is a mainframe: a hulking, brobdingnagian incarnation of ancient victories over mechanics, electrical engineering, and heat. Or maybe your baby is a farmette of Metaframe XP servers on Windows 2000 Server Advanced, erstwhile wonder of desktop abstraction and application management. But for many readers of this blog, your baby is vSphere.

vSphere disrupted the mid 2000's data center in ways we couldn't compare to other technologies at the time. Now, we'd likely call vSphere the Uber of the data center, or the Airbnb of the infrastructure. (Don't get lost in those examples and align them in the VM vs. container holy war). Or drop the metaphor entirely, and close your eyes and remember the last time you were as excited as when you vMotioned your first server.

Many of us saw virtualization for what it was: an opportunity to revolutionize the server delivery infrastructure. But let's be honest: it was also an opportunity to re-invent ourselves in the process. Personally, I went from being an Exchange and Windows geek to a vSphere geek nearly overnight. And the pride associated with building an agency's first vSphere cluster permanently attached a modicum of emotion to an otherwise apathetic project. Eight years ago, my baby was vSphere.

But in the immortal lifespan of IT, no single technology lasts long. The mainframe drove the need to create data centers half a century ago, cavernous, cold rooms which IBM ruled unopposed for decades. But when the microprocessor pushed the mainframe out, server sprawl consumed every bit of floor and rack space in the data center, and facilities managers world-wide rushed to send all of their idling big iron to the great surplus room in the sky (well, maybe they were just turned into razorblades, like battleships years after their violent tendencies succumbed to old age). In any event, revolutions occur in the data center seemingly overnight, and all it takes is one motivated engineer with an idea.

Once you've lived through a tech revolution, you acquire a new perspective on the rate of change in the modern data center. And you subconsciously carry that knowledge forward into every project you complete. You proceed to build sandcastles on the shore, knowing very well that one day a wave will break just right, swallow your castle, and carry it back into the ocean. If you're lucky, you'll get five years of usefulness out of your work.

This rate of change all but ensures that, in IT, nostalgia is irrelevant. IT serves the business, and when better, faster, more efficient methods of serving the business arrive, the instantly-obsolete method is decommissioned, surplussed, and forgotten. Mike Colson, currently with AWS but formerly with Solidfire, made this point in a Twitter thread two months ago:

Colson's correct; there's no room for nostalgia in IT.

Our industry thrives on what's next, what's ahead. The past is only relevant as a defense of the future state; because the past is old and dated, the future will be new and disruptive.

But there's a major flaw in this line of thinking: IT may principally serve business, but IT is built by people. And people are nothing if not nostalgic.

Over the next several days, I'll be posting vignettes on the people who were tasked with powering down and packing up various technologies that fell victim to extinction-level events in the contemporary data center. And maybe the next time we're ready to post another snarky tweet about the death of a vendor, or the passing of a product, we'll stop, take a deep breath, and remember that behind every product, there's a cluster of humans who poured the measure of their professional worth into that product's success.

Friday, October 21, 2016

When The Cloud is Down

Monday through Thursday, I leave the house well before dawn to make the 60 mile drive to my office. It's a quick and quiet drive, as long as I'm in the car by 5:45am. If you live outside of the Baltimore / Washington, D.C. region, you're probably thinking, "what in the holy hell are you doing driving 60 miles before the sun rises?" But my fellow DMV peeps certainly understand, and are likely nodding along with this declaration.

Friday, though, is different. I'm working from home, and my commute is shortened to about 30' as I shuffle to my desk and log in to start my day. A lurched, plaid caffeine addict inhabits the workspace, and like so many of you, I begin my day on Twitter.

Lately, I'm interested in the US general election (because I'm still hopeful we won't elect a raging narcissist puppet), so I scroll through political and press tweets about the latest debate debacles. I lol at #NastyWoman and #BadHombres, then get a sinking feeling that the center in American, possibly global, politics has been stolen by fanatics on both sides of the aisle.

So I move on to tech Twitter. The Nutanix IPO has me interested in that company again, although I'm still struggling to reconcile my admiration for their technology with my dislike for much of their corporate marketing. And their simmering beef with VMW evokes feelings of confusion and hopelessness, not unlike those emotions evinced by political observers this year. Is it ok to admire both Nutanix and VMware? Based on tweets from employees of both companies, you'd think the answer is no.

Fanatics and zealots, you're one or the other.

Is there an emoji for the utter lack of emotion due to shock?
Put that one here.
These Friday routines are sacred rites, accompanied by several large cups of coffee from a vendor's mug. Wake up, Twitter. Coffee, Twitter. Throughout the workday, Twitter. I didn't realize the extent to which I'm dependent on Twitter as a micro distraction until... it was down.

Twitter was down this morning, and it took about 10 minutes to accept that it wasn't a local problem.

Reflexively, and without thought, I attempted to share my frustration with the outage... on Twitter. "Oh, yeah," I thought, and, in quiet defeat, closed the Twitter client.

So what do I do? I don't use Facebook. I deleted Instagram from my phone again just last night because, well, I guess because I really don't like social media after all. And I'm decidedly too old for any of that Snapchat, WhatsApp, KiK, whatever-the-fuck-else social media apps are out there today. So when Twitter isn't an option, there is no option. It's vendor lock-in at a personal level.

Down Cloud

Clouds go up, clouds go down, such is the nature of technology. The most reliable system in the world can't meet 100%, no matter how loud the customers and CIOs scream. We all know this, and we're all happy to live in uptime denial nonetheless. "The free service I use extensively is down for a few minutes? UNBELIEVABLE CALL THE POLICE I'M DYING," we say. Or at least we think it.

To make things worse, one of the SaaS solutions we use at work, xMatters, also went offline this morning. And when xMatters isn't healthy, many of the system-generated alerts from various monitoring solutions do not get delivered. Of course, this is a paid service, so there's slightly less guilt involved.

Today is a much needed reminder that all services, on-prem, hybrid, cloud, shadow, you name it, are subject to failure. Moving your applications or services to another platform, or vendor, or solution, will never solve your problems, unless you are willing to honestly assess the faults in your own applications.

Where ever your apps go, there they are.

Monday, June 27, 2016

Moon Car

I love my car.

The black Volvo S40 that I lovingly refer to as "The Legend" joined our family eleven years ago. After we drove it home for the first time, the odometer displayed a dozen or so miles, which measured the distance from the dealership to our home. A pristine, beautiful, Swedish (well, kinda, with the exception of FoMoCo's under-the-hood meddling) thing. Flawless. Lustrous. Immaculate. All those tight adjectives applied, for The Legend was born in the summer of 2005.

Fast forward to 2016, and The Legend is still with us. But all of those youthful descriptors have been voided, the victims of time, weather, circumstance, and use. From a distance, it's the same small sedan as it ever was. But get closer, and you'll see what it's become.

The Legend is a mechanically functioning ghost of its former self. The radio works, but not the CD changer. Some speakers work, most do not. The power seat is powerless, stoic in its semi-reclined pose. (Luckily, it froze in an agreeable position for me.) The ceiling upholstery droops and sags, which is why I fired a few dozen staples upwards to delay the inevitable. And yesterday, after years of protracted wilting, the ceiling upholstery was unceremoniously ripped from the interior by your humble correspondent in a moment of clarity and/or rage. The trunk lid's light melted years ago. Don't ask. The glove compartment lock is broken, which created a time capsule of the late 2000s that will never be opened. Melted hard candies add muted color to an otherwise dreary gray back seat. The air conditioning taunts the car's occupants by secretly replacing cooled air with humid, warm air. Combined with a leaking moon roof, the devious AC creates a mobile rainforest in the interior of The Legend, an environment which is ideal for insects and certain fungal organisms that pose respiratory hazards for, well, humans.

Oh, and sometimes water shoots out of the floor vents.

Sic Transit Gloria

Persistence.
For a brief moment yesterday, I was ready to get rid of this car. The ensuing cascade of emotion started with anger, evolved into curiosity and excitement, morphed into resentment and disbelief, and finally settled for resignation. Kinda like this (warning, bad words ahead):

Fuck this car.
Ooo, it's time for a new car!
Fuck this, I don't want a new car.
It's fine. It's fine. I'll deal with it. It's fine.
Fuck it.

The Legend's glory faded long ago. The odometer currently indicates that the Legend has logged enough miles to get to the moon (at least when the moon is at its closest to the Earth, which is 225,623 miles). The Legend is the Moon Car. And it's got the Heritage Club emblems to prove it.

Roads

The problem with this car is that it's still working. It starts up every morning when I need to go to work. It starts up every evening when I want to go home. It doesn't complain. It just keeps running. Its primary function, transport, is reliably satisfied at every opportunity. Secondary functions, originally as reliable, are now just visiting, temporal pleasantries. But to say goodbye to this car now would be like burying your thirteen year-old cat while he still enjoys long naps in the summer sunshine. It's a grotesque thought that should make you recoil, not quicken your pulse with anticipation.

And so, The Legend lives on. Uncountable miles await ahead, laid out on familiar and unfamiliar roads. It keeps running. It keeps running. It keeps running.

Thursday, May 12, 2016

VMTools Installer Doesn't Appear on a Windows VM

Look. I know what you're thinking.

"Really? A blog post about VMTools? Good lord, man. Write about something interesting already."

Well, the truth is that what many of us consider boring, pedestrian, mundane, and trivial is actually a big deal to the vast majority of vSphere users out there. So shut up already. If you know everything, why are you even here in the first place? Shouldn't you be working on your second VCDX?

(NB: That's awfully defensive, I know. I'm probably sorry, if that makes you feel any better.)

On to the post.

Hey, VM admin! Have you ever encountered a VM that has a VMware Tools status of Not running (Not installed), selected "Install/Upgrade VMware Tools," and then wondered why the installer didn't launch on the VM's desktop? It's weird, right?

I recently ran into a strange configuration problem on three VMs that took me a few minutes to unravel. And because I haven't posted in a month, and because maybe someone else will run into the same issue and can save a minute, I'm sharing this story with you, dear readers.

The Symptom

You're logged in via Remote Desktop to a VM, and although you've initiated the VMTools installer from the vSphere Client (or Web Client, but you know you still use and love the vSphere Client), you don't see the nice little installer pop-up on the VM's desktop. You confirmed that the VMTools install is in progress from vSphere's perspective, and you can see that the ISO is connected and mapped to the image properly. And still, no trace of the installer on your VM.

In my case, it turns out that the virtual optical drive was disabled on these VMs. Further inspection revealed that the template had this configuration, so all VMs deployed from said template inherited this problem. Annoying.

The Fix

The fix is simple.

  1. Open Device Manager on your VM.
  2. Expand the DVD/CD-ROM drives element, and you'll see that the virtual devices is disabled (it's the little downward arrow that indicates a disabled state).
  3. Right-click that MFer and select Enable.
  4. Open Windows Explorer, and you'll see the VMTools image mounted and ready.

VMTools is good stuff. I mean, the drivers and all are great. And who doesn't love a little vmxnet3? But most importantly, you'll never have to CTRL + ALT out of your VM consoles ever again.

Sometimes, it's the little things.

Monday, April 11, 2016

HP VIB Depot Not Connected in Update Manager?

So I'm like way late to this party, but I thought I'd share anyway because I don't blog as often as I'd like these days because it's spring and the daffodils are blooming in the median strips and the sun shines just a little later in the day and because reasons.

vSphere Update Manager is awesome, let it be known. And you should spend more time looking at it, specifically the Admin View, because there is more to life than Compliance. And if you spend time looking at the Admin View, why not click over to the Configuration tab, and select Download Settings. Maybe, just maybe, you'll see something like this:


I've highlighted the interesting part. In this case, the Download Source for HP VIBs (that is, software bundles that are specific to your HP physical hosts) shows a Connectivity Status of "Not Connected." Oh, and bad news: it's been that way for a while now. Why? Because of the HP / HPE split that happened last fall. The fix is easy enough: just edit the Download Source and replace hp.com with hpe.com (and while you're at it, why not enable HTTP/S? Use https://vibsdepot.hpe.com/index.xml). Validate that URL, and you're back in the realm of the Connected.


Of course, the admins behind the old URL tried to tell you about the new URL, they just chose a really arcane method for doing it. Look carefully at the URL in the first screenshot: it's http://vibsdepot.hp.com//index.xml. Two slashes, yes. Go ahead and bring that up in a browser window.

Connected again, you may go forth and patch. Happy updating, intrepid vSphere administrator.

Tuesday, February 23, 2016

Microsoft SystemCenter 2012 R2: A Sour Suite

Microsoft SystemCenter 2012 R2 is a collection of systems management applications that aim to facilitate the administration of a Windows-centric technology infrastructure. Whether you're attempting to patch one thousand Windows Servers, or monitor the health of a dozen SQL servers, or just do some basic VM management on Hyper-V hosts, SystemCenter is a reasonable place to get these tasks done. At least, you'd think.

The truth is that the SystemCenter suite is quite sour; using the applications as standalone products is acceptable, but when you switch between several components during your work, you'll immediately get the feeling that Microsoft just blended a bunch of in-house tools with acquired products into a "suite" and rushed it out of the door. And I'm not just referring to the mismatched aesthetics of the thing. In some cases, what you would expect to be core functionality, like interoperability between SystemCenter components, is missing.

Example: Patch Management using SCCM and SCOM

First, a primer for those who are fortunate enough to not use SystemCenter on a regular basis.

  • SCCM - SystemCenter Configuration Manager. Think of this as a sophisticated software management tool. I'm not being sarcastic, for once. SCCM is decent.
  • SCOM - SystemCenter Operations Manager. Think of this as a sophisticated monitoring tool, minus the sophistication.
  • SCORCH - SystemCenter Orchestrator. It's okay to like this one, because it's functionally identical to pretty much every other orchestration tool that's on the market. And it's got a killer nickname to boot.
  • POSH - PowerShell. Wait, PowerShell isn't part of the SystemCenter suite, you say? Technically, you're right. But MSFT was smart to build modules for most of the SystemCenter components that lets a PowerShell-savvy engineer string cmdlets together to automate routine SystemCenter tasks. This is important, for reasons that will be totes obvs in just a minute.

So let's say that you've got 100 Windows Server that you're managing, which means you are responsible for the availability, performance, and security of these systems. Your Microsoft account team was slick enough to bundle SystemCenter into your EA, so you've got access to SCOM and SCCM. You enable SCOM monitoring for your servers, and you validate that you're getting health and performance data.

When it's time to roll out patches, you decide to lean on SCCM. You create a few collections, assign them to groups of servers, and you schedule the deployment.

Before you deploy the patches, however, you have a great idea: I'll go back into SCOM and enable Maintenance Mode  (hereafter MM because lazy) for all of my servers. That way, as they are being patched and rebooted, I won't be flooded with alerts from SCOM. And you can do that.

But you're going to run into a problem. Because you're a good engineer, and you like to automate stuff, and you like to work smart, and you like to use the tools you have to the best of their ability. So you think to yourself, "I bet I can configure SCCM in such a way as to put the servers into MM as part of the patch deployment." I mean, of course you think this. You'd be silly not to. Because you're working with two components of a single "suite." And after years of experience in IT, you know that suite means a collection of tools that are tightly integrated in order to manage the full spectrum of tasks.

The problem is that you can't do what you want to do in SystemCenter 2012R2.

You can't have SCCM put a server in MM, even though you will certainly fall for the trap MSFT has set for you:


The problem here is that these options, specifically the first, are described quite literally. In your haste, you may think that selecting the first option (Disable Operations Manager alerts while software updates run) will effectively put the servers in MM before patching. But no. This option will only prevent SCOM alerts while the updates are running. We all know that patch installations require reboots, and it's these reboots that will still cause SCOM to generate an alarm, even with this option selected.

Microsoft should have included an option to put the servers in MM while the patches are deployed. I mean, remember that SystemCenter is a suite. Its components should seamlessly interoperate. Is that a word? It should be.

POSH: Gluing Together SystemCenter's Components

PowerShell is amazing, for myriad reasons. Among these reasons: it's easy enough for anyone to learn, even people who are too lazy to learn Perl (and admit it: you've got to be pretty lazy to not know Perl).

Because there are PowerShell modules for SCCM and SCOM, you can pull together some one liners to do what the GUIs in these products can't: quickly and easily place groups of servers into MM for a duration of your choosing. The syntax of the command.... isn't the point of this post. I mean, if you really want to know, just google it. A dozen other posts can explain that for you. But you don't need that info; you're smart. Just do what I do and tab-complete your way to PowerShell god mode.

The point of this post is to highlight the obvious shortfall of the SystemCenter 2012 product: it's a suite of applications with half-assed interoperability and integrations that prevent you from working smartly and efficiently. And it depends heavily on PowerShell to glue these disjointed pieces together. Sure, you can use SCORCH as a band-aid, but you shouldn't need to.

Hope For The Future: SystemCenter 2016

"The next version will be better." ~every vendor, every release
2012R2 is a bit long in the tooth, and its replacement is preparing for its debut. With luck, Microsoft has addressed the core issue here: unify these components into a true suite of components that work well without any POSH magic. Time will tell.

And maybe this post is really just a bit of catharsis after wrestling with this stupid software for a few months now.

Tuesday, February 9, 2016

Temporary Maintenance

Pretty sure that the change manager who approved a "temporary maintenance" that collides with the VMware event today has been sacked.


Thursday, January 28, 2016

Snowbound: Thoughts on Automation

Baltimore, Washington, D.C., and the corridor received record snowfall over the weekend. I measured 28" in a 24 hour period, but measuring snowfall is notoriously difficult and inaccurate. Some locals reported paltry totals in the teens, while others likely exaggerated 36" depths and greater. In any event, the demi-mountains of snow piled along streets and roads confirm non-trivial totals, regardless of the accuracy of the measurement.

In previous years, when faced with the task of removing snow from the driveway, I had two options: retrieve the shovel from the garage and start hefting loads of snow; or cross my fingers and hope that the friendly neighbor would offer to clear the drive with his snow thrower. Usually, it'd be a combination of both: I'd start by struggling with the shovel, and in an act of pity, the neighbor would materialize at the end of the driveway to help out. Well, maybe not pity. He's genuinely friendly, the neighbor.

I haven't left the house since Friday.
This year, I decided it was time to stop relying on the kindness of neighbors; I joined the crowd of consumers who rushed to the local Sears to buy a snow thrower. It's not the best, not the worst, just a tool to facilitate the wintery ritual of moving snow from here to there. There's a photograph of the machine, with your humble correspondent dressed for the weather.

Moving snow from here to there with the snow thrower is stupid easy. Learning to use the machine took all of 10 minutes, after which time it was a matter of following a short litany of steps:

  1. Point machine at snow
  2. Engage auger
  3. Engage drive
  4. Keep up
Aside from totally confusing the hell out of my Apple Watch (which recorded the sloth-like advances of the machine and I as "exercise"), the process of moving snow with the machine was so easy that I found myself considering the state of automation in IT, which is something that no one is thinking, talking, tweeting, blogging, podcasting, speaking, or otherwise considering at all.

And yet.

The Machine and the Shovel

Before the snow thrower joined the growing collection of seasonal machines in my garage, there was the humble shovel. Humble as in: I bought it at the grocery store a few years ago. Its only metal component is the screw which fastens the handle to the blade; the rest is various plastics. It is an inexpensive thing, but it satisfies my requirements, so it earns its place in the garage.

The shovel served its purpose well. The problem was that I couldn't operate the shovel for the duration of the task: cleaning the driveway takes hours, and I'm not a millennial or anything. In past years when I did manage to complete the task without aid from the friendly neighbor, the job was 100% complete. Drive clear, mailbox accessible, front walk passable... basically, a peninsula of asphalt surrounded by a snowy sea. It was a thing to behold, if contrast and clean lines speak to you.

This past week, I observed, scientifically, that using the machine to move snow was approximately 10000% easier than using the shovel. What had taken hours in the past now took 30 minutes, and that's rounding up a bit. The machine is superior with regard to speed, and reduced my effort from humping 50 pound scoops to meandering up and down the drive. The shortfall of the machine, however, is in its ability to handle the details of the chore: when I finished with the machine, the job was about 90% complete. Drive clear, except for the bits the machine couldn't reach close to the garage, close to the front walk, and close to the mailbox. So I found myself putting the machine back into the garage and reaching for the shovel.

The machine made 90% of the work easy, and was utterly inept at the remaining 10%.

The Low A: 90% is Good Enough

The machine's 90% mark is good enough for an A grade, albeit on the low end. And its ineptitude is intentional, not a shortcoming. To make a single machine that is capable of completing 100% of this work would require every possible snow-moving scenario to be both considered and addressed; such an effort would consume massive quantities of time and money, and the cost of completing the remaining 10% would render the machine a financial failure.

We're in the realm of the Pareto principle, for certain.

Automation in IT is no different. Automating 90% of infrastructure tasks is easy. I mean, how many times do we deploy a switch, a VM, a container, an application, a whole infrastructure in the exact same way? Most of the time, I'd wager. So automating that chunk of work is worthwhile. Time well spent. Money invested. But automating the atypical configurations that only a small minority of use cases require? That's rarely worth either the time or money.

Let's be honest here: I'm talking about what happens to people in IT. We've heard the chicken little types for years now, clamoring on about the sky (cloud?) is falling and that we'll all be replaced by clever scripts. But that's not true at all. While much of our work will be automated, infrastructure workers will fall into two categories: the automation junkies, and the "10 percenters." To suggest that either group is more important is to have no sense of history, not just in tech but in Western civilization.

The invention of the snow thrower did not obsolete the snow shovel. And automation in IT will never obsolete the need for craft IT.