Tuesday, December 23, 2014

What We Talk About When We Talk About Storage

Thanks to a deluge of storage marketing in the last year or two, we have adopted the opinion that flash storage will push spinning disks out of the data center. We even picked up on the term “spinning rust” and “mechanical disks” that many vendors use when pejoratively referring to that IT stalwart, the hard disk drive. And with an all-flash world in our minds, we frame storage decisions in quantities of performance and capacity.

But we’re doing it wrong. And it’s not our fault.

Have we all been conditioned to ask the wrong questions when it comes to storage? Or more to the point, have we been conditioned to skip right to the bits that are required for placing an order: how much, and how fast? We don’t even consider what we’re storing.

The rate at which data is created is staggeringly unfathomable. No, really. Sure, you can read a statistic (from IBM, so it carries a significant amount of credibility) like, “…every day, 2.5 billion gigabytes of data is made around the world.” Yes, that’s 2.4 exabytes, and no, it’s almost impossible to understand how much data that is. You’d need to manufacture 2.4 million 1TB drives every day to just meet the current demand. At some point, and this point may already be in our rear-view mirror, data growth will outpace our ability to manufacture media.

We take this information with us when we jump to the following conclusion: we need more storage. We need bigger drives. We need bigger arrays. We need more compression and deduplication and virtualization and thin provisioning. We need a home for all this data.

But before we come up with capacity requirements, consider this: our data is valuable. It may be intellectual property, financial, personally identifiable, or any of the myriad other classifications we’ve devised over the years. We may have a dozen Office documents that hold the figurative keys to the corporate kingdom. We may have a few databases that contain information on every customer you’ve worked with. We might have some text files that have all of your IT department’s passwords. Just kidding, no one does that. :)

Now comes the hard part: we need to accept that non-trivial amounts of our data may be, well, worthless. Employees store all kinds of media that have no value to the business. In fact, some of that data may expose the business to unwanted attention from ambitious legal and law enforcement types.

Maybe instead of focusing on a tiering model that’s based primarily (if not exclusively) on performance, we should tier our storage based on data value. Maybe “where do I store all of this data?” is begging the question: do we need to store all of this data? After all, the data holds value. Storage is just the cost of holding that data.

Of course, a value-based tiering model requires quantifying our data’s value. Or, it might require that we have a storage solution that can identify types of data and categorize them automatically. Either way, we will need to take a wide-eyed, new look at our corporate data. And more importantly, we need to change what we talk about when we talk about storage.

NB: This post is part of the NexGen sponsored Tech Talk series and originally appeared on GestaltIT.com. For more information on this topic, please see the rest of the series HERE. To learn more about NexGen’s Architecture, please visit http://nexgenstorage.com/products/.

Monday, December 1, 2014

Perceived Security & The Home Depot

An NCR POS terminal at The Home Depot.
Lots of posts and tweets and chats about perception lately. Frank Denneman asked if people still perceived a single chassis as a risk and sparked a Twitter discussion that I was ill-prepared for at 5:00am my time. I wrote about seeming vs. being in my post about Hamlet and Coffee Cup Sleeves. And just last night, while I was making a last-minute dash to The Home Depot to get some more Christmas lights, I was left with the perception that Home Depot just doesn't get security. Just look at this photo --->

Yes, that's a self check-out POS terminal running Windows XP. And yes, The Home Depot frequently runs into trouble with the security of its POS terminals.

Of course I took a picture. What kind of blogger would I be if I let this one slide? Here we have one of the biggest companies in the country, fresh off one of the most well-documented and journalized credit card thefts to date, running a version of Windows that is older than Twitter and is no longer supported by the vendor. It screams irony, right?


This is where perceived security comes into play. I was quick to criticize HD for learning nothing from the last ~twelve months of headlines regarding credit card thefts due to infected POS terminals. Have you learned nothing? How can HD continue to use a known-vulnerable operating system to process payments? But @stvkpln reminded me that, for large corporations with tens of thousands of terminals across the country, upgrading and / or migrating to a new system is a non-trivial task. It's not just a patch you push out over the weekend. And he's right; not matter how urgent the need may be, there's nothing worse than rushing a solution into production before it's properly thought out. I call that duress-driven design, and I've had a post in Drafts for a month now on that topic.

We agreed, however, that HD and its business partners (NCR in this particular case) need to address the issue before they suffer another attack.

But this post is about perception, and in this case, it's the perception I'm left with from this experience that is arguably more important than the reality. Metaphysically, we'll never know reality; we live in a world of shadows cast on a cave wall. The reality may be that HD is actively working towards resolving their security issues. They may be blazing a new trail in information security and innovating standards for credit card protection. But all it took was a single terminal with the iconic bliss desktop image, and my perception was set: The Home Depot still doesn't get security.

Thursday, November 20, 2014

No Service Stands Alone

I recently had the pleasure of attending a "lessons learned" meeting with two dozen colleagues. We've just recovered from a service interruption at work, and our CIO arranged for us to spend some quality time discussing what worked well and what didn't. The discussion was very productive, thanks in part to some ground rules that prevented us from defending our actions. As far as two hour meetings go, this one wasn't too bad.

Credit: Danby @ BDN
I'll save you the sordid details of the service interruption. Suffice to say that members from many technical disciplines were present; this service in question required infrastructure from across the IT spectrum. The group observed that, because the service was distributed, restoring service required collaboration between teams. In Washington, D.C., we call that "reaching across the aisle."

At this point in the discussion, the CIO interrupted us to make a simple, seemingly obvious, statement.

No service stands alone.

It was one of those moments that shocked us into silence. Among her many skills, our CIO can command a silence like no one else. The silence lasted 5 seconds, but you'd think it was a year. Yeah, she's that good.

But her point is worth considering. In any enterprise, no service worth providing lives entirely in the orthogonal confines of an organizational chart. Services span teams and technologies, and live and die by the success of each individual component.

That's why technology professionals must change their perspective and see the services that they support. Your storage engineer can't be blind to the Exchange workloads hosted on the SAN. Your vSphere administrator can't be blissfully unaware of the impact of slow performance on hosted applications. And everyone who shares partial responsibility for a mission-critical service must be accountable for its availability.

Service Monitoring

A deceptively easy way to change your perspective is to leverage monitoring tools that are "service aware." That is, in addition to monitoring individual devices like switches and servers, these tools can group devices into a "service." In EM7 vernacular, this is called an IT Service. SolarWinds is going to tackle this notion with their AppStack solution. In fact, I'd bet that most modern monitoring solutions have a similar capability. It's a logical evolution, after all.

So stop pretending that the infrastructure you support exists for any other reason than to enable services. And start putting the health and performance of the service first.

Sunday, November 9, 2014

Unperceived Existence of Collected Data

Can something exist without being perceived?

No, this blog hasn't taken a turn into the maddening world of metaphysics. Not yet.

I'm talking about event and performance logging, naturally. In the infrastructure racket profession (well, I think it's a vocation, but I'll get to that in a later post), we're conditioned to set up logging for all of our systems. It's usually for the following reasons:
  1. You were told to do it.
  3. Security told you to do it.
So you dutifully, begrudgingly configure your remote log hosts, or you deploy your logging agents, or you do some other manner of configuration to enable logging to satisfy a requirement. And then you go about your job. Easy.

But what about that data? Where does it go? And what happens to it once it's there?

Do you use any tools to exploit that data? Or does it just consume blocks on a spinning piece of rust in your data center? Have I asked enough rhetorical questions yet?

*   *   *

The pragmatic engineer seeks to acquire knowledge of all of her or his systems, and in times of service degradation or outage, such knowledge can reduce downtime. But knowledge of a system typically requires an understanding of "normal" performance. And that understanding can only come from the analysis of collected events and performance data.

If you send your performance data, for example, to a logging system that is incapable of presenting and analyzing that data, then what's the point of logging in the first place? If you can't put that data to work, and exploit the data to make informed decisions about your infrastructure, what's the point? Why collect data if you have no intent (or capacity) to use it?

Dashboarding for Fun and Profit (but mostly for Fun)

One great way to make your data meaningful is to present it in the only way that those management-types know: dashboards. It's okay if you just rolled your eyes. The word "dashboard" was murdered by marketing in the last 10 years. And what a shame. Because we all stare at a dashboard while we're driving to and from work, and we likely don't realize how powerful it is to have all of the information we need to make decisions about how we drive right in front of us. The same should be true for your dashboards at work.

So here are a few tips for you, dear readers, to guide you in the creation of meaningful dashboards:

  1. Present the data you need, not the data you want. It's easy to start throwing every metric you have available at your dashboard. And most tools will allow you to do so. You certainly won't get an error that says, "dude, lay off the metrics." But just because you can display certain metrics, doesn't mean you should. For example, CPU and memory % utilization are dashboard stalwarts. Use them whenever you need a quick sense of health for a device. But do you really need to display your disk queue length for every system on the main dashboard? No.
  2. Less is more. Be selective not only in the types of data you present, but also in the quantity of data you present. Avoid filling every pixel with a gauge or bar chart; these aren't Victorian works, and horror vacui does not apply here. When you develop a dashboard, you're crossing into the realm of information architecture and design. Build your spaces carefully.
  3. Know your audience. You'll recall that I called out the "management-types" when talking about the intended audience for your dashboards. That was intentional. Hard-nosed engineers are often content with function over form; personally, I'll take a shell with grep, sed, and awk and I can make /var/log beg for mercy. But The Suits want form over function. So make the data work for them.
  4. Think services, not servers. When you spend your 8 hours a days managing hosts and devices, you tend to think about the infrastructure as a collection of servers, switches, storage, and software. But dashboards should focus on the services that these devices, when cooperating, provide. Again, The Suits don't care if srvw2k8r2xcmlbx01 is running at 100% CPU; they care that email for the Director's office just went down.
Don't ignore the dashboard functionality of your monitoring solution just because you're tired of hearing your account rep say "dashboard" so many times that the word loses all meaning. When used properly, and with a little bit of work on your part, a dashboard can put all of that event and performance data to work.

Sunday, October 19, 2014

TFDx Roundtable Discussion at Interop New York 2014

Did you catch the Tech Field Day Delegate Roundtable discussion from Interop New York this year? If not, you'll want to watch the replay. We discuss the state of technology conferences, career development for new IT professionals, and why asking the right question is so important.

Interop is the last large-scale technology conference that isn't explicitly tied to a single vendor. And that's significant; in the last two years, we've seen new tech conferences pop up that are hyper-focused on specific software. VeeamOn is a good example (I expect Veeam fanboys to furrow their brow with this example). We've seen events like Cisco Live and VMworld balloon into the chaotic frenzies they are. And we've seen independent conferences like Interop dwindle in participation in spite of the many wonderful opportunities to learn and network with great engineers from all over the IT spectrum.

Tech Field Day invited me (along with much more knowledgeable folks: Howard Marks, John Herbert, Tony Matke, Jason Edelman, and our indefatigable minder and moderator Tom Hollingsworth) to attend Interop and hear about innovations from Cisco, Glue Networks, Akamai, and HP Networking. The roundtable discussion is a bonus, and in my opinion it's what makes a TFD event unique.

Yes, yes: TFD paid my travel and lodging and meal expenses while in NYC. 

Saturday, October 18, 2014

How to Suppress Those Awful Ask.com Options When Installing Java

Let's start off with some categorical truths: everyone hates Java, and everyone hates Ask.com. Here's another one: everyone uses Java, and no one uses Ask.com. No one, that is, except for the unlucky souls who inadvertently install the Ask.com browser add-ons and set it to their default search engine during the Java installer. Poor bastards.

I only deal with Java these days because my kids play Minecraft. A lot. Not as much as this dude, but still. And since Java receives security patches on a frequent basis (we'll call it weekly, and we won't be far off), I spend time updating Java more often than I'd like. I grew tired of the gratuitous attempts by Oracle to install Ask.com software for you in an opt-out manner. And I'm not alone: many of us have banded together to petition Larry Ellison to stop bundling Ask Toolbar with the Java installer. Sadly, only ~20,000 people of the Internet have signed up. So.... yeah.

Until then, here's what you can do to prevent seeing that damned Ask.com screen during the installer.

  1. Open your Java Control Panel.
  2. Click the Advanced tab.
  3. Scroll down to the bottom, and find the Miscellaneous section.
  4. Click the "Suppress sponsor offers when installing or updating Java" box.
  5. Click OK.
Now you won't be bothered with that crap Oracle tries to force on you when you install or update future versions of Java.

You're welcome.

Wednesday, October 15, 2014

Dr. Vint Cerf's Keynote Address at the HHS IPv6 Symposium

ICYMI: Dr. Vint Cerf delivered a keynote address to the HHS IPv6 Symposium at the National Library of Medicine at NIH this morning. Dr. Cerf's contributions to the Internet are nearly unbelievable: he co-created TCP/IP, and can get away with saying things like, "When I turned the Internet on in January of 1983." Do yourself a favor and read his Google Research profile; he's currently Vice President and Chief Internet Evangelist at Google.

Dr. Cerf spoke about some of the barriers to IPv6 adoption in both the consumer and business / federal spaces. In both cases, it's a matter of need: if technologists don't pound on tables and demand IPv6 from Internet service providers and hardware / software vendors, we'll be discussing this migration for another 16 years. He cited Comcast as an example of an ISP who is leading the way with their ambitious IPv6 adoption program.

The Internet of Things (IoT for the initialization crowd) is suddenly a major driver for IPv6: with the future of sensors embedded in everything, addressing everything is only possible with IPv6. We exhausted IPv4 addresses with the devices of today; devices of tomorrow need v6.

After the address, Dr. Cerf was kind enough to field questions from the audience. Two questions in particular elicited interesting responses: the first was a question about whether IPv6 had baked-in security features that would serve to encourage faster adoption. I expected the answer to be yes, but Dr. Cerf said that this was not true, and instead it's a widely held assumption of IPv6. And the second question was, "what's next after IPv6?" The answer took an interesting turn: the development of an interplanetary network protocol, and why TCP/IP doesn't work with RTTs that exceed 7 minutes. Once again, in space, the speed of light is just too slow.

Finally, Dr. Cerf talked about his new project: the preservation of digital information. He cited some examples from our lifetime of data storage technology that is obsolete: 5 1/4" floppies, VHS cassettes, and (soon!) polycarbonate discs. In many cases, the challenge is more than just preserving the data; reading from the media, and being able to reliably interpret the data on the media, also pose great risks to the life of stored data.

I'll try to make it back this afternoon for the CIO's presentation on NIH's IPv6 Project. The monsoon-like rains may keep me at my desk. But never fear: the event is videocast here.

Tuesday, September 30, 2014

Interop New York 2014

As luck would have it, I once again find myself traveling to New York City for yet another Tech Field Day Extra event: Interop New York!

Helpful sign is helpful.
I've not attended Interop before, so I'm excited to see what a conference that isn't focused solely on virtualization has to offer. I'll admit that I'm already regretting bringing my VMware laptop bag; maybe I'll put a black Sharpie to good use and remedy that situation. But maybe I don't need to obscure my virtualization vendor of choice; a quick glance at Wednesday's session listing reveals titles such as "'Parley!' - The Rules of Network Automation" and "Network Function Virtualization." At this rate, one day we'll no longer be able to distinguish between network- and virtualization-centric conferences. They'll both be directed at programmers. Those of us who don't learn to program and script will be relegated to cranking out "Pepperidge Farm Remembers" memes while we lament the end of the sysadmin as we knew it.

Until then, and certainly for the next two days, I'm going to be learning as much as possible from my fellow delegates for TFDx. And of course, tomorrow afternoon's presentations from HP and Cisco promise to be intriguing, with possible topics that include intelligent WAN and the launch of the SDN App Store. Throw in a little roundtable discussion with friends, and you've got the recipe for TFDx!

My editor (ok, it's me) just pointed out that three exclamation marks punctuate this humble blog post. That's a good indication that I need sleep. So I'm off to find coffee instead. If you see me wandering around this week, grab me and say hello!

Friday, September 19, 2014

Hamlet & Coffee Cup Sleeves

“Seems,” madam? Nay, it is. I know not “seems.”

Remember that Marathon level? Yeah.
With these words, Prince Hamlet gives us a glimpse into his grand deception. Hamlet is, among many things, a play about seeming versus being. Is he truly mad with loss and anger, or is he acting mad in order to exact revenge? And does it even make a difference?

I'm not sure why, but this is one of those lines that goes through my mind on a daily basis. Being vs. seeming. Seeming vs. being. It's partially a remnant of the many wonderful literature courses I took at Maryland so long ago. But it's also because I go to the same shitty coffee shop every morning to get a cup of "coffee." And I have ritualized the process so effectively that I don't think about what I'm doing. I just get the largest cup in the stacks, grab a coffee cup sleeve, pour in some half&half (before the coffee so I don't have to stir it), then fill up with "coffee." Pop on a lid (careful to align the sipping portal with the cup's edge opposite of the seam). Pay and I'm off.

But the sleeve is worth a bit of consideration. Because it looks like a sleeve at Starbucks. It seems like a sleeve at Starbucks. But it is not a functional equivalent.


How Coffee Cup Sleeves Work

Sleeves are made from cardboard. But cardboard is a terrible insulator on its own. That's why you'll notice that Starbucks uses sleeves that are corrugated. And it's the corrugation that provides insulation via the trapped air within the fluting. It's the same principle as fiberglass insulation: glass is a terrible insulator (that's why glass coffee mugs never took off), but fiberglass traps a big layer of dead air that is a wonderful insulator. Air is the insulator. The corrugated cardboard just creates a layer of air.

But the shitty coffee shop that I frequent stocks coffee cup sleeves that are just a thin layer of cardboard; there's no corrugation. It's worth noting that, when cardboard is in direct contact with the coffee cup, it acts as a conductor, not an insulator. So using a sleeve that has no corrugation (or any other means of trapping a layer of air) has zero functional value. It seems to be a coffee sleeve, but it certainly is not.

What's the Point?

The point is that, in a rush to imitate, many products end up mimicking the form of a similar product, but completely miss the function. Seeming and being are very, very different states. Be mindful of mistaking one for the other.

And the other point is that the temperature of the coffee is too damn high.

Sunday, September 14, 2014

UCS M-Series: The Server, Deconstructed

Last week, I attended the Cisco #UCSGrandSlam product launch in NYC as a member of the Tech Field Day delegation. I'm sure you followed the excitement on social media; the @CiscoDC team really knows how to promote their events! And you've likely read some great blog posts from some of the other delegates, including Chris Wahl and Jason Edelman.

To sum up the event, Cisco unveiled two new breeds of server hardware: the M-Series and the C-Series. The C-series server is a beast of a rack-mounted server with enough drive bays that you'd be forgiven for calling it a drive tray. And the M-series represents, in Cisco's terms, the "dis-aggregated server." While the products were certainly provocative and riddled with innovation, I left the press event with one question: what is a server?

For some reason, the first thing that popped into my head during the press event was the Scion xB. Well, not just the Scion xB, but this dead-on review of the auto from slate.com. Specifically, the article correctly correlates the existence of the xB with the "[complete] deconstruction of the SUV." Briefly, the Scion xB has most of the attributes that those bulky boat-show SUVs have, but the sum of the parts is somehow not an SUV.

That's what the M-series is to me. It's the complete deconstruction of a server. Servers are no longer autonomous sheet metal monsters that have static and rigid identities. You can no longer walk into any modern data center and point to your server. It could be anywhere, including... not in your data center. So the M-series takes this notion to heart with its decoupling of CPU and memory from drive controllers and vNICs & vHBAs. Take a look at this thing with the cover off. You'll marvel at it and wonder what the hell is going on here. Where are the VICs? (Hint: there's only one, and it's a monster.) What are those four SSDs? (Hint: they're presented to the nodes as local storage.) How the hell am I going to manage this thing? (Hint: UCSM.)

Sorry I'm late getting this post out. But I was sidetracked when the latest TechReckoning arrived in my Inbox this week, and it was coincidentally titled "Not Your Grandmas x86 Box." Sidetracked, because that's pretty much what I wanted to say, too. Maybe Troyer is  right; maybe the term "server" is dead. Cisco calls them "nodes" and "cartridges" now. We've colloquially referred to them as boxes for years. Maybe it's time to put the "server" out to pasture.

Thursday, September 11, 2014

Query NTP Settings on ESXi 5.x Hosts

Short and sweet today. We needed to report on the NTP settings for our ESXi 5.0 hosts. We've got enough hosts to justify wrestling with PowerCLI a few minutes (okay, more like 30 minutes), as opposed to clicking through the dozens of screens necessary in the vSphere client.

Specifically, we wanted a list of the hosts, the NTP servers they were using for time sources, how the service was set to start (the Policy), and whether the service was running. Here it is:

The command will output a report with Name, NTP Server, Policy, and ServiceRunning columns for each VMHost in your vCenter. It's perfect if you're looking to verify that each host has been set identically.

Wednesday, September 10, 2014

Syslog on ESXi 5.0 Hosts

So you want to enable syslog on your ESXi 5.0 hosts? Or you just want to read a blogpost about doing that? Either way is cool with me.

Setting up syslog is easy to do. But doing so requires some attention to detail, because this change is enabled in several places via the vSphere Client. In short form:
  1. Configuring syslog in the host's Advanced Options screen
  2. Configuring the firewall rules for your ESXi hosts
I'll show you the click-happy way to do this, then we'll do a little PowerCLI that will do the same thing while you go get some coffee.

Configuring syslog in the host's Advanced Options screen

The Config.HostAgent.log.level screen
You'll need to configure, at a minimum, three advanced options for each host that will be sending syslog: the hostAgent log level, the Vpxa config log level, and the remote syslog host. Start by selecting your host, then clicking the Configuration tab, and dig into the juicy advanced settings.

Your first stop is to set your logging level for the Config.HostAgent.log.level property. See the image to the right for available options. I suggest using warning; info might be more logging information than you need. However, some environments may elect to capture as much data as possible and filter it at the syslog server level. That's fine, too. Just be prepared for a non-trivial increase in logging when you go to info or higher.
The Vpx.Vpxa.config.log.level screen

Next, we need to do the same for the vCenter agent (aka Vpx.Vpxa.config.log.level) logging level. Check the screenshot to the right for the exact location. The same advice applies here: set it to warning, unless you really need more logging information.

You'll be tempted to check your syslog host for log data at this point. Don't. You'll only be disappointed, and perhaps slightly confused. Because the trusty ESXi firewall is dutifully blocking syslog traffic. So let's fix that.

Configuring the firewall rules for your ESXi hosts

The trusty ESXi Firewall settings screen
On your Configuration tab, click the Security Profile option, and select the Firewall's Properties. Scroll down a bit, and you'll see that the option for syslog is unchecked. Easy fix: click the empty box to the left of syslog, and then click OK. You'll see a task in the vSphere client that says, "Opening firewall port." And when it's done, you're done. Easy. But damn that's a lot of clicking. There's got to be an easier way, right? Right?

The Easier Way - PowerCLI

You're damned right there's an easier way. Just launch PowerCLI (or alt-tab to it, since you shouldn't ever close PowerCLI anyway) and let's see how easy this is.

Here's the command you'll want to use to configure syslog. It will configure every host in your vCenter Server (via the get-vmhost cmdlet) to use the "warning" log level for your Host and vCenter agent logging, and will send syslog to the host at (probably not what you want, so make sure to change this to your syslog server).

Now, we just need a one-liner to open up that firewall rule for syslog. Easy.

Aaaaaaand you're done. But did you notice that warning in PowerCLI about the set-VMHostAdvancedConfiguration cmdlet being deprecated? Yeah, me too. But I'll write that up tomorrow. It's time for baseball practice.

Sunday, August 31, 2014

DataGravity - An Intelligence Platform for Your Data

By now, you've heard of DataGravity. They just won the Best of VMworld 2014 awards for two categories: New Technology and Best of Show. If you weren't able to attend VMworld in San Francisco this year, you should do yourself a favor and check out the DataGravity Discovery Series demo. Because this company is not just creating a new storage product, they're creating a new platform.

The core principle that drives DataGravity's development is this: you should expect more from your storage. But I like what I heard Paula Long say during DataGravity's presentation to the Tech Field Day Extra delegation on Monday: "You bought the storage, you should know what's in it."

The Discovery Series product provides the storage services like most other networked storage solutions: SMB, NFS, and iSCSI. Nothing too shocking there (well, not that I'll get into now). But the unique power of this product is in the data-aware platform they've created. We've been conditioned to ask only a few basic questions from our storage:

  • How much space are we consuming?
  • How much space do we have left?
  • How many IOPS can the storage handle?

We've limited the interrogation of our storage systems because these are the only questions for which they have answers. DataGravity's intelligence engine runs on the "passive" storage controller, and captures and analyzes meta-data on a dedicated pool of disks. That means that there's no performance penalty to be paid, and you'll have access to data about your data. Initially, the platform looks for five tags within your data: credit card numbers, social security numbers, URIs (or is it URLs?), IPv4 addresses, and email addresses. (Future releases will include the ability to create custom tags for your data). But think of what you can learn by just looking for these five bits of data.

Take this concept one step further: you can also create an audit trail of which users access which data (this requires the installation of an agent to associate SIDs with human-readable names). Now you know what's on your storage, and who is accessing it. For those of us who spend any measurable amount of time managing storage, this is a big pivot in how we think about data.

One final thought: Paula Long is very aware of where this technology is going. She put it this way: "We didn't build a bunch of features into the array. We actually built a bunch of capabilities into the array, which we can build features from." To me, that sounds like DataGravity knows their innovations will create a community of users who find new use cases for this technology. And that's when this platform will take off.

Thursday, August 28, 2014

EVO:Rail Announcement at VMworld 2014

An anagram for Love Air. :)
At Monday's VMworld General Session, VMware announced the next phase in the journey to the Software-Defined Data Center: EVO:Rail. It's kind of a big deal, too. Here's why.

Validation of Hyperconvergence

First and foremost, EVO:Rail's existence validates the trend in IT over the last two years to build self-contained, purpose-built platforms for virtualization. Many companies, such as SimpliVity and Nutanix, successfully merged compute and storage into a single hardware appliance. SimpliVity took that notion a step farther and bundled some networking magic (such as WAN optimization). But the networking stack was conspicuously absent from these converged solutions. And, no, VCE's VBlock doesn't belong in this conversation because it is NOT a converged anything.

EVO:Rail is different because it's not just hyperconverged at the hardware layer: it's hyperconverged software, too. You'll get vSphere and Virtual SAN, with a beautifully simple HTML5 interface for doing the initial configuration. For me, that means that I don't need to spin up my Fusion Windows 8.1 VM just to manage EVO:Rail. Win! (And thanks to Gabriel Chapman for correcting my error: EVO:Rack, and not EVO:Rail, that will ship later includes NSX.)

A New Unit for Infrastructure

A while back, I wrote a post about hyperconvergence: The Hyperconverged Hamburger. One of the main concepts in that post is that we need to stop treating virtualization as the collection of server, network, and storage hardware. Instead, we need to treat virtualization as a component, not a confluence of components. This is where EVO:Rail shines: you buy EVO:Rail, and you get virtualization. Period. You can quit investing in the storage dinosaur that is SAN. And you can quit wrestling with the integration between your blade chassis and your upstream switches.

To learn more, read through Duncan Epping's blog post here. Or check out the HOL at HOL-SDC-1428.

Saying Goodbye - VMworld Edition


It's over, isn't it? VMworld.

We congregated. We jubilated. We educated. We submarinated. We... inebriated.

But what we really did was connected. Engineer to consultant, marketer to executive: we connected.

VMworld isn't about VMware. It isn't about virtualization. It's isn't about whatever you think it's about.

It's about being in the right place at the right time. It's about listening. It's about putting your ego on pause long enough to listen to another member of the community. Whether that's a Cisco evangelist or a vSphere security guru: we listened to learn.

Everyone I spoke with this week agreed: finding your voice is paramount to success in this whole twitter / blogging thing. Ergo: I submit this post to you, LS.

Listen to the people around you. They're smarter than you. They're wiser than you. And they're more eager to share their experience with you.

Tuesday, August 26, 2014

Tech Field Day at Cisco UCS Grand Slam!

Next week, I'm headed to NYC with the Tech Field Day crew will for the Cisco UCS Grand Slam event. What is that, you ask? Cisco isn't sharing many details, but you can tell from their event description that they intend to, once again, disrupt the data center world.

Cisco UCS truly changed how we think about server and network in the context of data center virtualization. It took some time to grok the concept of separating the server from the hardware, but once we adopted that approach, UCS suddenly made sense. And other server vendors took notice; look at HP's and Dell's attempts to replicate UCS and its flagship features. I expect this event to unveil the next significant leap in data center "server" technology. But I don't have any more insight into this event that you do!

They're promoting the event heavily via social media channels, and with good reason: Cisco's CTO Padmasree Warrior will be presenting.

The good news is that you can watch the live stream of this event by registering here. Or follow along with #UCSGrandSlam on Twitter. Either way, be sure to tune in when this event gets started.

Disclosure: Tech Field Day and Cisco are sponsoring my travel and lodging for this event. Awesome.

Monday, August 25, 2014

Tech Field Day Extra - DataGravity, SolarWinds, Asigra

Well, it's finally here: today I'm joining a group of wicked smart people as part of the TFD Extra delegation at VMworld! We've got three companies (DataGravity, SolarWinds, and Asigra) lined up to present their latest products, solutions, and insight into the development of both.


DataGravity (formerly known as the "Secret Company" on Monday's TFDX session) will pretty much make you rethink storage. I can't possibly do their work justice in a single paragraph, so I'll just say that you'll wonder why no one else has thought of this before. The short version is that they are data-aware storage with analytics built-in. I spent an hour chatting with the Product Manager last night at the Solutions Exchange, and it was inspiring to hear how passionate he and the team is about this platform. I'm excited to see how customers use the data that's collected; especially since I spend my days consulting for the public sector. This is going to be KILLER.


Oh, SolarWinds. We've been BFFs for, I don't know, a decade? Everywhere I go, there's a SolarWinds product in the mix. Whether it's Orion NPM, SAM, VMAN, or Storage Manager: these are the tools I rely on to see what's happening on the infrastructure and application level. You may recall that I spent April of this month as a Thwack Ambassador, which was a great experience with the company and the community they're foster. Today they will share their perspective on performance in a hybrid environment (side note: EVERY environment is now a hybrid environment, IMO). This, like the previous session, will be KILLER.


Asigra's presentation will make you rethink how you protect your data. I agree with the concept: put the focus on recovery, not backup. And bundle all elements of data protection into a single suite, so you're not cobbling together a backup / dedupe / encryption / replication / recovery infrastructure with solutions from many, many vendors. I'm excited to learn more about Asigra, and I'll be certain to share what I've learned.

Stay tuned for updates and new posts after (if?) I've survived the infamous TFD party wagon!

Sunday, August 24, 2014

It's the People, Stupid

It's my second VMworld, and it couldn't be more different from my first appearance in 2012.

The first time around, I built a schedule of technical sessions and stuck to it. Ok, maybe I missed one because I didn't realize that travel time between building was deceptively... deceptive. But I really just stuck to the sessions. Sure, I hit up the big party Wednesday (to watch Jon Bon Jovi do whatever it is that he does nowadays). But I was naive to the real draw of this event.

It's the people, stupid.

Stroll around the Solutions Exchange and chat with the SEs and engineers who actually helped develop the solutions. You'll run into all kinds of true technology experts in random places (for example, I ran into Stephen Foskett, Tom Hollingsworth, and Chris Wahl (in that order, mind you) after the #vBrownBag opening acts wrapped up in City View). And wouldn't you know it? Everyone loves to make time to chat a bit, in spite of being terribly busy themselves. Plus I got to spend the afternoon catching up with Adi Lutvich, my occasional co-worker and a hell of a virtualization, storage, and networking engineer who will be cajoled into tweeting and blogging before the week is over.

After just one afternoon, I'm convinced that, while the conference sessions are clearly incredible opportunities to learn and develop your professional skills, it's the social networking events, put on by members of the community, that are the main attraction. (Yes, that was a lot of commas.)

I'm off to #VMunderground now to continue my streak of meeting and chatting with people that I've only interacted with on Twitter. See you there?

Saturday, August 9, 2014

Minecraft Server 1.7.9 on EC2 Ubuntu

My boys LOVE Minecraft. Their obsession has evolved from learning how to mine and place blocks to troubleshooting Forge, working with multiple mods, and downloading texture packs. They're smart kids, and they've surpassed my knowledge of the game. They've learned how to identify suspicious downloads (which you run into a lot in the weird world of ad.fly and mediafire-hosted files) and can work with compressed files and the quirks of the 8.1 Metro interface without much assistance.

But the one thing I've got over them is an EC2 instance running a minecraft server.

I launched it last fall to test out EC2. I figured I'd have zero credibility in the technical community if I'd never, you know, actually used AWS. So I launched a micro instance running Ubuntu 12.04 LTS, installed java, downloaded the minecraft server .jar file, and I was up and running.

But that was last fall, when 1.7.6 was the latest release, and it relied on whitelisting via a text file named whitelist.txt. That file was easy to maintain: each player that you want to whitelist gets its own line. Easy. Basic.

Enter 1.7.9. Whitelisting (and OPing, and banning) is now configured via JSON files. And the format of these files is significantly different from the old whitelist.txt files I was familiar with. So this post is about how to get whitelist.json and ops.json configured manually for at least one player. And remember, I'm running minecraft_server.1.7.9.jar from a linux server with no GUI. That's what makes this slightly more interesting.

Manually Editing Your 1.7.9 Whitelist

Here's what my whitelist.json file looks like. Follow this format for yours, too.

You'll notice that we've got more than just a username here. We've got the corresponding UUID along with the username. So your first task is to gather the UUID for the users you'll be adding to the whitelist. To do this, I just launched Minecraft on my Mac and tried to connect to the server. The connection failed, because I wasn't whitelisted quite yet. But with access to the server and its logs, I knew I'd see an entry in my /usr/minecraft_server/logs directory that listed my username and UUID along with an error. I copied that UUID into the whitelist.json file, saved it, restarted minecraft_server.1.7.9.jar, and I was able to connect. But I couldn't use the in-game console to add my boys to the server. It's because I wasn't an OP.

Manually Editing Your 1.7.9 OPS list

You won't be surprised to learn that this file is of the same format as whitelist.json. So you can cut and paste from that file into this one (assuming you want the users listed in your whitelist to also be server operators).

The only bit that's unique to this file is the level line. It's a measure of how much control an OP has over the game and server (see this article for the details). I gave myself 4 because awesome.

Next Steps

Now you're configured to log into your minecraft server and use console commands to manage these two lists from here on out. You have to admit: /whitelist add username is a hell of a lot easier that the manual method above. But on this early Saturday morning, while I'm listening to Gang Starr and drinking coffee out of a mug from the US House of Representatives, getting into some old-school vi trouble is just what I was looking for. :)

Tuesday, August 5, 2014

My Day In Pictures, Episode 1

I'm kinda tired of writing paragraphs about stuff. So here are two pictures to show you how my day has been (oh, and thanks for asking!).

Yes, you read that correctly. 281840 microseconds (281ms) of latency. 

I have issue with the words "Managed by your system administrator" here.

Friday, August 1, 2014

Blogging at VMworld 2014 in San Francisco!

I'm excited to return to VMworld this year, and this time around I've been selected to receive a blogger pass for the conference! That means I'll be cordoned off in the 31337 blogger section for the keynotes.... I think. Really, it means that the rest of the conference attendees won't need to put up with me furiously banging away on my MBA while you're trying to listen to the presentations.

I've booked a solid 4.5 days of sessions, community events, and the occasional party for VMworld this year. That'll give me dozens of topics to write about and share with you all. And share I will. When I haven't attended VMworld in the past, I've relied on accurate and timely blogposts to catch up on the events. This time, it'll be my opportunity to return the favor.

I'll be attending sessions on hipster stuff like DevOps (actually, is DevOps mainstream now, and Docker is hipster?), NSX, Storage Best Practices, and some PowerCLI stuff I'm into lately. So I'll be posting about those topics and more. You're excited, right? Right. I could tell.

If you're interested in learning more about VMworld, this link is for you.

If you've read enough, and you're ready to register, well THIS link is for you.

Hope to see you in San Francisco! And thanks for reading the #eager0.

Thursday, July 31, 2014

Tech Field Day Extra: VMworld 2014 in San Francisco!

I'm very happy to share with you all that I've been selected as a delegate for Tech Field Day Extra at VMworld 2014 in San Francisco!

The Tech Field Day Extra event spans three days, and will give us a chance to learn about new solutions from companies such as SolarWinds, EMC, and SanDisk. And perhaps more importantly, it's a unique opportunity to ask good questions in order to dive deep into the inner workings of these things. It's certainly not a lunch'n'learn led by a salesperson who can't go beyond the glossy exterior of the product packaging.

Each day will have a different selection of delegates; I'm excited to be included on Monday's panel. That means I'll be getting smart(er?) on SolarWinds, Asigra, and the infamous SECRET COMPANY. Most of you know that I'm a regular contributor at Thwack!, the SolarWinds community forum. I was even fortunate enough to take a turn as a Thwack Ambassador in April of this year. I'm doing my homework now on Asigra; I just might have a two part question to ask. And secret company... more later.

You'll likely recall the discussion that occurred on Duncan Epping's blogpost about a TFD from earlier this year. I think it was a great conversation, especially my dialogue with Hans De Leenheer. I was happy to see that we'll be delegates for Monday's session.

And that's really the point, isn't it? The esprit de corps that springs into existence when you get some social geeks who would otherwise only interact via comments and tweets into the same room... that's truly an amazing opportunity.

And for that reason, I'd like to sincerely thank Stephen Foskett, Tom Hollingsworth, and Claire Chaplais for selecting me to participate, and for covering my travel and lodging for the event. I mean really, I'm just a guy with a blog and a Twitter account. But maybe that's the point. Maybe TFD is for all of us.

See you in SanFran.

Monday, July 21, 2014

PowerCLI cmdlets not loading in ISE?

Since I'm spending most of my time in PowerCLI these days, I thought I'd share a fix for a strange problem I ran into this morning. The problem is that the PowerCLI cmdlets weren't loading automatically when I launched the ISE from a PowerCLI prompt. The PowerCLI cmdlets loaded just fine when I launched ISE from the Start menu.

Here's why this happens. (Spoiler alert: I was doing something wrong.)

One of the first things you learn about PowerCLI is how to enable the cmdlets in the PowerShell ISE. You do this so you can take advantage of the native PowerShell scripting environment while working on your PowerCLI scripts. At first, you may just issue the command to add these cmdlets manually:

Add-PSSnapin VMware.VimAutomation.Core

And with that, PowerCLI cmdlets are now available in the ISE. But you quickly learn that you need to issue this command each time you launch the ISE. So you discover PowerShell Profiles, and you learn that creating a profile allows you to load these cmdlets at the start of each ISE session automatically. Pretty sweet. You end up with something like this:

And it lives in your home directory at My Documents \ WindowsPowerShell. Each time you lauch ISE from the Start Menu, the script runs (you'll notice a status message in the lower left of your ISE screen that says "Running script / selection. Press Control+Break to stop." You're good to go.

But every once in a while, when you launch ISE from anywhere other than the Start Menu, you find that your PowerCLI cmdlets are nowhere to be found. That's the problem I ran into, and it turns out that we need to go back and read more about those PowerShell Profiles from earlier.

When you created your PowerShell profile, you may have not noticed that the profile was specific to your user account. That means that if you launch ISE with any other account, your profile will not load. If you want the same profile to apply to all ISE sessions on your machine, you need to create a different profile. The file itself is identical to the one you created before, but this file needs to be in a different location: C:\Windows\System32\WindowsPowerShell\v1.0. Just copy&paste your profile.ps1 from earlier into this location, and every ISE session on your machine will load the PowerCLI cmdlets for you.

Sure, that sounds great. But what was really going on here, dude?

As it turns out, this behavior (in which my profile ran sometimes but not always) was the result of me running the PowerCLI shell as my administrative user, while using my non-privileged account to log into my workstation. With PowerCLI running as someone else (or more specifically, not the account that I created a $profile for), my personal profile wasn't loaded when ISE was launched (since ISE inherited the credentials of the user associated with the PowerCLI process). The system-wide profile that I created fixed this problem, but I could have just copied my profile to the correct location in the profile path for my administrative account.

tl;dr - I am now slightly less stupid than I was this morning.

Sunday, July 13, 2014

On Writing with Conviction

Writing well is a skill that requires study and practice. By no measure am I an expert on writing, but I'd like to think I've got a decent background in the field; I earned my BA in English (specifically language, writing, and rhetoric) from the University of Maryland, College Park, I've worked as a copy editor for a financial firm, and I spent time as a technical writer... oh about fifteen years ago. And the modest frequency with which I post here at the #eager0, along with some side work at Thwack and the short-form writing at Bugs In My Back Yard, provides me with lots of practice. I think I'm on my way; I'll let you know in thirty years.

Part of the craft of writing is having the audacity to subject your thoughts and opinions to public (and, as is often the case, anonymous) scrutiny. This is by no means an easy task. It's the pen-and-ink (anachronistic, sure, but okay because nostalgic) equivalent to stage fright. And to avoid this scrutiny, we employ all sorts of trickery. We're vague, but disguise it through humor, esoteric intimation, and sarcasm (a wit we're meant to grown out of, not into). We wiggle out of declarations with woulds, coulds, and shoulds. And sometimes, we even resort to... snark.

I recently re-discovered this poem by Taylor Mali. It's titled "Totally like whatever, you know?" and it's required viewing / reading / listening for, you know, like, everyone?

These words should inspire you to write with that same conviction. Again, it's not an easy task. And it's a lesson I'm re-learning on a regular basis; no need to scour the #eager0 for posts that violate this rule.

Have the audacity to make clear, concise statements. Play loud, and make big mistakes. Put yourself out there.

Oh, and if you're looking for better advice from a better writer, check out my friend Oliver Gray's post at Literature & Libation. I blame him for making me work on my writing these days.

Thursday, July 3, 2014

The Calculated Community

It can't be just me. I can't be the only one who is getting a bit suspicious about all of the tech communities out there, and the ever-growing number of evangelist designations that we publicly love put privately loathe. Some of us try to collect them all, just to earn the right to stroll the exhibit halls covered in logos like a race car while tweeting an endless barrage of vendor-inspired hashtags.

As members of these communities (and the absurd meta-community that they form, Voltron-style), we IT professionals pride ourselves on being a smart group of people. But if we're so smart, why is it we fall into the same trap over and over again? "What trap," you ask? The Calculated Community trap.

Deploying from Template

This story starts the same way most IT stories start: by blaming Microsoft. Blame them for the MVP program they launched in 1993 (even though MVP itself was modeled on previous groupings of IT rock stars). VMware recognized the marketing potential of these community-oriented programs and paid a compliment to Microsoft by creating the vExpert program. Clearly, John Troyer (incidentally, have you subscribed to TechReckoning yet? DO IT.) hit that one out of the park; the vExpert title is a highly sought-after recognition of one's contributions to the VMware community. (FD: I'm a vExpert for 2014, and was rightfully rejected in 2013).

Microsoft's MVP program created the template for recognizing individual contributors in the technical community. VMware, EMC, Cisco, and Citrix deployed their programs from that template, with only a few guest customizations. :)

The problem is these communities follow that template so well, you have to squint to see the differences. That's fine when you're deploying VMs; you want them to be identical. But these communities aren't populated with virtual machines; they're populated with people.

Pets versus Livestock, Round Two

You're likely familiar with the pets vs. livestock metaphor that we like to use when it comes to systems management (Joe Baguely's VMUG presentation on this topic is a classic). You can treat large collections of VMs identically because they're not sentient, intelligent bags of bones. But people are.

People don't like to be treated as simply a member of a group. People want to be treated as individuals. Sure, there's camaraderie and solidarity to be realized in groups of like-minded people. And there's a lot of time to be saved when you meet a fellow geek and can quickly negotiate a common technical competency by identifying which communities you both participate in. But after the users groups, after the conferences, after the vBeers, we're individuals. I know I've left a few of these events thinking, "is this a users group, or a thinly veiled marketing campaign?"

What's the Point?

The point is this: the explosion of technical communities, and their striking similarities to every other technical community, is getting out of hand. It's next-level community fatigue. Unless Twitter gives us a higher character count, we'll run out of space for content by the time we hashtag all of the "titles" we've "earned." And slowly, the engineers and administrators and practitioners that truly comprise the technical community at-large will recognize the cookie cutter nature of these communities, and the value of membership will plummet.

It's a community bubble in every sense of the word.

Thursday, June 26, 2014

Potomac Regional VMUG User Conference 2014

It's VMUG User Conference season!

You know the drill: brainy keynote, lots of exhibitors, unlimited free coffee, and dozens of great technical breakout sessions. The only problem is that you end up running into so many people you know, that you pretty much miss all that other stuff. But maybe that's the point.

The only breakout session I attended was on vCAC and Heterogenous Cloud Management. It's funny how some sessions really speak to you. I'm working on a project to modernize the virtualization infrastructure at the moment, and vCAC is likely the tool I've been looking for. I've been familiar with the product for a while, but it was never relevant to my work until now. But I hadn't been looking to integrate so many disparate solutions in a complex environment before. And all of a sudden, I can't complete a sentence at work without using "workflow" and "blueprint" a few times.

So expect to see a few vCAC and integration posts from me in the near future. I haven't been posting highly technical articles lately; that will change. I'm starting a project where I'll be rolling up my sleeves and getting into the guts of a virtualization infrastructure again. I'm looking forward to it!

A final note: many thanks are in order for the leaders of the MD and DC VMUGs. They put in more effort than we realize to make these events successful. Do yourself a favor and join one or both groups if you're in the Baltimore/DC area.

Wednesday, June 18, 2014

Please Use Other Door

Illegible because iPhone 4.

Four glass doors mark the entrance to my office building. Each day, up to one thousand people pass through these doors on their way to work, to lunch, to smoke, to walk, or just to get out for a moment. These doors are busy, for certain.

These doors are also broken.

See the door on the left? The sign says "Please Use Other Door." And that sign has been Scotch taped to the glass since at least December, when I started work here. It didn't bother me at first; doors break all the time. The world is an imperfect place. But after a few months, I started to wonder if the door would ever be fixed.

Now, seven months later, it seems that this door is doomed for dysfunction. What's worse: no one even notices it anymore.

*   *   *

In IT infrastructure, we spend time two ways: building things and fixing things. Personally, I like building, but love fixing. It's a chance to stretch your brain a bit as you detect, diagnose, and dispatch a problem. Usually, a problem arises, we sort it out, and we move on. But what about the problems that you don't fix right away?

Given enough time, you stop seeing these problems. You develop a blind spot; the problem still exists, you just no longer recognize it as a problem. It's like the old adage: the worst thing you can do with a problem is ignore it.

Dear readers, let this be a reminder to you: broken doors won't fix themselves. Stop making signs, and start solving problems.

Tuesday, June 10, 2014

The Hyperconverged Hamburger

Le Big Mac.
A few ground rules for readers of this post: don't get lost in the metaphor here. I'm sure you've got a really clever angle on the McDonaldization of virtualization, or the paleo-inspired server diet you're promoting (and my my, aren't you fit?). But temper your white-hot desire to boast; we're all sure you're terribly smart.

Now. On to the meat of this post.

O, Big Mac. We love to hate you, but my god how we love to love you, too. Consumers of fast food world-wide see you as a whole, not just a collection of manufactured ingredients. As we drift into the smooth undulations of the drive-through lanes, we politely shout "BIG MAC" into a brown box with relentless red LEDs that transcribe our order. Never in the history of mankind (it's a documented fact) has a consumer made the following demands in this situation:
I'd like a seasame seed bun, a club, two burgers, some slices of cheese, lettuce, onions, pickles, catsup, and some "secret" sauce, please.
It's because the Big Mac isn't just some collection of things. It, itself, is the thing. It's the unit of measurement with regard to hunger. Well, sometimes.

It's also a perfect metaphor for hyperconvergence.

Believe the Hype(rconvergence)

We've got to stop thinking about virtualization as a collection of things. Rather, it, itself, is the thing. Cisco squinted hard years ago and saw hyperconvergence on the horizon, and Project California begat Cisco UCS. UCS converges server and network (and storage, too, with the Whiptail acquisition from last year). Nutanix implements convergence of server and storage resources, which avoids the storage networking problem altogether. Dell's VRTX tries to do all three, but to date I've neither seen nor heard of a single implementation of this product (though to be honest, whether I've seen or heard of something is indicative of nothing).

The point is that no one treats virtualization like the product of multiple, discrete resources. It's not storage + server + network. It's virtualization.

The Hyperconverged Org Chart

I've written about the perils of applying yesterday's org charts to today's technology and how isolating your engineers can imperil problem solving. But over the weekend, while we drank copious amounts of cheap beer and jumped into the hot/cold waters of the Chesapeake Bay, a good friend echoed my opinions on org chart silos and the negative impact on IT. So I'm here to tell you again: quit it.

Stop segmenting your virtualization infrastructure into Server | Storage | Network teams. It does not work.

Wednesday, May 28, 2014

Some Advice on Testing

I've run into a few oddities lately regarding testing, and thought I'd fire off a post on the subject before I clear /tmp in /dev/brain for the day.

Testing a Negative

When you're creating a connectivity test, you should test with the expectation of receiving a response. For example, write your script to not just ping a host, but to expect a response for each ping. Do NOT write your test script to expect NO response. If you do, you're introducing a reachability problem into your testing. I've seen a firewall testing script written this way; it failed all the time (pretty sure it was trying to ping a site with bewbs), but the operations team expected it to fail, and the failure was considered a success. This approach backfired when the firewall was offline for a few days before anyone realized it. Oops.

A Valid Test

If you're going to test, make sure the test is valid. Recently, I've seen a web application test plan that neglected to test the new www server via https. The test passed, so the change was put into production. And wouldn't you know it, mod_ssl was missing from the new web server. Hilarity ensued. I mean panic and thrashing.

That's it. A quickie today.

Wednesday, May 14, 2014

Post Hoc Ergo Propter Hoc

This guy, too.
You really should have taken Latin in school.

You'd have undoubtably discovered the phrase "post hoc ergo propter hoc." And you'd use those words every day in your IT career. It means, "after this, therefore because of this." It's a logical fallacy (which means it's a great way to pepper your conversation at hipster dinner parties) that concerns perceived causality between sequential events.

Why they hell are we talking about this?

Because in IT, when something breaks, the first thing we wonder aloud is, "what recently changed?" It's step 1 in the Troubleshooting 101 handbook. And for good reason: changes frequently have unintended consequences. So knowing the recent changes can help you find the source of the problem.

But here's the dark side of that logic. If you put a change into production that another engineer disapproves of, you're in for trouble. You can bet that every incident and outage afterwards will be blamed on your change (and, transitively, you). Now that thinking from above comes back to bite you: "well, the outage occurred after that change you put in, so your change caused the outage." Post hoc, ergo propter hoc.

An Example

Years ago, when it was not uncommon to have dozens of Windows Server 2003 VMs in your production environment, I worked as a systems engineer in an applications hosting shop. I noticed lots of events in the application logs that indicated problems closing registry handles when users were logging off of their RDP sessions. I'd seen this problem before, so I prepared a change request to install the User Profile Hive Cleanup Service (UPHClean) on these VMs to fix the problem. Easy. Basic.

The proposed change was met with bemused hand-wringing. "Why are we doing this?" "Can't we just ignore those errors?" But the change had been tested and approved in our non-production environment, so the change manager OK'd the request. And sure enough, all of those registry errors went away.

A week later, we had an outage on a SQL server that took out an application. Immediately, the UPHClean process was blamed. "Well, the outage happened after your change, so your change caused the outage." Post hoc, ergo propter hoc.

The Point

Don't fall for this trap. It's perfectly acceptable to ask "what changed?" when troubleshooting a problem, but be careful about making the leap from "what changed" to "the change must be the problem." It can lead to thrashing and flailing, and can obfuscate the true root cause of the outage.

Wednesday, May 7, 2014

Building a New Home Lab

IT engineers and administrators loathe discussions about budgets. To us, it's just something that management should worry about. We just want to keep the systems up and running, and to play with implement some cool technology so our skills don't stagnate. We think, "You worry about the dollars, we'll worry about the infrastructure."

Except it doesn't work that way.

If you've ever wanted some perspective on IT budgets, challenge yourself to build and fund a new home lab. All of a sudden, every dollar matters.

I'm in the process of building a new lab, so I'm doing some research to see what I'll need to purchase in order to have a decent VSAN environment at my disposal. This is a well covered topic: Duncan Epping discusses hardware selection for VSAN here, and Chris Wahl covered some options, too. But as you read through these articles, you'll soon realize that the technology is the easy part.

Start pricing out some of the solutions and you'll learn that a VSAN lab is significantly more costly than a vanilla vSphere lab. For a vSphere home lab, you could pick up a pair of servers from craigslist for a few hundred bucks. Hell, you could even pick up a NetApp FAS2020 on eBay with a diskshelf for about $600. Add in a few networking devices, and you could have a serious home lab for under $1,000. But VSAN depends on new technologies that prevent you from just selecting a server based on the number of drive bays.

You'll need to refer to the VSAN HCL early and often so that your disk controllers, HDDs, and SSDs are VSAN ready. Sure, you can get VSAN to run on devices that aren't listed there, but if you're going to invest in a home lab, you should invest in the certified hardware. Otherwise, just go to VMware's HOL and whet your VSAN appetite there.

You'll quickly learn that budget is everything. Because now it's YOUR money. It's not just a line item on a spreadsheet that lives on a network share.

So over the next few weeks, I'll be sharing my research and designing my lab environment. And this time, the biggest design constraint will be cost.