Saturday, November 23, 2013

Ever the New Guy

My penmanship sank years ago.
Show of hands: who among you has a hilarious anecdote about a FNG at your job? You know, that annoyingly enthusiastic technologist who wants to share their experience from other workplaces?

An often overlooked benefit of being a consultant in the IT world is that you get to see how technology is used in many diverse business settings. Your eyes open to the variety of functional and non-functional requirements that can shape the design of things like vSphere and UCS. Your assumptions about how technology is used will be challenged, and you'll end developing your consulting skills (and probably your coping skills, too).

But these are all benefits for you, dear consultant. What about the client? What do they gain from some consultant showing up for a few weeks, maybe even a few months?

Consultants are the bumblebees of the IT world. We cross-pollinate infrastructures by sharing our knowledge of what works and what doesn't. And for many organizations with a low turn-over rate, we provide a much needed infusion of fresh ideas and new approaches to problem solving.

Of course, the ultimate responsibility for adopting change falls on the customer. I can share the best, most current thinking on business concepts like enterprise mobility, client virtualization, and BYOD (if you're looking for BYOB, check out my doppelganger's blog here). But if the existing staff is hell-bent on convincing me why they can't do things the right way, or why their outdated, ill-informed understanding of technology should be the basis for an agency's enterprise mobility strategy, change will be... difficult.

tl;dr - Consultants are ever the new guy, and new ideas are worth listening to.

PS - Looking back at this post, it seems to be a bit passive aggressive. It's really not, though. Sure, it was the product of a difficult debate at the office this week, but I did my best to turn a frustrating argument into a positive experience. And the guy I was arguing with is a good engineer, so the debate was every bit technical as it was rhetorical. Of course, I'm an IT professional with a degree in rhetoric, so he didn't stand a chance. :)

Tuesday, November 19, 2013

Cooperative Puzzle Solving

Most of us have completed a lesson in cooperative puzzle solving while we were in grade school. I recall my third grade class being divided into groups, and each group was given a set of puzzle pieces. When each group tried to complete the puzzle, they ended up with a piece that didn't fit into the last space. Then you realized that every group had a leftover piece and a missing piece. And the only way to solve the puzzles was to trade pieces with other groups.

It's a great lesson in cooperation and divergent thinking. Students are tempted to look for a problem within the limits of their group, but they need to think about solutions that are external to their group to find the right answer.

Now let's talk about how information technology professionals, and the technology they manage, is typically "organized." We've got a box for server people, a box for network people, a box for storage people, and then there's that weird "developers" box that is, like, totally on a different org chart. If server, network, and storage were continents, developers would be the moon.

Humans are hard-wired to sort things. It's what we do. The need to sort people based on skills seems straightforward and harmless enough. But before you know it, you've created the antithesis of the goal-oriented engineer: THE SILO. (See my post on a related org chart malady.)

Silos trap not only engineers in neat little boxes, but also tend to isolate applications and functionality as well.

Earlier today, I was discussing strategy for the adoption of cloud computing when someone pointed out that many of the custom applications in use had their own authentication mechanisms, in contrast to using a centralized directory service like Active Directory. Migrating to the cloud is surprisingly easy when you're using centralized authentication. However, migration to the cloud is crippled by multiple, discrete authentication sources. It's possible, just messy.

In this example, the organizational silos were effectively interfering with the agency's ability to quickly move to the cloud. What's most frustrating here is that all of the pieces to this puzzle are distributed among the various IT groups, but those groups aren't willing to share their pieces, nor are they open to the idea that all groups benefit from cooperation and collaboration. If developers and server teams could work together to address access and authentication to applications via Active Directory, the agency would be moving towards a more portable and secure application infrastructure. Developers would save time by not having to build auth from scratch, and server admins would maintain complete control over all users in the environment. Divergent thinking would prevail.

Please keep this in mind as you go about your career in IT. The solutions to your problems might not be within your box, but they're rarely more than a few inches to the left or right on your org chart.

Friday, November 15, 2013

Cisco Champions for Data Center

Looks like a Mantis, therefore cool.
I had just slinked into the driver's seat of my beloved S40 at the end of a long day at work. I always glance at my email before hitting the road to make sure I don't have any grocery store requests to satisfy en route, so I unlocked my phone and did a double-take. Something about "Welcome to Cisco Champ..." but the rest was truncated. I had assumed it was an acknowledgement of my interest in the program, and headed home.

Fast forward to later that night.  At my desk, I read the full text of the message. After a few careful reads (and after dismissing thoughts of being an unintended recipient), I finally realized that I had been accepted into the Cisco Champions Program for Data Center! Not too shabby for what many coworkers dismiss as a "server guy." :)

I'll save the litany of thanks for an evening at DuClaw with friends. But I do want to thank David Yarashus of Chesapeake NetCraftsmen for encouraging me to be active in the community, and supporting my efforts to "reach across the aisle" in order to break free from the server vs. network approach to IT that I've ranted about in the past. David is wicked smart, and while I can't quite call him a co-worker any longer, I am fortunate to call him a friend. Thank you, David!

One of the guidelines for the CCDC program says to continue to be yourself. So expect more photos of insects, the occasional technical break-fix article, and plenty of posts from an English major with an IT addiction. Hell, maybe I'll toss in a few unicorns just so you know I'm legit.

Tuesday, November 5, 2013

Wrestling OpenNMS

All jesting aside, I love OpenNMS.
I've been using OpenNMS since before it was cool. Though to be honest, so has everyone since OpenNMS is still not cool. But it is a great open-source NMS solution, and it has matured significantly in the ten years since my first exposure to the software.

The current release of OpenNMS is much easier to implement than previous versions, even more so when deploying via yum. You no longer have to treat each component as individual installations; the installer takes cares of everything from the postgres database up through the web application. (This is not an insignificant matter. Years ago, simply getting OpenNMS up and running was a mark of honor.)

Last week, I was asked to troubleshoot an issue with OpenNMS. The issue was that it was broken.

Turns out that the application couldn't run because the postgres instance wasn't running. Like any three-tiered web app, if the database layer is down, then the upper layers can't function. So I dug into the logs to find out what the matter was.

If you're not familiar with troubleshooting on linux, let me tell you about a great command to use when diagnosing a problem with a system service: systemctl <processname> status. In this case, I issued systemctl postgresql status and got the following:

FATAL: could not create shared memory segment: Invalid argument 
DETAIL: Failed system call was shmget(key=5432001, size=4194304, 03600).
HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded your kernel's SHMMAX parameter. You can either reduce the request size or reconfigure the kernel with larger SHMMAX.

You can see pretty easily what the problem is. Postgres is looking for more shared memory than is currently available on the VM (shared memory is different from "physical memory"). So how do we fix that? It's surprisingly simple. Just drop this line into /etc/sysctl:

kernel.shmmax=4194304

Now a reboot, and you're back in business. Technically, you could issue a different command (sysctl -w kernel.shmmax=4194304) and then start postgres, but I like to confirm that the change will survive a reboot so there are no surprises down the road.

You might be wondering why postgres stopped working. Ask the guy who made some changes to the config file without understanding the effect of his changes. (He owes me a few beers for this one. :))
Mastodon