Wednesday, May 28, 2014

Some Advice on Testing

I've run into a few oddities lately regarding testing, and thought I'd fire off a post on the subject before I clear /tmp in /dev/brain for the day.

Testing a Negative

When you're creating a connectivity test, you should test with the expectation of receiving a response. For example, write your script to not just ping a host, but to expect a response for each ping. Do NOT write your test script to expect NO response. If you do, you're introducing a reachability problem into your testing. I've seen a firewall testing script written this way; it failed all the time (pretty sure it was trying to ping a site with bewbs), but the operations team expected it to fail, and the failure was considered a success. This approach backfired when the firewall was offline for a few days before anyone realized it. Oops.

A Valid Test

If you're going to test, make sure the test is valid. Recently, I've seen a web application test plan that neglected to test the new www server via https. The test passed, so the change was put into production. And wouldn't you know it, mod_ssl was missing from the new web server. Hilarity ensued. I mean panic and thrashing.

That's it. A quickie today.