Good automation vs bad automation

I recently eliminated a bit of code that was supposed to handle upgrading our build infrastructure from using one distribution (e.g., Ubuntu 8.04 LTS) to another (e.g., Ubuntu 10.04 LTS). That got me thinking about how to decide (and then explain) when it's a good idea to automate and when it isn't.

Since writing and maintaining any piece of software comes with a cost, you always have to weigh the costs against the benefits.

I boiled it down to the following:

• Good automation: the best kind of automation can reliably handle tasks that it is slow, error prone and labor intensive to perform manually. That kind of automation is usually a good idea because once it works well enough you can basically forget about it and enjoy the benefits from an accelerated development cycle (AKA tightened developer feedback loop), reduced amount of friction from debugging human mistakes. It basically frees up your mind and precious labor for other tasks.
• Bad automation: the worst kind of automation handles in an unreliable, error-prone way tasks that are infrequently performed and simple to handle manually. Not only do you have to pay for the cost of developing and maintaining this kind of bad automation, you also get an overall decrease in productivity for your efforts because the automation mechanism will require more maintenance and attention then the tasks it supposedly handles.

A litmus test for good vs bad automation is the ability to test. For example, if you try to automate something that happens infrequently there is a good chance that the assumptions embedded in your automation will not hold over time.

So case in point, since there is no way to test that a bit of automation will work for a future release transition, it's safe to assume it probably won't.