Documentation as a Path to DevOps Automation

Unless you work on a dream team with lots of time and expertise, going from a little to a lot of automation can be hard. Sometimes, it doesn’t even seem like an option. Most definitions of DevOps preach automation as a core tenant, but the jump from idealistic philosophy to implementation in the real world can be a soaring leap. It doesn’t matter if we know it helps reduce toil, yields repeatable processes, and gets us on the path to more reliable systems.

Whether the targeted tasks are special cases that make it hard to automate, or the organization is so understaffed that the team is too busy manually resolving issues to craft beautiful automation, getting started on putting an automation goal into practice can be a more daunting task than many engineers want to admit.

Many organizations moving towards DevOps have realized that it is all about incremental change. Mind shifting cultural change cannot be made overnight, which is the same for automation. So, what is the first incremental step towards automation for an organization wishing to practice DevOps? Documentation!

The First Step Towards Automation

If you don’t have clear documented tasks, not only do you have no idea what is going on, but you also don’t know how to automate it. In the job description for the Site Reliability Engineer role at GitLab, they say that “as an SRE you will document every action so your findings turn into repeatable actions–and then into automation.” Automation isn’t the sole domain of SREs - developers across an engineering organization automate various tasks - and regardless of the automation project, documentation is a useful starting point.

How do you know what and how to automate something unless you write the steps down? In software testing, test scripts, which are a set of instructions that are performed to test that a system is working as expected, have been written for decades now. Before any test automation is created, you write a test script.

Organizations newer to automation might find it useful to first focus on continuous delivery. The goal of continuous delivery is to make deployments a predictable, routine affair, including running automated tests and packaging up successful builds to prepare for a release to production. Often this means automating most of the release process. Automating processes like this can help make “Works on my machine!” irrelevant and break down a wall between development and operations.

Benefits of using documentation as the first step towards automation

There are three significant benefits to using documentation as the first step towards automation:

1. Documentation breaks down tasks into pieces, which are easier to write code to automate later and more efficiently.
2. Documentation helps prioritize what should be automated first.
3. Documentation encourages collaboration between developers and operations.

I am using a broad definition of “documentation.” For example, these are all forms of documentation: A checklist, runbook, design document, specification, wiki page.

1. Breaking down tasks

We’ve written about how useful checklists can be when going through a task. It can help you improve the way you approach specific tasks with consistency and better future output, but it is not only useful for manual tasks too.

Documenting runbooks and other maintenance tasks is a great example of moving teams toward automation. Here at Transposit, we like to think about runbooks and the actions related to them a lot. For example, what steps do we have to go from doing something manually to human-in-the-loop automation, where humans intersect at critical decision points? Often this makes sense for things like runbooks and in some areas of incident response.

It does not always make some tasks feel less like toil, but as the blog post on “do-nothing scripts” says:

• “It’s now much less likely that you’ll lose your place and skip a step. This makes it easier to maintain focus and power through the slog.”
• “Each step of the procedure is now encapsulated in a function, which makes it possible to replace the text in any given step with code that performs the action automatically.”
• “Over time, you’ll develop a library of useful steps, which will make future automation tasks more efficient.”

I especially find the idea that by breaking down the steps in documentation, it is easier to replace the steps with automated code later. It lowers the activation energy for having more automation, which can help eliminate some of the toil that engineers may be experiencing and make future automation more efficient.

2. Prioritization

When your organization has limited time and expertise, how do you decide what tasks to automate? There are some ways that documentation can help you decide.

For teams that don’t have much automation in place, build, test, deploy, and provisioning automation, as we mentioned before, continuous deployment is an excellent place to start. So the first question is, what are the step-by-step processes like for these tasks on your team?

If it doesn’t exist or many steps are missing, it is good to start here. This type of documentation also encourages developers, testers, and operators to work together, which spreads the DevOps ethos. For example, maybe once you write out the steps for a runbook, you might realize that human intervention in the task is critical because there are multiple decision points along the way that would be hard to automate fully. Depending on your goals of automation, this might lower its priority.

Another example is writing documentation for release checklists. Release checklists help teams release software faster and with higher quality for users. Sometimes they feel like toil and are good candidates for automation. Still, you need to write the documentation that lays everything in the checklist before you can compare it to other potential automation projects.

3. Collaboration

Simply put, documentation encourages collaboration. Google published some of the common pitfalls of deployment automation, including poor collaboration between teams. It says that “the deployment automation process must be created by developers and IT operations working together. This approach ensures that both teams can understand, maintain, and evolve deployment automation.”

Documentation, especially when it comes to operating software, is crucial to success. The opening of The DevOps Handbook says, “Imagine a world where product owners, Development, QA, IT Operations, and Infosec work together, not only to help each other, but also to ensure that the overall organization succeeds.” In many cases, documentation can help teams help each other. It takes that tribal knowledge on how to solve problems during an incident or a routine maintenance task and puts it on paper (or rather our screens.)

So, when it comes time to automate something, the document formed through collaboration can help guide the way.

Overcoming the DevOps tools trap

It is easy to fall into the trap of thinking you need a whole new set of tooling to practice DevOps within your organization. The trap is common within automation. Many organizations already have existing tooling that could help teams move closer to DevOps automation.

An excellent way to avoid this is through going through what could potentially be automated. Again, the documentation of potential automation steps and actions comes into play. While reviewing it, you may even realize that there’s existing tooling that could help move towards automation, or maybe a more cultural shift needs to occur. If you can use existing tooling within your organization, it becomes easier to adopt more DevOps practices in a shorter period, since you aren’t having to onboard and train with a whole new set of tools.

Moving Forward

So, where do you go from here? Hopefully, some automation is the next step. As I said before, it is an incremental process, so you have to figure out what tradeoffs there are for different potential automation projects and start with only part of them. If automation still isn’t possible, you may consider the idea of do-nothing scripting. Or, in some cases, implementing human-in-the-loop automation when possible, which is still automating part of a process.

If you are concerned about what you shouldn’t automate, I highly recommend this talk by J. Paul Reed on revisiting the idea of “automate all the things!” And if you are looking for ways to reward documentation within your teams, I recently wrote about it on our blog.

Lastly, what have been your experiences with increasing the amount of documentation before going head first into automation projects within your organization? Feel free to tweet at me your answer at @taylor_atx.