Managing integration systems using clone playbooks
You totally can test stuff, fix bugs and check out how to implement new features on production!
If you’re a masochist, that is.
Usually a production system is a no-no for anything development related. Even if bugs happen on production, the best practice is to reproduce those bug on systems other than production.
These systems are usually called integration, development, acceptance or QA with various understandings from project to project.
But how can you reproduce a bug on a system that is different from production?
How can you bring the database, data and application files from production to a different system.
At my employer, kps we started doing that using Bash scripts. That worked quite well until we had more demands concerning speed, flexibility, complexity and – let’s be honest – readability.
To conquer those we moved away from Bash scripts and to Ansible playbooks.
In case you lived among a flock of sheeps for the last ten years, Ansible is a software belonging to the „Configuration Management System“ category. It is used to describe how a bunch of servers should be installed and configured. It’s written in Python and describes itself as „agentless“ (although it actually needs an SSH access to your system and an installed Python environment).
Ansible works using so called playbooks consisting of multiple plays against target systems. Plays have a bunch of tasks that are run on these systems.
There’s a vast list of modules available providing tasks for every flavor. There are file system tasks, database tasks (for all sorts of databases) and so forth. Check out their documentation for details.
But I’m not talking about my professional life today. Instead, let me tell you that I’m active in an open air theater association called the „Waldbühne Heessen“ (sorry, german only website). I started as an actor there, but you know the common way for people working in the IT business. So I’m now managing their IT stuff together with two fellow actors.
One of the systems I’m responsible for is a site the association uses for organizing auditions, shows and the other stuff that happens throughout the year. Additionally, it contains a bulletin board for discussing things.
Now we needed to migrate from Elgg 2.3 to Elgg 3.2, which was a major jump with several breaking changes and breaking a lot of plugins. As a good SRE I knew, I had to build a proper integration system to prepare that move and to use it as a showcase for my fellow members.
It’s actually quite easy to build a clone playbook, because it simply mimics what you would do manually. So to build my playbook, I checked out what I had to do to clone my production data to an integration system.
From my experience with the clones from my employer I knew the basic steps usually required:
- Dump the database
- Augment the database dump file to match the integration system (This usually includes changing URLs or paths)
- Restore the database onto the target database host
- Copy the application files
- Copy the data files
- Augment the application configuration file to match the integration system (This obviously includes the database settings, paths and some minor stuff)
- Additional works (i.e. Setting an announcement banner to warn the user, that they’re on the integration system)
As an additional topping, I wanted to include all stuff required to update the system to the new version as well. But I wanted to do that dynamically, so I could just switch it off when I had migrated the production system.
Transforming the manual steps into a clone playbook was mostly just checking for fitting modules in the Ansible docs and figuring out how they needed to be configured (which was taking the most time frankly).
For the database tasks, the mysql_db task was used. It supports dumping and restoring databases and also running SQL scripts. The file tasks mostly compress folders using the archive task and copies them around. Augmenting the database dump or application settings, I used the replace and lineinfile tasks.
The playbook itself is rather static. It’s basically just a list of tasks in a Yaml file, that Ansible runs from top to bottom.
Parametrizing the playbook is done using inventory files. They hold the connections to the target hosts, but also parameters like database username and paths.
The inventory files can be written in an ini and yaml flavor. After I initially started using the ini flavor because we used the same at my employer’s I finally switched to yaml format because I needed to configure lists and object values for parameters.
That way I could add an external task file with the update tasks into the inventory, which is included into the main playbook using include_tasks Ansible task and could then clone the integration server and migrate in one step.
For sensitive data like passwords, Ansible brings a vault solution, that encrypts the values in the files using AES (or other cryptos you configure). You encrypt the values using a passphrase and simply use the same passphrase when running the playbook.
Of course, this all wasn’t a one shot. I had to use several tests to finally finalize the playbook and have it working correctly. Depending of the size of the system, this can take some time. So prepare yourself with enough coffee for the task.
As a reference, if you’re interested, I open sourced the whole stuff on the association’s github organization. If you have questions, don’t hesitate to ask.
I don’t know about you but I don’t like to do stuff twice. To develop the site and plugins we’re using a local containerized development environment based on Docker with mounted database and application paths. So why not use the clone playbook to clone production for the local environment?
Ansible supports us there quite well by accepting docker connections to target systems using the local docker daemon.
So I wrapped up a compose file that starts a database and an application server (we created our own Elgg-compatible application server image). This servers are then filled using Ansible and I can simply tar the mounted volumes, bring them to my server and start my development environment, also using a compose file.
(Cover Image: Night. Clone by AdNorrel)
Originally published to dev.to
- November 2019
- Oktober 2019
- Juli 2019
- Juni 2019
- Mai 2019
- April 2019
- März 2019
- September 2018
- August 2018
- Juli 2018
- März 2018
- Januar 2018
- Dezember 2017
- September 2017
- März 2017
- Februar 2017
- Januar 2017
- August 2016
- Mai 2016
- Dezember 2015
- November 2015
- August 2015
- März 2015
- Dezember 2014
- September 2014
- August 2014
- Juli 2014
- Februar 2014
- Oktober 2013
- September 2013
- August 2013
- Juli 2013
- Juni 2013
- Mai 2013
- April 2013
- November 2012
- Oktober 2012
- September 2012
- August 2012
- Juni 2012
- Mai 2012
- März 2012
- Februar 2012
- Januar 2012
- November 2011
- Juli 2011
- Juni 2011
- März 2011
- Februar 2011
- Januar 2011
- Dezember 2010
- November 2010
- April 2010
- Februar 2010