Testing Ansible in LXC Containers
Wiktionary defines yak-shaving as
Any apparently useless activity which, by allowing you to overcome intermediate difficulties, allows you to solve a larger problem.
When folks use the term yak shaving, usually these “useless activities” are numerous. We’ve all experienced this! It’s very common in work involving computers, and frustratingly the useless activities always seem to pile up the most in work done in one’s free time.
I’m doing a lot of my own yak shaving lately on the way to completing a few projects. One of these projects involves needing a CI/CD server. Before I can get into that server and return to the “larger problem” I’m interested in solving I need to provision a virtual machine with all the required users, keys, dependencies, and finally with the CI/CD server and its associated agents. To get that virtual machine provisioned I can make a few Ansible roles and playbooks to automate away most of the setup. Writing Ansible roles and playbooks requires some incremental testing in an environment that is not live, and unfortunately that also means I have to set up test infrastructure for myself.
It’s around that point in a recursive task list where you started to wonder what the heck you were trying to do in the first place, and why you are surrounded by piles of yak hair with pair of electric clippers in your hand next to a very patient half-shaven Tibetan yak. He’s kind of cute; I think I’ll call him Ralph.
Forgetting the rest of this yak-shaving call stack for a moment, I want to focus on one particular pain point I had during some of this yak shaving and how I solved it: testing Ansible playbooks.
It took a little time to get there, but I am now using lightweight LXC containers as targets for my playbooks while doing local testing. I want to explain briefly what they are, why I like them for testing, and then outline how you can get started with them.
What are LXC containers?
LXC is short for “Linux Container”. An LXC style container is a “machine container” that aims to create an environment as close as possible to a normal Linux installation. Docker containers are generally focused on a single process, but LXC containers are focused on the entire system. In effect, LXC containers are very similar to virtual machines. However, there is one important difference between an LXC container and a full virtual machine. A full VM has it’s own kernel that is separate from the host system’s kernel. LXC containers, much like Docker containers, share the host system’s kernel. This makes them very low-overhead when compared to virtual machines.
Mark Shuttleworth of Canonical talked about the rationale of LXC containers at Container Camp in 2016. I first heard of them in the book Linux in Action by David Clinton, and was very surprised to learn how long they have been around, and that Docker even supported them for a time.
Why I like to use them
LXC containers are the closest approximation I can have to the full virtual machines I will be using in production without actually running a full virtual machine locally. Because of the small footprint, they make great local testing companions. Similar to Docker containers, I can run several LXC containers on the same machine without running into resource problems.
When testing my plays using LXC containers I make extensive use of LXC container snapshots. Having snapshots my toolkit empowers me to experiment and try different approaches to playbook implementation quickly and easily. I know I can easily get a clean environment if I decide to rework things.
I can also leverage these snapshots as a sort of checkpoint when
developing multiple roles within a play (or even multiple plays) to save
time in testing. For example, if I am developing two different Ansible
B and I know that
A works perfectly, but
still needs some work, I can take a snapshot of the container after
A role, and use that as a starting point for testing my
B role. It wouldn’t make sense to repeatedly apply the
A role over
and over while developing the
B role. Snapshots can be a wonderful
time saver in this type of situation.
When working on a playbook using LXC containers my workflow begins by adding tasks to a role, and applying that role to the test container through a playbook. As I add tasks, I can SSH into the container to poke around and verify things are working as I like. Often I change my mind about how something should be done, and I can then restore my test container to a clean snapshot, then re-apply the play I am developing. My test container becomes a sort of REPL1 as I continue to try ideas in the playbook, check on things inside the container, and continue to refine the play and its associated roles. Ansible developing becomes iterative.
LXC containers also have persistent storage across starting and stopping containers, as well as across system reboots. This is great for me as I can jump in and jump out of Ansible work and not worry about my container state evaporating.
LXC basics: Creating a container using a profile
In this article, I will not outline instructions for installing the
system container manager LXD. It is available for installation through
snap package manager. It’s snapcraft.io entry can be found
here. I am currently running version
4.11. The LXD snap packages includes the LXD daemon, as well as the
lxc client tool which I will be using to perform tasks within this
Creating an LXC container named
local-testing running Ubuntu 20.04 is
done through the launch command:
lxc launch ubuntu:20.04 local-testing
This will download the requisite Ubuntu “Focal Fossa” image and create a
local-testing. Many other images are available
through remote image
lxc ls will show you a list of all containers, their
status (Running, Stopped), IPv4 and IPv6 addresses, the container type
and the number of snapshots. If everything went smoothly, running
shows us that our container
local-testing has an IP! Great! I can tell
our friend Ralph the yak is very pleased with this development. However,
by default we’re missing the ability to SSH into a container without
using a password. In our case, Ubuntu images create a user called
ubuntu (UID 1000), and that user’s password is the same as its
username. It’s possible to run Ansible plays as this user if the
--ask-pass flags are provided, but that is not ideal. It requires me
to add a flag to my scripts or commands that I won’t be using in my
normal “production” application of these playbooks.
One possible solution to this situation is to leverage container
profiles. Profiles are an easy way to provide cloud-config, as well as
configuration for things like network interfaces used by containers. By
default, Every LXC container by default is assigned a profile called
default. To view its contents, the show command is used:
lxc profile show default`
Pretty boring! I want to SSH into my container without a password. I
need to create a profile that includes cloud-configuration containing a
ssh_authorized_keys. To do this, I clone the default profile,
and modify it with cloud-config.
To clone the
default profile into a profile called
I execute the copy command:
lxc profile copy default local-testing
I can verify the
local-testing profile was created running
lxc profile list. It will have the same contents as the default. I am able
to monkey around with the profile’s configuration keys with the
lxc profile set command, but first I need to create a file with the
cloud-config I wish to assign to my new profile. I will create a
cloud-config YAML file called
~/tmp/lxc-profile-config.yml to hold my
#cloud-config ssh_authorized_keys: - ssh-rsa ... email@example.com # replace with your own public key
I can then modify the
local-testing profile through
lxc profile set
to leverage this cloud configuration through the following magic spell
that redirects the input of the
set command to my YAML file:
lxc profile set local-testing \ user.user-data - < ~/tmp/lxc-profile-config.yml
Wicked. If the above snippet looks a bit frightening, the bash
reference manual might sooth your
may not though, it’s bash after all. Anyway, I can verify this change
has taken effect by examining the output of
lxc profile show local-testing. The cloud-configuration from the YAML file I created
will be visible in the profile. I can now create a new container using
lxc launch -p local-testing ubuntu:20.04 local-ansible-testing
The application of the profile
local-testing to the container will
allow me to ssh into the newly created container with the default
ubuntu user without a password, the provided public key will be used
instead. Now I don’t have to mess around and change any scripts I use to
run plays to have an
Please note that this new container
local-ansible-testing has a
different name than the one we previously created. I’ll be referring to
local-ansible-testing container only from now on. To clean up the
first container I made that is not using the fancy new profile, run the
lxc stop local-testing # stops the container called local-testing lxc delete local-testing # deletes the container called local-testing
Taking an initial LXC container snapshot
So I have a container, what now?
The first thing I do whenever I stand up a container is take a snapshot of its clean, newly-initialized state. This affords me the luxury of being able to restore a clean snapshot if I pollute the container with junk while developing Ansible plays against it.
To create a snapshot of the container
local-ansible-testing from the
previous section, run the
snapshot create command:
lxc snapshot create \ local-ansible-testing \ local-ansible-testing--clean
You can name your own snapshot whatever you like, but a little thought
into a naming scheme pays off the longer the container is in use. The
first snapshot I take of every container I create is the name of the
--clean as a suffix. In the above example my initial
snapshot for the container
local-ansible-testing--clean. Following this convention, or a similar
one, really pays off the more plays and containers you have in use. I
know that whatever container I am working with has a clean snapshot.
Restoring the state of the container to the initial clean snapshot is done with the restore command:
lxc snapshot restore \ local-ansible-testing \ local-ansible-testing--clean
This will remove any changes made since the snapshot was taken. Having a
--clean snapshot lets me jump back in time to a fresh container nearly
instantaneously, it’s great! To list the snapshots associated to a
container, run the info command:
lxc info local-ansible-testing.
I mentioned earlier that snapshots can also be layered. I don’t always
need multiple snapshots when making plays if the play is a single role,
but snapshots can make your life a lot easier when you have plays that
are made of multiple roles that build on top of each other - especially
when some take a long time to run. For example, say I am writing a
playbook to install and configure
nginx (that’s a common theme lately
here at radicalmatt.com). I might split this into two roles:
nginx, configure file/directory ownership, firewall rules, etc
nginxconfiguration, create a symlink to sites-enabled, restart the service
The second role where the deployed configuration is made live requires the first role to have been applied. After finishing development on the first role, and getting it fully applied to the container, I can take a snapshot to use as a new starting point while working on the second role.
If you’re a forgetful person like me, it would be a smart idea to create a bash or python script (or whatever script!) for yourself that creates a container and also creates an initial snapshot for that container. Something to the effect of the following example in bash might be enough to get started with:
#!/bin/bash set -u PROFILE=$1 NAME=$2 # Create an ubuntu container with the given name and profile lxc launch -p "$PROFILE" ubuntu:20.04 "$NAME" # Create a --clean suffixed snapshot of that container lxc snapshot "$NAME" "$NAME--clean"
This little script can be invoked with two positional arguments, the profile you wish to create your container with, and then the name you wish to give it. There’s a lot of opportunity with LXC containers to get creative with lightweight scripting while hacking. You might also explore the scripting of profile creation, or more robust ways to create different types of containers. I think all this is starting to really impress Ralph, here. He’s making a lot of yak noises.
Integrating a test container into your Ansible Inventory
I will return to the land of make-believe where I’m developing a playbook to set up and configure the nginx web server on a couple remote hosts. A flexible way to manage what hosts a play will run against is to maintain a separate inventory file for every environment. Both inventory files share the same groups, but different hosts depending on what environment you are targeting.
For the purposes of this Nginx example, I would define a production
inventory file called
inventory.ini with an
[nginx] group, and
associate a number of host names or IP addresses to the group:
[nginx] production-hostname-1 production-hostname-2
Inside my playbook
host parameter would be set to
--- - name: Nginx server provisioning become: true hosts: nginx ...
The playbook would then be run via:
ansible-playbook \ nginx.yml \ -i inventory.ini # Production inventory
and Ansible would take care of running the play against both the
production-hostname-2 remote hosts.
In order to test my Ansible play against using my new LXC container
local-ansible-testing, a second inventory file called
inventory-test.ini can be created using the same
[nginx] group, but
this time the sole group member will be my LXC container:
[nginx] 10.55.104.21 # The IP of the `local-ansible-testing` container
The IP in this inventory file is the IP of the test container
local-ansible-testing. The IP address of a container can be fund by
lxc ls, and inspecting the table output. With this inventory
file created, running my play against the test environment made up of
my single LXC container becomes a matter of changing the value of the
-i argument of the
ansible-playbook invocation to my test inventory
ansible-playbook \ nginx.yml \ -i inventory-test.ini # Test inventory
Setting up your own plays to work from environment based inventory files
is a great way to ensure consistency in how you apply your plays to
hosts. The change in how the play is applied is limited to a single
argument in the
ansible-playbook command. I think Ralph agrees that
this is a good idea, I can see his tail wagging.
Using LXC containers is a great lightweight way to stand up a quick test environment for your playbooks. They approximate a VM without the full overhead of an actual VM, they support snapshots, and integrate wonderfully into well established Ansible workflows. These containers are a boon for local development. Even in a CI/CD context containers could be created fresh and torn down with much more speed than an ordinary virtual machine.
If you’re interested in learning more about LXC containers, and using them on your own machine, documentation for LXD can be found here. Additionally a worthwhile getting started guide is also available. Both of these resources will provide a much more in depth look at LXC containers and the tools that LXD provides.
Now that my excellent friend Ralph3 is thoroughly at least half shaven and sufficiently impressed at our new magic powers, I can get back to whatever the heck it as that I was originally doing. I just need to sweep up all this yak hair first.
Read-Eval-Print-Loop - I use this term more in the Python REPL sense than the real-ultimate-power LISP sense. All apologies to King Guido. ↩︎
The distinction between “LXD” and “lxc” can be confusing. What you need to know is that “lxc” is simply the client interface to the LXD daemon. This is very similar to “docker” being the client interface to “dockerd”. ↩︎
Here is a photo of Ralph in his natural habitat before his haircut. Good dog, Ralphie. ↩︎
Hi! Welcome to my old school comment section. We don't do comment boxes here at Radicalmatt, Inc. If you read this article and have a question, comment, or have some criticism to share about the article then feel free to reach out. To do so, email me with the title of this post as the subject.
Responses may take a while, but I'll do my best. Your comment could even become the source of another post!