Testing Ansible in LXC Containers

tag icon

Wiktionary defines yak-shaving as

Any apparently useless activity which, by allowing you to overcome intermediate difficulties, allows you to solve a larger problem.

When folks use the term yak shaving, usually these “useless activities” are numerous. We’ve all experienced this! It’s very common in work involving computers, and frustratingly the useless activities always seem to pile up the most in work done in one’s free time.

I’m doing a lot of my own yak shaving lately on the way to completing a few projects. One of these projects involves needing a CI/CD server. Before I can get into that server and return to the “larger problem” I’m interested in solving I need to provision a virtual machine with all the required users, keys, dependencies, and finally with the CI/CD server and its associated agents. To get that virtual machine provisioned I can make a few Ansible roles and playbooks to automate away most of the setup. Writing Ansible roles and playbooks requires some incremental testing in an environment that is not live, and unfortunately that also means I have to set up test infrastructure for myself.

It’s around that point in a recursive task list where you started to wonder what the heck you were trying to do in the first place, and why you are surrounded by piles of yak hair with pair of electric clippers in your hand next to a very patient half-shaven Tibetan yak. He’s kind of cute; I think I’ll call him Ralph.

Forgetting the rest of this yak-shaving call stack for a moment, I want to focus on one particular pain point I had during some of this yak shaving and how I solved it: testing Ansible playbooks.

It took a little time to get there, but I am now using lightweight LXC containers as targets for my playbooks while doing local testing. I want to explain briefly what they are, why I like them for testing, and then outline how you can get started with them.

What are LXC containers?

LXC is short for “Linux Container”. An LXC style container is a “machine container” that aims to create an environment as close as possible to a normal Linux installation. Docker containers are generally focused on a single process, but LXC containers are focused on the entire system. In effect, LXC containers are very similar to virtual machines. However, there is one important difference between an LXC container and a full virtual machine. A full VM has it’s own kernel that is separate from the host system’s kernel. LXC containers, much like Docker containers, share the host system’s kernel. This makes them very low-overhead when compared to virtual machines.

Mark Shuttleworth of Canonical talked about the rationale of LXC containers at Container Camp in 2016. I first heard of them in the book Linux in Action by David Clinton, and was very surprised to learn how long they have been around, and that Docker even supported them for a time.

Why I like to use them

LXC containers are the closest approximation I can have to the full virtual machines I will be using in production without actually running a full virtual machine locally. Because of the small footprint, they make great local testing companions. Similar to Docker containers, I can run several LXC containers on the same machine without running into resource problems.

When testing my plays using LXC containers I make extensive use of LXC container snapshots. Having snapshots my toolkit empowers me to experiment and try different approaches to playbook implementation quickly and easily. I know I can easily get a clean environment if I decide to rework things.

I can also leverage these snapshots as a sort of checkpoint when developing multiple roles within a play (or even multiple plays) to save time in testing. For example, if I am developing two different Ansible roles A and B and I know that A works perfectly, but B is still needs some work, I can take a snapshot of the container after applying the A role, and use that as a starting point for testing my B role. It wouldn’t make sense to repeatedly apply the A role over and over while developing the B role. Snapshots can be a wonderful time saver in this type of situation.

When working on a playbook using LXC containers my workflow begins by adding tasks to a role, and applying that role to the test container through a playbook. As I add tasks, I can SSH into the container to poke around and verify things are working as I like. Often I change my mind about how something should be done, and I can then restore my test container to a clean snapshot, then re-apply the play I am developing. My test container becomes a sort of REPL1 as I continue to try ideas in the playbook, check on things inside the container, and continue to refine the play and its associated roles. Ansible developing becomes iterative.

LXC containers also have persistent storage across starting and stopping containers, as well as across system reboots. This is great for me as I can jump in and jump out of Ansible work and not worry about my container state evaporating.

LXC basics: Creating a container using a profile

In this article, I will not outline instructions for installing the system container manager LXD. It is available for installation through the snap package manager. It’s snapcraft.io entry can be found here. I am currently running version 4.11. The LXD snap packages includes the LXD daemon, as well as the lxc client tool which I will be using to perform tasks within this article2.

Creating an LXC container named local-testing running Ubuntu 20.04 is done through the launch command:

lxc launch ubuntu:20.04 local-testing

This will download the requisite Ubuntu “Focal Fossa” image and create a container called local-testing. Many other images are available through remote image servers. Executing the command lxc ls will show you a list of all containers, their status (Running, Stopped), IPv4 and IPv6 addresses, the container type and the number of snapshots. If everything went smoothly, running ls shows us that our container local-testing has an IP! Great! I can tell our friend Ralph the yak is very pleased with this development. However, by default we’re missing the ability to SSH into a container without using a password. In our case, Ubuntu images create a user called ubuntu (UID 1000), and that user’s password is the same as its username. It’s possible to run Ansible plays as this user if the -k or --ask-pass flags are provided, but that is not ideal. It requires me to add a flag to my scripts or commands that I won’t be using in my normal “production” application of these playbooks.

One possible solution to this situation is to leverage container profiles. Profiles are an easy way to provide cloud-config, as well as configuration for things like network interfaces used by containers. By default, Every LXC container by default is assigned a profile called default. To view its contents, the show command is used:

lxc profile show default`

Pretty boring! I want to SSH into my container without a password. I need to create a profile that includes cloud-configuration containing a list of ssh_authorized_keys. To do this, I clone the default profile, and modify it with cloud-config.

To clone the default profile into a profile called local-testing, I execute the copy command:

lxc profile copy default local-testing

I can verify the local-testing profile was created running lxc profile list. It will have the same contents as the default. I am able to monkey around with the profile’s configuration keys with the lxc profile set command, but first I need to create a file with the cloud-config I wish to assign to my new profile. I will create a cloud-config YAML file called ~/tmp/lxc-profile-config.yml to hold my authorized_ssh_keys list:

#cloud-config
ssh_authorized_keys:
	- ssh-rsa ... you@email.com # replace with your own public key

I can then modify the local-testing profile through lxc profile set to leverage this cloud configuration through the following magic spell that redirects the input of the set command to my YAML file:

lxc profile set local-testing \ 
	user.user-data - < ~/tmp/lxc-profile-config.yml

Wicked. If the above snippet looks a bit frightening, the bash reference manual might sooth your heart. It may not though, it’s bash after all. Anyway, I can verify this change has taken effect by examining the output of lxc profile show local-testing. The cloud-configuration from the YAML file I created will be visible in the profile. I can now create a new container using the local-testing profile:

lxc launch -p local-testing ubuntu:20.04 local-ansible-testing

The application of the profile local-testing to the container will allow me to ssh into the newly created container with the default ubuntu user without a password, the provided public key will be used instead. Now I don’t have to mess around and change any scripts I use to run plays to have an --ask-password flag.

Please note that this new container local-ansible-testing has a different name than the one we previously created. I’ll be referring to the local-ansible-testing container only from now on. To clean up the first container I made that is not using the fancy new profile, run the following:

lxc stop local-testing    # stops the container called local-testing 
lxc delete local-testing  # deletes the container called local-testing 

Taking an initial LXC container snapshot

So I have a container, what now?

The first thing I do whenever I stand up a container is take a snapshot of its clean, newly-initialized state. This affords me the luxury of being able to restore a clean snapshot if I pollute the container with junk while developing Ansible plays against it.

To create a snapshot of the container local-ansible-testing from the previous section, run the snapshot create command:

lxc snapshot create \
	local-ansible-testing \
	local-ansible-testing--clean

You can name your own snapshot whatever you like, but a little thought into a naming scheme pays off the longer the container is in use. The first snapshot I take of every container I create is the name of the container with --clean as a suffix. In the above example my initial snapshot for the container local-ansible-testing is local-ansible-testing--clean. Following this convention, or a similar one, really pays off the more plays and containers you have in use. I know that whatever container I am working with has a clean snapshot.

Restoring the state of the container to the initial clean snapshot is done with the restore command:

lxc snapshot restore \
	local-ansible-testing \
	local-ansible-testing--clean

This will remove any changes made since the snapshot was taken. Having a --clean snapshot lets me jump back in time to a fresh container nearly instantaneously, it’s great! To list the snapshots associated to a container, run the info command: lxc info local-ansible-testing.

I mentioned earlier that snapshots can also be layered. I don’t always need multiple snapshots when making plays if the play is a single role, but snapshots can make your life a lot easier when you have plays that are made of multiple roles that build on top of each other - especially when some take a long time to run. For example, say I am writing a playbook to install and configure nginx (that’s a common theme lately here at radicalmatt.com). I might split this into two roles:

  1. Install nginx, configure file/directory ownership, firewall rules, etc
  2. Deploy nginx configuration, create a symlink to sites-enabled, restart the service

The second role where the deployed configuration is made live requires the first role to have been applied. After finishing development on the first role, and getting it fully applied to the container, I can take a snapshot to use as a new starting point while working on the second role.

If you’re a forgetful person like me, it would be a smart idea to create a bash or python script (or whatever script!) for yourself that creates a container and also creates an initial snapshot for that container. Something to the effect of the following example in bash might be enough to get started with:

#!/bin/bash
set -u

PROFILE=$1
NAME=$2

# Create an ubuntu container with the given name and profile
lxc launch -p "$PROFILE" ubuntu:20.04 "$NAME"

# Create a --clean suffixed snapshot of that container
lxc snapshot "$NAME" "$NAME--clean"

This little script can be invoked with two positional arguments, the profile you wish to create your container with, and then the name you wish to give it. There’s a lot of opportunity with LXC containers to get creative with lightweight scripting while hacking. You might also explore the scripting of profile creation, or more robust ways to create different types of containers. I think all this is starting to really impress Ralph, here. He’s making a lot of yak noises.

Integrating a test container into your Ansible Inventory

I will return to the land of make-believe where I’m developing a playbook to set up and configure the nginx web server on a couple remote hosts. A flexible way to manage what hosts a play will run against is to maintain a separate inventory file for every environment. Both inventory files share the same groups, but different hosts depending on what environment you are targeting.

For the purposes of this Nginx example, I would define a production inventory file called inventory.ini with an [nginx] group, and associate a number of host names or IP addresses to the group:

[nginx]
production-hostname-1
production-hostname-2

Inside my playbook nginx.yml, the host parameter would be set to nginx:

---
- name: Nginx server provisioning
  become: true
  hosts: nginx
  
  ...

The playbook would then be run via:

ansible-playbook \
	nginx.yml \
	-i inventory.ini    # Production inventory

and Ansible would take care of running the play against both the production-hostname-1 and production-hostname-2 remote hosts.

In order to test my Ansible play against using my new LXC container local-ansible-testing, a second inventory file called inventory-test.ini can be created using the same [nginx] group, but this time the sole group member will be my LXC container:

[nginx]
10.55.104.21    # The IP of the `local-ansible-testing` container

The IP in this inventory file is the IP of the test container local-ansible-testing. The IP address of a container can be fund by running lxc ls, and inspecting the table output. With this inventory file created, running my play against the test environment made up of my single LXC container becomes a matter of changing the value of the -i argument of the ansible-playbook invocation to my test inventory file:

ansible-playbook \
	nginx.yml \
	-i inventory-test.ini    # Test inventory

Setting up your own plays to work from environment based inventory files is a great way to ensure consistency in how you apply your plays to hosts. The change in how the play is applied is limited to a single argument in the ansible-playbook command. I think Ralph agrees that this is a good idea, I can see his tail wagging.

In Conclusion

Using LXC containers is a great lightweight way to stand up a quick test environment for your playbooks. They approximate a VM without the full overhead of an actual VM, they support snapshots, and integrate wonderfully into well established Ansible workflows. These containers are a boon for local development. Even in a CI/CD context containers could be created fresh and torn down with much more speed than an ordinary virtual machine.

If you’re interested in learning more about LXC containers, and using them on your own machine, documentation for LXD can be found here. Additionally a worthwhile getting started guide is also available. Both of these resources will provide a much more in depth look at LXC containers and the tools that LXD provides.

Now that my excellent friend Ralph3 is thoroughly at least half shaven and sufficiently impressed at our new magic powers, I can get back to whatever the heck it as that I was originally doing. I just need to sweep up all this yak hair first.


  1. Read-Eval-Print-Loop - I use this term more in the Python REPL sense than the real-ultimate-power LISP sense. All apologies to King Guido. ↩︎

  2. The distinction between “LXD” and “lxc” can be confusing. What you need to know is that “lxc” is simply the client interface to the LXD daemon. This is very similar to “docker” being the client interface to “dockerd”. ↩︎

  3. Here is a photo of Ralph in his natural habitat before his haircut. Good dog, Ralphie. ↩︎

Comments

Hi! Welcome to my old school comment section. We don't do comment boxes here at Radicalmatt, Inc. If you read this article and have a question, comment, or have some criticism to share about the article then feel free to reach out. To do so, email me with the title of this post as the subject.

Responses may take a while, but I'll do my best. Your comment could even become the source of another post!