docker compose generator v2 release

For the last ~7 years I have been using an Ansible role to manage my docker compose files. The premise was simple, manage my compose files declaratively and have them sanitised so that they could be shared publicly in a git repo.

Note that v1 to v2 is a completely breaking change. v1 repos will need to be entirely refactored to be compatible.

In today's post I will outline why v1 came to be, why v2 became an obvious evolution of that process, and a bit about how to use v2 as well.

A quick v1 history lesson

I, along with many contributors, built up and ever more elaborate Jinja2 based template structure which was full of if and for statements. This actually worked rather well, but always left me feeling that it was unnecessarily clunky when it came to constructing the containers list variable.

Take Jellyfin, for example.

# group_vars/morphnix.yaml
containers:
  - service_name: jellyfin
    active: true
    image: jellyfin/jellyfin
    hostname: us-rdu
    devices:
      - /dev/dri:/dev/dri
    labels:
      - traefik.enable=true
      - "traefik.http.routers.jellyfin.rule=Host(`jf.{{ wd_domain_me }}`)"
      - traefik.http.services.jellyfin.loadbalancer.server.port=8096
    ports:
      - 2285:8096
    volumes:
      - "{{ appdata_path }}/mediaservers/jellyfin:/config"
      - "{{ storage_path }}:/data:ro"
      - "{{ bigrust18_path }}/media:/bigrust18/media:ro"
    environment:
      - "JELLYFIN_PublishedServerUrl=jf.{{ wd_domain_me }}"
    include_global_env_vars: true
    restart: unless-stopped

This looks almost like a docker compose file doesn't it? But it isn't. This is an entirely bespoke construction that had to be assembled for each and every container I wanted to run through the compose generator mechanism.

Sure. We get some niceities like being able to update entire compose files worth of variables in one spot. For example, if you renamed the underlying {{ appdata_path }} directory. And being able to use Ansible Vault for secret management was a particularly nice feature. But it still felt off.

Why do I have to take a perfectly functional compose file from a projects website and then convert it to this esoteric format that is not portable?

Wouldn't it be better if I could just copy / paste compose files at random from the internet and have the automation ingest them?

Late one holiday evening talking with a friend, we came up with just such a solution. Enter docker-compose-generator v2!

v2

The idea was simple. Supply a directory of standard compose.yaml files and have the automation slurp them up and spit out a single compose file per host. So that's what I built.

In the root of your Ansible git repo, create a directory named services/hostname - see Node naming for more info on hostname specifics. All compose files live here, separated by a directory per remote host.

Take the following directory structure as an example:

└── services
    ├── morphnix
    │   ├── 01-traefik
    │   │   └── compose.yaml
    │   ├── 02-media-servers
    │   │   └── compose.yaml
    │   ├── 03-arrmatey
    │   │   └── compose.yaml
    │   └── 04-monitoring
    │       └── compose.yaml
    └── nix-nvllama
        ├── 01-traefik
        │   └── compose.yaml
        ├── 02-immich-ml
        │   └── compose.yaml
        └── 03-ai
            └── compose.yaml

Each of the compose.yaml files is a fully working, standalone file. But in most cases on my servers I want those files concatenated together into a single larger file which is then managed further with the standard docker compose tool chain.

A few moments with claude later and I was able to happily figure out the required regex syntax to do just that. v2 largely eschews large complex j2 template files but it still relies on Jinja2 for variable substitution and interpolation. Therefore, technically these YAML files are actually treated by the role as .j2 files. However, in order for the interoperability and portability goal I set forth with the rewrite to be achieved we want to treat them as YAMLs.

The role itself walks the filetree using a neat Ansible plugin named filetree and slurps up all the files named compose.yaml in a directory that matches the Ansible magic variable {{ inventory_hostname }}. Next, a bit of regex strips parts of the files we don't want to concatenate (services: for example) and then iterates over each file until we have a rendered output.

All standard docker compose features are supported, the one caveat is that your file must begin with services: at the top. Any other configurations like volumes or network configurations must be placed at the bottom of the last file to be ingested. I plan to work on making this a bit less brittle moving forward. To ensure the file containing the extra configurations ends up in the right place number it something like 99-services (it must start with services: even if it contains none - I know, I know).

After all this regex'ing is complete, the rendered output file is created using the standard Ansible template module (we inherit all the goodies there like Ansible Vault support and variable interpolation j2 style).

Node naming

This one has tripped up a couple of my early testers. In an Ansible hosts file you'll probably have a couple of lines that look like this:

[morphnix]
morphnix ansible_ssh_user=alex

[deepthought]
deepthought ansible_ssh_user=alex

The node names here happen to match the [group names] but sometimes that isn't the case. Sometimes you'll call your hosts via an IP address thus breaking the hostname lookup in the role.

If this is you, specify docker_compose_hostname in group_vars or host_vars to match the name of the directory you placed your compose.yaml files in under services/.

Here's a working example for you. Here, u/fuzzymistborn uses the group_vars/all.yaml feature to automatically apply the hostname variable (which he sets elsewhere by hand) to match the naming convention he wanted for the directories structure under the services directory.

Disabling services

Sometimes you want to disable a service - be it for testing or any other number of reasons. The role can handle this too by creating a list variable in group_vars or host_vars thus:

disabled_compose_files:
  - directory-name1 # e.g. 01-traefik
  - directory-name2 # e.g. 02-apps

This will disable the compose.yaml file found in the named directory. Here's a working example.

Conclusion

Overall, in my opinion this is a big step up in removing friction for trying out new projects whilst maintaining the original goals of santised, safe files for sharing via GitHub. There are a couple of small gotchas which I will work out in time but for now, I've switched over all my primary infra to this v2 of the role.

Let me know your thoughts below and if you're so inclined, I'll happily take PRs on the GitHub repo too. Thanks, and enjoy!