Security of Docker Images

We use containers in production and for our self-hosted package. This document explains how we handle the security of our containers. We implement the following measures to make sure that our deployments are secure.

The native security of Docker images

Build once, Run anywhere

We do not deploy our software directly on machines, but instead depend on container runtimes like Docker or ContainerD. These runtimes are responsible for the security of the containers, and are also responsible for the security of the underlying host.

These runtimes requires container images. A container image is a multi-layered Linux filesystem. To build a container image, we use an OS image containing all the dependencies needed to run a specific application, also known as "base image". We can add our own layers to the base image to build the complete container image, which is then pushed to a container registry.

A machine running with containers.

Using this technology, we are able to make sure that the behavior of the containers is the same regardless of the underlying host.

Host-Container security

Since we use container runtimes, the host is not directly affected by the vulnerabilities of the container images. The software is isolated from the host.

It's the responsibility of the container runtimes to deny unwanted privileges access to the host, and it is our responsibility to correctly configure the container runtimes to disable these privileges.

Secure base images

Since we depend on base images, we make sure that they are secure.

Frontend

For the frontend, we build static assets and copy them on the nginx:<version>-alpine-slim container image, a stripped down version of nginx, drastically reducing the attack surface and the maintenance cost, which is itself built upon alpine:3.

The alpine:3 is known for the very low attack surface and low vulnerability. Coupled with the stripped down version of nginx, there is rarely any vulnerability.

Python projects

We use the ghcr.io/astral-sh/uv:python<py-version>-<debian-version>-slim base image, which is itself built upon python:<py-version>-<debian-version>-slim, a stripped down version of python, which is itself built upon debian:<version>-slim, a stripped down version of debian.

The debian:<version>-slim is known for having a good track of their vulnerabilities. But, due to their release process, some vulnerabilities may show in reports. However, this is mostly false positives, or unexploitable vulnerabilities.

You can check the Debian Security Tracker for more information.

NodeJS projects

We use the node:24-alpine base image, which itself is built upon alpine:3.

Rust projects

We use the debian:bookworm-slim and alpine:3 base images depending on the requirements of the project. Some dependencies requires C libraries which may or may not be available in alpine:3.

Go projects

We do not use any base image for Go projects, and run from scratch. Sometimes, we use busybox to add small debugging tools.

Go projects are statically compiled using the embedded C library inside the Go runtime. CGO is not enabled.

This offer the advantage of having no attack surface based on the base image.

Summary

The containerization of our applications allow us to make sure the underlying host is not affected by the vulnerabilities of the container images.

To avoid a unmaintainable infrastructure, we use container images to offer the required dependencies, but also to easily update the base image when needed.

The chosen base images are known to be secure and maintained by a trusted source.

Container Image Management

Trusted container registries

We mainly use Red Hat Quay Container Registry to host our container images and OCI artifacts like Helm Charts. This registry is managed by the Toucan team, and is automatically scanned by Quay.

Quay Scan Results

We also depend on third-party container images, which are hosted on Docker Hub or GitHub Container Registry. Docker Hub also scans their images, and we tend to avoid the GitHub Container Registry unless we know the image is trusted.

This way, the container images are always available and trusted.

Continuous Container Image Scanning

While Quay scans the images automatically, we also use Trivy to scan the images on our side, without the need to wait for Quay.

The scan occurs daily and per-commit, and is run on the CI/CD infrastructure.

Depending on the release cycle and the security policy of the project, we might not be able to offer a fix for a vulnerability immediatly. Generally, the vulnerability reports usually concern third-party dependencies and it is recommended to read the CVE as false positives may be reported.

In any case, we implement strict security policies at runtime to avoid the exploitation of vulnerabilities.

Deployment details

Least privileges during deployment

Images are never pulled manually and uses Robots Accounts with read-only access to the registry, avoiding potential Supply Chain Attacks. We also use GitOps and a Zero-Trust infrastructure to deploy the services.

No untrusted users have access to the deployment server, and deployments are never done manually.

Least privileges at runtime

While images are naturally built with rootless in mind, we also force the container runtime to remove privileges:

  • User is always non-root and cannot escalate in privileges.

  • Filesystems are read-only.

  • Linux capabilities are dropped.

Strict monitoring

We keep track of metrics and logs of applications to detect undesirable behavior. We use alerts to detect potential vulnerabilities or unexpected behavior.

High available and self-healing

The services are replicated on multiple machines to avoid single points of failure. In case of DDoS, the unhealthy services are automatically restarted to their default state.

Attack detection and network hardening

We use CrowdSec to detect attacks and block them. Attackers with suspect behavior are automatically blocked.

Besides public services, internal services are never exposed to the public internet, and all communication is encrypted using TLS, or, if possible, using mTLS.

Deployment rollouts and rollbacks

Finally, unhealthy services are never served, and the new versions of services are only served when all services are healthy.

This makes sure that the deployment of services is always safe, and broken releases are never deployed.

Conclusion

Using secure container images with least privileges at runtime is the best way to make sure our infrastructure is secure. No privileges can be gain, no unexpected files can be written, and in case of incident, the infrastructure will automatically heal itself.

We keep track of vulnerabilities, and monitor constantly the health of the infrastructure to offer the best possible experience to our users.

The pipeline is secured from the beginning to the end, and we can be sure that the infrastructure is always secure.

An extract of our CI/CD pipeline.

Last updated

Was this helpful?