⛴️Deploy Toucan in Air-Gapped Environment

In this section, we will deploy Toucan in an air-gapped environment using Helm Charts. We'll assume this configuration:

• Traffic is only exposed internally:

  • The machine can only be contacted through private networks (VPC or VPN).

  • The machine cannot be reached from the internet and is stricly blocked by the firewall.

• A Private DNS is configured to forward toucan.example.com to the machine IP.

This guide assumes a strict air-gapped environment:

• At first, you have a networked environment. You have access to the internet and can download files to put on a USB stick.

• Then, you deploy the Toucan Stack on your air-gapped environment.

NOTE: This guide helps you deploy a simple "one-shot" "all-in-one" Toucan Stack, which might not be suitable for production.

We heavily recommend in using an external PostgreSQL database as the one embedded might not be suitable for production. Please follow the following guide to connect to your external database: Toucan - External Database

Description and additional requirements

This guide does NOT cover the deployment of Kubernetes in an Air-Gapped environment. If you are interested, we recommend you to read the k3s - Air-Gap Install guide. We recommend using the Private Registry method.

In this guide, we plan to follow that method:

1. In a networked environment, you have access to the internet and will download files.

2. In an air-gapped environment, you have no access to the internet and will need to put these files on Kubernetes:

   1. Helm Charts will be hosted on the deployment server containing the tools for installing the Toucan Stack.

   2. The Docker images will be hosted on a local registry.

   3. The deployment will use that local registry to pull the images.

Therefore, you will need:

• A storage to transfer files from the networked environment to the air-gapped environment. Recommended size is 10GB.

• After uploading the container images in the local registry, container layers will be uncompressed. The registry will requires at least the double.

Preparations in the networked environment

1. Download the Private Registry

If you are using minikube, k3s, or another Kubernetes distribution, it's very possible there is already a private registry, or a way to load container images directly on the container runtime. You should check the documentation of your Kubernetes distribution.

If not, in this guide, we'll install registry as a private registry.

1. Download the container image:

   shell: user@home:/work

   docker pull "docker.io/library/registry:2"
   docker save "docker.io/library/registry:2" | gzip > "registry-2.tar.gz"

2. Download Toucan-Stack

1. Download the Helm Chart

   shell: user@home:/work

   helm pull oci://quay.io/toucantoco/charts/toucan-stack

2. Download the container images:

   shell: user@home:/work

   helm template toucan-stack oci://quay.io/toucantoco/charts/toucan-stack --skip-tests | grep 'image:' | awk '{print $2}' | sort | uniq | while read image; do
     image=$(echo "$image" | sed 's/"//g' | sed "s/'//g")
     docker pull "$image"
   done

You should have every files required for the air-gapped installation!

Installation in the air-gapped environment

1. Deploy the Private Registry

Since there is no registry to host the container registry image (chicken-egg problem), we need to deploy a private registry manually.

1. Transfer the registry container image file directly on the Kubernetes node.

   shell: user@home:/work

   scp ./registry-2.tar.gz root@<node-ip>:registry-2.tar.gz

   If you are in a multi-node setup, you should do this for all the nodes. Since the registry requires a volume, it is better to stick the registry to a single node using nodeSelectors and use a local-path/hostPath volume.

2. Import the image in the container runtime:

   shell: root@node-0:/home/user

   gunzip registry-2.tar.gz
   
   # containerd
   ctr --namespace k8s.io image import registry-2.tar
   
   # OR, docker
   docker load -i registry-2.tar

   You might need to pass --address to ctr since some distributions move the containerd.sock:

   • Minikube: /run/docker/containerd/containerd.sock (docker runtime)

   • k0s: /run/k0s/containerd.sock

3. Deploy the registry using Kubectl:

   shell: user@home:/work

   kubectl apply -f - <<EOF
     apiVersion: v1
     kind: Service
     metadata:
       name: registry-nodeport-service
       labels:
         app: docker-registry
     spec:
       type: NodePort
       selector:
         app: docker-registry
       ports:
         - name: registry-port
           port: 5000
           targetPort: 5000
           nodePort: 32000  # Port 32000 exposed on the node
     ---
     apiVersion: apps/v1
     kind: StatefulSet
     metadata:
       name: registry
     spec:
       serviceName: "registry-nodeport-service"
       replicas: 1
       selector:
         matchLabels:
           app: docker-registry
       template:
         metadata:
           labels:
             app: docker-registry
         spec:
           # Stick the registry to a single node
           nodeSelector:
             kubernetes.io/hostname: node-0
           containers:
           - name: registry
             image: docker.io/library/registry:2
             imagePullPolicy: Never # Air-gapped
             ports:
             - containerPort: 5000
               name: registry-port
             volumeMounts:
             - name: registry-data
               mountPath: /var/lib/registry
       volumeClaimTemplates:
       - metadata:
           name: registry-data
         spec:
           accessModes: [ "ReadWriteOnce" ]
           storageClassName: "local-path"
           resources:
             requests:
               storage: 10Gi
     EOF

2. Transfer the images to the Private Registry

1. Edit the /etc/docker/daemon.json to indicates that <node-0 ip>:32000 is not secured by TLS:

   json: user@home:/etc/docker/daemon.json

   {
     "insecure-registries": ["<node-0 ip>:32000"]
   }

   And run:

   shell: user@home:/work

   sudo systemctl restart docker

2. Using the images you've pulled from earlier steps, tag them to the private registry:

   shell: user@home:/work

   docker tag <registry>/<repo>/<image>:<tag> <node-0 ip>:32000/<repo>/<image>:<tag>

   If you don't remember which images you've pulled, you can use the following command to push all the images to the private registry:

   shell: user@home:/work

   remove_registry_hostname() {
     image_name="$1"
     slash_count=$(echo "$image_name" | grep -o "/" | wc -l)
   
     if [ "$slash_count" -ge 2 ]; then
       echo "$image_name" | cut -d '/' -f2-
     else
       echo "$image_name"
     fi
   }
   
   helm template toucan-stack ./toucan-stack*.tgz --skip-tests | grep 'image:' | awk '{print $2}' | sort | uniq | while read image; do
     image=$(echo "$image" | sed 's/"//g' | sed "s/'//g")
     docker tag "$image" "<node-0 ip>:32000/$(remove_registry_hostname "$image")"
     docker push "<node-0 ip>:32000/$(remove_registry_hostname "$image")"
   done

   Replace <node-0 ip> with the IP address of the node where you've deployed the registry.

3. Deploy Toucan Stack

1

Create a namespace

Create a Kubernetes namespace to deploy the Toucan Stack Helm charts in.

shell: /work/

kubectl create namespace toucan

Namespaces are used to avoid name conflicts between different projects. Since we are deploying a stack of services, we can use the same namespace for all of them, and avoid conflicting with your own projects.

2

Deploy the Helm charts

Since we are using Helm, we can patch the necessary values to inject the credentials and secrets. We also need to expose the service to the external network and secure it with TLS.

1. Create the values file, which will override the default values.

   shell: /work/

   touch values.override.yaml # You can name it whatever you want

2. (optional) For strict air-gapped environments, assuming you have transferred the container images to your air-gapped container registry, add these lines to values.override.yaml override the registry:

   yaml: /work/values.override.yaml

   global:
     imageRegistry: 'localhost:32000'
     security:
       allowInsecureImages: true
   
   # Gotenberg doesn't use global.imageRegistry
   gotenberg:
     image:
       repository: localhost:32000/gotenberg/gotenberg

3. Add this line to select your storage provisioner:

   yaml: /work/values.override.yaml

   global:
     # ...
     defaultStorageClass: local-path
   # ...

   You can fetch the available storage classes with:

   shell: /work/

   kubectl get storageclass

   You should see something like this:

   shell: /work/

   NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
   local-path           rancher.io/local-path   Delete          WaitForFirstConsumer   false                  121d

4. (Optional) Override the volume size:

   yaml: /work/values.override.yaml

   # ...
   laputa:
     persistence:
       size: 10Gi
   
   keycloak:
     persistence:
       size: 8Gi
   
   postgresql:
     primary:
       persistence:
         size: 10Gi
   
   mongodb:
     persistence:
       size: 8Gi
   
   garage:
     persistence:
       meta:
         size: 8Gi
       data:
         size: 8Gi

   NOTE: This is only useful if you are using a storage provisioner which handles "sizing". The local-path-provisioner does NOT! So these values doesn't mean anything, but many cloud provider and block storage provisioners do.

5. Configure TLS for the Toucan Stack and deploy the routes:

Using Gateway API

SUGGESTION: We recommend using cert-manager to issue TLS certificates, which is able to rotate certificates on a regular basis.

You can also use external-secrets, to fetch TLS certificates from a secret manager like AWS Secrets Manager, Hashicorp Vault, etc.

Create these files:

yaml: /work/gateway/tls-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: toucan.example.com-tls
  namespace: envoy-gateway-system # NOTICE THIS
stringData:
  tls.crt: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
  tls.key: |
    -----BEGIN RSA PRIVATE KEY-----
    ...
    -----END RSA PRIVATE KEY-----
  ca.crt: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
type: kubernetes.io/tls

Deploy the certificates with:

shell: /work/

kubectl apply -f gateway/tls-secret.yaml

Now, add a new listener to the gateway:

yaml: /work/gateway/gateway.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg
  namespace: envoy-gateway-system
spec:
  gatewayClassName: envoy-gateway
  # ...
  listeners:
    # ...
    - name: toucan
      protocol: HTTPS
      port: 443
      hostname: toucan.example.com
      tls:
        mode: Terminate
        certificateRefs:
          - name: toucan.example.com-tls
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              kubernetes.io/metadata.name: <namespace>

Re-deploy the gateway with:

shell: /work/

kubectl apply -f gateway/gateway.yaml

Check the status of the gateway with:

shell: /work/

kubectl get gateway -n envoy-gateway-system eg
# You can add [-o yaml] to get more details

Now, add a new route to the gateway:

yaml: /work/values.override.yaml

global:
  # ...

  ## global.hostname configures the helm chart to use toucan.example.com as "public" domain.
  hostname: toucan.example.com

canopee:
  httpRoute:
    enabled: true
    parentRefs:
      - name: eg
        namespace: envoy-gateway-system

Using Ingress API

SUGGESTION: We recommend using cert-manager to issue TLS certificates, which is able to rotate certificates on a regular basis.

You can also use external-secrets, to fetch TLS certificates from a secret manager like AWS Secrets Manager, Hashicorp Vault, etc.

Create these files:

yaml: tls-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: toucan.example.com-tls
stringData:
  tls.crt: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
  tls.key: |
    -----BEGIN RSA PRIVATE KEY-----
    ...
    -----END RSA PRIVATE KEY-----
  ca.crt: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
type: kubernetes.io/tls

Deploy the certificates with:

shell: /work/

kubectl apply -n <namespace> -f tls-secret.yaml

Expose the Toucan Stack by adding these lines:

yaml: /work/values.override.yaml

global:
  # ...

  ## global.hostname configures the helm chart to use toucan.example.com as "public" domain.
  hostname: toucan.example.com

canopee:
  ingress:
    enabled: true
    ingressClassName: nginx
    tls: true
    extraTls:
      - hosts:
          - toucan.example.com
        secretName: 'toucan.example.com-tls'

1. At this point, your values.override.yaml should looks like (minus the volume size overrides):

yaml: /work/values.override.yaml

global:
  imageRegistry: 'localhost:32000'
  security:
    allowInsecureImages: true
  imagePullSecrets:
    - dockerconfigjson
  defaultStorageClass: local-path
  hostname: toucan.example.com

canopee:
  # If Gateway API:
  httpRoute:
    enabled: true
    parentRefs:
      - name: eg
        namespace: envoy-gateway-system

  # If Ingress API:
  ingress:
    enabled: true
    ingressClassName: nginx
    annotations:
      cert-manager.io/cluster-issuer: private-cluster-issuer
    tls: true
    extraTls:
      - hosts:
          - toucan.example.com
        secretName: toucan.example.com-tls

gotenberg:
  image:
    repository: localhost:32000/gotenberg/gotenberg

1. Deploy the Toucan Stack:

shell: /work/

helm upgrade --install toucan-stack ./toucan-stack*.tgz \
  --namespace toucan \
  --values ./values.override.yaml

If the installation fails with:

shell: /work/

Error: INSTALLATION FAILED: failed post-install: 1 error occurred:
        * timed out waiting for the condition

You should check the health of the deployment. Use kubectl get <deployments/statefulsets/pods> -n toucan to check the status of the deployment. And use kubectl logs <pod-name> -c <container-name> -n toucan to check the logs of the deployment.

We highly recommend using a Kubernetes GUI for troubleshooting like for example Headlamp.

1. To get the Admin password, run the following command:

shell: /work/

kubectl get secret --namespace toucan toucan-stack-auth -o jsonpath='{.data.toucan-admin-password}' | base64 --decode

1. You should be able to access the Toucan Stack at https://toucan.example.com and login with the admin credentials. Enter [e-mail] for the username and the password you got from the previous step.