YouPrep documentationHelp centerGet a demo

⛴️Deploy Toucan in Air-Gapped Environment

In this section, we will deploy Toucan in an air-gapped environment using Helm Charts. We'll assume this configuration:

  • Traffic is only exposed internally:

    • The machine can only be contacted through private networks (VPC or VPN).

    • The machine cannot be reached from the internet and is stricly blocked by the firewall.

  • A Private DNS is configured to forward auth-toucan.example.com and toucan.example.com to the machine IP. The Kubernetes internal DNS will be used as an example.

Also, for simplification reasons of this guide, we consider that the Docker images and Helm Charts are still accessible.

If you are in a strict air-gapped environment, you should:

  1. Self-host a private registry like Harbor, zot, or the official container registry.

  2. Download the Docker images and Helm Charts, in which you have verified the security, integrity and origin of the image. (Container image signature and Helm Chart signature are not available yet. It is planned.)

    Here's a list of Docker images and Helm Charts to download:

    shell
    # Helm Charts
quay.io/toucantoco/charts/toucan-stack:latest

# The following container images are used by the Toucan Stack Helm Chart,
# but is probably outdated.
#
# Check the list of images, using:
#
#   helm template --set curity.config.license.secretName=dummy release-name oci://quay.io/toucantoco/charts/toucan-stack | grep 'image:' | sort | uniq
# Toucan Services:
quay.io/toucantoco/frontend:v145.0.1
quay.io/toucantoco/ngx-auth-module:v1.6.0-toucan.5-debian12-ngx1.27.4
quay.io/toucantoco/dataset_service:v1.11.0
quay.io/toucantoco/impersonate-token-service:v0.3.1
quay.io/toucantoco/backend:v145.0.1
quay.io/toucantoco/layout-service:v1.11.0

# Third party services:
curity.azurecr.io/curity/idsvr:10.1.0
docker.io/bitnami/redis:7.4.2-debian-12-r6
gotenberg/gotenberg:8.15.3
docker.io/bitnami/nginx:1.27.4
docker.io/library/mongo:8.0.8
ghcr.io/authzed/spicedb:v1.38.0
docker.io/bitnami/vault:1.19.0-debian-12-r0
docker.io/bitnami/postgresql:17.4.0-debian-12-r11

# Utilities images:
docker.io/library/busybox:1.37
docker.io/library/alpine:3.21
docker.io/alpine/curl:8.21.1
ghcr.io/authzed/zed:v0.27.0-debug
docker.io/hairyhenderson/gomplate:v4.3.2-alpine

Use helm pull to download the latest Helm Charts from the Toucan Toco's Quay registry in .tar.gz, in which you can push it to your air-gapped OCI registry using helm push.

Use docker pull to download the latest Docker images from the container registry, in which you can push it to your air-gapped container registry using docker tag combined with docker push.

  1. Transfer these images and Helm Charts to your air-gapped OCI registry.

Lastly, we assume you have installed k3s and NGINX Ingress Controller on your air-gapped environment. Here's a guide on how to install k3s on your air-gapped environment: K3s - Air-Gap Install.

Configure TLS with cert-manager

This guide uses cert-manager to manage the TLS certificates for the ingress controller. cert-manager automatically fetches certificates from ACME (Automatic Certificate Management Environment) servers, such as Let's Encrypt.

If your prefer to self-manage certificates, you can use kubectl to import secrets directly. However, we heavily recommend to use cert-manager to manage the private and public TLS certificates for the ingress controller.

shell: /work/
# (in the networked environment) You can pull the tar.gz before you install cert-manager
helm pull --repo https://charts.jetstack.io cert-manager

# (in the air-gapped environment)
helm install \
  cert-manager ./cert-manager-v<version>.tgz \
  --namespace cert-manager \
  --create-namespace \
  --set crds.enabled=true

We are assuming you have imported the cert-manager Docker image to your air-gapped container registry.

You can list the images using:

helm template --kube-version 1.31 --repo https://charts.jetstack.io cert-manager  | grep 'image:'

Which are:

quay.io/jetstack/cert-manager-cainjector:v1.17.1
quay.io/jetstack/cert-manager-controller:v1.17.1
quay.io/jetstack/cert-manager-webhook:v1.17.1
quay.io/jetstack/cert-manager-startupapicheck:v1.17.1

To generate certificates, we need a ClusterIssuer resource. We are assuming you have no Certificate Authority (CA) in your air-gapped environment, and we need to create one.

  1. Create a self-signed issuer:

yaml: /work/selfsigned-issuer.yaml
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned-issuer
  namespace: cert-manager
spec:
  selfSigned: {}

This self-signed issuer will be used to create a Certificate Authority. Deploy it:

shell: /work/
kubectl apply -f selfsigned-issuer.yaml

  1. Create a Certificate for the Certificate Authority:

yaml: /work/ca-certificate.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ca-certificate
  namespace: cert-manager
spec:
  secretName: root-ca
  issuerRef:
    name: selfsigned-issuer
    kind: Issuer
  isCA: true
  duration: 43800h # 5 year
  renewBefore: 720h # 30 days before expiry
  privateKey:
    algorithm: RSA
    encoding: PKCS1
    size: 2048
  subject:
    organizations: [Toucan Toco]
    countries: [FR]
    organizationalUnits: [IT]
    localities: [Paris]
  commonName: Toucan Root CA

Deploy it:

shell: /work/
kubectl apply -f ca-certificate.yaml

You should share the CA's certificate to your users. Use kubectl get secret root-ca -n cert-manager -o jsonpath='{.data.ca\.crt}' | base64 -d to get the certificate.

Users will need to import it in their browser. See this guide.

  1. Create a ClusterIssuer that can be used by the whole cluster to fetch certificate from the CA:

yaml: /work/cluster-issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: private-cluster-issuer
  namespace: cert-manager
spec:
  ca:
    secretName: root-ca

Deploy it:

shell: /work/
kubectl apply -f cluster-issuer.yaml

Deploy Toucan Stack

1

Create a namespace

Create a Kubernetes namespace to deploy the Toucan Stack Helm charts in.

shell: /work/
kubectl create namespace toucan

Namespaces are used to avoid name conflicts between different projects. Since we are deploying a stack of services, we can use the same namespace for all of them, and avoid conflicting with your own projects.

2

Deploy the registry credentials

Since we are assuming you have access to Quay for "simplicity", we need to create a secret to access it:

shell: /work/
# Replace <username> and <password> with your credentials
# docker-registry: The type of secret to create.
# --namespace: The namespace to create the secret in.
# dockerconfigjson: The name of the secret to create.
# --docker-server: The server address of the registry.
# --docker-username: The username for the registry.
# --docker-password: The password for the registry.
kubectl create secret docker-registry --namespace toucan dockerconfigjson \
  --docker-server=quay.io \
  --docker-username=<username> \
  --docker-password=<password>

To fetch your Quay credentials, you can generate an encrypted password on Quay.io:

  1. Go to Account Settings.

  2. Go to the "Gear Menu" on the left side menu.

  3. Click on "Generate Encrypted Password"

    Fetching Quay encrypted password

    Use the encrypted password and username in the --docker-username and --docker-password flags.

In a strict air-gapped environment, the secret would be the credentials to access your air-gapped registry.

3

Deploy the Curity secret

You should have a JSON file in this format:

json: Subscription_YYY-MM-DD.json
{
  "Company": "[email protected]",
  "Edition": "Community",
  "Environment": "",
  "Expires": "2025-12-13",
  "Feature": "",
  "GracePeriod": "30",
  "Groups": "1",
  "Issued": "2024-12-13",
  "Licence": "ey...",
  "License": "ey...",
  "Product": "Identity Server",
  "Tier": "Subscription",
  "Type": "Production",
  "Version": "4.3",
  "expired": false
}

Copy the value from the License or Licence field, and create the secret with:

shell: /work/
# Replace <value> with your license
# generic: The type of secret to create.
# --namespace: The namespace to create the secret in.
# toucan-curity: The name of the secret to create.
# --from-literal: The key and value of the secret to create.
kubectl create secret generic --namespace toucan toucan-curity \
  --from-literal=license=<value>
4

Deploy the Helm charts

Since we are using Helm, we can patch the necessary values to inject the credentials and secrets. We also need to expose the service to the external network and secure it with TLS.

  1. Create the values file, which will override the default values.

shell: /work/
touch values.override.yaml # You can name it whatever you want

  1. (optional) For strict air-gapped environments, assuming you have transferred the container images to your air-gapped container registry in this way:

    docker.io/org/image:tag -> my-registry/org/image:tag

    Add these lines to values.override.yaml override the registry:

yaml: /work/values.override.yaml
global:
  imageRegistry: 'my-registry'
  security:
    allowInsecureImages: true

If you need to override the image precisely, look into the values and override the image field. For example, if you want to override the laputa image to use a different one, check the values:

shell: /work/
helm show values oci://quay.io/toucantoco/charts/toucan-stack | less

Look for the field laputa: and image: (type / to search in less), you'll something similar to:

yaml: helm show values oci://quay.io/toucantoco/charts/toucan-stack | less
laputa:
  # ... (truncated)
  image:
    registry: quay.io
    repository: toucantoco/backend
    # renovate: image=quay.io/toucantoco/backend
    tag: 'v144.0.2'
    digest: ''
    ## Specify a imagePullPolicy
    ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent'
    ## ref: https://kubernetes.io/docs/concepts/containers/images/#pre-pulled-images
    ##
    pullPolicy: IfNotPresent
    ## Optionally specify an array of imagePullSecrets.
    ## Secrets must be manually created in the namespace.
    ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
    ## e.g:
    ## pullSecrets:
    ##   - myRegistryKeySecretName
    ##
    pullSecrets: []

Copy and replace accordingly in the values.override.yaml file.

  1. Add this line to values.override.yaml inject the registry credentials:

yaml: /work/values.override.yaml
global:
  imagePullSecrets:
    - dockerconfigjson

  1. Add this line to inject the Curity secret:

yaml: /work/values.override.yaml
# ...

curity:
  config:
    license:
      secretName: toucan-curity
      secretKey: license

  1. Add this line to select your storage provisioner:

yaml: /work/values.override.yaml
global:
  # ...
  defaultStorageClass: local-path
# ...

You can fetch the available storage classes with:

shell: /work/
kubectl get storageclass

You should see something like this:

shell: /work/
NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
local-path           rancher.io/local-path   Delete          WaitForFirstConsumer   false                  121d

  1. (Optional) Override the volume size:

yaml: /work/values.override.yaml
# ...
laputa:
  persistence:
    size: 10Gi

curity:
  # ...
  admin:
    persistence:
      size: 8Gi

postgresql:
  primary:
    persistence:
      size: 10Gi

mongodb:
  persistence:
    size: 8Gi

NOTE: This is only useful if you are using a storage provisioner which handles "sizing". The local-path-provisioner does NOT! So these values doesn't mean anything, but many cloud provider and block storage provisioners do.

  1. Expose the Toucan Stack by adding these lines:

yaml: /work/values.override.yaml
global:
  # ...

  ## global.hostname configures the helm chart to use toucan.example.com as "public" domain.
  hostname: toucan.example.com

nginx:
  ingress:
    enabled: true
    ingressClassName: nginx
    annotations:
      cert-manager.io/cluster-issuer: private-cluster-issuer # private-cluster-issuer references the previously created ClusterIssuer
    extraTls:
      - hosts:
          - toucan.example.com
        secretName: 'toucan.example.com-cert' # This secret will be generated.

 curity:
   # ...
   runtime:
     ingress:
       enabled: true
       ingressClassName: nginx
       hostname: auth-toucan.example.com
       annotations:
         cert-manager.io/cluster-issuer: private-cluster-issuer # private-cluster-issuer references the previously created ClusterIssuer
       extraTls:
         - hosts:
             - auth-toucan.example.com
           secretName: 'auth-toucan.example.com' # This secret will be generated.

Annotations are used by controllers like cert-manager to trigger side effects.

  1. (ONLY DURING BETA) Curity's theme are downloaded at runtime during the beta phase, so you can't prefetch them. So, for now, disable them:

yaml: /work/values.override.yaml
curity:
  config:
    theme:
      enabled: false

  1. Lastly, you need to inject the CA's certificate to the internal services that uses toucan.example.com:

yaml: /work/values.override.yaml
laputa:
  config:
    common:
      REQUESTS_CA_BUNDLE: /etc/ssl/certs/ca-certificates.crt

  extraVolumes:
    - name: spicedb-certs
      secret:
        secretName: '{{ template "toucan-stack.spicedb.tls.secretName" . }}'
        items:
          - key: ca.crt
            path: ca.crt
    - name: ca-bundle
      secret:
        secretName: 'toucan.example.com-cert'
        items:
          - key: ca.crt
            path: ca-certificates.crt

  extraVolumeMounts:
    - name: spicedb-certs
      mountPath: /spicedb-certs
    - name: ca-bundle
      mountPath: /etc/ssl/certs

layout:
  extraEnvVars:
    - name: NODE_EXTRA_CA_CERTS
      value: /etc/ssl/certs/ca-certificates.crt

  extraVolumes:
    - name: spicedb-certs
      secret:
        secretName: '{{ template "toucan-stack.spicedb.tls.secretName" . }}'
        items:
          - key: ca.crt
            path: ca.crt
    - name: ca-bundle
      secret:
        secretName: 'toucan.example.com-cert'
        items:
          - key: ca.crt
            path: ca-certificates.crt

  extraVolumeMounts:
    - name: spicedb-certs
      mountPath: /spicedb-certs
    - name: ca-bundle
      mountPath: /etc/ssl/certs

dataset:
  extraEnvVars:
    - name: SSL_CERT_FILE # For httpx
      value: /etc/ssl/certs/ca-certificates.crt
    - name: REQUESTS_CA_BUNDLE # For requests
      value: /etc/ssl/certs/ca-certificates.crt

  extraVolumes:
    - name: spicedb-certs
      secret:
        secretName: '{{ template "toucan-stack.spicedb.tls.secretName" . }}'
        items:
          - key: ca.crt
            path: ca.crt
    - name: ca-bundle
      secret:
        secretName: 'toucan.example.com-cert'
        items:
          - key: ca.crt
            path: ca-certificates.crt

  extraVolumeMounts:
    - name: spicedb-certs
      mountPath: /spicedb-certs
    - name: ca-bundle
      mountPath: /etc/ssl/certs

impersonate:
  extraVolumes:
    - name: ca-bundle
      secret:
        secretName: 'toucan.example.com-cert'
        items:
          - key: ca.crt
            path: ca-certificates.crt

  extraVolumeMounts:
    - name: ca-bundle
      mountPath: /etc/ssl/certs

vault:
  server:
    extraVolumes:
      - name: ca-bundle
        secret:
          secretName: 'toucan.example.com-cert'
          items:
            - key: ca.crt
              path: ca-certificates.crt

    extraVolumeMounts:
      - name: ca-bundle
        mountPath: /etc/ssl/certs/

  1. At this point, your values.override.yaml should looks like (minus the volume size overrides):

yaml: /work/values.override.yaml
global:
  ## Uncomment if you are using a custom registry.
  # imageRegistry: 'my-registry'
  # security:
  #   allowInsecureImages: true
  imagePullSecrets:
    - dockerconfigjson
  defaultStorageClass: local-path
  hostname: toucan.example.com

nginx:
  ingress:
    enabled: true
    ingressClassName: nginx
    annotations:
      cert-manager.io/cluster-issuer: http01
    extraTls:
      - hosts:
          - toucan.example.com
        secretName: 'toucan.example.com-cert'

laputa:
  config:
    common:
      REQUESTS_CA_BUNDLE: /etc/ssl/certs/ca-certificates.crt

  extraVolumes:
    - name: spicedb-certs
      secret:
        secretName: '{{ template "toucan-stack.spicedb.tls.secretName" . }}'
        items:
          - key: ca.crt
            path: ca.crt
    - name: ca-bundle
      secret:
        secretName: 'toucan.example.com-cert'
        items:
          - key: ca.crt
            path: ca-certificates.crt

  extraVolumeMounts:
    - name: spicedb-certs
      mountPath: /spicedb-certs
    - name: ca-bundle
      mountPath: /etc/ssl/certs

layout:
  extraEnvVars:
    - name: NODE_EXTRA_CA_CERTS
      value: /etc/ssl/certs/ca-certificates.crt

  extraVolumes:
    - name: spicedb-certs
      secret:
        secretName: '{{ template "toucan-stack.spicedb.tls.secretName" . }}'
        items:
          - key: ca.crt
            path: ca.crt
    - name: ca-bundle
      secret:
        secretName: 'toucan.example.com-cert'
        items:
          - key: ca.crt
            path: ca-certificates.crt

  extraVolumeMounts:
    - name: spicedb-certs
      mountPath: /spicedb-certs
    - name: ca-bundle
      mountPath: /etc/ssl/certs

dataset:
  extraEnvVars:
    - name: SSL_CERT_FILE # For httpx
      value: /etc/ssl/certs/ca-certificates.crt
    - name: REQUESTS_CA_BUNDLE # For requests
      value: /etc/ssl/certs/ca-certificates.crt

  extraVolumes:
    - name: spicedb-certs
      secret:
        secretName: '{{ template "toucan-stack.spicedb.tls.secretName" . }}'
        items:
          - key: ca.crt
            path: ca.crt
    - name: ca-bundle
      secret:
        secretName: 'toucan.example.com-cert'
        items:
          - key: ca.crt
            path: ca-certificates.crt

  extraVolumeMounts:
    - name: spicedb-certs
      mountPath: /spicedb-certs
    - name: ca-bundle
      mountPath: /etc/ssl/certs

impersonate:
  extraVolumes:
    - name: ca-bundle
      secret:
        secretName: 'toucan.example.com-cert'
        items:
          - key: ca.crt
            path: ca-certificates.crt

  extraVolumeMounts:
    - name: ca-bundle
      mountPath: /etc/ssl/certs

vault:
  server:
    extraVolumes:
      - name: plugins
        emptyDir: {}
      - name: bootstrap
        configMap:
          name: '{{ printf "%s-bootstrap" (include "vault.server.fullname" .) | trunc 63 | trimSuffix "-" }}'
          defaultMode: 448 # 0700
      - name: vault-init-logs
        emptyDir: {}
      - name: tmp
        emptyDir: {}
      - name: ca-bundle
        secret:
          secretName: 'toucan.example.com-cert'
          items:
            - key: ca.crt
              path: ca-certificates.crt

    extraVolumeMounts:
      - mountPath: /vault/plugins
        name: plugins
        readOnly: true
      - mountPath: /bootstrap
        name: bootstrap
      - mountPath: /logs
        name: vault-init-logs
      - mountPath: /tmp
        name: tmp
      - name: ca-bundle
        mountPath: /etc/ssl/certs/

curity:
  config:
    license:
      secretName: toucan-curity
      secretKey: license

    # Temporary disable the theme, not ready for air-gapped
    theme:
      enabled: false

  runtime:
    ingress:
      enabled: true
      ingressClassName: nginx
      hostname: auth-toucan.example.com
      annotations:
        cert-manager.io/cluster-issuer: http01
      extraTls:
        - hosts:
            - auth-toucan.example.com
          secretName: 'auth-toucan.example.com'

  1. Deploy the Toucan Stack:

shell: /work/
helm upgrade --install toucan-stack oci://quay.io/toucantoco/charts/toucan-stack \
  --namespace toucan \
  --values ./values.override.yaml

If the installation fails with:

shell: /work/
Error: INSTALLATION FAILED: failed post-install: 1 error occurred:
        * timed out waiting for the condition

You should check the health of the deployment. Use kubectl get <deployments/statefulsets/pods> -n toucan to check the status of the deployment. And use kubectl logs <pod-name> -c <container-name> -n toucan to check the logs of the deployment.

We highly recommend using a Kubernetes GUI for troubleshooting like for example Headlamp.

If you want to lock the version, simply set the --version flag:

shell: /work/
helm upgrade --install toucan-stack oci://quay.io/toucantoco/charts/toucan-stack \
  --namespace toucan \
  --version v1.0.0 \
  --values ./values.override.yaml

If you want to customize the values, you can fetch the default values with:

shell: /work/
helm show values oci://quay.io/toucantoco/charts/toucan-stack | less

It's quite long, so we recommend using a YAML editor able to "group" the values.

  1. To get the Admin password, run the following command:

shell: /work/
kubectl get secret --namespace toucan toucan-stack-auth -o jsonpath='{.data.toucan-admin-password}' | base64 --decode

  1. You should be able to access the Toucan Stack at https://toucan.example.com and login with the admin credentials. Enter [email protected] for the username and the password you got from the previous step.

Last updated

Was this helpful?