⛴️Deploy Toucan in Air-Gapped Environment
In this section, we will deploy Toucan in an air-gapped environment using Helm Charts. We'll assume this configuration:
Traffic is only exposed internally:
The machine can only be contacted through private networks (VPC or VPN).
The machine cannot be reached from the internet and is stricly blocked by the firewall.
A Private DNS is configured to forward
auth-toucan.example.com
andtoucan.example.com
to the machine IP. The Kubernetes internal DNS will be used as an example.
Also, for simplification reasons of this guide, we consider that the Docker images and Helm Charts are still accessible.
If you are in a strict air-gapped environment, you should:
Self-host a private registry like Harbor, zot, or the official container registry.
Download the Docker images and Helm Charts, in which you have verified the security, integrity and origin of the image. (Container image signature and Helm Chart signature are not available yet. It is planned.)
Here's a list of Docker images and Helm Charts to download:
# Helm Charts quay.io/toucantoco/charts/toucan-stack:latest # The following container images are used by the Toucan Stack Helm Chart, # but is probably outdated. # # Check the list of images, using: # # helm template --set curity.config.license.secretName=dummy release-name oci://quay.io/toucantoco/charts/toucan-stack | grep 'image:' | sort | uniq # Toucan Services: quay.io/toucantoco/frontend:v145.0.1 quay.io/toucantoco/ngx-auth-module:v1.6.0-toucan.5-debian12-ngx1.27.4 quay.io/toucantoco/dataset_service:v1.11.0 quay.io/toucantoco/impersonate-token-service:v0.3.1 quay.io/toucantoco/backend:v145.0.1 quay.io/toucantoco/layout-service:v1.11.0 # Third party services: curity.azurecr.io/curity/idsvr:10.1.0 docker.io/bitnami/redis:7.4.2-debian-12-r6 gotenberg/gotenberg:8.15.3 docker.io/bitnami/nginx:1.27.4 docker.io/library/mongo:8.0.8 ghcr.io/authzed/spicedb:v1.38.0 docker.io/bitnami/vault:1.19.0-debian-12-r0 docker.io/bitnami/postgresql:17.4.0-debian-12-r11 # Utilities images: docker.io/library/busybox:1.37 docker.io/library/alpine:3.21 docker.io/alpine/curl:8.21.1 ghcr.io/authzed/zed:v0.27.0-debug docker.io/hairyhenderson/gomplate:v4.3.2-alpine
Transfer these images and Helm Charts to your air-gapped OCI registry.
Lastly, we assume you have installed k3s and NGINX Ingress Controller on your air-gapped environment. Here's a guide on how to install k3s on your air-gapped environment: K3s - Air-Gap Install.
Configure TLS with cert-manager
This guide uses cert-manager to manage the TLS certificates for the ingress controller. cert-manager automatically fetches certificates from ACME (Automatic Certificate Management Environment) servers, such as Let's Encrypt.
If your prefer to self-manage certificates, you can use kubectl
to import secrets directly. However, we heavily recommend to use cert-manager to manage the private and public TLS certificates for the ingress controller.
# (in the networked environment) You can pull the tar.gz before you install cert-manager
helm pull --repo https://charts.jetstack.io cert-manager
# (in the air-gapped environment)
helm install \
cert-manager ./cert-manager-v<version>.tgz \
--namespace cert-manager \
--create-namespace \
--set crds.enabled=true
To generate certificates, we need a ClusterIssuer
resource. We are assuming you have no Certificate Authority (CA) in your air-gapped environment, and we need to create one.
Create a self-signed issuer:
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: selfsigned-issuer
namespace: cert-manager
spec:
selfSigned: {}
This self-signed issuer will be used to create a Certificate Authority. Deploy it:
kubectl apply -f selfsigned-issuer.yaml
Create a
Certificate
for the Certificate Authority:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: ca-certificate
namespace: cert-manager
spec:
secretName: root-ca
issuerRef:
name: selfsigned-issuer
kind: Issuer
isCA: true
duration: 43800h # 5 year
renewBefore: 720h # 30 days before expiry
privateKey:
algorithm: RSA
encoding: PKCS1
size: 2048
subject:
organizations: [Toucan Toco]
countries: [FR]
organizationalUnits: [IT]
localities: [Paris]
commonName: Toucan Root CA
Deploy it:
kubectl apply -f ca-certificate.yaml
Create a
ClusterIssuer
that can be used by the whole cluster to fetch certificate from the CA:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: private-cluster-issuer
namespace: cert-manager
spec:
ca:
secretName: root-ca
Deploy it:
kubectl apply -f cluster-issuer.yaml
Deploy Toucan Stack
Create a namespace
Create a Kubernetes namespace to deploy the Toucan Stack Helm charts in.
kubectl create namespace toucan
Namespaces are used to avoid name conflicts between different projects. Since we are deploying a stack of services, we can use the same namespace for all of them, and avoid conflicting with your own projects.
Deploy the registry credentials
Since we are assuming you have access to Quay for "simplicity", we need to create a secret to access it:
# Replace <username> and <password> with your credentials
# docker-registry: The type of secret to create.
# --namespace: The namespace to create the secret in.
# dockerconfigjson: The name of the secret to create.
# --docker-server: The server address of the registry.
# --docker-username: The username for the registry.
# --docker-password: The password for the registry.
kubectl create secret docker-registry --namespace toucan dockerconfigjson \
--docker-server=quay.io \
--docker-username=<username> \
--docker-password=<password>
In a strict air-gapped environment, the secret would be the credentials to access your air-gapped registry.
Deploy the Curity secret
You should have a JSON file in this format:
{
"Company": "[email protected]",
"Edition": "Community",
"Environment": "",
"Expires": "2025-12-13",
"Feature": "",
"GracePeriod": "30",
"Groups": "1",
"Issued": "2024-12-13",
"Licence": "ey...",
"License": "ey...",
"Product": "Identity Server",
"Tier": "Subscription",
"Type": "Production",
"Version": "4.3",
"expired": false
}
Copy the value from the License
or Licence
field, and create the secret with:
# Replace <value> with your license
# generic: The type of secret to create.
# --namespace: The namespace to create the secret in.
# toucan-curity: The name of the secret to create.
# --from-literal: The key and value of the secret to create.
kubectl create secret generic --namespace toucan toucan-curity \
--from-literal=license=<value>
Deploy the Helm charts
Since we are using Helm, we can patch the necessary values to inject the credentials and secrets. We also need to expose the service to the external network and secure it with TLS.
Create the values file, which will override the default values.
touch values.override.yaml # You can name it whatever you want
(optional) For strict air-gapped environments, assuming you have transferred the container images to your air-gapped container registry in this way:
docker.io/org/image:tag -> my-registry/org/image:tag
Add these lines to
values.override.yaml
override the registry:
global:
imageRegistry: 'my-registry'
security:
allowInsecureImages: true
If you need to override the image precisely, look into the values and override the image
field. For example, if you want to override the laputa
image to use a different one, check the values:
helm show values oci://quay.io/toucantoco/charts/toucan-stack | less
Look for the field laputa:
and image:
(type / to search in less
), you'll something similar to:
laputa:
# ... (truncated)
image:
registry: quay.io
repository: toucantoco/backend
# renovate: image=quay.io/toucantoco/backend
tag: 'v144.0.2'
digest: ''
## Specify a imagePullPolicy
## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent'
## ref: https://kubernetes.io/docs/concepts/containers/images/#pre-pulled-images
##
pullPolicy: IfNotPresent
## Optionally specify an array of imagePullSecrets.
## Secrets must be manually created in the namespace.
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
## e.g:
## pullSecrets:
## - myRegistryKeySecretName
##
pullSecrets: []
Copy and replace accordingly in the values.override.yaml
file.
Add this line to
values.override.yaml
inject the registry credentials:
global:
imagePullSecrets:
- dockerconfigjson
Add this line to inject the Curity secret:
# ...
curity:
config:
license:
secretName: toucan-curity
secretKey: license
Add this line to select your storage provisioner:
global:
# ...
defaultStorageClass: local-path
# ...
(Optional) Override the volume size:
# ...
laputa:
persistence:
size: 10Gi
curity:
# ...
admin:
persistence:
size: 8Gi
postgresql:
primary:
persistence:
size: 10Gi
mongodb:
persistence:
size: 8Gi
Expose the Toucan Stack by adding these lines:
global:
# ...
## global.hostname configures the helm chart to use toucan.example.com as "public" domain.
hostname: toucan.example.com
nginx:
ingress:
enabled: true
ingressClassName: nginx
annotations:
cert-manager.io/cluster-issuer: private-cluster-issuer # private-cluster-issuer references the previously created ClusterIssuer
extraTls:
- hosts:
- toucan.example.com
secretName: 'toucan.example.com-cert' # This secret will be generated.
curity:
# ...
runtime:
ingress:
enabled: true
ingressClassName: nginx
hostname: auth-toucan.example.com
annotations:
cert-manager.io/cluster-issuer: private-cluster-issuer # private-cluster-issuer references the previously created ClusterIssuer
extraTls:
- hosts:
- auth-toucan.example.com
secretName: 'auth-toucan.example.com' # This secret will be generated.
Lastly, you need to inject the CA's certificate to the internal services that uses
toucan.example.com
:
laputa:
config:
common:
REQUESTS_CA_BUNDLE: /etc/ssl/certs/ca-certificates.crt
extraVolumes:
- name: spicedb-certs
secret:
secretName: '{{ template "toucan-stack.spicedb.tls.secretName" . }}'
items:
- key: ca.crt
path: ca.crt
- name: ca-bundle
secret:
secretName: 'toucan.example.com-cert'
items:
- key: ca.crt
path: ca-certificates.crt
extraVolumeMounts:
- name: spicedb-certs
mountPath: /spicedb-certs
- name: ca-bundle
mountPath: /etc/ssl/certs
layout:
extraEnvVars:
- name: NODE_EXTRA_CA_CERTS
value: /etc/ssl/certs/ca-certificates.crt
extraVolumes:
- name: spicedb-certs
secret:
secretName: '{{ template "toucan-stack.spicedb.tls.secretName" . }}'
items:
- key: ca.crt
path: ca.crt
- name: ca-bundle
secret:
secretName: 'toucan.example.com-cert'
items:
- key: ca.crt
path: ca-certificates.crt
extraVolumeMounts:
- name: spicedb-certs
mountPath: /spicedb-certs
- name: ca-bundle
mountPath: /etc/ssl/certs
dataset:
extraEnvVars:
- name: SSL_CERT_FILE # For httpx
value: /etc/ssl/certs/ca-certificates.crt
- name: REQUESTS_CA_BUNDLE # For requests
value: /etc/ssl/certs/ca-certificates.crt
extraVolumes:
- name: spicedb-certs
secret:
secretName: '{{ template "toucan-stack.spicedb.tls.secretName" . }}'
items:
- key: ca.crt
path: ca.crt
- name: ca-bundle
secret:
secretName: 'toucan.example.com-cert'
items:
- key: ca.crt
path: ca-certificates.crt
extraVolumeMounts:
- name: spicedb-certs
mountPath: /spicedb-certs
- name: ca-bundle
mountPath: /etc/ssl/certs
impersonate:
extraVolumes:
- name: ca-bundle
secret:
secretName: 'toucan.example.com-cert'
items:
- key: ca.crt
path: ca-certificates.crt
extraVolumeMounts:
- name: ca-bundle
mountPath: /etc/ssl/certs
vault:
server:
extraVolumes:
- name: ca-bundle
secret:
secretName: 'toucan.example.com-cert'
items:
- key: ca.crt
path: ca-certificates.crt
extraVolumeMounts:
- name: ca-bundle
mountPath: /etc/ssl/certs/
At this point, your
values.override.yaml
should looks like (minus the volume size overrides):
global:
## Uncomment if you are using a custom registry.
# imageRegistry: 'my-registry'
# security:
# allowInsecureImages: true
imagePullSecrets:
- dockerconfigjson
defaultStorageClass: local-path
hostname: toucan.example.com
nginx:
ingress:
enabled: true
ingressClassName: nginx
annotations:
cert-manager.io/cluster-issuer: http01
extraTls:
- hosts:
- toucan.example.com
secretName: 'toucan.example.com-cert'
laputa:
config:
common:
REQUESTS_CA_BUNDLE: /etc/ssl/certs/ca-certificates.crt
extraVolumes:
- name: spicedb-certs
secret:
secretName: '{{ template "toucan-stack.spicedb.tls.secretName" . }}'
items:
- key: ca.crt
path: ca.crt
- name: ca-bundle
secret:
secretName: 'toucan.example.com-cert'
items:
- key: ca.crt
path: ca-certificates.crt
extraVolumeMounts:
- name: spicedb-certs
mountPath: /spicedb-certs
- name: ca-bundle
mountPath: /etc/ssl/certs
layout:
extraEnvVars:
- name: NODE_EXTRA_CA_CERTS
value: /etc/ssl/certs/ca-certificates.crt
extraVolumes:
- name: spicedb-certs
secret:
secretName: '{{ template "toucan-stack.spicedb.tls.secretName" . }}'
items:
- key: ca.crt
path: ca.crt
- name: ca-bundle
secret:
secretName: 'toucan.example.com-cert'
items:
- key: ca.crt
path: ca-certificates.crt
extraVolumeMounts:
- name: spicedb-certs
mountPath: /spicedb-certs
- name: ca-bundle
mountPath: /etc/ssl/certs
dataset:
extraEnvVars:
- name: SSL_CERT_FILE # For httpx
value: /etc/ssl/certs/ca-certificates.crt
- name: REQUESTS_CA_BUNDLE # For requests
value: /etc/ssl/certs/ca-certificates.crt
extraVolumes:
- name: spicedb-certs
secret:
secretName: '{{ template "toucan-stack.spicedb.tls.secretName" . }}'
items:
- key: ca.crt
path: ca.crt
- name: ca-bundle
secret:
secretName: 'toucan.example.com-cert'
items:
- key: ca.crt
path: ca-certificates.crt
extraVolumeMounts:
- name: spicedb-certs
mountPath: /spicedb-certs
- name: ca-bundle
mountPath: /etc/ssl/certs
impersonate:
extraVolumes:
- name: ca-bundle
secret:
secretName: 'toucan.example.com-cert'
items:
- key: ca.crt
path: ca-certificates.crt
extraVolumeMounts:
- name: ca-bundle
mountPath: /etc/ssl/certs
vault:
server:
extraVolumes:
- name: plugins
emptyDir: {}
- name: bootstrap
configMap:
name: '{{ printf "%s-bootstrap" (include "vault.server.fullname" .) | trunc 63 | trimSuffix "-" }}'
defaultMode: 448 # 0700
- name: vault-init-logs
emptyDir: {}
- name: tmp
emptyDir: {}
- name: ca-bundle
secret:
secretName: 'toucan.example.com-cert'
items:
- key: ca.crt
path: ca-certificates.crt
extraVolumeMounts:
- mountPath: /vault/plugins
name: plugins
readOnly: true
- mountPath: /bootstrap
name: bootstrap
- mountPath: /logs
name: vault-init-logs
- mountPath: /tmp
name: tmp
- name: ca-bundle
mountPath: /etc/ssl/certs/
curity:
config:
license:
secretName: toucan-curity
secretKey: license
runtime:
ingress:
enabled: true
ingressClassName: nginx
hostname: auth-toucan.example.com
annotations:
cert-manager.io/cluster-issuer: http01
extraTls:
- hosts:
- auth-toucan.example.com
secretName: 'auth-toucan.example.com'
Deploy the Toucan Stack:
{% code title="shell: /work/" overflow="wrap" %}
helm upgrade --install toucan-stack oci://quay.io/toucantoco/charts/toucan-stack \ --namespace toucan \ --values ./values.override.yaml
{% endcode %}
{% hint style="info" %}
If the installation fails with:
{% code title="shell: /work/" overflow="wrap" %}
Error: INSTALLATION FAILED: failed post-install: 1 error occurred: * timed out waiting for the condition
{% endcode %}
You should check the health of the deployment. Use
kubectl get <deployments/statefulsets/pods> -n toucan
to check the status of the deployment. And usekubectl logs <pod-name> -c <container-name> -n toucan
to check the logs of the deployment.We highly recommend using a Kubernetes GUI for troubleshooting like for example Headlamp.
{% endhint %}
{% hint style="info" %}
If you want to lock the version, simply set the
--version
flag:{% code title="shell: /work/" overflow="wrap" %}
helm upgrade --install toucan-stack oci://quay.io/toucantoco/charts/toucan-stack \ --namespace toucan \ --version v1.0.0 \ --values ./values.override.yaml
{% endcode %}
If you want to customize the values, you can fetch the default values with:
{% code title="shell: /work/" overflow="wrap" %}
helm show values oci://quay.io/toucantoco/charts/toucan-stack | less
{% endcode %}
It's quite long, so we recommend using a YAML editor able to "group" the values.
{% endhint %}
To get the Admin password, run the following command:
{% code title="shell: /work/" overflow="wrap" %}
kubectl get secret --namespace toucan toucan-stack-auth -o jsonpath='{.data.toucan-admin-password}' | base64 --decode
{% endcode %}
You should be able to access the Toucan Stack at https://toucan.example.com and login with the admin credentials. Enter
[email protected]
for the username and the password you got from the previous step.
Last updated
Was this helpful?