# Deploy Toucan in Air-Gapped Environment

In this section, we will deploy Toucan in an air-gapped environment using Helm Charts. We'll assume this configuration:

* Traffic is only exposed internally:
  * The machine can only be contacted through private networks (VPC or VPN).
  * The machine cannot be reached from the internet and is stricly blocked by the firewall.
* A **Private DNS** is configured to forward `auth-toucan.example.com` and `toucan.example.com` to the machine IP.

This guide assumes a strict air-gapped environment:

* At first, you have a networked environment. You have access to the internet and can download files to put on a USB stick.
* Then, you deploy the Toucan Stack on your air-gapped environment.

{% hint style="danger" %}
**NOTE**: This guide helps you deploy a simple "one-shot" "all-in-one" Toucan Stack, which might not be suitable for production.

We **heavily** recommend in using an external PostgreSQL database as the one embedded might not be suitable for production. Please follow the following guide to connect to your external database: [Toucan - External Database](/self-hosted-toucan/configuration/external-database.md)
{% endhint %}

## Description and additional requirements

This guide does NOT cover the deployment of Kubernetes in an Air-Gapped environment. If you are interested, we recommend you to read the [k3s - Air-Gap Install](https://docs.k3s.io/installation/airgap) guide. We recommend using the Private Registry method.

In this guide, we plan to follow that method:

1. In a networked environment, you have access to the internet and will download files.
2. In an air-gapped environment, you have no access to the internet and will need to put these files on Kubernetes:
   1. Helm Charts will be hosted on the deployment server containing the tools for installing the Toucan Stack.
   2. The Docker images will be hosted on a local registry.
   3. The deployment will use that local registry to pull the images.

Therefore, you will need:

* A storage to transfer files from the networked environment to the air-gapped environment. Recommended size is 10GB.
* After uploading the container images in the local registry, container layers will be uncompressed. The registry will requires at least the double.

## Preparations in the networked environment

### 1. Download the Private Registry

If you are using minikube, k3s, or another Kubernetes distribution, it's very possible there is already a private registry, or a way to load container images directly on the container runtime. You should check the documentation of your Kubernetes distribution.

If not, in this guide, we'll install [zot](https://zotregistry.dev) as a private registry.

1. Download the Helm Chart

   <pre class="language-shell" data-title="shell: user@home:/work" data-overflow="wrap"><code class="lang-shell">helm pull --repo https://zotregistry.dev/helm-charts zot
   </code></pre>
2. Download the container image:

   <pre class="language-shell" data-title="shell: user@home:/work" data-overflow="wrap"><code class="lang-shell">helm template --repo https://zotregistry.dev/helm-charts zot zot --skip-tests | grep 'image:' | awk '{print $2}' | sort | uniq | while read image; do
     image=$(echo "$image" | sed 's/"//g' | sed "s/'//g")
     echo "Transferring $image"
     docker pull "$image"
     mkdir -p "$(dirname "$image")"
     docker save "$image" | gzip > "$(echo $image | sed 's/:/-/')".tar.gz
   done
   </code></pre>

### 2. Download Toucan-Stack

1. Download the Helm Chart

   <pre class="language-shell" data-title="shell: user@home:/work" data-overflow="wrap"><code class="lang-shell">helm pull oci://quay.io/toucantoco/charts/toucan-stack
   </code></pre>
2. Download the container images:

   <pre class="language-shell" data-title="shell: user@home:/work" data-overflow="wrap"><code class="lang-shell">helm template --set curity.config.license.secretName=dummy toucan-stack oci://quay.io/toucantoco/charts/toucan-stack --skip-tests | grep 'image:' | awk '{print $2}' | sort | uniq | while read image; do
     image=$(echo "$image" | sed 's/"//g' | sed "s/'//g")
     docker pull "$image"
   done
   </code></pre>

You should have every files required for the air-gapped installation!

## Installation in the air-gapped environment

### 1. Deploy the Private Registry

Since there is no registry to host the container registry image (chicken-egg problem), we need to deploy a private registry manually.

1. Transfer the zot container image file directly on the Kubernetes node.

   <pre class="language-shell" data-title="shell: user@home:/work" data-overflow="wrap"><code class="lang-shell">scp ./ghcr.io/project-zot/zot-linux-amd64-v*.tar.gz root@&#x3C;node-ip>:zot-linux-amd64.tar.gz
   </code></pre>

   If you are in a multi-node setup, you should do this for all the nodes. Since the registry requires a volume, it is better to stick the registry to a single node using `nodeSelectors` and use a `local-path`/`hostPath` volume.
2. Import the image in the container runtime:

   <pre class="language-shell" data-title="shell: root@node-0:/home/user" data-overflow="wrap"><code class="lang-shell">gunzip zot-linux-amd64.tar.gz

   # containerd
   ctr --namespace k8s.io image import zot-linux-amd64.tar

   # OR, docker
   docker load -i zot-linux-amd64.tar
   </code></pre>

   <div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><p>You might need to pass <code>--address</code> to <code>ctr</code> since some distributions move the <code>containerd.sock</code>:</p><ul><li>Minikube: <code>/run/docker/containerd/containerd.sock</code></li><li>k0s: <code>/run/k0s/containerd.sock</code></li></ul></div>
3. Deploy the registry using Helm:

   <pre class="language-shell" data-title="shell: user@home:/work" data-overflow="wrap"><code class="lang-shell">helm upgrade --install zot ./zot*.tgz \
     --set persistence=true \
     --set pvc.create=true \
     --set pvc.storageClassName=local-path \
     --set service.nodePort=32000 \
     --set nodeSelector."kubernetes\.io/hostname"=node-0
   </code></pre>

### 2. Transfer the images to the Private Registry

1. Edit the `/etc/docker/daemon.json` to indicates that `<node-0 ip>:32000` is not secured by TLS:

   <pre class="language-json" data-title="json: user@home:/etc/docker/daemon.json" data-overflow="wrap"><code class="lang-json">{
     "insecure-registries": ["&#x3C;node-0 ip>:32000"]
   }
   </code></pre>

   And run:

   <pre class="language-shell" data-title="shell: user@home:/work" data-overflow="wrap"><code class="lang-shell">sudo systemctl restart docker
   </code></pre>
2. Using the images you've pulled from earlier steps, tag them to the private registry:

   <pre class="language-shell" data-title="shell: user@home:/work" data-overflow="wrap"><code class="lang-shell">docker tag &#x3C;registry>/&#x3C;repo>/&#x3C;image>:&#x3C;tag> &#x3C;node-0 ip>:32000/&#x3C;repo>/&#x3C;image>:&#x3C;tag>
   </code></pre>

   <div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><p>If you don't remember which images you've pulled, you can use the following command to push all the images to the private registry:</p><pre class="language-shell" data-title="shell: user@home:/work" data-overflow="wrap"><code class="lang-shell">remove_registry_hostname() {
     image_name="$1"
     slash_count=$(echo "$image_name" | grep -o "/" | wc -l)

   if \[ "$slash\_count" -ge 2 ]; then
   echo "$image\_name" | cut -d '/' -f2-
   else
   echo "$image\_name"
   fi
   }

   helm template --set curity.config.license.secretName=dummy toucan-stack ./toucan-stack\*.tgz --skip-tests | grep 'image:' | awk '{print $2}' | sort | uniq | while read image; do
   image=$(echo "$image" | sed 's/"//g' | sed "s/'//g")
   docker tag "$image" "\<node-0 ip>:32000/$(remove\_registry\_hostname "$image")"
   docker push "\<node-0 ip>:32000/$(remove\_registry\_hostname "$image")"
   done </code></pre><p>Replace <code>\<node-0 ip></code> with the IP address of the node where you've deployed the registry.</p></div>

### 3. Deploy Toucan Stack

{% stepper %}
{% step %}
**Create a namespace**

Create a Kubernetes namespace to deploy the Toucan Stack Helm charts in.

{% code title="shell: /work/" overflow="wrap" %}

```shell
kubectl create namespace toucan
```

{% endcode %}

Namespaces are used to avoid name conflicts between different projects. Since we are deploying a stack of services, we can use the same namespace for all of them, and avoid conflicting with your own projects.
{% endstep %}

{% step %}
**Deploy the Curity secret**

You should have a JSON file in this format:

{% code title="json: Subscription\_YYY-MM-DD.json" overflow="wrap" %}

```json
{
  "Company": "user@example.com",
  "Edition": "Community",
  "Environment": "",
  "Expires": "2025-12-13",
  "Feature": "",
  "GracePeriod": "30",
  "Groups": "1",
  "Issued": "2024-12-13",
  "Licence": "ey...",
  "License": "ey...",
  "Product": "Identity Server",
  "Tier": "Subscription",
  "Type": "Production",
  "Version": "4.3",
  "expired": false
}
```

{% endcode %}

Copy the value from the `License` or `Licence` field, and create the secret with:

{% code title="shell: /work/" overflow="wrap" %}

```shell
# generic: The type of secret to create.
# --namespace: The namespace to create the secret in.
# curity-secret: The name of the secret to create.
# --from-literal: The key and value of the secret to create.
kubectl create secret generic --namespace toucan curity-secret \
  --from-literal=CURITY_LICENSE_KEY=<value>
```

{% endcode %}

Replace `<value>` with your with the value from the JSON file, i.e. the `License` or `Licence` field.
{% endstep %}

{% step %}
**Deploy the Helm charts**

Since we are using Helm, we can patch the necessary values to inject the credentials and secrets. We also need to expose the service to the external network and secure it with TLS.

1. Create the values file, which will override the default values.

   <pre class="language-shell" data-title="shell: /work/" data-overflow="wrap"><code class="lang-shell">touch values.override.yaml # You can name it whatever you want
   </code></pre>
2. (optional) For strict air-gapped environments, assuming you have transferred the container images to your air-gapped container registry, add these lines to `values.override.yaml` override the registry:

   <pre class="language-yaml" data-title="yaml: /work/values.override.yaml"><code class="lang-yaml">global:
     imageRegistry: 'localhost:32000'
     security:
       allowInsecureImages: true

   # Gotenberg doesn't use global.imageRegistry
   gotenberg:
     image:
       repository: localhost:32000/gotenberg/gotenberg
   </code></pre>
3. Add these lines to disable password checking since it requires an internet connection:

   <pre class="language-yaml" data-title="yaml: /work/values.override.yaml"><code class="lang-yaml">curity:
     config:
       credentialPolicy:
         # Since we're air-gapped, we cannot download the dictionary.
         dictionary:
           enabled: false
   </code></pre>
4. Add this line to inject the Curity secret:

   <pre class="language-yaml" data-title="yaml: /work/values.override.yaml"><code class="lang-yaml"># ...

   curity:
     config:
       license:
         secretName: curity-secret
         secretKey: CURITY_LICENSE_KEY
   </code></pre>
5. Add this line to select your storage provisioner:

   <pre class="language-yaml" data-title="yaml: /work/values.override.yaml"><code class="lang-yaml">global:
     # ...
     defaultStorageClass: local-path
   # ...
   </code></pre>

   <div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><p>You can fetch the available storage classes with:</p><pre class="language-shell" data-title="shell: /work/" data-overflow="wrap"><code class="lang-shell">kubectl get storageclass
   </code></pre><p>You should see something like this:</p><pre class="language-shell" data-title="shell: /work/"><code class="lang-shell">NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
   local-path           rancher.io/local-path   Delete          WaitForFirstConsumer   false                  121d
   </code></pre></div>
6. (Optional) Override the volume size:

   <pre class="language-yaml" data-title="yaml: /work/values.override.yaml"><code class="lang-yaml"># ...
   laputa:
     persistence:
       size: 10Gi

   curity:
     # ...
     admin:
       persistence:
         size: 8Gi

   postgresql:
     primary:
       persistence:
         size: 10Gi

   mongodb:
     persistence:
       size: 8Gi
   </code></pre>

   <div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><p><strong>NOTE</strong>: This is only useful if you are using a storage provisioner which handles "sizing". The <code>local-path-provisioner</code> does NOT! So these values doesn't mean anything, but many cloud provider and block storage provisioners do.</p></div>
7. Configure TLS for the Toucan Stack:

   <div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><p><strong>SUGGESTION</strong>: We recommend using <a href="https://cert-manager.io/">cert-manager</a> to issue TLS certificates, which is able to rotatecertificates on a regular basis.</p><p>You can also use <a href="https://github.com/external-secrets/external-secrets">external-secrets</a>, to fetch TLS certificates from a secretmanager like AWS Secrets Manager, Hashicorp Vault, etc.</p></div>

   Create these files:

   <pre class="language-yaml" data-title="yaml: tls-secret.yaml" data-overflow="wrap"><code class="lang-yaml">apiVersion: v1
   kind: Secret
   metadata:
     name: toucan.example.com-tls
   stringData:
     tls.crt: |
       -----BEGIN CERTIFICATE-----
       ...
       -----END CERTIFICATE-----
     tls.key: |
       -----BEGIN RSA PRIVATE KEY-----
       ...
       -----END RSA PRIVATE KEY-----
     ca.crt: |
       -----BEGIN CERTIFICATE-----
       ...
       -----END CERTIFICATE-----
   type: kubernetes.io/tls
   ---
   apiVersion: v1
   kind: Secret
   metadata:
     name: auth-toucan.example.com-tls
   stringData:
     tls.crt: |
       -----BEGIN CERTIFICATE-----
       ...
       -----END CERTIFICATE-----
     tls.key: |
       -----BEGIN RSA PRIVATE KEY-----
       ...
       -----END RSA PRIVATE KEY-----
     ca.crt: |
       -----BEGIN CERTIFICATE-----
       ...
       -----END CERTIFICATE-----
   type: kubernetes.io/tls
   </code></pre>

   Deploy the certificates with:

   <pre class="language-shell" data-title="shell: /work/" data-overflow="wrap"><code class="lang-shell">kubectl apply -n &#x3C;namespace> -f tls-secret.yaml
   </code></pre>
8. Expose the Toucan Stack by adding these lines:

   <pre class="language-yaml" data-title="yaml: /work/values.override.yaml"><code class="lang-yaml">global:
     # ...

     ## global.hostname configures the helm chart to use toucan.example.com as "public" domain.
     hostname: toucan.example.com

   canopee:
     ingress:
       enabled: true
       ingressClassName: nginx
       tls: true
       extraTls:
         - hosts:
             - toucan.example.com
           secretName: 'toucan.example.com-tls' # This secret will be generated.

   curity:
     # ...
     runtime:
       ingress:
         enabled: true
         ingressClassName: nginx
         hostname: auth-toucan.example.com
         tls: true
         extraTls:
           - hosts:
               - auth-toucan.example.com
             secretName: 'auth-toucan.example.com-tls' # This secret will be generated.
   </code></pre>

   <div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><p>Annotations are used by controllers like cert-manager to trigger side effects.</p></div>
9. Lastly, you need to inject the CA's certificate to the internal services that uses `toucan.example.com`:

   <pre class="language-yaml" data-title="yaml: /work/values.override.yaml"><code class="lang-yaml">laputa:
     config:
       common:
         REQUESTS_CA_BUNDLE: /etc/ssl/certs/ca-certificates.crt

     extraVolumes:
       - name: ca-bundle
         secret:
           secretName: 'toucan.example.com-cert' # Change this
           items:
             - key: ca.crt
               path: my-ca.crt # (optional) To change

     extraVolumeMounts:
       - name: ca-bundle
         mountPath: /usr/local/share/ca-certificates/my-ca.crt # Must match secret path
         subPath: my-ca.crt # Must match secret path

   layout:
     extraEnvVars:
       - name: NODE_EXTRA_CA_CERTS
         value: /etc/ssl/certs/ca-certificates.crt

     extraVolumes:
       - name: ca-bundle
         secret:
           secretName: 'auth-toucan.example.com-tls' # Change this
           items:
             - key: ca.crt
               path: ca-certificates.crt

     extraVolumeMounts:
       - name: ca-bundle
         mountPath: /etc/ssl/certs

   dataset:
     extraEnvVars:
       - name: SSL_CERT_FILE # For httpx
         value: /etc/ssl/certs/ca-certificates.crt
       - name: REQUESTS_CA_BUNDLE # For requests
         value: /etc/ssl/certs/ca-certificates.crt

     extraVolumes:
       - name: ca-bundle
         secret:
           secretName: 'auth-toucan.example.com-tls' # Change this
           items:
             - key: ca.crt
               path: ca-certificates.crt

     extraVolumeMounts:
       - name: ca-bundle
         mountPath: /etc/ssl/certs

   dataexecution:
     api:
       extraVolumes:
         - name: ca-bundle
           secret:
             secretName: 'auth-toucan.example.com-tls' # Change this
             items:
               - key: ca.crt
                 path: my-ca.crt # (optional) To change

       extraVolumeMounts:
         - name: ca-bundle
           mountPath: /usr/local/share/ca-certificates/my-ca.crt # Must match secret path
           subPath: my-ca.crt # Must match secret path

     worker:
       extraVolumes:
         - name: ca-bundle
           secret:
             secretName: 'auth-toucan.example.com-tls' # Change this
             items:
               - key: ca.crt
                 path: my-ca.crt # (optional) To change

       extraVolumeMounts:
         - name: ca-bundle
           mountPath: /usr/local/share/ca-certificates/my-ca.crt # Must match secret path
           subPath: my-ca.crt # Must match secret path

   impersonate:
     extraVolumes:
       - name: ca-bundle
         secret:
           secretName: 'auth-toucan.example.com-tls' # Change this
           items:
             - key: ca.crt
               path: ca-certificates.crt

     extraVolumeMounts:
       - name: ca-bundle
         mountPath: /etc/ssl/certs

   vault:
     server:
       extraVolumes:
         - name: ca-bundle
           secret:
             secretName: 'auth-toucan.example.com-tls' # Change this
             items:
               - key: ca.crt
                 path: ca-certificates.crt

       extraVolumeMounts:
         - name: ca-bundle
           mountPath: /etc/ssl/certs/
   </code></pre>
10. At this point, your `values.override.yaml` should looks like (minus the volume size overrides):

{% code title="yaml: /work/values.override.yaml" %}

```yaml
global:
  imageRegistry: 'localhost:32000'
  security:
    allowInsecureImages: true
  imagePullSecrets:
    - dockerconfigjson
  defaultStorageClass: local-path
  hostname: toucan.example.com

canopee:
  ingress:
    enabled: true
    ingressClassName: nginx
    annotations:
      cert-manager.io/cluster-issuer: private-cluster-issuer
    tls: true

laputa:
  config:
    common:
      REQUESTS_CA_BUNDLE: /etc/ssl/certs/ca-certificates.crt

  extraVolumes:
    - name: ca-bundle
      secret:
        secretName: 'toucan.example.com-tls'
        items:
          - key: ca.crt
            path: ca-certificates.crt

  extraVolumeMounts:
    - name: ca-bundle
      mountPath: /etc/ssl/certs

layout:
  extraEnvVars:
    - name: NODE_EXTRA_CA_CERTS
      value: /etc/ssl/certs/ca-certificates.crt

  extraVolumes:
    - name: ca-bundle
      secret:
        secretName: 'toucan.example.com-tls'
        items:
          - key: ca.crt
            path: ca-certificates.crt

  extraVolumeMounts:
    - name: ca-bundle
      mountPath: /etc/ssl/certs

dataset:
  extraEnvVars:
    - name: SSL_CERT_FILE # For httpx
      value: /etc/ssl/certs/ca-certificates.crt
    - name: REQUESTS_CA_BUNDLE # For requests
      value: /etc/ssl/certs/ca-certificates.crt

  extraVolumes:
    - name: ca-bundle
      secret:
        secretName: 'toucan.example.com-tls'
        items:
          - key: ca.crt
            path: ca-certificates.crt

  extraVolumeMounts:
    - name: ca-bundle
      mountPath: /etc/ssl/certs

dataexecution:
  api:
    extraVolumes:
      - name: ca-bundle
        secret:
          secretName: 'toucan.example.com-tls'
          items:
            - key: tls.crt
              path: ca-certificates.crt

    extraVolumeMounts:
      - name: ca-bundle
        mountPath: /etc/ssl/certs/

  worker:
    extraVolumes:
      - name: ca-bundle
        secret:
          secretName: 'toucan.example.com-tls'
          items:
            - key: tls.crt
              path: ca-certificates.crt

    extraVolumeMounts:
      - name: ca-bundle
        mountPath: /etc/ssl/certs/

impersonate:
  extraVolumes:
    - name: ca-bundle
      secret:
        secretName: 'toucan.example.com-tls'
        items:
          - key: ca.crt
            path: ca-certificates.crt

  extraVolumeMounts:
    - name: ca-bundle
      mountPath: /etc/ssl/certs

gotenberg:
  image:
    repository: localhost:32000/gotenberg/gotenberg

vault:
  server:
    extraVolumes:
      - name: ca-bundle
        secret:
          secretName: 'toucan.example.com-tls'
          items:
            - key: ca.crt
              path: ca-certificates.crt

    extraVolumeMounts:
      - name: ca-bundle
        mountPath: /etc/ssl/certs/

curity:
  config:
    license:
      secretName: curity-secret
      secretKey: CURITY_LICENSE_KEY

    credentialPolicy:
      dictionary:
        enabled: false

  runtime:
    ingress:
      enabled: true
      ingressClassName: nginx
      hostname: auth-toucan.example.com
      annotations:
        cert-manager.io/cluster-issuer: private-cluster-issuer
      tls: true
```

{% endcode %}

11. Deploy the Toucan Stack:

{% code title="shell: /work/" overflow="wrap" %}

```shell
helm upgrade --install toucan-stack ./toucan-stack*.tgz \
  --namespace toucan \
  --values ./values.override.yaml
```

{% endcode %}

{% hint style="info" %}
If the installation fails with:

{% code title="shell: /work/" overflow="wrap" %}

```shell
Error: INSTALLATION FAILED: failed post-install: 1 error occurred:
        * timed out waiting for the condition
```

{% endcode %}

You should check the health of the deployment. Use `kubectl get <deployments/statefulsets/pods> -n toucan` to check the status of the deployment. And use `kubectl logs <pod-name> -c <container-name> -n toucan` to check the logs of the deployment.

We **highly recommend** using a Kubernetes GUI for troubleshooting like for example [Headlamp](https://headlamp.dev).
{% endhint %}

12. To get the Admin password, run the following command:

{% code title="shell: /work/" overflow="wrap" %}

```shell
kubectl get secret --namespace toucan toucan-stack-auth -o jsonpath='{.data.toucan-admin-password}' | base64 --decode
```

{% endcode %}

13. You should be able to access the Toucan Stack at <https://toucan.example.com> and login with the admin credentials. Enter `admin@example.com` for the username and the password you got from the previous step.
    {% endstep %}
    {% endstepper %}


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs-v3.toucantoco.com/self-hosted-toucan/getting-started/air-gapped.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.