🔧Troubleshooting Bootstraping

This guide will help you debug common bootstraping issues.

Issues appearing here only happen on bootstrap. Other issues happening at runtime are not documented here.

Post-install job is failing

The post-install job is failing.

The issue is a configuration issue, or a network issue.

Detection

bash

kubectl get jobs --all-namespaces

With jobs looking like:

Example

NAMESPACE       NAME                              STATUS     COMPLETIONS   DURATION   AGE
<namespace>     toucan-stack-postinstall          Failed     0/1           1m         1d

Possible causes

The post-install job initializes:

• The admin account.

• The permissions of the admin account.

• The permissions of the backend.

Therefore, the issues could be linked to:

• The reachability of the authentication server Keycloak.

• The reachability of the authorization server SpiceDB.

• A programming error.

The first issue is very likely caused by configuration or network issue.

The second issue is very unlikely because the authorization server can only be accessible internally. Meaning, the Helm Chart should have already been configured for the authorization server to be reachable.

Investigation

bash

kubectl logs -n demo 'toucan-stack-postinstall-...' -c <create-relationship/init-workspace>

Logs should indicates the errors. Search the errors on Google.

Issues could be linked to (but not limited to):

• A programming error.

Mitigation

Contact the support for help.

Vault is in CrashLoopBackOff state

The vault container is crashing repeatedly, stuck in crash loop.

The issue is a configuration issue, or a network issue.

Detection

bash

kubectl get pods --all-namespaces
kubectl get events --all-namespaces

With events looking like:

Example

Events:
  Type     Reason               Age                 From               Message
  ----     ------               ----                ----               -------
  Warning  FailedPostStartHook  35s (x3 over 117s)  kubelet            PostStartHook failed
  Normal   Killing              35s (x3 over 117s)  kubelet            FailedPostStartHook
  Warning  BackOff              16s (x7 over 86s)   kubelet            Back-off restarting failed container server in pod toucan-stack-vault-server-0_demo(7b241ae3-4338-4dbb-9017-a74b22d6bfb4)

Possible causes

On bootstrap, if the vault container is crashing, it's very likely linked to the PostStart lifecycle Hook.

The lifecycle Hook is responsible for:

1. Fetching or inserting the vault bootstraping secret.

2. Setup the KV2 secrets engine.

Step 1 can fail if RBAC has been disabled: configuration issue.

Lastly, it can also fail due to implementation errors.

Investigation

bash

# Check the log of the lifecycle hook.
# NOTE: It might not show anything.
kubectl logs toucan-stack-vault-server-0 -c tail-vault-init -n <namespace>
kubectl describe pod toucan-stack-vault-server-0 -n <namespace>

Mitigation

Failed at step 1

Check if the RBAC is properly configured:

• A Role and RoleBinding both named toucan-stack-vault-server-secret-manager (or similar) must be present.

• A ServiceAccount named toucan-stack-vault-server (or similar) must be present.

Unknown failures

Contact the support.