# Configure an external S3

## Layout

Toucan requires 2 buckets:

* `dataexecution-cache`, the cache for the data execution service.
* `toucan-data`, the main bucket used to store your data when users drop files as data sources.

It is recommended to set up 3 keys:

* `dataexecution` which has Read-Write access to the `dataexecution-cache` bucket.
* `toucan_ro` which has Read access to the `toucan-data` bucket.
* `toucan` which has Read-Write access to the `toucan-data` bucket.

## Configuration

{% stepper %}
{% step %}
**Disable the embedded S3**

Set these parameters in your values file:

{% code title="yaml: values.override.yaml" %}

```yaml
garage:
  enabled: false
```

{% endcode %}
{% endstep %}

{% step %}
**Set up the credentials**

Set these parameters in your values file, so that Toucan can use the credentials:

{% code title="yaml: values.override.yaml" %}

```yaml
global:
  s3:
    keys:
      dataexecution:
        id: <AWS_ACCESS_KEY_ID>

        secret: <AWS_SECRET_ACCESS_KEY>
        # OR
        existingSecret:
          name: '<K8S Secret Name>'
          key: '<K8S Secret Key>'

      toucan_ro:
        id: <AWS_ACCESS_KEY_ID>

        secret: <AWS_SECRET_ACCESS_KEY>
        # OR
        existingSecret:
          name: '<K8S Secret Name>'
          key: '<K8S Secret Key>'

      toucan:
        id: <AWS_ACCESS_KEY_ID>

        secret: <AWS_SECRET_ACCESS_KEY>
        # OR
        existingSecret:
          name: '<K8S Secret Name>'
          key: '<K8S Secret Key>'
```

{% endcode %}

{% hint style="info" %}
You do not need to fill the `name`, `expiration` and `neverExpires` fields.
{% endhint %}
{% endstep %}

{% step %}
**Replace references to garage**

Set these parameters in your values file, so that Toucan can connect to the external S3:

{% code title="yaml: values.override.yaml" %}

```yaml
laputa:
  config:
    s3_storage:
      bucket_name: '<your-toucan-data-bucket>' # 'toucan-data'
      region_name: '<your-aws-region>' # 'fr-par', check your S3 provider
      endpoint_url: '<your-external-s3-endpoint-url>' # 'https://<your-external-s3-endpoint-url>'
      verify: true # Check TLS certificate.
    common:
      TOUCAN_S3_TOUCAN_URI_AUTH_ENCODED_RW: 's3://{{ .Values.global.s3.keys.toucan.id }}:$(TOUCAN_S3_TOUCAN_SECRET_KEY)@<your-toucan-data-bucket>'
      TOUCAN_S3_TOUCAN_URI_AUTH_ENCODED_RO: 's3://{{ .Values.global.s3.keys.toucan_ro.id }}:$(TOUCAN_S3_TOUCAN_RO_SECRET_KEY)@<your-toucan-data-bucket>'

dataexecution:
  config:
    specific:
      bucket_name: '<your-dataexecution-cache-bucket>' # 'dataexecution-cache'
      region: '<your-aws-region>' # 'fr-par', check your S3 provider
      endpoint: '<your-external-s3-endpoint-url>' # 'https://<your-external-s3-endpoint-url>'

vault:
  bootstrap:
    s3:
      # Sadly, this is hardcoded inside the dataset service code.
      # If you wish to change it, feel free to send us a feedback.
      path: secret/{{ .Values.dataset.config.environment }}/{{ .Values.global.tenantID }}/{{ .Values.global.workspaceID }}/s3_ro
      uri: 's3://<your-toucan-data-bucket>' # 's3://toucan-data'
      region: '<your-aws-region>' # 'fr-par', check your S3 provider
      endpoint: '<your-external-s3-endpoint-url>' # 'https://<your-external-s3-endpoint-url>'

dataset:
  config:
    specific:
      vault_secret_paths:
        s3_datasource_upload_path: s3_ro # See vault.bootstrap.s3.path
```

{% endcode %}
{% endstep %}

{% step %}
**Install**

At this point, your `values.override.yaml` should looks like:

{% code title="yaml: /work/values.override.yaml" %}

```yaml
# ...

global:
  s3:
    keys:
      dataexecution:
        id: <AWS_ACCESS_KEY_ID>

        secret: <AWS_SECRET_ACCESS_KEY>
        # OR
        existingSecret:
          name: '<K8S Secret Name>'
          key: '<K8S Secret Key>'

      toucan_ro:
        id: <AWS_ACCESS_KEY_ID>

        secret: <AWS_SECRET_ACCESS_KEY>
        # OR
        existingSecret:
          name: '<K8S Secret Name>'
          key: '<K8S Secret Key>'

      toucan:
        id: <AWS_ACCESS_KEY_ID>

        secret: <AWS_SECRET_ACCESS_KEY>
        # OR
        existingSecret:
          name: '<K8S Secret Name>'
          key: '<K8S Secret Key>'

laputa:
  config:
    s3_storage:
      bucket_name: '<your-toucan-data-bucket>' # 'toucan-data'
      region_name: '<your-aws-region>' # 'fr-par', check your S3 provider
      endpoint_url: '<your-external-s3-endpoint-url>' # 'https://<your-external-s3-endpoint-url>'
      verify: true # Check TLS certificate.

dataexecution:
  config:
    specific:
      bucket_name: '<your-dataexecution-cache-bucket>' # 'dataexecution-cache'
      region: '<your-aws-region>' # 'fr-par', check your S3 provider
      endpoint: '<your-external-s3-endpoint-url>' # 'https://<your-external-s3-endpoint-url>'

vault:
  bootstrap:
    s3:
      # Sadly, this is hardcoded inside the dataset service code.
      # If you wish to change it, feel free to send us a feedback.
      path: secret/{{ .Values.dataset.config.environment }}/{{ .Values.global.tenantID }}/{{ .Values.global.workspaceID }}/s3_ro
      uri: 's3://<your-toucan-data-bucket>' # 's3://toucan-data'
      region: '<your-aws-region>' # 'fr-par', check your S3 provider
      endpoint: '<your-external-s3-endpoint-url>' # 'https://<your-external-s3-endpoint-url>'

dataset:
  config:
    specific:
      vault_secret_paths:
        s3_datasource_upload_path: s3_ro # (must match the path: /secret)
```

{% endcode %}

{% code title="shell: /work/" %}

```shell
helm upgrade --install toucan-stack oci://quay.io/toucantoco/charts/toucan-stack \
  --namespace toucan \
  --values ./values.override.yaml
```

{% endcode %}
{% endstep %}
{% endstepper %}