💾Backup and Restore

Infrastructure overview

Here's are the storage worth backing up:

  • PostgreSQL

  • Laputa Storage & MongoDB

  • Curity config

  • S3

PostgreSQL

Backup

You can run a Kubernetes CronJob to dump the database. Set these parameters in the values.yaml:

yaml: values.override.yaml
postgresql:
  backup:
    ## @param backup.enabled Enable the logical dump of the database "regularly"
    enabled: false
    cronjob:
      ## @param backup.cronjob.schedule Set the cronjob parameter schedule
      schedule: '@daily'
      ## @param backup.cronjob.timeZone Set the cronjob parameter timeZone
      timeZone: ''

      ## @param backup.cronjob.command Set backup container's command to run
      command:
        - /bin/bash
        - -c
        - PGPASSWORD="${PGPASSWORD:-$(< "$PGPASSWORD_FILE")}" pg_dumpall --clean --if-exists --load-via-partition-root --quote-all-identifiers --no-password --file="${PGDUMP_DIR}/pg_dumpall-$(date '+%Y-%m-%d-%H-%M').pgdump"

      storage:
        ## @param backup.cronjob.storage.enabled Enable using a `PersistentVolumeClaim` as backup data volume
        ##
        enabled: true
        ## @param backup.cronjob.storage.existingClaim Provide an existing `PersistentVolumeClaim` (only when `architecture=standalone`)
        ## If defined, PVC must be created manually before volume will be bound
        ##
        existingClaim: ''
        ## @param backup.cronjob.storage.resourcePolicy Setting it to "keep" to avoid removing PVCs during a helm delete operation. Leaving it empty will delete PVCs after the chart deleted
        ##
        resourcePolicy: ''
        ## @param backup.cronjob.storage.storageClass PVC Storage Class for the backup data volume
        ## If defined, storageClassName: <storageClass>
        ## If set to "-", storageClassName: "", which disables dynamic provisioning
        ## If undefined (the default) or set to null, no storageClassName spec is
        ## set, choosing the default provisioner.
        ##
        storageClass: ''
        ## @param backup.cronjob.storage.accessModes PV Access Mode
        ##
        accessModes:
          - ReadWriteOnce
        ## @param backup.cronjob.storage.size PVC Storage Request for the backup data volume
        ##
        size: 8Gi
        ## @param backup.cronjob.storage.annotations PVC annotations
        ##
        annotations: {}
        ## @param backup.cronjob.storage.mountPath Path to mount the volume at
        ##
        mountPath: /backup/pgdump
        ## @param backup.cronjob.storage.subPath Subdirectory of the volume to mount at
        ## and one PV for multiple services.
        ##
        subPath: ''
        ## Fine tuning for volumeClaimTemplates
        ##
        volumeClaimTemplates:
          ## @param backup.cronjob.storage.volumeClaimTemplates.selector A label query over volumes to consider for binding (e.g. when using local volumes)
          ## A label query over volumes to consider for binding (e.g. when using local volumes)
          ## See https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#labelselector-v1-meta for more details
          ##
          selector: {}
Example with S3

If you want to backup to S3, you would use the mountpoint-s3-csi-driver and mount the S3 bucket at /backup/pgdump:

yaml: persistentvolume.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: postgres-backups-pv
spec:
  capacity:
    storage: 50Gi
  accessModes:
    - ReadWriteMany
  storageClassName: ''
  mountOptions:
    - prefix postgres-backups/
    - endpoint-url https://s3.fr-par.scw.cloud
    - uid=1001
    - gid=1001
    - allow-other
  csi:
    driver: s3.csi.aws.com
    volumeHandle: postgres-backups-pv
    volumeAttributes:
      bucketName: backups
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-backups-pvc
  namespace: postgresql
spec:
  resources:
    requests:
      storage: 50Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  storageClassName: ''
  volumeName: postgres-backups-pv
yaml: values.override.yaml
postgresql:
  backup:
    enabled: true
    cronjob:
      storage:
        enabled: true
        existingClaim: postgres-backups-pvc

Restore

  1. Find a way to connect to the database. You can forward the service port if you have admin access to the Kubernetes cluster:

bash
kubectl port-forward -n <namespace> svc/toucan-stack-postgresql-primary 5432:5432
  1. Restore the backup using the command:

    psql -h localhost -p 5432 -U toucan-admin -d "$DATABASE" -f "<backup file>" -v ON_ERROR_STOP=1

MongoDB

No backup utilities are available for MongoDB at the moment, and it is not planned as well.

We plan to deprecated Laputa and MongoDB, which are legacy services from Toucan v2.

It's possible that the utilities in v2 and expect to work the same. However, this guide will use proper tools.

This guide will guide you through the steps required to backup and restore Toucan's MongoDB database outside of Kubernetes.

A better approach is to use a Kubernetes CronJob to dump the database, in which you won't need to run the restore command manually and won't need to forward the port.

Backup

  1. Get the MongoDB credentials:

bash
kubectl get secret --namespace toucan toucan-stack-mongodb -o jsonpath='{.data.mongodb-root-password}' | base64 --decode
  1. Forward the MongoDB port if you have admin access to the Kubernetes cluster:

bash
kubectl port-forward -n <namespace> svc/toucan-stack-mongodb 27017:27017
  1. Mongo dump:

bash
mongodump --uri="mongodb://admin:<password>@localhost:27017" --gzip --archive="<backup file>"

Restore

  1. Get the MongoDB credentials:

bash
kubectl get secret --namespace toucan toucan-stack-mongodb -o jsonpath='{.data.mongodb-root-password}' | base64 --decode
  1. Forward the MongoDB port if you have admin access to the Kubernetes cluster:

bash
kubectl port-forward -n <namespace> svc/toucan-stack-mongodb 27017:27017
  1. Mongo restore:

bash
mongorestore --verbose --uri="mongodb://admin:<password>@localhost:27017" --gzip --archive="<backup file>"

Use --dryRun to test before restoring.

Persistent Volume backup (Laputa Storage, MongoDB, Curity config, PostgreSQL)

Persistent Volume backups are filesystem snapshots, and not logical backups.

Backup

By using Kubernetes, you have used a storage provisioner to provision a volume for Laputa. Most storage provisioner can backup the data automatically.

See either your cloud provider or the storage provisioner documentation for more information.

Here's a non-exhaustive list of storage provisioners with backup capabilities:

You can also look at external-snapshotter if you want a Kubernetes-native way to handle snapshots. In which, some provider implements its API:

To create a snapshot using external-snapshotter,

  1. Create a VolumeSnapshotClass (if not present):

yaml: volumesnapshotclass.yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: csi-cinder-snapclass-in-use-v1
deletionPolicy: Delete
driver: cinder.csi.openstack.org # OVH or OpenStack
parameters: # Read the CSI driver documentation for more information
  force-create: 'true'

You can verify your snapshot class by using kubectl get volumesnapshotclass.

  1. Create a VolumeSnapshot:

yaml: volumesnapshot.yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: postgresql-snapshot
  namespace: toucan
spec:
  volumeSnapshotClassName: csi-cinder-snapclass-in-use-v1
  source:
    persistentVolumeClaimName: postgresql-pvc

After creating the snapshot, you should get a VolumeSnapshotContent.

Restore

You can restore a snapshot using the Web UI of your cloud provider.

If you are using external-snapshotter, to restore a snapshot, you need to reference the VolumeSnapshot in the PersistentVolumeClaim. Set these parameters in the values to restore the snapshot:

yaml: values.override.yaml
postgresql:
  persistence:
    enabled: true
    storageClass: csi-cinder-high-speed
    dataSource:
      name: postgresql-snapshot
      kind: VolumeSnapshot
      apiGroup: snapshot.storage.k8s.io
    size: 10Gi

If you fear losing data, remember to set the retainPolicy to Retain on the PersistentVolume. You should delete the existing StatefulSet and PersistentVolumeClaim before upgrading the chart. If the retainPolicy is set to Delete, the previous data will be lost.

Last updated

Was this helpful?