Use Cases
      Back to home
      1. Support
      2. Use Cases

      Setting up GitLab Backup with bucketZero Storage

      This KB introduces basic concept and configuration for gitlab-backup using bZs. Because bZs is an S3-compatible storage, users can follow GitLab reference documents about S3-compatible storage for backup. GitLab reference documents explains bac

      GitLab Backup with bZs

      This document introduces basic concept and configuration for gitlab-backup using bZs. Because bZs is an S3-compatible storage, users can follow GitLab reference documents about S3-compatible storage for backup. GitLab reference documents explains backup for Postgres database, repositories, blobs, container registry, configuration files, and others. This document explains how to backup repositories into bZs. Users should follow the instruction in the GitLab reference documents to backup the other data.

      Backup Storage Configuration

      As S3-compatible storage, bZs access information and credentials should be configured in /etc/gitlab/gitlab.rb

      gitlab_rails['backup_upload_connection'] = {'provider' => 'AWS','region' => 'BZS_REGION','aws_access_key_id' => 'BZS_ACCESS_KEY_ID','aws_secret_access_key' => 'BZS_SECRET_ACCESS_KEY','endpoint' => 'BZS_ENDPOINT_URL','path_style'=> true,# # If IAM profile use is enabled, remove aws_access_key_id and aws_secret_access_key# 'use_iam_profile' => false}gitlab_rails['backup_upload_remote_directory'] = 'BZS_BUCKET/PATH_FOR_BACKUPS'gitlab_rails['backup_multipart_chunk_size'] = 104857600

      You should configure ‘backup_upload_connection’.

      • provider: should be AWS.
      • region: bZs region. Default value is us-east-1.
      • aws_access_key_id: bZs access key ID.
      • aws_secret_access_key: bZs secret access key.
      • endpoint: bZs endpoint url.
      • path_style: should be true.
      • use_iam_profile: should be false.
      • backup_upload_remote_directory: You should specify bucket and path for the backups. For example, if bucket name is ‘my-gitlab-backup-bucket’ and you want to store backups in ‘weekly-backup’ folder, you should set ‘my-gitlab-backup-bucket/weekly-backup’.
      • backup_multipart_chunk_size: S3 API recommends multipart upload API to upload a large file. This specifies the size of part. The staging backup file (e.g. tar file) will be uploaded as 100MB parts and aggregated. 100MB part size is default value and strongly recommended.

      If you update bZs configuration to /etc/gitlab/gitlab.rb correctly, you are ready to backup repositories.

      Create Backups

      The GitLab reference document recommends Postgres DB needs to be backup separately if database size is larger than 100GB. In this document, we will skip database backup. Users should backup the database following vendor’s instruction.

      gitlab-backup create SKIP=db

      This command will create a backup staging file at /var/opt/gitlab/backups/ before uploading the staging file to bZs. Thus, users must make sure the local storage has enough spaces to store the staging file. If the staging file is created at the local disk, the staging file will be uploaded to the configured bZs.

      Output sample.

      root@dev-gitlab-ce:~# gitlab-backup create SKIP=db2024-06-23 22:43:05 UTC -- Dumping database ... [SKIPPED]2024-06-23 22:43:05 UTC -- Dumping repositories ...{"command":"create","gl_project_path":"myota/test-1","level":"info","msg":"started create","pid":582171,"relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git","storage_name":"default","time":"2024-06-23T22:43:05.444Z"}{"command":"create","gl_project_path":"myota/test-1","level":"info","msg":"completed create","pid":582171,"relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git","storage_name":"default","time":"2024-06-23T22:43:05.468Z"}{"command":"create","gl_project_path":"myota/test-1.wiki","level":"info","msg":"started create","pid":582171,"relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.wiki.git","storage_name":"default","time":"2024-06-23T22:43:05.550Z"}{"command":"create","gl_project_path":"myota/client-api","level":"info","msg":"started create","pid":582171,"relative_path":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git","storage_name":"default","time":"2024-06-23T22:43:05.554Z"}{"command":"create","gl_project_path":"myota/test-1.wiki","level":"info","msg":"completed create","pid":582171,"relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.wiki.git","storage_name":"default","time":"2024-06-23T22:43:05.558Z"}{"command":"create","gl_project_path":"myota/client-api.wiki","level":"info","msg":"started create","pid":582171,"relative_path":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.wiki.git","storage_name":"default","time":"2024-06-23T22:43:05.558Z"}{"command":"create","gl_project_path":"myota/client-api.wiki","level":"info","msg":"completed create","pid":582171,"relative_path":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.wiki.git","storage_name":"default","time":"2024-06-23T22:43:05.573Z"}{"command":"create","gl_project_path":"myota/desktop","level":"info","msg":"started create","pid":582171,"relative_path":"@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce.git","storage_name":"default","time":"2024-06-23T22:43:05.573Z"}{"command":"create","gl_project_path":"myota/client-api","level":"info","msg":"completed create","pid":582171,"relative_path":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git","storage_name":"default","time":"2024-06-23T22:43:06.685Z"}{"command":"create","gl_project_path":"myota/desktop.wiki","level":"info","msg":"started create","pid":582171,"relative_path":"@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce.wiki.git","storage_name":"default","time":"2024-06-23T22:43:06.685Z"}{"command":"create","gl_project_path":"myota/desktop.wiki","level":"info","msg":"completed create","pid":582171,"relative_path":"@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce.wiki.git","storage_name":"default","time":"2024-06-23T22:43:06.704Z"}{"command":"create","gl_project_path":"myota/desktop","level":"info","msg":"completed create","pid":582171,"relative_path":"@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce.git","storage_name":"default","time":"2024-06-23T22:43:08.707Z"}2024-06-23 22:43:08 UTC -- Dumping repositories ... done2024-06-23 22:43:08 UTC -- Dumping uploads ...2024-06-23 22:43:08 UTC -- Dumping uploads ... done2024-06-23 22:43:08 UTC -- Dumping builds ...2024-06-23 22:43:08 UTC -- Dumping builds ... done2024-06-23 22:43:08 UTC -- Dumping artifacts ...2024-06-23 22:43:08 UTC -- Dumping artifacts ... done2024-06-23 22:43:08 UTC -- Dumping pages ...2024-06-23 22:43:08 UTC -- Dumping pages ... done2024-06-23 22:43:08 UTC -- Dumping lfs objects ...2024-06-23 22:43:08 UTC -- Dumping lfs objects ... done2024-06-23 22:43:08 UTC -- Dumping terraform states ...2024-06-23 22:43:08 UTC -- Dumping terraform states ... done2024-06-23 22:43:08 UTC -- Dumping container registry images ... [DISABLED]2024-06-23 22:43:08 UTC -- Dumping packages ...2024-06-23 22:43:08 UTC -- Dumping packages ... done2024-06-23 22:43:08 UTC -- Dumping ci secure files ...2024-06-23 22:43:08 UTC -- Dumping ci secure files ... done2024-06-23 22:43:08 UTC -- Creating backup archive: 1719182585_2024_06_23_16.10.7_gitlab_backup.tar ...2024-06-23 22:43:09 UTC -- Creating backup archive: 1719182585_2024_06_23_16.10.7_gitlab_backup.tar ... done2024-06-23 22:43:09 UTC -- Uploading backup archive to remote storage demo-1/gitlab-backup ...2024-06-23 22:43:28 UTC -- Uploading backup archive to remote storage demo-1/gitlab-backup ... done2024-06-23 22:43:28 UTC -- Deleting old backups ... [SKIPPED]2024-06-23 22:43:28 UTC -- Deleting tar staging files ...2024-06-23 22:43:28 UTC -- Cleaning up /var/opt/gitlab/backups/backup_information.yml2024-06-23 22:43:28 UTC -- Cleaning up /var/opt/gitlab/backups/repositories2024-06-23 22:43:28 UTC -- Cleaning up /var/opt/gitlab/backups/uploads.tar.gz2024-06-23 22:43:28 UTC -- Cleaning up /var/opt/gitlab/backups/builds.tar.gz2024-06-23 22:43:28 UTC -- Cleaning up /var/opt/gitlab/backups/artifacts.tar.gz2024-06-23 22:43:28 UTC -- Cleaning up /var/opt/gitlab/backups/pages.tar.gz2024-06-23 22:43:28 UTC -- Cleaning up /var/opt/gitlab/backups/lfs.tar.gz2024-06-23 22:43:28 UTC -- Cleaning up /var/opt/gitlab/backups/terraform_state.tar.gz2024-06-23 22:43:28 UTC -- Cleaning up /var/opt/gitlab/backups/packages.tar.gz2024-06-23 22:43:28 UTC -- Cleaning up /var/opt/gitlab/backups/ci_secure_files.tar.gz2024-06-23 22:43:28 UTC -- Deleting tar staging files ... done2024-06-23 22:43:28 UTC -- Deleting backups/tmp ...2024-06-23 22:43:28 UTC -- Deleting backups/tmp ... done2024-06-23 22:43:28 UTC -- Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive dataand are not included in this backup. You will need these files to restore a backup.Please back them up manually.2024-06-23 22:43:28 UTC -- Backup YOUR_BACKUP_ID is done.2024-06-23 22:43:28 UTC -- Deleting backup and restore PID file ... done

      Restore

      The GitLab reference document recommends Postgres DB needs to be backup separately if database size is larger than 100GB. Please follow instruction from the database service vendor and restore database properly. The details of the Gitlab instruction is available at https://docs.gitlab.com/ee/administration/backup_restore/restore_gitlab.html. This document explains how to restore repositories only.

      gitlab-backup restore BACKUP=YOUR_BACKUP_ID SKIP=db

      Output sample.

      root@dev-gitlab-ce:~# gitlab-backup restore BACKUP=1719235211_2024_06_24_16.10.7 SKIP=db2024-06-24 13:29:33 UTC -- Unpacking backup ...2024-06-24 13:29:33 UTC -- Unpacking backup ... done2024-06-24 13:29:33 UTC -- Restoring repositories ...{"command":"restore","gl_project_path":"myota/test-1","level":"info","msg":"started restore","pid":614068,"relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git","storage_name":"default","time":"2024-06-24T13:29:33.872Z"}{"command":"restore","gl_project_path":"myota/test-1.wiki","level":"info","msg":"started restore","pid":614068,"relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.wiki.git","storage_name":"default","time":"2024-06-24T13:29:33.978Z"}{"backup_id":"1719235211_2024_06_24_16.10.7","error":"no refs in backup","level":"warning","msg":"unable to reset refs. Proceeding with a normal restore","pid":614068,"relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.wiki.git","storage":"default","time":"2024-06-24T13:29:33.979Z"}{"command":"restore","gl_project_path":"myota/test-1.wiki","level":"info","msg":"completed restore","pid":614068,"relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.wiki.git","storage_name":"default","time":"2024-06-24T13:29:33.987Z"}{"command":"restore","gl_project_path":"myota/client-api","level":"info","msg":"started restore","pid":614068,"relative_path":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git","storage_name":"default","time":"2024-06-24T13:29:33.987Z"}{"command":"restore","gl_project_path":"myota/test-1","level":"info","msg":"completed restore","pid":614068,"relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git","storage_name":"default","time":"2024-06-24T13:29:34.022Z"}{"command":"restore","gl_project_path":"myota/client-api.wiki","level":"info","msg":"started restore","pid":614068,"relative_path":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.wiki.git","storage_name":"default","time":"2024-06-24T13:29:34.022Z"}{"backup_id":"1719235211_2024_06_24_16.10.7","error":"no refs in backup","level":"warning","msg":"unable to reset refs. Proceeding with a normal restore","pid":614068,"relative_path":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.wiki.git","storage":"default","time":"2024-06-24T13:29:34.022Z"}{"command":"restore","gl_project_path":"myota/client-api.wiki","level":"info","msg":"completed restore","pid":614068,"relative_path":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.wiki.git","storage_name":"default","time":"2024-06-24T13:29:34.029Z"}{"command":"restore","gl_project_path":"myota/client-api","level":"info","msg":"completed restore","pid":614068,"relative_path":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git","storage_name":"default","time":"2024-06-24T13:29:34.123Z"}2024-06-24 13:29:34 UTC -- Restoring repositories ... done2024-06-24 13:29:34 UTC -- Restoring uploads ...2024-06-24 13:29:34 UTC -- Restoring uploads ... done2024-06-24 13:29:34 UTC -- Restoring builds ...2024-06-24 13:29:34 UTC -- Restoring builds ... done2024-06-24 13:29:34 UTC -- Restoring artifacts ...2024-06-24 13:29:34 UTC -- Restoring artifacts ... done2024-06-24 13:29:34 UTC -- Restoring pages ...2024-06-24 13:29:34 UTC -- Restoring pages ... done2024-06-24 13:29:34 UTC -- Restoring lfs objects ...2024-06-24 13:29:34 UTC -- Restoring lfs objects ... done2024-06-24 13:29:34 UTC -- Restoring terraform states ...2024-06-24 13:29:34 UTC -- Restoring terraform states ... done2024-06-24 13:29:34 UTC -- Restoring packages ...2024-06-24 13:29:34 UTC -- Restoring packages ... done2024-06-24 13:29:34 UTC -- Restoring ci secure files ...2024-06-24 13:29:34 UTC -- Restoring ci secure files ... doneThis task will now rebuild the authorized_keys file.You will lose any data stored in the authorized_keys file.Do you want to continue (yes/no)? yes 2024-06-24 13:30:18 UTC -- Deleting tar staging files ...2024-06-24 13:30:18 UTC -- Cleaning up /var/opt/gitlab/backups/backup_information.yml2024-06-24 13:30:18 UTC -- Cleaning up /var/opt/gitlab/backups/db2024-06-24 13:30:18 UTC -- Cleaning up /var/opt/gitlab/backups/repositories2024-06-24 13:30:18 UTC -- Cleaning up /var/opt/gitlab/backups/uploads.tar.gz2024-06-24 13:30:18 UTC -- Cleaning up /var/opt/gitlab/backups/builds.tar.gz2024-06-24 13:30:18 UTC -- Cleaning up /var/opt/gitlab/backups/artifacts.tar.gz2024-06-24 13:30:18 UTC -- Cleaning up /var/opt/gitlab/backups/pages.tar.gz2024-06-24 13:30:18 UTC -- Cleaning up /var/opt/gitlab/backups/lfs.tar.gz2024-06-24 13:30:18 UTC -- Cleaning up /var/opt/gitlab/backups/terraform_state.tar.gz2024-06-24 13:30:18 UTC -- Cleaning up /var/opt/gitlab/backups/packages.tar.gz2024-06-24 13:30:18 UTC -- Cleaning up /var/opt/gitlab/backups/ci_secure_files.tar.gz2024-06-24 13:30:18 UTC -- Deleting tar staging files ... done2024-06-24 13:30:18 UTC -- Deleting backups/tmp ...2024-06-24 13:30:18 UTC -- Deleting backups/tmp ... done2024-06-24 13:30:18 UTC -- Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive dataand are not included in this backup. You will need to restore these files manually.2024-06-24 13:30:18 UTC -- Restore task is done.2024-06-24 13:30:18 UTC -- Deleting backup and restore PID file ... done

      References

      Large reference architecture: https://docs.gitlab.com/17.1/ee/administration/backup_restore/backup_large_reference_architectures.html

      Backup storage configuration: https://docs.gitlab.com/ee/administration/backup_restore/backup_gitlab.html#s3-encrypted-buckets

      Backup commands: https://docs.gitlab.com/ee/administration/backup_restore/backup_gitlab.html#backup-command

      Restore:
      https://docs.gitlab.com/ee/administration/backup_restore/restore_gitlab.html