Backup and Restore kubernetes cluster using Velero
Installing or Upgrade Velero client on Mac OS:
1 brew install velero
2 HOMEBREW_NO_AUTO_UPDATE=1 brew upgrade velero #upgrade to the latest version if maybe
The Velero should use object store save the snapshot. so we use Minio as kubernetes object store.Minio launched as part of docker-compose.yaml
1 minio:
2 container_name: "minio"
3 image: minio/minio
4 command: "server /data"
5 ports:
6 - "9000:9000"
7 environment:
8 MINIO_ACCESS_KEY: “xxxxxxx”
9 MINIO_SECRET_KEY: “xxxxxxx”
10 volumes:
11 - "./minio/data:/data"
12 networks:
13 - easy-mock
Create a env file is named: velero-env. because the Velero need the similar aws credentials access to Minio service storage backup object.
1 [default]
2 aws_access_key_id = xxxxxx
3 aws_secret_access_key = xxxxxx
if you setup Velero before. maybe delete custom resource definitions in k8s:
1#such as
2 kubectl delete crd velero.io/*
3 kubectl delete all --all -n velero
Execute velero server end install command:
1 velero install \
2 --provider aws \
3 --plugins velero/velero-plugin-for-aws:v1.0.0 \
4 --bucket local-backup \
5 --secret-file ./velero-env \
6 --use-volume-snapshots=true \
7 --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://example.local.com:9000 \
8 --snapshot-location-config region=minio \
9 --use-restic=true
the velero install command can repeat execute. when was problem
usually the velero command example:
1 velero create backup k8s-all-backup # creae backup
2 velero backup describe k8s-all-backup # view detail
3 velero backup logs k8s-all-backup # view logs
4 #schedule backups
5 velero schedule create k8s-all-backup --schedule "0 0 * * *"
6
7 # specific annotations for resources backup snapshot
8 kubectl -n kube-system annotate pod/etcd-master backup.velero.io/backup-volumes=etcd-certs,etcd-data
9 velero backup create etcd-master-backup-with-pv --include-namespaces kube-system
10
11 velero backup describe etcd-master-backup-with-pv
12 velero backup logs etcd-master-backup-with-pv
13
14 kubectl -n kubesphere-system annotate pod/mysql-66df969d-wx8z9 backup.velero.io/backup-volumes=db-persistent-storage
15 velero backup create kubesphere-mysql-backup-with-pv --include-namespaces kubesphere-system
16# you can also specific snapshot locations
17#
18 velero snapshot-location create office-minio --provider aws --config region=minio
19 velero backup create kubesphere-mysql-backup-with-pv --include-namespaces kubesphere-system --volume-snapshot-locations office-minio
20 velero schedule create kubesphere-system-all-backup --schedule "0 0 * * *" --volume-snapshot-locations office-minio
testing backup and restore
1 kubectl create ns test # create namespace
2 helm install nginx apphub/nginx -n test # install example resource nginx
3 velero backup create nginx-backup --include-namespaces test
4 kubectl delete namespaces test # delete test namespace include all restource
5 velero restore create --from-backup nginx-backup
6 velero restore describe nginx-backup-20200914164752 # veiw detail
7 velero restore get # the status is Completed.
You can also using Helm chart install Velero server end. the Helm repo looks like following:
1 helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts # add repo
2 ## other some repo if testing in local.
3 #apphub https://apphub.aliyuncs.com/
4 #vmware-tanzu https://vmware-tanzu.github.io/helm-charts
5 #harbor https://helm.goharbor.io
6 #choerodon https://openchart.choerodon.com.cn/choerodon/c7n
7 #rancher-latest https://releases.rancher.com/server-charts/latest
8 #jetstack https://charts.jetstack.io
9 #nuclio https://nuclio.github.io/nuclio/charts
10 #bitnami https://charts.bitnami.com/bitnami
reference to: https://github.com/vmware-tanzu/helm-charts/blob/main/charts/velero/README.md
backup kubernetes master etcd snapshot and pki files
1#!/usr/bin/env bash
2sudo cp -r /etc/kubernetes/pki ./backup/
3sudo docker run --rm -v /data1/k8s/backup:/backup \
4 --network host \
5 -v /etc/kubernetes/pki/etcd:/etc/kubernetes/pki/etcd \
6 --env ETCDCTL_API=3 \
7 registry.aliyuncs.com/google_containers/etcd:3.4.3-0 \
8 etcdctl --endpoints=https://127.0.0.1:2379 \
9 --cacert=/etc/kubernetes/pki/etcd/ca.crt \
10 --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
11 --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
12 snapshot save ./backup/etcd-snapshot-latest.db
use helm install velero server and backup aws EKS
- first install velero Helm chat use terraform
1resource "kubernetes_namespace" "velero-system" {
2 metadata {
3 annotations = {
4 name = "velero-system"
5 }
6 name = "velero-system"
7 }
8}
9
10resource "helm_release" "velero" {
11 name = "velero"
12 chart = "../../charts/velero"
13 namespace = kubernetes_namespace.velero-system.metadata.0.name
14
15 values = [
16 <<-EOF
17 configuration:
18 backupStorageLocation:
19 bucket: "${local.velero.bucket}"
20 prefix: "${local.velero.prefix}"
21 provider: aws
22 volumeSnapshotLocation:
23 name: aws-volumesnapshot
24 provider: aws
25 config:
26 region: "${local.region}"
27 credentials:
28 useSecret: false
29 initContainers:
30 - name: velero-plugin-for-aws
31 image: velero/velero-plugin-for-aws:v1.2.0
32 volumeMounts:
33 - mountPath: /target
34 name: plugins
35 EOF
36 ]
37
38 depends_on = [
39 kubernetes_namespace.velero-system
40 ]
41}
- second use velero client command operation backup and restore
because velero not have install default namespace, so need
-nparameter refer to namespace
1# check version client and server
2velero version -n velero-system
3# get backup
4velero backup get -n velero-system
5# get backup locaction
6velero backup-location get -n velero-system
7# create backup
8velero backup create 2022-12-28-backup -n velero-system
9# view backup logs
10velero backup logs 2022-12-28-backup -n velero-system
11# delete backup
12velero backup delete 2022-12-28-backup -n velero-system
13# backup full eks cluster and refer to snapshot :
14velero backup create 2023-2-3-backup-02 -n backup-system --snapshot-volumes