Offline install and setup kubernetes cluster in internal network
Sometimes enterprise is ineternal network not has internet network in production The offline installing and setup k8s environment and resources is necessery
Prepare the terraform environment
- Download the Terraform binary terraform You can also download the HELM binary to operate the k8s cluster.
1wget https://releases.hashicorp.com/terraform/0.15.4/terraform_0.15.4_linux_amd64.zip
2unzip terraform_0.15.4_linux_amd64.zip
3mv terraform /usr/local/bin/
4chmod +x /usr/local/bin/terraform
5terraform version
-
Prepare the Terraform syntax tips plugins in idea IDE
https://plugins.jetbrains.com/plugin/7808-hashicorp-terraform--hcl-language-support -
Because the Terraform plugins and providers Offline to local directory and push to CVS repo so needs to install git lfs plugins
1git lfs install #the binary in git repo need to lfs plugin
2cd app/dev/ #enter the app dir
3terraform init -backend-config=./backend.tfvars #init terraform environment
4terraform fmt # format the HCL code
5terraform validate #validation the HCL syntax
6terraform plan -var-file=./dev.tfvars #view the IAC information and changed
7terraform apply -var-file=./dev.tfvars #asign to change
- Init the directories level
1├── charts # storage the helm charts
2│ ├── eck-operator
3│ ├── etcd
4│ ├── fluentd
5│ ├── ingress-nginx
6│ ├── kube-prometheus-stack
7│ ├── kubernetes-dashboard
8│ ├── postgresql
9│ └── myapp-skeleton # yourself the helm chart
10├── iac # storage the terraform HCL code and environment config
11│ ├── base # the public global settings
12│ ├── base-us # the other special global settings
13│ ├── exec.sh # quickly create environment structure
14│ ├── modules # the public modules reference to othe environment
15│ ├── dev # the developement env
16│ └── qa # the qa testing env
17├── README.md
18└── tests # some test suite and so on
- The
basedirectory include global resources definitions has different*.tfvarsin each environment dir. such as: dev/dev.tfvars qa/qa.tfvars
101_main.tf
202_base.tf
303_ceph_ingress.tf
404_operators.tf
505_other.tf
6backend.tfvars
7base.tfvars
8outputs.tf
9terraform.tfstate
10variables.tf
They have the same structure, but different values
1cat base/base.tfvars
2k8s = {
3 config_path : "~/.kube/config"
4 config_context : "kubernetes-admin@kubernetes"
5}
6nodes_ip = [
7 "10.10.1.1",
8 "10.10.1.2",
9 "10.10.1.3"
10 ]
11private_registry_prefix = "registry.me"
12custom_namespaces = ["qa", "dev"]
13component = {
14 redis = {
15 enabled: true
16 }
17 postgres = {
18 enabled: true
19 }
20 elk = {
21 enabled: true
22 es_storage_size: "20Gi"
23 expose_es_endpoint: true
24 kibana_node_count: 1
25 master_node_count: 2
26 data_node_count: 2
27 apm_node_count: 2
28 }
29 fluentd = {
30 enabled: true
31 }
32}
33core_domains = {
34 grafana : "kb.k8s.test.com"
35 alert_manager : "alert.k8s.test.com"
36 ceph_ingress : "ceph.k8s.test.com"
37 dashboard : "admin.k8s.test.com"
38 pg_op_ui: "pg-op-ui.k8s.test.com"
39 redis_insight: "redis-ui.k8s.test.com"
40 kibana: "kibana.k8s.test.com"
41 prometheus: "metric.k8s.test.com"
42 es_master: "es-master.k8s.test.com"
43 es_data: "es-data.k8s.test.com"
44}
45storage_class_name = "rook-cephfs"
46s3_config = {
47 access_key: "xxxxxxxxxxxxxxxxxxxxxxxxx"
48 secret_key: "xxxxxxxxxxxxxxxxxxxxxxxxx"
49}
- The
dev/dev.tfvars
1k8s = {
2 config_path : "~/.kube/config"
3 config_context : "kubernetes-admin@kubernetes"
4}
5private_registry_prefix = "registry.me"
6storage_class_name = "rook-cephfs"
7namespace = "dev"
8domains = {
9 rocketmq_console : "rq.k8s.test.com"
10 etcd_ui: "etcd-ui.k8s.test.com"
11}
12middleware = {
13 rocketmq = {
14 enabled = true
15 broker_extra_config = <<-EOT
16 brokerIP1=10.10.1.1
17 brokerIP2=10.10.1.1
18 EOT
19 broker_size = 1
20 broker_memory_set = "-Xms1g -Xmx1g -Xmn1g"
21 }
22 redis = {
23 enabled : true
24 mode : "standalone"
25 }
26 postgres = {
27 enabled : true
28 team_id : "saas"
29 cluster_name : "postgres"
30 }
31 etcd = {
32 enabled: true
33 }
34}
- All environment reference the
basedirectory’s01_main.tfsymbolical link to it’s
1terraform {
2 required_version = ">= 0.13"
3 required_providers {
4 kubernetes = {
5 source = "hashicorp/kubernetes"
6 version = "2.2.0"
7 }
8 helm = {
9 source = "hashicorp/helm"
10 version = "2.1.2"
11 }
12 kubectl = {
13 source = "gavinbunney/kubectl"
14 version = "1.11.1"
15 }
16 template = {
17 source = "hashicorp/template"
18 version = "2.2.0"
19 }
20 }
21
22 backend "s3" {
23 region = "main"
24 skip_requesting_account_id = true
25 skip_credentials_validation = true
26 skip_get_ec2_platforms = true
27 skip_metadata_api_check = true
28 skip_region_validation = true
29 force_path_style = true
30 }
31
32}
33
34provider "kubernetes" {
35 config_path = var.k8s.config_path
36 config_context = var.k8s.config_context
37}
38
39provider "helm" {
40 kubernetes {
41 config_path = var.k8s.config_path
42 }
43}
44
45provider "kubectl" {
46 config_path = var.k8s.config_path
47 config_context = var.k8s.config_context
48}
- Terraform state manager through s3 storage
config in
backend.tfvars. useminioreplace in non AWS environment
1endpoint = "http://s3.test.com"
2bucket = "terraform-state"
3key = "k8s-dev/terraform.tfstate"
4access_key = "xxxxxxx"
5secret_key = "xxxxxxx"
- At any time to backup the state file to local
1terraform state pull > ./terraform.tfstate
2#the terraform.tfstate is ignored in git repo
- When initialization the new base environment use the raipd command
1./exec.sh base base-test-pipeline
2cd base-test-pipeline && terraform init -backend-config=./backend.tfvars
3terraform validate
4terraform plan -var-file=./base.tfvars
- When init app environment based to parent base environment
1./exec.sh app app-test-01
2cd app-test-01 && terraform init -backend-config=./backend.tfvars
3terraform validate
4terraform plan -var-file=./env.tfvars
- The simple
exec.shscript look like this following
1#!/usr/bin/env bash
2set -e
3
4[ "$#" -eq 2 ] || {
5 echo "Create a base or app environment directory layout"
6 echo "Please refer to Environment directory name: $0 base|app EnvironmentName"
7 exit 1
8}
9declare -r env_type="${1:-base}"
10declare -r env_dir="${2:-sample}"
11
12mkdir -p ${env_dir}/.terraform
13cd ${env_dir}/.terraform && ln -s ../../base/.terraform/providers ./providers
14cd ..
15ln -s ../base/.terraform.lock.hcl .terraform.lock.hcl
16ln -s ../base/01_main.tf ./01_main.tf
17cp ../base/backend.tfvars ./backend.tfvars
18sed -i "s#^key.*#key = \"${env_dir}/terraform.tfstate\"#g" backend.tfvars
19
20
21if [[ "${env_type}" == "base" ]]; then
22 ln -s ../base/02_base.tf ./02_base.tf
23 ln -s ../base/03_ceph_ingress.tf ./03_ceph_ingress.tf
24 ln -s ../base/04_operators.tf ./04_operators.tf
25 ln -s ../base/variables.tf ./variables.tf
26 touch base.tfvars
27 echo "Please copy the keys to base.tfvars file don't include the values!!!"
28fi
29
30if [[ "${env_type}" == "app" ]]; then
31 ln ../saas-dev/variables.tf ./variables.tf
32 touch env.tfvars
33 echo "Please copy the app environment file keys to env.tfvars don't include the values!!!"
34fi
- Probably going to prepare docker images import to local private registry
1registry.me/public/grafana/grafana-image-renderer latest 601MB
2registry.me/public/prometheus-operator/prometheus-config-reloader v0.47.1 12.4MB
3registry.me/public/prometheus-operator/prometheus-operator v0.47.1 43.9MB
4registry.me/public/grafana/grafana 7.5.5 204MB
5registry.me/public/kube-state-metrics/kube-state-metrics v2.0.0 33.4MB
6registry.me/public/prometheus/prometheus v2.26.0 169MB
7registry.me/public/kiwigrid/k8s-sidecar 1.10.7 88.8MB
8registry.me/public/prometheus/node-exporter v1.1.2 26MB
9registry.me/public/kubernetesui/dashboard v2.2.0 225MB
10registry.me/public/jettech/kube-webhook-certgen v1.5.1 44.7MB
11registry.me/public/kubernetesui/metrcis-scraper v1.0.6 34.5MB
12registry.me/public/kubernetesui/metrics-scraper v1.0.6 34.5MB
13registry.me/public/curlimages/curl 7.73.0 11.1MB
14registry.me/public/prometheus/alertmanager v0.21.0 55.5MB
15registry.me/public/busybox 1.31.1 1.22MB
16registry.me/public/defaultbackend-amd64 1.5 5.13MB
17registry.me/public/bats/bats v1.1.0 15MB