Launch Consul+Nomad+Vault through Ansible Role

Oct 20, 2021 · · consul nomad vault ansible ·

Sometimes we need to manage distributed task scheduling and secure clandestine transport, so introducing HashiStack is a good choice

Prepare Ansible script and playbook

I choice the following playbook roles for stacks,But I changed things to suit my own needs

Consul: github url
Nomad: github url
Vault use both repo community-ansibleand MiteshSharma/AnsiblevaultRole

 1# The playbook folder level
 2├── site.yml
 3├── nomad-dev.local
 4├── ansible.cfg
 5├── roles
 6│   ├── consul
 7│   └── nomad
 8│   └── vault
 9└── group_vars
10    ├── consul
11    └── nomad
12    └── vault

the ansible.cfg content like following

1[defaults]
2host_key_checking = False
3roles_path = ./roles
4gathering = smart
5
6[ssh_connection]
7ssh_args = -o ControlMaster=auto -o ControlPersist=600s
8pipelining = True

the site.yml playbook

 1---
 2
 3- name: install consul server
 4  hosts: servers
 5  roles:
 6    - role: "consul"
 7      vars:
 8        consul_node_role: "server"
 9  tags:
10    - servers
11    - consul
12
13- name: install consul clients
14  hosts: clients
15  roles:
16    - role: "consul"
17      vars:
18        consul_node_role: "client"
19  tags:
20    - clients    
21    - consul
22
23- name: install nomad server
24  hosts: servers
25  roles:
26    - role: "nomad"
27      vars:
28        nomad_server_enabled: true
29  tags:
30    - servers
31    - nomad
32
33- name: install nomad clients
34  hosts: clients
35  roles:
36    - role: "nomad"
37      vars:
38        nomad_user_managed: false
39        nomad_user: root
40        nomad_group: root
41  tags:
42    - clients    
43    - nomad
44
45- name: install vault
46  hosts: servers
47  roles:
48    - role: "vault"
49      vars:
50        is_operator_init: false
51        is_unseal: false
52  tags:
53    - vault

And the host inventory hosts We deployed the server and client cluster nodes separately,but the consul’s server and client must together running only

 1host_001 ansible_host=10.0.0.16
 2host_002 ansible_host=10.0.0.13
 3host_003 ansible_host=10.0.0.14
 4host_004 ansible_host=10.0.0.7
 5host_005 ansible_host=10.0.0.15
 6host_006 ansible_host=10.0.0.14
 7
 8[all:vars]
 9ansible_user=root
10ansible_password=xxxx
11ansible_ssh_common_args='-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ProxyCommand="ssh -W %h:%p -q ues.proxy"'
12
13[servers]
14host_001 
15host_002 
16host_003 
17
18[clients]
19host_004 
20host_005 
21host_006 
22
23[nomad:children]
24servers
25clients
26
27[consul:children]
28servers
29clients
30
31[vault:children]
32servers

The Consul group varsgroup_vars/consul

 1consul_version: 1.10.3
 2consul_data_path: "/var/lib/consul"
 3consul_datacenter: bj-1
 4consul_bootstrap_expect: true
 5consul_bootstrap_expect_value: 3
 6consul_autopilot_enable: true
 7consul_autopilot_cleanup_dead_Servers: true
 8consul_autopilot_last_contact_threshold: 10s
 9consul_group_name: "servers"
10consul_servers: "{{ groups['servers'] }}"
11consul_acl_policy: false
12consul_acl_enable: true
13consul_acl_default_policy: "deny"
14consul_client_address: "0.0.0.0"
15consul_acl_token: 'xxxxxx-xxxxx-xxx-xxx-xxxxxx'
16consul_acl_master_token: 'xxxxxx-xxxxx-xxx-xxx-xxxxxx'

The Nomad group varsgroup_vars/nomad

 1nomad_general_config:
 2  region: bj
 3  datacenter: bj-1
 4  bind_addr: 0.0.0.0
 5  advertise:
 6    rpc: "{{ ansible_default_ipv4.address }}:4647"
 7    serf: "{{ ansible_default_ipv4.address }}:4648"
 8  consul:
 9    token: 'xxxxxx-xxxxx-xxx-xxx-xxxxxx'
10nomad_server_config:
11  bootstrap_expect: 3
12nomad_client_config:
13  reserved:
14    cpu: 500
15    memory: 512
16    disk: 1024

The vault group variables

1vault_version: "1.9.0"
2vault_install_remotely: true
3vault_enable_log: true
4vault_enable_logrotate: true
5vault_group_name: servers
6vault_cluster_name: us-1
7vault_datacenter: us-1
8vault_consul_token: 'xxxxxxx-xxx-xxx-xxxxx-xxxx'

Test apply playbook

1# play the consul servers and clients node
2ansible-playbook site.yml -i nomad-dev.local  --tags consul
3# play the nomad nodes book
4ansible-playbook site.yml -i nomad-dev.local  --tags nomad --skip-tags consul
5# install vault
6ansible-playbook site.yml -i nomad-dev.local -e "is_operator_init=true is_unseal=true" --tags vault --skip-tags consul,nomad

View the web ui through servers one ip and port

1# the consul web
2curl http://10.0.0.16:8500
3# the nomad web
4curl http://10.0.0.16:4646

The Nomad can auto discovery the Consul in local node and register
Create token or policy can through the web ui and consul command

1# create policy
2consul acl policy create -name "ui-policy" \
3  -description "Necessary permissions for UI functionality" \
4  -rules 'key_prefix"" { policy = "write" } node_prefix "" { policy = "read" } service_prefix "" { policy = "read" }'
5
6# and create token attach the polciy name or id
7consul acl token create -description "UI Token" -policy-name "ui-policy"

Terraform Nomad provider module nomad provider

Issues

Make sure all consul nodes hostname different Don’t have the same name
Make sure the all Consul node has same encrypt key in /var/lib/consul/serf/local.keyring becaues the consul agent may diferent keys through ansible roles(it’s bug)
Open the tcp port in consul nodes in firewall

1# allowlist the ports in server nodes
28500/tcp 8600/tcp 8301/tcp 8302/tcp 8300/tcp 8301/udp 8302/udp 8600/udp
3# allowlist the ports in agent nodes
48600/tcp 8301/tcp 8600/udp 8301/udp 8500/tcp

The /var/lib/consul folder permission assign to consul:consul user and group
If the Ansible and python is 2.X version the python package needs low version etc

1pip:
2  name: netaddr==0.7.19

It is best to pack all package to Docker image in Ansible container

Highly recommend cancel the default token in consul config.json change to agent

1#change tokens.default to tokens.agent in servers node
2"tokens": {
3    "agent": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx",
4    "master": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx",
5    "replication": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx"
6}

if download the vault is slow,you can pre download to

1wget https://releases.hashicorp.com/vault/1.9.0/vault_1.9.0_linux_amd64.zip -P /tmp/vault

You should the init and unseal vault if ansible init and unseal failed,because cluster ip is external ip address not localhost

1# execute init command in any one host
2vault operator init --address http://external.ip:8200
3# save the unseal keys and root key to secure place
4# and unseal the vault in every host three times use 3 unseal keys
5vault operator unseal --address http://external-host1.ip:8200
6# use same unseal keys in other hosts
7vault operator unseal --address http://external-host2.ip:8200

-Enabled nomad acl system

 1# change config file add acl stanza
 2# for hcl config file format
 3#acl {
 4#  enabled = true
 5#}
 6#or json
 7#"acl": { "eanbled": true }
 8systemctl restart nomad
 9#first bootstrap the acl in nomad
10nomad acl bootstrap
11export NOMAD_TOKEN="BOOTSTRAP_SECRET_ID"
12nomad status
13
14#reset the acl token 
15echo 7 >> /nomad-data-dir/server/acl-bootstrap-reset
16nomad acl bootstrap
17
18#

Configure Nomad secrets engine in Vault

1vault secrets enable nomad
2vault mount nomad
3vault write nomad/config/access address=http://127.0.0.1:4646

Backup consul

1export CONSUL_HTTP_TOKEN=xxxxxxxx-xxxxx-xxxx-xxxxxxxx-xxxx
2consul snapshot save /data/consul-20211230.snap