Skip to content

Commit f51bfa0

Browse files
MichaelCadeMichaelCade
authored
Merge pull request MichaelCade#481 from mocdaniel/day27
Adds resources for day 27
2 parents 62bc346 + 4fb4742 commit f51bfa0

1 file changed

Lines changed: 340 additions & 0 deletions

File tree

‎2024/day27.md‎

Lines changed: 340 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,340 @@
1+
# Day 27: 90DaysofDevOps
2+
3+
## From Automated to Automatic - Event-Driven Infrastructure Management with Ansible
4+
5+
**Daniel Bodky**
6+
- [Twitter](https://twitter.com/d_bodky)
7+
- [LinkedIn](https://linkedin.com/in/daniel-bodky)
8+
- [Blog](https://dbodky.me)
9+
10+
## Overview
11+
12+
A universal truth and recurring theme in the DevOps world is automation. From providing infrastructure to testing code to deploying to production, many parts of the DevOps lifecycle get automated already. One popular technology for managing infrastructure and configuration in an automated way is Ansible, but are we fully utilizing its capabilities yet?
13+
14+
This presentation will give a broad overview of Ansible and its architecture and use-cases, before exploring a relatively new feature, Event-driven Ansible (EDA). Analzying applications of event-driven Ansible, participants will see that automated management is nice, but automatic management is awesome, not just regarding DevOps principles, but also in terms of reaction times, the human tendency for minor mistakes, and toil for operators.
15+
16+
Participants will get first-hand insights into Ansible, its strengths, weaknesses, and the potential of event-driven automation within the DevOps world.
17+
18+
> [!NOTE]
19+
> The below content is a copy of the [lab repository's](https://github.com/mocdaniel/lab-event-driven-ansible) README for convenience.
20+
21+
---
22+
# Event-Driven Ansible Lab
23+
24+
This is a lab designed to demonstrate Ansible and how Event-Driven Ansible (**EDA**) builds on top of its capabilities.
25+
26+
The setup is done with Ansible, too. It will install **Ansible, EDA, Prometheus**, and **Alertmanager** on a VM to demonstrate some of the capabilities of EDA.
27+
28+
## Prerequisites
29+
30+
To follow along with this lab in its entirety, you will need three VMs:
31+
32+
> [!NOTE]
33+
> If you want to skip Ansible basics and go straight to EDA, you'll need just the `eda-controller.example.com` VM and can skip the others.
34+
35+
| VM name | OS |
36+
|--------------------|-------------|
37+
| eda-controller.example.com | CentOS/Rocky 8.9 |
38+
| company.example.com | CentOS/Rocky 8.9 |
39+
| webshop.example.com | Ubuntu 22.04 |
40+
41+
**You'll need to be able to SSH to each of these VMs as root using SSH keys.**
42+
43+
## Lab Setup
44+
45+
### Clone the repository and create a Python virtual environment
46+
47+
```bash
48+
git clone https://github.com/mocdaniel/lab-event-driven-ansible.git
49+
cd lab-event-driven-ansible
50+
python3 -m venv .venv
51+
source .venv/bin/activate
52+
```
53+
54+
### Install Ansible and other dependencies
55+
56+
```bash
57+
pip install -r requirements.txt
58+
```
59+
60+
### Create the inventory file
61+
62+
```yaml
63+
---
64+
# hosts.yml
65+
webservers:
66+
hosts:
67+
webshop.example.com:
68+
ansible_host: <ip-address>
69+
webserver: apache2
70+
company.example.com:
71+
ansible_host: <ip-address>
72+
webserver: httpd
73+
eda_controller:
74+
hosts:
75+
eda-controller.example.com:
76+
ansible_host: <ip-address>
77+
```
78+
79+
### Install Needed Roles and Collections
80+
81+
```bash
82+
ansible-galaxy install -r requirements.yml
83+
```
84+
85+
### Run the Setup Playbook
86+
87+
After you created the inventory file and filled in the IP addresses, you can run the setup playbook:
88+
89+
```bash
90+
ansible-playbook playbooks/setup.yml
91+
```
92+
93+
> [!CAUTION]
94+
> Due to a known bug with Python on MacOS, you need to run `export NO_PROXY="*"` on MacOS before running the playbook
95+
96+
---
97+
98+
## Demos
99+
100+
### Lab 1: Ansible Basics
101+
102+
<details>
103+
104+
<summary>Ansible from the CLI via ansible</summary>
105+
106+
#### Ansible from the CLI via `ansible`
107+
108+
The first example installs a webserver on all hosts in the `webservers` group. The installed webserver is defined as a **host variable** in the inventory file `hosts.yml` (*see above*).
109+
110+
```console
111+
ansible \
112+
webservers \
113+
-m package \
114+
-a 'name="{{ webserver }}"' \
115+
--one-line
116+
```
117+
118+
Afterwards, we can start the webserver on all hosts in the `webservers` group.
119+
120+
```console
121+
ansible \
122+
webservers \
123+
-m service \
124+
-a 'name="{{ webserver }}" state=started' \
125+
--one-line
126+
```
127+
128+
Go on and check if the web servers are running on the respective hosts.
129+
130+
> [!HINT]
131+
> Ansible is **idempotent** - try running the commands again and see how the output differs.
132+
133+
</details>
134+
135+
<details>
136+
137+
<summary>Ansible from the CLI via ansible-playbook</summary>
138+
139+
#### Ansible from the CLI via `ansible-playbook`
140+
141+
The second example utilizes the following **playbook** to **gather** and **display information** for all hosts in the `webservers` group, utilizing the **example** role from the lab repository.
142+
143+
```yaml
144+
---
145+
- name: Example role
146+
hosts: webservers
147+
gather_facts: false
148+
vars:
149+
greeting: "Hello World!"
150+
pre_tasks:
151+
- name: Say Hello
152+
ansible.builtin.debug:
153+
msg: "{{ greeting }}"
154+
roles:
155+
- role: example
156+
post_tasks:
157+
- name: Say goodbye
158+
ansible.builtin.debug:
159+
msg: Goodbye!
160+
```
161+
162+
```console
163+
ansible-playbook \
164+
playbooks/example.yml
165+
```
166+
167+
</details>
168+
169+
### Lab 2: Event-Driven Ansible
170+
171+
<details>
172+
173+
<summary>Receive Generic Events via Webhook</summary>
174+
175+
#### Receive Generic Events via Webhook
176+
177+
If you followed the setup instructions for the EDA lab, you should already have a running EDA instance on the `eda-controller.example.com` VM.
178+
179+
If you navigate to `/etc/edacontroller/rulebook.yml` on the VM, you'll see the following rulebook:
180+
181+
```yaml
182+
---
183+
- name: Listen to webhook events
184+
hosts: all
185+
sources:
186+
- ansible.eda.webhook:
187+
host: 0.0.0.0
188+
port: 5000
189+
rules:
190+
- name: Debug event output
191+
condition: 1 == 1
192+
action:
193+
debug:
194+
msg: "{{ event }}"
195+
196+
- name: Listen to Alertmanager alerts
197+
hosts: all
198+
sources:
199+
- ansible.eda.alertmanager:
200+
host: 0.0.0.0
201+
port: 9000
202+
data_alerts_path: alerts
203+
data_host_path: labels.instance
204+
data_path_separator: .
205+
rules:
206+
- name: Restart MySQL server
207+
condition: event.alert.labels.alertname == 'MySQL not running' and event.alert.status == 'firing'
208+
action:
209+
run_module:
210+
name: ansible.builtin.service
211+
module_args:
212+
name: mysql
213+
state: restarted
214+
- name: Debug event output
215+
condition: 1 == 1
216+
action:
217+
debug:
218+
msg: "{{ event }}"
219+
220+
```
221+
222+
For this part of the lab, the **first rule** is the one we're interested in: It listens to a generic webhook on port `5000` and prints the event's **metadata** to its logs.
223+
224+
To test this, we can use the `curl` command to send a `POST` request to the webhook `/endpoint` from the VM itself:
225+
226+
```console
227+
curl \
228+
-X POST \
229+
-H "Content-Type: application/json" \
230+
-d '{"foo": "bar"}' \
231+
http://localhost:5000/endpoint
232+
```
233+
234+
If you now check the logs of the EDA controller, you should see the following output:
235+
236+
```console
237+
journalctl -fu eda-controller
238+
239+
Jan 11 16:35:29 eda-controller ansible-rulebook[56882]: {'payload': {'foo': 'bar'}, 'meta': {'endpoint': 'endpoint',
240+
'headers': {'Host': 'localhost:5000', 'User-Agent': 'curl/7.76.1', 'Accept': '*/*', 'Content-Length': '21',
241+
'Content-Type': 'application/x-www-form-urlencoded'}, 'source': {'name': 'ansible.eda.webhook', 'type': 'ansible.eda.webhook'},
242+
'received_at': '2024-01-11T15:35:29.798401Z', 'uuid': '6ebf8dd2-60a2-455a-9383-97b81f535366'}}
243+
```
244+
245+
A rule that always evaluates to `true` is not very useful, so let's change the rule to only print the the value of `foo` if the `foo` key is present in the event's payload, and `no foo :(` otherwise:
246+
247+
```yaml
248+
---
249+
- name: Listen to webhook events
250+
hosts: all
251+
sources:
252+
- ansible.eda.webhook:
253+
host: 0.0.0.0
254+
port: 5000
255+
rules:
256+
- name: Foo
257+
condition: event.payload.foo is defined
258+
action:
259+
debug:
260+
msg: "{{ event.payload.foo }}"
261+
- name: No foo
262+
condition: 1 == 1
263+
action:
264+
debug:
265+
msg: "no foo :("
266+
```
267+
268+
Send the same `curl` request again and check the logs, you should see a line saying `bar` now.
269+
270+
Let's also try a `curl` request with a different payload:
271+
272+
```console
273+
curl \
274+
-X POST \
275+
-H "Content-Type: application/json" \
276+
-d '{"bar": "baz"}' \
277+
http://localhost:5000/endpoint
278+
```
279+
280+
This time, the output should be `no foo :(`.
281+
282+
</details>
283+
284+
<details>
285+
286+
<summary>Restarting Services Automatically with EDA</summary>
287+
288+
#### Restarting Services Automatically with EDA
289+
290+
The last lab is more of a demo - it shows how you can use EDA to automatically react on events observed by **Prometheus** and **Alertmanager**.
291+
292+
For this demo, the second **ruleset** in our rulebook is the one we're interested in:
293+
294+
```yaml
295+
- name: Listen to Alertmanager alerts
296+
hosts: all
297+
sources:
298+
- ansible.eda.alertmanager:
299+
host: 0.0.0.0
300+
port: 9000
301+
data_alerts_path: alerts
302+
data_host_path: labels.instance
303+
data_path_separator: .
304+
rules:
305+
- name: Restart MySQL server
306+
condition: event.alert.labels.alertname == 'MySQL not running' and event.alert.status == 'firing'
307+
action:
308+
run_playbook:
309+
name: ./playbook.yml
310+
- name: Debug event output
311+
condition: 1 == 1
312+
action:
313+
debug:
314+
msg: "{{ event }}"
315+
```
316+
317+
With this rule, we can restart our MySQL server if it's not running! But how do we get the event to trigger? With **Prometheus** and **Alertmanager**!
318+
319+
When you ran the setup playbook, it installed **Prometheus** and **Alertmanager** on the `eda-controller.example.com` VM. You can access the **Prometheus** UI at `http://<eda-controller-ip>:9090` and the **Alertmanager** UI at `http://<eda-controller-ip>:9093`.
320+
321+
It also installed a **Prometheus exporter** for the **MySQL** database that runs on the server.
322+
323+
With this setup, we can now shut down our MySQL server and see what happens - make sure to watch the output of the EDA controller's logs:
324+
325+
```console
326+
systemctl stop mysql
327+
journalctl -fu edacontroller
328+
```
329+
330+
331+
Within 30-90 seconds, you should see EDA running our **playbook** and restarting the MySQL server. You can track that process by watching the Prometheus/Alertmanager UIs for firing alerts.
332+
333+
Once you see the playbook being executed in the logs, you can check the MySQL state once more:
334+
335+
```console
336+
systemctl status mysql
337+
```
338+
339+
MySQL should be up and running again!
340+
</details>

0 commit comments

Comments
 (0)