Skip to content

Commit beb2ad5

Browse files
LaurenceJJonesLaurenceJJones
authored
enhance: Getting started add whitelist section (#617)
* enhance: Add a whitelist step in getting started so people can whitelist themselves as a general flow on first time install * fix: review 1 * fix: review 2
1 parent e951ffc commit beb2ad5

3 files changed

Lines changed: 242 additions & 3 deletions

File tree

‎crowdsec-docs/sidebarsUnversioned.js‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -454,6 +454,7 @@ module.exports = {
454454
"getting_started/post_installation/console_hub",
455455
],
456456
},
457+
"getting_started/post_installation/whitelists",
457458
{
458459
type: "category",
459460
label: "Acquisition",

‎crowdsec-docs/unversioned/getting_started/next_steps.md‎

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,18 @@ The CrowdSec Console is a web-based interface provided by CrowdSec, offering a w
1111

1212
See the dedicated [CrowdSec Console](/getting_started/post_installation/console.mdx) guide for more information.
1313

14-
### 2. Acquisitions
14+
15+
### 2. Whitelists
16+
17+
:::info
18+
Whitelists are a way to tell CrowdSec to ignore certain events or IP addresses.
19+
:::
20+
21+
By default CrowdSec will whitelist private LAN IP addresses, however you may want to whitelist additional IP addresses or events.
22+
23+
See the dedicated [Whitelists](/getting_started/post_installation/whitelists.mdx) guide for more information.
24+
25+
### 3. Acquisitions
1526

1627
:::info
1728
Acquisitions are sources of logs that CrowdSec can analyze.
@@ -21,7 +32,7 @@ By default when CrowdSec is installed it will attempt to detect the running serv
2132

2233
See the dedicated [Acquisition](/getting_started/post_installation/acquisition.mdx) guide for more information.
2334

24-
### 3. Profiles
35+
### 4. Profiles
2536

2637
:::info
2738
Profiles are a set of rules that drives what decisions will be taken by CrowdSec.
@@ -31,7 +42,7 @@ CrowdSec comes with a default profile that is suitable for most use cases. Howev
3142

3243
See the dedicated [Profiles](/getting_started/post_installation/profiles.mdx) guide for more information.
3344

34-
### 4. Metrics
45+
### 5. Metrics
3546

3647
:::info
3748
Metrics are a way to monitor the behavior of CrowdSec.

‎crowdsec-docs/unversioned/getting_started/post_installation/whitelists.mdx‎

Lines changed: 227 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,227 @@
1+
---
2+
id: whitelists
3+
title: Whitelists
4+
pagination_prev: getting_started/next_steps
5+
pagination_next: getting_started/next_steps
6+
---
7+
8+
# Whitelists
9+
10+
Whitelists are a way to tell CrowdSec to ignore certain events or IP addresses. This can be useful if you have a static IP address that you know is safe, or if you have a service that _could_ generates a lot of false triggers by loading alot of thumbnails, images or fonts.
11+
12+
By default CrowdSec will whitelist private LAN IP addresses via [this parser](https://app.crowdsec.net/hub/author/crowdsecurity/configurations/whitelists), however you may want to whitelist additional IP addresses or events.
13+
14+
In this guide we will cover how to create some whitelists and how to apply them to your CrowdSec instance.
15+
16+
However, we can't cover every possible use case, so if you have a specific one that isn't covered here, please refer to [detailed whitelist documentation](/docs/next/whitelist/intro)
17+
18+
:::info
19+
We use "event" as a term to describe a log line that is currently being processed by CrowdSec. An "event" can be a log line from a log file, or an "overflow" from a scenario.
20+
:::
21+
22+
## Whitelist Types
23+
24+
There are two different types of whitelists in CrowdSec:
25+
- Parser (at enrich stage)
26+
- Postoverflow
27+
28+
Its important to note where these whitelists are applied in the pipeline, as this will affect the behavior and the context of the whitelists.
29+
30+
### Parser
31+
32+
Parser based whitelists are applied whilst the event is being Enriched. This means that the whitelists will prevent the event (log line) from hitting the scenario stage, leading to better memory management.
33+
34+
typically these files are located in the following locations depending on your OS:
35+
- Linux: `/etc/crowdsec/parsers/s02-enrich/`
36+
- Freebsd: `/usr/local/etc/crowdsec/parsers/s02-enrich/`
37+
- Windows: `c:/programdata/crowdsec/config/parsers/s02-enrich/`
38+
39+
### Postoverflow
40+
41+
Postoverflow whitelists are applied after a scenario has been triggered. You may want use this stage to do _"expensive"_ whitelists, such as DNS lookups.
42+
43+
:::info
44+
We deem a whitelist as _"expensive"_ if it requires a network call or a lookup that could potentially slow down the processing of the event.
45+
:::
46+
47+
typically these files are located in the following locations depending on your OS:
48+
- Linux: `/etc/crowdsec/postoverflows/s01-whitelist/`
49+
- Freebsd: `/usr/local/etc/crowdsec/postoverflows/s01-whitelist/`
50+
- Windows: `c:/programdata/crowdsec/config/postoverflows/s01-whitelist/`
51+
52+
*Postoverflow whitelist folders do not exist by default so you **MUST** manually create them*
53+
54+
### Which one should I use?
55+
56+
If you know which IP or event pattern you want to whitelist (for example a URI), then you should use a Parser whitelist. If you want to do a more complex whitelist, such as a DNS/rDNS lookup, then you should use the Postoverflow Whitelist.
57+
58+
In short, enricher whitelists are applied to **every** event (log line), whereas postoverflow whitelists are only applied to **triggered** scenarios.
59+
60+
## Should I create a whitelist?
61+
62+
Whilst configuring CrowdSec for the first time you may want to create a whitelist for the following reasons:
63+
- You have a static IP address that you want to ensure is never banned.
64+
- You have a service that could generate a lot of false triggers.
65+
66+
:::info
67+
An example of a service that could generate a lot of false triggers is a web application that loads a lot of resources such a thumbnails, images, or fonts.
68+
:::
69+
70+
If you are unsure if you need a whitelist, you can monitor the logs and see if there are any false positives that you want to prevent.
71+
72+
However, please note if you should get a decision made agaisnt your IP address and using remediation components such as `iptables` or `nftables` then this will prevent you from accessing **ANY** service on the server.
73+
74+
Within our enterprise offering we have a feature called [Decision Management](console/decisions/decisions_management.md) that allows you to manage decisions via the [console](https://app.crowdsec.net/).
75+
76+
## Creating your first whitelist.
77+
78+
Depending on your criteria, you may want to create a whitelist for a specific IP address, a range of IP addresses, or a specific event pattern.
79+
80+
We will provide examples for each of these scenarios.
81+
82+
:::note
83+
The example location shown is for Linux, you will need to adjust the path based on your OS as noted above.
84+
:::
85+
86+
### Static IP address
87+
88+
If you want to whitelist a specific IP address for example `192.168.1.1`, you can create a file in the Enricher folder with the following content:
89+
90+
```yaml title="/etc/crowdsec/parsers/s02-enrich/01-my-whitelist.yaml"
91+
name: my/whitelist ## Must be unqiue
92+
description: "Whitelist events from my IP"
93+
whitelist:
94+
reason: "My IP"
95+
ip:
96+
- "192.168.1.1"
97+
```
98+
99+
Once you have created the file you will need to restart the CrowdSec service for the changes to take effect.
100+
101+
```bash title="Restart CrowdSec"
102+
sudo systemctl restart crowdsec
103+
```
104+
105+
### IP Range
106+
107+
If you want to whitelist a range of IP addresses, for example `192.168.1.0/24` you can create a file in the Enricher folder with the following content:
108+
109+
```yaml title="/etc/crowdsec/parsers/s02-enrich/01-my-whitelist.yaml"
110+
name: my/whitelist ## Must be unqiue
111+
description: "Whitelist events from my IP range"
112+
whitelist:
113+
reason: "My IP range"
114+
cidr:
115+
- "192.168.1.0/24"
116+
```
117+
118+
Once you have created the file you will need to restart the CrowdSec service for the changes to take effect.
119+
120+
```bash title="Restart CrowdSec"
121+
sudo systemctl restart crowdsec
122+
```
123+
124+
### Expression pattern
125+
126+
If you want to whitelist a specific event pattern, for example http log line that is a healthcheck so typically a `GET` request to `/health` you can create a file in the Enricher folder with the following content:
127+
128+
```yaml title="/etc/crowdsec/parsers/s02-enrich/01-my-whitelist.yaml"
129+
name: my/whitelist ## Must be unqiue
130+
description: "Whitelist events with GET /health"
131+
filter: "evt.Meta.service == 'http' && evt.Meta.log_type in ['http_access-log', 'http_error-log']"
132+
whitelist:
133+
reason: "GET /health"
134+
expression:
135+
- "evt.Meta.http_verb == 'GET' && evt.Meta.http_path == '/health'"
136+
```
137+
138+
:::note
139+
This will discard any event that has a `http_verb` of `GET` and a `http_path` of `/health` no matter the origin.
140+
:::
141+
142+
Once you have created the file you will need to restart the CrowdSec service for the changes to take effect.
143+
144+
```bash title="Restart CrowdSec"
145+
sudo systemctl restart crowdsec
146+
```
147+
148+
Expression whitelists are very powerful and can be used to whitelist based on any field in the event. You can find a more detailed version of the [expression guide here](/docs/next/whitelist/create_expr) which will showcase how you can find which fields are available to you based on the log line you are processing.
149+
150+
### Dynamic IP address
151+
152+
If you want to whitelist an IP address that is not static, you need to use a external DDNS service to resolve the IP address and then use the Postoverflow whitelist to whitelist the resolved IP address.
153+
154+
:::note
155+
This is a postoverflow whitelist as it requires a network call to resolve the IP address.
156+
:::
157+
158+
```yaml title="/etc/crowdsec/postoverflows/s01-whitelist/01-my-whitelist.yaml"
159+
name: my/whitelist ## Must be unqiue
160+
description: "Whitelist events from my dynamic IP"
161+
whitelist:
162+
reason: "My dynamic IP"
163+
expression:
164+
- evt.Overflow.Alert.Source.IP in LookupHost("foo.com")
165+
```
166+
167+
:::warning
168+
Please read the [LookupHost](/docs/next/expr/ip_helpers/#lookuphosthost-string-string) function documentation before altering the current example.
169+
:::
170+
171+
## Testing your whitelist
172+
173+
Once you have created your whitelist, you can test it by running a log line through the parser and checking the output.
174+
175+
:::info
176+
In the example we using Nginx logs, you will need to adjust the testing to point towards your log file.
177+
:::
178+
179+
```bash title="Test an IP/Range whitelist"
180+
grep 192.168.1.1 /var/log/nginx/access.log | tail -n 1 | sudo cscli explain -f- --type nginx
181+
```
182+
183+
```bash title="Test an Expression whitelist"
184+
grep /health /var/log/nginx/access.log | tail -n 1 | sudo cscli explain -f- --type nginx
185+
```
186+
187+
```bash title="Example Output"
188+
line: 192.168.1.1 - - [05/Sep/2024:18:07:25 +0000] "GET /health? HTTP/2.0" 200 42 "-" "curl/7.68.0"
189+
├ s00-raw
190+
| └ 🟢 crowdsecurity/non-syslog (+5 ~8)
191+
├ s01-parse
192+
| └ 🟢 crowdsecurity/nginx-logs (+23 ~2)
193+
├ s02-enrich
194+
| ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~2)
195+
| ├ 🟢 crowdsecurity/geoip-enrich (+13)
196+
| ├ 🟢 crowdsecurity/http-logs (+8 ~1)
197+
#highlight-next-line
198+
| └ 🟢 my/whitelist (~2 [whitelisted])
199+
#highlight-next-line
200+
└-------- parser success, ignored by whitelist (<Reason will vary here>) 🟢
201+
```
202+
203+
:::info
204+
Currently postoverflows are not supported by `cscli explain` so you will need to check the logs to see if the whitelist is working.
205+
:::
206+
207+
Example logs line you will see:
208+
209+
```bash title="Example logs"
210+
time="07-07-2020 17:11:09" level=info msg="Ban for 192.168.1.1 whitelisted, reason [My dynamic IP]" id=cold-sunset name=my/whitelist stage=s01
211+
```
212+
213+
## Whitelisted but there still a decision?
214+
215+
Whitelisting an IP address or event will prevent the events from triggering **new** decisions, however, any existing decisions will still be applied.
216+
217+
You must manually remove the decisions by running:
218+
219+
```bash title="Remove decisions"
220+
sudo cscli decisions delete --ip 192.168.1.1
221+
```
222+
223+
## Next Steps?
224+
225+
If you are still looking for examples on how to create a whitelist, you can find more detailed documentation [here](/docs/next/whitelist/intro).
226+
227+
If you have created your first whitelist, tested it and happy to continue then [click here](getting_started/next_steps.md#2-whitelists).

0 commit comments

Comments
 (0)