cmc/cleberg.net

My personal web garden & blog. blog personal website https://cleberg.net

Commit 1b12d8a947

1b12d8a947cb771a7acf68dccc0418856c720de1

parent: bc885dc1cf

Unsigned

cmc <hello@cleberg.net> · 2024-09-20 18:51 UTC 

add prometheus-grafana-cloud post

Layout: unified · split

content/blog/2024-09-19-prometheus-grafana-cloud.org added +347
@@ -0,0 +1,347 @@
1#+date: <2024-09-20 Fri 13:38:52>
2#+title: Linux Observability with Self-Hosted Prometheus and Grafana Cloud
3#+description: Learn how to self-host a Prometheus data collection tool with Docker and visualize the results with Grafana Cloud.
4#+filetags: :linux:grafana:
5#+slug: prometheus-grafana-cloud
6
7This tutorial will guide you through the process of:
8
91. Configuring a free Grafana cloud account.
102. Installing Prometheus to store metrics.
113. Installing Node Exporter to export machine metrics for Prometheus.
124. Installing Nginx Exporter to export Nginx metrics for Prometheus.
135. Visualizing data in Grafana dashboards.
146. Configure alerts based on Grafana metrics.
15
16* Grafana Cloud
17
18To get started, visit the [[https://grafana.com/auth/sign-up/create-user][Grafana website]] and create a free account.
19
20** Prometheus Data Source
21
22By default, a Prometheus data source should exist in your data sources page
23(=$yourOrg.grafana.net/connections/datasources=). If not, add a new data source
24using the Prometheus type.
25
26Once you have a valid Prometheus data source, open the data source and note the
27following items:
28
29| Data | Example |
30|-----------------------+----------------------------------------------------------------|
31| Prometheus Server URL | https://prometheus-prod-13-prod-us-east-0.grafana.net/api/prom |
32|-----------------------+----------------------------------------------------------------|
33| User | 1234567 |
34|-----------------------+----------------------------------------------------------------|
35| Password | configured |
36
37** Cloud Access Policy Token
38
39Now let's create an access token in Grafana. Navigate to the Administration
40> Users and Access > Cloud Access Policies page and create an access policy.
41
42The =metrics > write= scope must be enabled within the access policy you choose.
43
44Once you have an access policy with the correct scope, click the Add Token
45button and be sure to copy and save the token since it will disappear once the
46modal window is closed.
47
48** Dashboards
49
50Finally, let's create a couple dashboards so that we can easily explore the data
51that we will be importing from the server.
52
53I recommend importing the following dashboards:
54
55- [[https://grafana.com/grafana/dashboards/1860-node-exporter-full/][Node Exporter Full]]
56- [[https://github.com/nginxinc/nginx-prometheus-exporter/blob/main/grafana][nginx-prometheus-exporter]]
57- Prometheus 2.0 Stats
58
59Refer to the bottom of the post for dashboard screenshots!
60
61* Docker
62
63On the machine that you want to observe, make sure Docker and Docker Compose are
64installed. This tutorial will be using Docker Compose to create a group of
65containers that will work together to send metrics to Grafana.
66
67Let's start by creating a working directory.
68
69#+begin_src shell
70mkdir ~/prometheus && \
71cd ~/prometheus && \
72nano compose.yml
73#+end_src
74
75Within the =compose.yml= file, let's paste the following:
76
77#+begin_src yaml
78# compose.yml
79
80networks:
81 monitoring:
82 driver: bridge
83
84volumes:
85 prometheus_data: {}
86
87services:
88 nginx-exporter:
89 image: nginx/nginx-prometheus-exporter
90 container_name: nginx-exporter
91 restart: unless-stopped
92 command:
93 - '--nginx.scrape-uri=http://host.docker.internal:8080/stub_status'
94 expose:
95 - 9113
96 networks:
97 - monitoring
98 extra_hosts:
99 - host.docker.internal:host-gateway
100
101 node-exporter:
102 image: prom/node-exporter:latest
103 container_name: node-exporter
104 restart: unless-stopped
105 volumes:
106 - /proc:/host/proc:ro
107 - /sys:/host/sys:ro
108 - /:/rootfs:ro
109 command:
110 - '--path.procfs=/host/proc'
111 - '--path.rootfs=/rootfs'
112 - '--path.sysfs=/host/sys'
113 - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
114 expose:
115 - 9100
116 networks:
117 - monitoring
118
119 prometheus:
120 image: prom/prometheus:latest
121 container_name: prometheus
122 restart: unless-stopped
123 volumes:
124 - ./prometheus.yml:/etc/prometheus/prometheus.yml
125 - prometheus_data:/prometheus
126 command:
127 - '--config.file=/etc/prometheus/prometheus.yml'
128 - '--storage.tsdb.path=/prometheus'
129 - '--web.console.libraries=/etc/prometheus/console_libraries'
130 - '--web.console.templates=/etc/prometheus/consoles'
131 - '--web.enable-lifecycle'
132 expose:
133 - 9090
134 networks:
135 - monitoring
136#+end_src
137
138#+begin_src shell
139sudo docker compose up -d
140#+end_src
141
142> I'm not sure if it made a difference but I also whitelisted port 8080 on my
143> local firewall with =sudo ufw allow 8080=.
144
145Next, let's create a =prometheus.yml= configuration file.
146
147,#+begin_src sh
148nano prometheus.yml
149#+end_src
150
151#+begin_src yaml
152# prometheus.yml
153
154global:
155 scrape_interval: 1m
156
157scrape_configs:
158 - job_name: 'prometheus'
159 scrape_interval: 1m
160 static_configs:
161 - targets: ['localhost:9090']
162
163 - job_name: 'node'
164 static_configs:
165 - targets: ['node-exporter:9100']
166
167 - job_name: 'nginx'
168 scrape_interval: 5s
169 static_configs:
170 - targets: ['nginx-exporter:9113']
171
172remote_write:
173 - url: 'https://prometheus-prod-13-prod-us-east-0.grafana.net/api/prom/push'
174 basic_auth:
175 username: 'prometheus-grafana-username'
176 password: 'access-policy-token'
177#+end_src
178
179** Nginx
180
181To enable to the Nginx statistics we need for the nginx-exporter container, we
182need to modify the Nginx configuration on the host.
183
184More specifically, we need to create a path for the =stub_status= to be returned
185when we query port 8080 on our localhost.
186
187#+begin_src shell
188sudo nano /etc/nginx/conf.d/default.conf
189#+end_src
190
191#+begin_src conf
192server {
193 listen 8080;
194 listen [::]:8080;
195
196 location /stub_status {
197 stub_status;
198 }
199}
200#+end_src
201
202#+begin_src shell
203sudo systemctl restart nginx.service
204#+end_src
205
206** Debugging
207
208At this point, everything should be running smoothly. If not, here are a few
209areas to check and see if any obvious errors exist.
210
211Nginx: Curl the stub_status from the Nginx web server on the host machine to see
212if Nginx and stub_status are working properly.
213
214#+begin_src shell
215curl http://127.0.0.1:8080/stub_status
216
217# EXPECTED RESULTS:
218Active connections: 101
219server accepts handled requests
220 7510 7510 9654
221Reading: 0 Writing: 1 Waiting: 93
222#+end_src
223
224Nginx-Exporter: Curl the exported Nginx metrics.
225
226#+begin_src sh
227# Figure out the IP address of the Docker container
228sudo docker network inspect grafana_monitoring
229
230...
231"Name": "nginx-exporter",
232"EndpointID": "ef999a53eb9e0753199a680f8d78db7c2a8d5f442626df0b1bb945f03b73dcdd",
233"MacAddress": "02:42:c0:a8:40:02",
234"IPv4Address": "192.168.64.2/20",
235...
236
237# Curl the exported Nginx metrics
238curl 192.168.64.2:9113/metrics
239
240# EXPECTED RESULTS:
241...
242# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
243# TYPE go_gc_duration_seconds summary
244go_gc_duration_seconds{quantile="0"} 2.9927e-05
245go_gc_duration_seconds{quantile="0.25"} 4.24e-05
246go_gc_duration_seconds{quantile="0.5"} 4.8531e-05
247...
248#+end_src
249
250Node-Exporter: Curl the exporter node machine metrics.
251
252#+begin_src shell
253# Curl the exported Node metrics
254curl 192.168.64.3:9100/metrics
255
256# EXPECTED RESULTS:
257...
258# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
259# TYPE promhttp_metric_handler_requests_total counter
260promhttp_metric_handler_requests_total{code="200"} 47
261promhttp_metric_handler_requests_total{code="500"} 0
262promhttp_metric_handler_requests_total{code="503"} 0
263...
264#+end_src
265
266Grafana: Open the Explore panel and look to see if any metrics are coming
267through the Prometheus data source. If not, something on the machine is
268preventing data from flowing through.
269
270* Alerts & IRM
271
272Now that we have our data connected and visualized, we can define alerting rules
273and determine what Grafana should do when an alert is triggered.
274
275** OnCall
276
277#+caption: OnCall
278[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/oncall.png]]
279
280Within the Alerts & IRM section of Grafana (=/alerts-and-incidents=), open the
281Users page.
282
283The Users page allows you to configure user connections such as:
284
285- Mobile App
286- Slack
287- Telegram
288- MS Teams
289- iCal
290- Google Calendar
291
292In addition to the connections of each user, you can specify how each user or
293team is alerted for Default Notifications and Important Notifications.
294
295Finally, you can access the Schedules page within the OnCall module to schedule
296users and teams to be on call for specific date and time ranges. For my
297purposes, I put myself on-call 24/7 so that I receive all alerts.
298
299#+caption: User Information
300[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/irm_user_info.png]]
301
302** Alerting
303
304#+caption: Alerting Insights
305[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/alerting_insights.png]]
306
307Now that we have defined users and team associated with an on-call schedule and
308configured to receive the proper alerts, let's define a rule that will generate
309alerts.
310
311Within the Alerting section of the Alerts & IRM module, you can create alert
312rules, contact points, and notification policies.
313
314Let's start by opening the Alert Rules page and click the New Alert Rule button.
315
316As shown in the image below, we will create an alert for high CPU temperature by querying the =node_hwmon_temp_celsius= metric from our Prometheus data source.
317
318Next, we will set the threshold to be anything above 50 (degrees Celsius).
319Finally, we will tell Grafana to evaluate this every 1 minute via our Default
320evaluation group. This is connected to our Grafana email, but can be associated
321with any notification policy.
322
323#+caption: New Alert Rule
324[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/new_alert.png]]
325
326When the alert fires, it will generate an email (or whatever notification policy
327you assigned) and will look something like the following image.
328
329#+caption: Alerting Example
330[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/email_alert.png]]
331
332** Dashboards
333
334As promised above, here are some dashboard screenshots based on the
335configurations above.
336
337#+caption: Nginx Dashboard
338[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/dashboard_nginx.png]]
339
340#+caption: Node Dashboard
341[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/dashboard_node.png]]
342
343#+caption: OnCall Dashboard
344[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/dashboard_oncall.png]]
345
346#+caption: Prometheus Dashboard
347[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/dashboard_prometheus.png]]
theme/templates/index.html +6 −4
@@ -8,6 +8,12 @@ uid [ultimate] Christian Cleberg &lt;hello@cleberg.net&gt;</pre>
8<section> 8<section>
9 <h2>Recent Blog Posts</h2> 9 <h2>Recent Blog Posts</h2>
10 10
11 <div class="post">
12 <time datetime="2024-09-20 13:38:52">2024-09-20</time>
13 <a href="/blog/prometheus-grafana-cloud.html"
14 >Linux Observability with Self-Hosted Prometheus and Grafana Cloud</a
15 >
16 </div>
11 <div class="post"> 17 <div class="post">
12 <time datetime="2024-08-25 09:45:30">2024-08-25</time> 18 <time datetime="2024-08-25 09:45:30">2024-08-25</time>
13 <a href="/blog/n8n-sentiment-analysis.html" 19 <a href="/blog/n8n-sentiment-analysis.html"
@@ -20,10 +26,6 @@ uid [ultimate] Christian Cleberg &lt;hello@cleberg.net&gt;</pre>
20 >My Top 8 Most Used Org-Mode Features</a 26 >My Top 8 Most Used Org-Mode Features</a
21 > 27 >
22 </div> 28 </div>
23 <div class="post">
24 <time datetime="2024-07-11 20:24:02">2024-07-11</time>
25 <a href="/blog/emacs-on-ipad.html">Emacs on iPadOS</a>
26 </div>
27 29
28 <br /> 30 <br />
29 <a href="/blog/">All Posts →</a> 31 <a href="/blog/">All Posts →</a>