cmc/cleberg.net
My personal web garden & blog.
clone: git clone https://gitbay.org/cmc/cleberg.net.git
main: content/blog/2024-09-20-prometheus-grafana-cloud.org · raw
1#+date: [2024-09-20 Fri 13:38:52]
2#+title: Prometheus and Grafana Cloud for Server Monitoring
3#+description: Monitoring Linux servers with Prometheus and visualizing in Grafana Cloud.
4#+slug: prometheus-grafana-cloud
5#+filetags: :linux:self-hosting:
6
7This tutorial will guide you through the process of:
8
91. Configuring a free Grafana cloud account.
102. Installing Prometheus to store metrics.
113. Installing Node Exporter to export machine metrics for Prometheus.
124. Installing Nginx Exporter to export Nginx metrics for Prometheus.
135. Visualizing data in Grafana dashboards.
146. Configure alerts based on Grafana metrics.
15
16* Grafana Cloud
17
18To get started, visit the [[https://grafana.com/auth/sign-up/create-user][Grafana website]] and create a free account.
19
20** Prometheus Data Source
21
22By default, a Prometheus data source should exist in your data sources page
23(=$yourOrg.grafana.net/connections/datasources=). If not, add a new data source
24using the Prometheus type.
25
26Once you have a valid Prometheus data source, open the data source and note the
27following items:
28
29| Data | Example |
30|-----------------------+---------------------------------------------------------------------|
31| Prometheus Server URL | https://prometheus-prod-13-prod-us-east-0.grafana.net/api/prom/push |
32|-----------------------+---------------------------------------------------------------------|
33| User | 1234567 |
34|-----------------------+---------------------------------------------------------------------|
35| Password | configured |
36
37** Cloud Access Policy Token
38
39Now let's create an access token in Grafana. Navigate to the Administration
40> Users and Access > Cloud Access Policies page and create an access policy.
41
42The =metrics > write= scope must be enabled within the access policy you choose.
43
44Once you have an access policy with the correct scope, click the Add Token
45button and be sure to copy and save the token since it will disappear once the
46modal window is closed.
47
48** Dashboards
49
50Finally, let's create a couple dashboards so that we can easily explore the data
51that we will be importing from the server.
52
53I recommend importing the following dashboards:
54
55- [[https://grafana.com/grafana/dashboards/1860-node-exporter-full/][Node Exporter Full]]
56- [[https://github.com/nginxinc/nginx-prometheus-exporter/blob/main/grafana][nginx-prometheus-exporter]]
57- Prometheus 2.0 Stats
58
59Refer to the bottom of the post for dashboard screenshots!
60
61* Docker
62
63On the machine that you want to observe, make sure Docker and Docker Compose are
64installed. This tutorial will be using Docker Compose to create a group of
65containers that will work together to send metrics to Grafana.
66
67Let's start by creating a working directory.
68
69#+begin_src sh
70mkdir ~/prometheus && \
71cd ~/prometheus && \
72nano compose.yml
73#+end_src
74
75Within the =compose.yml= file, let's paste the following:
76
77#+begin_src yaml
78# compose.yml
79
80networks:
81 monitoring:
82 driver: bridge
83
84volumes:
85 prometheus_data: {}
86
87services:
88 nginx-exporter:
89 image: nginx/nginx-prometheus-exporter
90 container_name: nginx-exporter
91 restart: unless-stopped
92 command:
93 - '--nginx.scrape-uri=http://host.docker.internal:8080/stub_status'
94 expose:
95 - 9113
96 networks:
97 - monitoring
98 extra_hosts:
99 - host.docker.internal:host-gateway
100
101 node-exporter:
102 image: prom/node-exporter:latest
103 container_name: node-exporter
104 restart: unless-stopped
105 volumes:
106 - /proc:/host/proc:ro
107 - /sys:/host/sys:ro
108 - /:/rootfs:ro
109 command:
110 - '--path.procfs=/host/proc'
111 - '--path.rootfs=/rootfs'
112 - '--path.sysfs=/host/sys'
113 - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
114 expose:
115 - 9100
116 networks:
117 - monitoring
118
119 prometheus:
120 image: prom/prometheus:latest
121 container_name: prometheus
122 restart: unless-stopped
123 volumes:
124 - ./prometheus.yml:/etc/prometheus/prometheus.yml
125 - prometheus_data:/prometheus
126 command:
127 - '--config.file=/etc/prometheus/prometheus.yml'
128 - '--storage.tsdb.path=/prometheus'
129 - '--web.console.libraries=/etc/prometheus/console_libraries'
130 - '--web.console.templates=/etc/prometheus/consoles'
131 - '--web.enable-lifecycle'
132 expose:
133 - 9090
134 networks:
135 - monitoring
136#+end_src
137
138#+begin_src sh
139sudo docker compose up -d
140#+end_src
141
142#+begin_quote
143I'm not sure if it made a difference but I also whitelisted port 8080 on my
144local firewall with =sudo ufw allow 8080=.
145#+end_quote
146
147Next, let's create a =prometheus.yml= configuration file.
148
149#+begin_src sh
150nano prometheus.yml
151#+end_src
152
153#+begin_src yaml
154# prometheus.yml
155
156global:
157 scrape_interval: 1m
158
159scrape_configs:
160 - job_name: 'prometheus'
161 scrape_interval: 1m
162 static_configs:
163 - targets: ['localhost:9090']
164
165 - job_name: 'node'
166 static_configs:
167 - targets: ['node-exporter:9100']
168
169 - job_name: 'nginx'
170 scrape_interval: 5s
171 static_configs:
172 - targets: ['nginx-exporter:9113']
173
174remote_write:
175 - url: 'https://prometheus-prod-13-prod-us-east-0.grafana.net/api/prom/push'
176 basic_auth:
177 username: 'prometheus-grafana-username'
178 password: 'access-policy-token'
179#+end_src
180
181** Nginx
182
183To enable to the Nginx statistics we need for the nginx-exporter container, we
184need to modify the Nginx configuration on the host.
185
186More specifically, we need to create a path for the =stub_status= to be returned
187when we query port 8080 on our localhost.
188
189#+begin_src sh
190sudo nano /etc/nginx/conf.d/default.conf
191#+end_src
192
193#+begin_src conf
194server {
195 listen 8080;
196 listen [::]:8080;
197
198 location /stub_status {
199 stub_status;
200 }
201}
202#+end_src
203
204#+begin_src sh
205sudo systemctl restart nginx.service
206#+end_src
207
208** Debugging
209
210At this point, everything should be running smoothly. If not, here are a few
211areas to check and see if any obvious errors exist.
212
213Nginx: Curl the stub_status from the Nginx web server on the host machine to see
214if Nginx and stub_status are working properly.
215
216#+begin_src sh
217curl http://127.0.0.1:8080/stub_status
218
219# EXPECTED RESULTS:
220Active connections: 101
221server accepts handled requests
222 7510 7510 9654
223Reading: 0 Writing: 1 Waiting: 93
224#+end_src
225
226Nginx-Exporter: Curl the exported Nginx metrics.
227
228#+begin_src sh
229# Figure out the IP address of the Docker container
230sudo docker network inspect grafana_monitoring
231
232...
233"Name": "nginx-exporter",
234"EndpointID": "ef999a53eb9e0753199a680f8d78db7c2a8d5f442626df0b1bb945f03b73dcdd",
235"MacAddress": "02:42:c0:a8:40:02",
236"IPv4Address": "192.168.64.2/20",
237...
238
239# Curl the exported Nginx metrics
240curl 192.168.64.2:9113/metrics
241
242# EXPECTED RESULTS:
243...
244# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
245# TYPE go_gc_duration_seconds summary
246go_gc_duration_seconds{quantile="0"} 2.9927e-05
247go_gc_duration_seconds{quantile="0.25"} 4.24e-05
248go_gc_duration_seconds{quantile="0.5"} 4.8531e-05
249...
250#+end_src
251
252Node-Exporter: Curl the exporter node machine metrics.
253
254#+begin_src sh
255# Curl the exported Node metrics
256curl 192.168.64.3:9100/metrics
257
258# EXPECTED RESULTS:
259...
260# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
261# TYPE promhttp_metric_handler_requests_total counter
262promhttp_metric_handler_requests_total{code="200"} 47
263promhttp_metric_handler_requests_total{code="500"} 0
264promhttp_metric_handler_requests_total{code="503"} 0
265...
266#+end_src
267
268Grafana: Open the Explore panel and look to see if any metrics are coming
269through the Prometheus data source. If not, something on the machine is
270preventing data from flowing through.
271
272* Alerts & IRM
273
274Now that we have our data connected and visualized, we can define alerting rules
275and determine what Grafana should do when an alert is triggered.
276
277** OnCall
278
279#+caption: OnCall
280#+attr_html: :alt View of the Grafana OnCall dashboard with metrics.
281[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/oncall.webp]]
282
283Within the Alerts & IRM section of Grafana (=/alerts-and-incidents=), open the
284Users page.
285
286The Users page allows you to configure user connections such as:
287
288- Mobile App
289- Slack
290- Telegram
291- MS Teams
292- iCal
293- Google Calendar
294
295In addition to the connections of each user, you can specify how each user or
296team is alerted for Default Notifications and Important Notifications.
297
298Finally, you can access the Schedules page within the OnCall module to schedule
299users and teams to be on call for specific date and time ranges. For my
300purposes, I put myself on-call 24/7 so that I receive all alerts.
301
302#+caption: User Information
303#+attr_html: :alt View of the grafanafd88 user with notification preferences.
304[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/irm_user_info.webp]]
305
306** Alerting
307
308#+caption: Alerting Insights
309#+attr_html: :alt A dashboard with the current grafana-managed alert rules.
310[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/alerting_insights.webp]]
311
312Now that we have defined users and team associated with an on-call schedule and
313configured to receive the proper alerts, let's define a rule that will generate
314alerts.
315
316Within the Alerting section of the Alerts & IRM module, you can create alert
317rules, contact points, and notification policies.
318
319Let's start by opening the Alert Rules page and click the New Alert Rule button.
320
321As shown in the image below, we will create an alert for high CPU temperature by querying the =node_hwmon_temp_celsius= metric from our Prometheus data source.
322
323Next, we will set the threshold to be anything above 50 (degrees Celsius).
324Finally, we will tell Grafana to evaluate this every 1 minute via our Default
325evaluation group. This is connected to our Grafana email, but can be associated
326with any notification policy.
327
328#+caption: New Alert Rule
329#+attr_html: :alt All available seetings when creating a new alert.
330[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/new_alert.webp]]
331
332When the alert fires, it will generate an email (or whatever notification policy
333you assigned) and will look something like the following image.
334
335#+caption: Alerting Example
336#+attr_html: :alt An email showing that "High CPU Temps" alert is firing.
337[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/email_alert.webp]]
338
339** Dashboards
340
341As promised above, here are some dashboard screenshots based on the
342configurations above.
343
344#+caption: Nginx Dashboard
345#+attr_html: :alt Updated dashboard based on the new alert created above.
346[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/dashboard_nginx.webp]]
347
348#+caption: Node Dashboard
349#+attr_html: :alt Metrics of server nodes, including CPU, network, and memory.
350[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/dashboard_node.webp]]
351
352#+caption: OnCall Dashboard
353#+attr_html: :alt Full OnCall dashboard with metrics for alerts, mean time to respond, and more.
354[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/dashboard_oncall.webp]]
355
356#+caption: Prometheus Dashboard
357#+attr_html: :alt Dashboard showing Prometheus metrics, including scrape duration, head shunks, reload count, and more.
358[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/dashboard_prometheus.webp]]