content/blog/2024-09-20-prometheus-grafana-cloud.org
349 lines · 10850 bytes
1#+date: <2024-09-20 Friday 13:38:52>
2#+title: Configuration of Linux Server Monitoring Using Prometheus and Grafana Cloud
3#+description: Stepwise method for deploying Prometheus with Docker to monitor Linux servers and integrating Grafana Cloud for visualization and analysis of collected metrics.
4#+slug: prometheus-grafana-cloud
5#+filetags: :monitoring:prometheus:grafana:
6
7This tutorial will guide you through the process of:
8
91. Configuring a free Grafana cloud account.
102. Installing Prometheus to store metrics.
113. Installing Node Exporter to export machine metrics for Prometheus.
124. Installing Nginx Exporter to export Nginx metrics for Prometheus.
135. Visualizing data in Grafana dashboards.
146. Configure alerts based on Grafana metrics.
15
16* Grafana Cloud
17
18To get started, visit the [[https://grafana.com/auth/sign-up/create-user][Grafana website]] and create a free account.
19
20** Prometheus Data Source
21
22By default, a Prometheus data source should exist in your data sources page
23(=$yourOrg.grafana.net/connections/datasources=). If not, add a new data source
24using the Prometheus type.
25
26Once you have a valid Prometheus data source, open the data source and note the
27following items:
28
29| Data | Example |
30|-----------------------+---------------------------------------------------------------------|
31| Prometheus Server URL | https://prometheus-prod-13-prod-us-east-0.grafana.net/api/prom/push |
32|-----------------------+---------------------------------------------------------------------|
33| User | 1234567 |
34|-----------------------+---------------------------------------------------------------------|
35| Password | configured |
36
37** Cloud Access Policy Token
38
39Now let's create an access token in Grafana. Navigate to the Administration
40> Users and Access > Cloud Access Policies page and create an access policy.
41
42The =metrics > write= scope must be enabled within the access policy you choose.
43
44Once you have an access policy with the correct scope, click the Add Token
45button and be sure to copy and save the token since it will disappear once the
46modal window is closed.
47
48** Dashboards
49
50Finally, let's create a couple dashboards so that we can easily explore the data
51that we will be importing from the server.
52
53I recommend importing the following dashboards:
54
55- [[https://grafana.com/grafana/dashboards/1860-node-exporter-full/][Node Exporter Full]]
56- [[https://github.com/nginxinc/nginx-prometheus-exporter/blob/main/grafana][nginx-prometheus-exporter]]
57- Prometheus 2.0 Stats
58
59Refer to the bottom of the post for dashboard screenshots!
60
61* Docker
62
63On the machine that you want to observe, make sure Docker and Docker Compose are
64installed. This tutorial will be using Docker Compose to create a group of
65containers that will work together to send metrics to Grafana.
66
67Let's start by creating a working directory.
68
69#+begin_src sh
70mkdir ~/prometheus && \
71cd ~/prometheus && \
72nano compose.yml
73#+end_src
74
75Within the =compose.yml= file, let's paste the following:
76
77#+begin_src yaml
78# compose.yml
79
80networks:
81 monitoring:
82 driver: bridge
83
84volumes:
85 prometheus_data: {}
86
87services:
88 nginx-exporter:
89 image: nginx/nginx-prometheus-exporter
90 container_name: nginx-exporter
91 restart: unless-stopped
92 command:
93 - '--nginx.scrape-uri=http://host.docker.internal:8080/stub_status'
94 expose:
95 - 9113
96 networks:
97 - monitoring
98 extra_hosts:
99 - host.docker.internal:host-gateway
100
101 node-exporter:
102 image: prom/node-exporter:latest
103 container_name: node-exporter
104 restart: unless-stopped
105 volumes:
106 - /proc:/host/proc:ro
107 - /sys:/host/sys:ro
108 - /:/rootfs:ro
109 command:
110 - '--path.procfs=/host/proc'
111 - '--path.rootfs=/rootfs'
112 - '--path.sysfs=/host/sys'
113 - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
114 expose:
115 - 9100
116 networks:
117 - monitoring
118
119 prometheus:
120 image: prom/prometheus:latest
121 container_name: prometheus
122 restart: unless-stopped
123 volumes:
124 - ./prometheus.yml:/etc/prometheus/prometheus.yml
125 - prometheus_data:/prometheus
126 command:
127 - '--config.file=/etc/prometheus/prometheus.yml'
128 - '--storage.tsdb.path=/prometheus'
129 - '--web.console.libraries=/etc/prometheus/console_libraries'
130 - '--web.console.templates=/etc/prometheus/consoles'
131 - '--web.enable-lifecycle'
132 expose:
133 - 9090
134 networks:
135 - monitoring
136#+end_src
137
138#+begin_src sh
139sudo docker compose up -d
140#+end_src
141
142#+begin_quote
143I'm not sure if it made a difference but I also whitelisted port 8080 on my
144local firewall with =sudo ufw allow 8080=.
145#+end_quote
146
147Next, let's create a =prometheus.yml= configuration file.
148
149#+begin_src sh
150nano prometheus.yml
151#+end_src
152
153#+begin_src yaml
154# prometheus.yml
155
156global:
157 scrape_interval: 1m
158
159scrape_configs:
160 - job_name: 'prometheus'
161 scrape_interval: 1m
162 static_configs:
163 - targets: ['localhost:9090']
164
165 - job_name: 'node'
166 static_configs:
167 - targets: ['node-exporter:9100']
168
169 - job_name: 'nginx'
170 scrape_interval: 5s
171 static_configs:
172 - targets: ['nginx-exporter:9113']
173
174remote_write:
175 - url: 'https://prometheus-prod-13-prod-us-east-0.grafana.net/api/prom/push'
176 basic_auth:
177 username: 'prometheus-grafana-username'
178 password: 'access-policy-token'
179#+end_src
180
181** Nginx
182
183To enable to the Nginx statistics we need for the nginx-exporter container, we
184need to modify the Nginx configuration on the host.
185
186More specifically, we need to create a path for the =stub_status= to be returned
187when we query port 8080 on our localhost.
188
189#+begin_src sh
190sudo nano /etc/nginx/conf.d/default.conf
191#+end_src
192
193#+begin_src conf
194server {
195 listen 8080;
196 listen [::]:8080;
197
198 location /stub_status {
199 stub_status;
200 }
201}
202#+end_src
203
204#+begin_src sh
205sudo systemctl restart nginx.service
206#+end_src
207
208** Debugging
209
210At this point, everything should be running smoothly. If not, here are a few
211areas to check and see if any obvious errors exist.
212
213Nginx: Curl the stub_status from the Nginx web server on the host machine to see
214if Nginx and stub_status are working properly.
215
216#+begin_src sh
217curl http://127.0.0.1:8080/stub_status
218
219# EXPECTED RESULTS:
220Active connections: 101
221server accepts handled requests
222 7510 7510 9654
223Reading: 0 Writing: 1 Waiting: 93
224#+end_src
225
226Nginx-Exporter: Curl the exported Nginx metrics.
227
228#+begin_src sh
229# Figure out the IP address of the Docker container
230sudo docker network inspect grafana_monitoring
231
232...
233"Name": "nginx-exporter",
234"EndpointID": "ef999a53eb9e0753199a680f8d78db7c2a8d5f442626df0b1bb945f03b73dcdd",
235"MacAddress": "02:42:c0:a8:40:02",
236"IPv4Address": "192.168.64.2/20",
237...
238
239# Curl the exported Nginx metrics
240curl 192.168.64.2:9113/metrics
241
242# EXPECTED RESULTS:
243...
244# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
245# TYPE go_gc_duration_seconds summary
246go_gc_duration_seconds{quantile="0"} 2.9927e-05
247go_gc_duration_seconds{quantile="0.25"} 4.24e-05
248go_gc_duration_seconds{quantile="0.5"} 4.8531e-05
249...
250#+end_src
251
252Node-Exporter: Curl the exporter node machine metrics.
253
254#+begin_src sh
255# Curl the exported Node metrics
256curl 192.168.64.3:9100/metrics
257
258# EXPECTED RESULTS:
259...
260# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
261# TYPE promhttp_metric_handler_requests_total counter
262promhttp_metric_handler_requests_total{code="200"} 47
263promhttp_metric_handler_requests_total{code="500"} 0
264promhttp_metric_handler_requests_total{code="503"} 0
265...
266#+end_src
267
268Grafana: Open the Explore panel and look to see if any metrics are coming
269through the Prometheus data source. If not, something on the machine is
270preventing data from flowing through.
271
272* Alerts & IRM
273
274Now that we have our data connected and visualized, we can define alerting rules
275and determine what Grafana should do when an alert is triggered.
276
277** OnCall
278
279#+caption: OnCall
280[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/oncall.png]]
281
282Within the Alerts & IRM section of Grafana (=/alerts-and-incidents=), open the
283Users page.
284
285The Users page allows you to configure user connections such as:
286
287- Mobile App
288- Slack
289- Telegram
290- MS Teams
291- iCal
292- Google Calendar
293
294In addition to the connections of each user, you can specify how each user or
295team is alerted for Default Notifications and Important Notifications.
296
297Finally, you can access the Schedules page within the OnCall module to schedule
298users and teams to be on call for specific date and time ranges. For my
299purposes, I put myself on-call 24/7 so that I receive all alerts.
300
301#+caption: User Information
302[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/irm_user_info.png]]
303
304** Alerting
305
306#+caption: Alerting Insights
307[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/alerting_insights.png]]
308
309Now that we have defined users and team associated with an on-call schedule and
310configured to receive the proper alerts, let's define a rule that will generate
311alerts.
312
313Within the Alerting section of the Alerts & IRM module, you can create alert
314rules, contact points, and notification policies.
315
316Let's start by opening the Alert Rules page and click the New Alert Rule button.
317
318As shown in the image below, we will create an alert for high CPU temperature by querying the =node_hwmon_temp_celsius= metric from our Prometheus data source.
319
320Next, we will set the threshold to be anything above 50 (degrees Celsius).
321Finally, we will tell Grafana to evaluate this every 1 minute via our Default
322evaluation group. This is connected to our Grafana email, but can be associated
323with any notification policy.
324
325#+caption: New Alert Rule
326[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/new_alert.png]]
327
328When the alert fires, it will generate an email (or whatever notification policy
329you assigned) and will look something like the following image.
330
331#+caption: Alerting Example
332[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/email_alert.png]]
333
334** Dashboards
335
336As promised above, here are some dashboard screenshots based on the
337configurations above.
338
339#+caption: Nginx Dashboard
340[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/dashboard_nginx.png]]
341
342#+caption: Node Dashboard
343[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/dashboard_node.png]]
344
345#+caption: OnCall Dashboard
346[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/dashboard_oncall.png]]
347
348#+caption: Prometheus Dashboard
349[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/dashboard_prometheus.png]]