Commit `1b12d8a947`

1b12d8a947cb771a7acf68dccc0418856c720de1

parent: bc885dc1cf

Unsigned

cmc <hello@cleberg.net> · 2024-09-20 18:51 UTC

add prometheus-grafana-cloud post

Layout: unified · split

content/blog/2024-09-19-prometheus-grafana-cloud.org added +347

		@@ -0,0 +1,347 @@
	1	#+date: <2024-09-20 Fri 13:38:52>
	2	#+title: Linux Observability with Self-Hosted Prometheus and Grafana Cloud
	3	#+description: Learn how to self-host a Prometheus data collection tool with Docker and visualize the results with Grafana Cloud.
	4	#+filetags: :linux:grafana:
	5	#+slug: prometheus-grafana-cloud
	6
	7	This tutorial will guide you through the process of:
	8
	9	1. Configuring a free Grafana cloud account.
	10	2. Installing Prometheus to store metrics.
	11	3. Installing Node Exporter to export machine metrics for Prometheus.
	12	4. Installing Nginx Exporter to export Nginx metrics for Prometheus.
	13	5. Visualizing data in Grafana dashboards.
	14	6. Configure alerts based on Grafana metrics.
	15
	16	* Grafana Cloud
	17
	18	To get started, visit the [[https://grafana.com/auth/sign-up/create-user][Grafana website]] and create a free account.
	19
	20	** Prometheus Data Source
	21
	22	By default, a Prometheus data source should exist in your data sources page
	23	(=$yourOrg.grafana.net/connections/datasources=). If not, add a new data source
	24	using the Prometheus type.
	25
	26	Once you have a valid Prometheus data source, open the data source and note the
	27	following items:
	28
	29	\| Data \| Example \|
	30	\|-----------------------+----------------------------------------------------------------\|
	31	\| Prometheus Server URL \| https://prometheus-prod-13-prod-us-east-0.grafana.net/api/prom \|
	32	\|-----------------------+----------------------------------------------------------------\|
	33	\| User \| 1234567 \|
	34	\|-----------------------+----------------------------------------------------------------\|
	35	\| Password \| configured \|
	36
	37	** Cloud Access Policy Token
	38
	39	Now let's create an access token in Grafana. Navigate to the Administration
	40	> Users and Access > Cloud Access Policies page and create an access policy.
	41
	42	The =metrics > write= scope must be enabled within the access policy you choose.
	43
	44	Once you have an access policy with the correct scope, click the Add Token
	45	button and be sure to copy and save the token since it will disappear once the
	46	modal window is closed.
	47
	48	** Dashboards
	49
	50	Finally, let's create a couple dashboards so that we can easily explore the data
	51	that we will be importing from the server.
	52
	53	I recommend importing the following dashboards:
	54
	55	- [[https://grafana.com/grafana/dashboards/1860-node-exporter-full/][Node Exporter Full]]
	56	- [[https://github.com/nginxinc/nginx-prometheus-exporter/blob/main/grafana][nginx-prometheus-exporter]]
	57	- Prometheus 2.0 Stats
	58
	59	Refer to the bottom of the post for dashboard screenshots!
	60
	61	* Docker
	62
	63	On the machine that you want to observe, make sure Docker and Docker Compose are
	64	installed. This tutorial will be using Docker Compose to create a group of
	65	containers that will work together to send metrics to Grafana.
	66
	67	Let's start by creating a working directory.
	68
	69	#+begin_src shell
	70	mkdir ~/prometheus && \
	71	cd ~/prometheus && \
	72	nano compose.yml
	73	#+end_src
	74
	75	Within the =compose.yml= file, let's paste the following:
	76
	77	#+begin_src yaml
	78	# compose.yml
	79
	80	networks:
	81	monitoring:
	82	driver: bridge
	83
	84	volumes:
	85	prometheus_data: {}
	86
	87	services:
	88	nginx-exporter:
	89	image: nginx/nginx-prometheus-exporter
	90	container_name: nginx-exporter
	91	restart: unless-stopped
	92	command:
	93	- '--nginx.scrape-uri=http://host.docker.internal:8080/stub_status'
	94	expose:
	95	- 9113
	96	networks:
	97	- monitoring
	98	extra_hosts:
	99	- host.docker.internal:host-gateway
	100
	101	node-exporter:
	102	image: prom/node-exporter:latest
	103	container_name: node-exporter
	104	restart: unless-stopped
	105	volumes:
	106	- /proc:/host/proc:ro
	107	- /sys:/host/sys:ro
	108	- /:/rootfs:ro
	109	command:
	110	- '--path.procfs=/host/proc'
	111	- '--path.rootfs=/rootfs'
	112	- '--path.sysfs=/host/sys'
	113	- '--collector.filesystem.mount-points-exclude=^/(sys\|proc\|dev\|host\|etc)($$\|/)'
	114	expose:
	115	- 9100
	116	networks:
	117	- monitoring
	118
	119	prometheus:
	120	image: prom/prometheus:latest
	121	container_name: prometheus
	122	restart: unless-stopped
	123	volumes:
	124	- ./prometheus.yml:/etc/prometheus/prometheus.yml
	125	- prometheus_data:/prometheus
	126	command:
	127	- '--config.file=/etc/prometheus/prometheus.yml'
	128	- '--storage.tsdb.path=/prometheus'
	129	- '--web.console.libraries=/etc/prometheus/console_libraries'
	130	- '--web.console.templates=/etc/prometheus/consoles'
	131	- '--web.enable-lifecycle'
	132	expose:
	133	- 9090
	134	networks:
	135	- monitoring
	136	#+end_src
	137
	138	#+begin_src shell
	139	sudo docker compose up -d
	140	#+end_src
	141
	142	> I'm not sure if it made a difference but I also whitelisted port 8080 on my
	143	> local firewall with =sudo ufw allow 8080=.
	144
	145	Next, let's create a =prometheus.yml= configuration file.
	146
	147	,#+begin_src sh
	148	nano prometheus.yml
	149	#+end_src
	150
	151	#+begin_src yaml
	152	# prometheus.yml
	153
	154	global:
	155	scrape_interval: 1m
	156
	157	scrape_configs:
	158	- job_name: 'prometheus'
	159	scrape_interval: 1m
	160	static_configs:
	161	- targets: ['localhost:9090']
	162
	163	- job_name: 'node'
	164	static_configs:
	165	- targets: ['node-exporter:9100']
	166
	167	- job_name: 'nginx'
	168	scrape_interval: 5s
	169	static_configs:
	170	- targets: ['nginx-exporter:9113']
	171
	172	remote_write:
	173	- url: 'https://prometheus-prod-13-prod-us-east-0.grafana.net/api/prom/push'
	174	basic_auth:
	175	username: 'prometheus-grafana-username'
	176	password: 'access-policy-token'
	177	#+end_src
	178
	179	** Nginx
	180
	181	To enable to the Nginx statistics we need for the nginx-exporter container, we
	182	need to modify the Nginx configuration on the host.
	183
	184	More specifically, we need to create a path for the =stub_status= to be returned
	185	when we query port 8080 on our localhost.
	186
	187	#+begin_src shell
	188	sudo nano /etc/nginx/conf.d/default.conf
	189	#+end_src
	190
	191	#+begin_src conf
	192	server {
	193	listen 8080;
	194	listen [::]:8080;
	195
	196	location /stub_status {
	197	stub_status;
	198	}
	199	}
	200	#+end_src
	201
	202	#+begin_src shell
	203	sudo systemctl restart nginx.service
	204	#+end_src
	205
	206	** Debugging
	207
	208	At this point, everything should be running smoothly. If not, here are a few
	209	areas to check and see if any obvious errors exist.
	210
	211	Nginx: Curl the stub_status from the Nginx web server on the host machine to see
	212	if Nginx and stub_status are working properly.
	213
	214	#+begin_src shell
	215	curl http://127.0.0.1:8080/stub_status
	216
	217	# EXPECTED RESULTS:
	218	Active connections: 101
	219	server accepts handled requests
	220	7510 7510 9654
	221	Reading: 0 Writing: 1 Waiting: 93
	222	#+end_src
	223
	224	Nginx-Exporter: Curl the exported Nginx metrics.
	225
	226	#+begin_src sh
	227	# Figure out the IP address of the Docker container
	228	sudo docker network inspect grafana_monitoring
	229
	230	...
	231	"Name": "nginx-exporter",
	232	"EndpointID": "ef999a53eb9e0753199a680f8d78db7c2a8d5f442626df0b1bb945f03b73dcdd",
	233	"MacAddress": "02:42:c0:a8:40:02",
	234	"IPv4Address": "192.168.64.2/20",
	235	...
	236
	237	# Curl the exported Nginx metrics
	238	curl 192.168.64.2:9113/metrics
	239
	240	# EXPECTED RESULTS:
	241	...
	242	# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
	243	# TYPE go_gc_duration_seconds summary
	244	go_gc_duration_seconds{quantile="0"} 2.9927e-05
	245	go_gc_duration_seconds{quantile="0.25"} 4.24e-05
	246	go_gc_duration_seconds{quantile="0.5"} 4.8531e-05
	247	...
	248	#+end_src
	249
	250	Node-Exporter: Curl the exporter node machine metrics.
	251
	252	#+begin_src shell
	253	# Curl the exported Node metrics
	254	curl 192.168.64.3:9100/metrics
	255
	256	# EXPECTED RESULTS:
	257	...
	258	# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
	259	# TYPE promhttp_metric_handler_requests_total counter
	260	promhttp_metric_handler_requests_total{code="200"} 47
	261	promhttp_metric_handler_requests_total{code="500"} 0
	262	promhttp_metric_handler_requests_total{code="503"} 0
	263	...
	264	#+end_src
	265
	266	Grafana: Open the Explore panel and look to see if any metrics are coming
	267	through the Prometheus data source. If not, something on the machine is
	268	preventing data from flowing through.
	269
	270	* Alerts & IRM
	271
	272	Now that we have our data connected and visualized, we can define alerting rules
	273	and determine what Grafana should do when an alert is triggered.
	274
	275	** OnCall
	276
	277	#+caption: OnCall
	278	[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/oncall.png]]
	279
	280	Within the Alerts & IRM section of Grafana (=/alerts-and-incidents=), open the
	281	Users page.
	282
	283	The Users page allows you to configure user connections such as:
	284
	285	- Mobile App
	286	- Slack
	287	- Telegram
	288	- MS Teams
	289	- iCal
	290	- Google Calendar
	291
	292	In addition to the connections of each user, you can specify how each user or
	293	team is alerted for Default Notifications and Important Notifications.
	294
	295	Finally, you can access the Schedules page within the OnCall module to schedule
	296	users and teams to be on call for specific date and time ranges. For my
	297	purposes, I put myself on-call 24/7 so that I receive all alerts.
	298
	299	#+caption: User Information
	300	[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/irm_user_info.png]]
	301
	302	** Alerting
	303
	304	#+caption: Alerting Insights
	305	[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/alerting_insights.png]]
	306
	307	Now that we have defined users and team associated with an on-call schedule and
	308	configured to receive the proper alerts, let's define a rule that will generate
	309	alerts.
	310
	311	Within the Alerting section of the Alerts & IRM module, you can create alert
	312	rules, contact points, and notification policies.
	313
	314	Let's start by opening the Alert Rules page and click the New Alert Rule button.
	315
	316	As shown in the image below, we will create an alert for high CPU temperature by querying the =node_hwmon_temp_celsius= metric from our Prometheus data source.
	317
	318	Next, we will set the threshold to be anything above 50 (degrees Celsius).
	319	Finally, we will tell Grafana to evaluate this every 1 minute via our Default
	320	evaluation group. This is connected to our Grafana email, but can be associated
	321	with any notification policy.
	322
	323	#+caption: New Alert Rule
	324	[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/new_alert.png]]
	325
	326	When the alert fires, it will generate an email (or whatever notification policy
	327	you assigned) and will look something like the following image.
	328
	329	#+caption: Alerting Example
	330	[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/email_alert.png]]
	331
	332	** Dashboards
	333
	334	As promised above, here are some dashboard screenshots based on the
	335	configurations above.
	336
	337	#+caption: Nginx Dashboard
	338	[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/dashboard_nginx.png]]
	339
	340	#+caption: Node Dashboard
	341	[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/dashboard_node.png]]
	342
	343	#+caption: OnCall Dashboard
	344	[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/dashboard_oncall.png]]
	345
	346	#+caption: Prometheus Dashboard
	347	[[https://img.cleberg.net/blog/20240920-prometheus-grafana-cloud/dashboard_prometheus.png]]

theme/templates/index.html +6 −4

		@@ -8,6 +8,12 @@ uid [ultimate] Christian Cleberg <hello@cleberg.net></pre>
8	8	<section>
9	9	<h2>Recent Blog Posts</h2>
10	10
	11	<div class="post">
	12	<time datetime="2024-09-20 13:38:52">2024-09-20</time>
	13	<a href="/blog/prometheus-grafana-cloud.html"
	14	>Linux Observability with Self-Hosted Prometheus and Grafana Cloud</a
	15	>
	16	</div>
11	17	<div class="post">
12	18	<time datetime="2024-08-25 09:45:30">2024-08-25</time>
13	19	<a href="/blog/n8n-sentiment-analysis.html"
		@@ -20,10 +26,6 @@ uid [ultimate] Christian Cleberg <hello@cleberg.net></pre>
20	26	>My Top 8 Most Used Org-Mode Features</a
21	27	>
22	28	</div>
23		<div class="post">
24		<time datetime="2024-07-11 20:24:02">2024-07-11</time>
25		<a href="/blog/emacs-on-ipad.html">Emacs on iPadOS</a>
26		</div>
27	29
28	30	<br />
29	31	<a href="/blog/">All Posts →</a>

Commit 1b12d8a947

Commit `1b12d8a947`