cmc/cleberg.net
My personal web garden & blog.
clone: git clone https://gitbay.org/cmc/cleberg.net.git
main: content/blog/2022-07-01-git-server.org · raw
1#+date: [2022-07-01 Fri 00:00:00]
2#+title: Self-Hosting Guide: Git and cgit
3#+description: How to self-host Git repos and browse them with cgit.
4#+slug: git-server
5#+filetags: :linux:self-hosting:
6
7* My Approach to Self-Hosting Git
8
9I have often tried to self-host my Git repositories, but have always fallen
10short when I tried to find a suitable web interface to show on the front-end.
11
12After a few years, I have finally found a combination of methods that allow me
13to easily self-host my projects, view them on the web, and access them from
14anywhere.
15
16Before I dive into the details, I want to state a high-level summary of my
17self-hosted Git approach:
18
19- This method uses the =ssh://= (read & write) and =git://= (read-only)
20 protocols for push and pull access.
21 - For the =git://= protocol, I create a =git-daemon-export-ok= file in any
22 repository that I want to be cloneable by anyone.
23 - The web interface I am using (=cgit=) allows simple HTTP cloning by default.
24 I do not disable this setting as I want beginners to be able to clone one of
25 my repositories even if they don't know the proper method.
26- I am not enabling Smart HTTPS (Hypertext Transfer Protocol Secure) for any
27 repositories. Updates to repositories must be pushed via SSH (Secure Shell
28 Protocol).
29- Beyond the actual repository management, I am using =cgit= for the front-end
30 web interface.
31 - If you use the =scan-path=<path>= configuration in the =cgitrc=
32 configuration file to automatically find repositories, you can't exclude a
33 repository from =cgit= if it's stored within the path that =cgit= reads. To
34 host private repositories, you'd need to set up another directory that
35 =cgit= can't read.
36
37* Assumptions
38
39For the purposes of this walkthrough, I am assuming you have a URL
40(=git.example.com=) or internet protocol (IP) address (=207.84.26.991=)
41addressed to the server that you will be using to host your git repositories.
42
43* Adding a Git User
44
45In order to use the SSH method associated with Git, we will need to add a user
46named =git=. If you have used the SSH method for other git hosting sites, you
47are probably used to the following syntax:
48
49#+begin_src sh
50git clone [user@]server:project.git
51#+end_src
52
53The syntax above is an =scp=-like syntax for using SSH on the =git= user on the
54server to access your repository.
55
56Let's delete any remnants of an old =git= user, if any, and create the new user
57account:
58
59#+begin_src sh
60sudo deluser --remove-home git
61sudo adduser git
62#+end_src
63
64** Import Your SSH Keys to the Git User
65
66Once the =git= user is created, you will need to copy your public SSH key on
67your local development machine to the =git= user on the server.
68
69If you don't have an SSH key yet, create one with this command:
70
71#+begin_src sh
72ssh-keygen
73#+end_src
74
75Once you create the key pair, the public should be saved to =~/.ssh/id_rsa.pub=.
76
77If your server still has password-based authentication available, you can copy
78it over to your user's home directory like this:
79
80#+begin_src sh
81ssh-copy-id git@server
82#+end_src
83
84Otherwise, copy it over to any user that you can access.
85
86#+begin_src sh
87scp ~/.ssh/id_rsa.pub your_user@your_server:
88#+end_src
89
90Once on the server, you will need to copy the contents into the =git= user's
91=authorized_keys= file:
92
93#+begin_src sh
94cat id_rsa.pub > /home/git/.ssh/authorized_keys
95#+end_src
96
97** (Optional) Disable Password-Based SSH
98
99If you want to lock down your server and ensure that no one can authenticate in
100via SSH with a password, you will need to edit your SSH configuration.
101
102#+begin_src sh
103sudo nano /etc/ssh/sshd_config
104#+end_src
105
106Within this file, find the following settings and set them to the values I am
107showing below:
108
109#+begin_src conf
110PermitRootLogin no
111PasswordAuthentication no
112AuthenticationMethods publickey
113#+end_src
114
115You may have other Authentication Methods required in your personal set-up, so
116the key here is just to ensure that =AuthenticationMethods= does not allow
117passwords.
118
119*** Setting up the Base Directory
120
121Now that we have set up a =git= user to handle all transport methods, we need to
122set up the directory that we will be using as our base of all repositories.
123
124In my case, I am using =/git= as my source folder. To create this folder and
125assign it to the user we created, execute the following commands:
126
127#+begin_src sh
128sudo mkdir /git
129sudo chown -R git:git /git
130#+end_src
131
132*** Creating a Test Repository
133
134On your server, switch over to the =git= user in order to start managing git
135files.
136
137#+begin_src sh
138su git
139#+end_src
140
141Once logged-in as the =git= user, go to your base directory and create a test
142repository.
143
144#+begin_src sh
145cd /git
146mkdir test.git && cd test.git
147git init --bare
148#+end_src
149
150If you want to make this repo viewable/cloneable to the public via the =git://=
151protocol, you need to create a =git-daemon-export-ok= file inside the
152repository.
153
154#+begin_src sh
155touch git-daemon-export-ok
156#+end_src
157
158* Change the Login Shell for =git=
159
160To make sure that the =git= user is only used for git operations and nothing
161else, you need to change the user's login shell. To do this, simply use the
162=chsh= command:
163
164#+begin_src sh
165sudo chsh git
166#+end_src
167
168The interactive prompt will ask which shell you want the =git= user to use. You
169must use the following value:
170
171#+begin_src sh
172/usr/bin/git-shell
173#+end_src
174
175Once done, no one will be able to SSH to the =git= user or execute commands
176other than the standard git commands.
177
178* Opening the Firewall
179
180Don't forget to open up ports on the device firewall and network firewall if you
181want to access these repositories publicly. If you're using default ports,
182forward ports =22= (ssh) and =9418= (git) from your router to your server's IP
183address.
184
185If your server also has a firewall, ensure that the firewall allows the same
186ports that are forwarded from the router. For example, if you use =ufw=:
187
188#+begin_src sh
189sudo ufw allow 22
190sudo ufw allow 9418
191#+end_src
192
193** Non-Standard SSH Ports
194
195If you use a non-standard port for SSH, such as =9876=, you will need to create
196an SSH configuration file on your local development machine in order to connect
197to your server's git repositories.
198
199To do this, you'll need to define your custom port on your client machine in
200your =~/.ssh/config= file:
201
202#+begin_src sh
203nano ~/.ssh/config
204#+end_src
205
206#+begin_src conf
207Host git.example.com
208 # HostName can be a URL or an IP address
209 HostName git.example.com
210 Port 9876
211 User git
212#+end_src
213
214** Testing SSH
215
216There are two main syntaxes you can use to manage git over SSH:
217
218- =git clone [user@]server:project.git=
219- =git clone ssh://[user@]server/project.git=
220
221I prefer the first, which is an =scp=-like syntax. To test it, try to clone the
222test repository you set up on the server:
223
224#+begin_src sh
225git clone git@git.example.com:/git/test.git
226#+end_src
227
228* Enabling Read-Only Access
229
230If you want people to be able to clone any repository where you've placed a
231=git-daemon-export-ok= file, you will need to start the git daemon.
232
233To do this on a system with =systemd=, create a service file:
234
235#+begin_src sh
236sudo nano /etc/systemd/system/git-daemon.service
237#+end_src
238
239Inside the =git-daemon.service= file, paste the following:
240
241#+begin_src conf
242[Unit]
243Description=Start Git Daemon
244
245[Service]
246ExecStart=/usr/bin/git daemon --reuseaddr --base-path=/git/ /git/
247
248Restart=always
249RestartSec=500ms
250
251StandardOutput=syslog
252StandardError=syslog
253SyslogIdentifier=git-daemon
254
255User=git
256Group=git
257
258[Install]
259WantedBy=multi-user.target
260#+end_src
261
262Once created, enable and start the service:
263
264#+begin_src sh
265sudo systemctl enable git-daemon.service
266sudo systemctl start git-daemon.service
267#+end_src
268
269To clone read-only via the =git://= protocol, you can use the following syntax:
270
271#+begin_src sh
272git clone git://git.example.com/test.git
273#+end_src
274
275* Migrating Repositories
276
277At this point, we have a working git server that works with both SSH and
278read-only access.
279
280For each of the repositories I had hosted a different provider, I executed the
281following commands in order to place a copy on my server as my new source of
282truth:
283
284Server:
285
286#+begin_src sh
287su git
288mkdir /git/<REPOSITORY_NAME>.git && cd /git/<REPOSITORY_NAME>.git
289git init --bare
290
291# If you want to make this repo viewable/cloneable to the public
292touch git-daemon-export-ok
293#+end_src
294
295Client:
296
297#+begin_src sh
298git clone git@<PREVIOUS_HOST>:<REPOSITORY_NAME>
299git remote set-url origin git@git.EXAMPLE.COM:/git/<REPOSITORY_NAME>.git
300git push
301#+end_src
302
303* Optional Web View: =cgit=
304
305If you want a web viewer for your repositories, you can use various tools, such
306as =gitweb=, =cgit=, or =klaus=. I chose =cgit= due to its simple interface and
307fairly easy set-up (compared to others). Not to mention that the [[https://git.kernel.org/][Linux kernel
308uses =cgit=]].
309
310** Docker Compose
311
312Instead of using my previous method of using a =docker run= command, I've
313updated this section to use =docker-compose= instead for an easier installation
314and simpler management and configuration.
315
316In order to use Docker Compose, you will set up a =docker-compose.yml= file to
317automatically connect resources like the repositories, =cgitrc=, and various
318files or folders to the =cgit= container you're creating:
319
320#+begin_src sh
321mkdir ~/cgit && cd ~/cgit
322nano docker-compose.yml
323#+end_src
324
325#+begin_src conf
326# docker-compose.yml
327version: '3'
328
329services:
330 cgit:
331 image: invokr/cgit
332 volumes:
333 - /git:/git
334 - ./cgitrc:/etc/cgitrc
335 - ./logo.png:/var/www/htdocs/cgit/logo.png
336 - ./favicon.png:/var/www/htdocs/cgit/favicon.png
337 - ./filters:/var/www/htdocs/cgit/filters
338 ports:
339 - "8763:80"
340 restart: always
341#+end_src
342
343Then, just start the container:
344
345#+begin_src sh
346sudo docker-compose up -d
347#+end_src
348
349Once it's finished installing, you can access the site at =<SERVER_IP>:8763= or
350use a reverse-proxy service to forward =cgit= to a URL, such as
351=git.example.com=. See the next section for more details on reverse proxying a
352URL to a local port.
353
354** Nginx Reverse Proxy
355
356I am using Nginx as my reverse proxy so that the =cgit= Docker container can use
357=git.example.com= as its uniform resource locator (URL). To do so, I simply
358created the following configuration file:
359
360#+begin_src sh
361sudo nano /etc/nginx/sites-available/git.example.com
362#+end_src
363
364#+begin_src conf
365server {
366 listen 80;
367 server_name git.example.com;
368
369 if ($host = git.example.com) {
370 return 301 https://$host$request_uri;
371 }
372
373 return 404;
374}
375
376server {
377 server_name git.example.com;
378 listen 443 ssl http2;
379
380 location / {
381 # The final `/` is important.
382 proxy_pass http://localhost:8763/;
383 add_header X-Frame-Options SAMEORIGIN;
384 add_header X-XSS-Protection "1; mode=block";
385 proxy_redirect off;
386 proxy_buffering off;
387 proxy_set_header Host $host;
388 proxy_set_header X-Real-IP $remote_addr;
389 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
390 proxy_set_header X-Forwarded-Proto $scheme;
391 proxy_set_header X-Forwarded-Port $server_port;
392 }
393
394 # INCLUDE ANY SSL CERTS HERE
395 include /etc/letsencrypt/options-ssl-nginx.conf;
396 ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
397}
398#+end_src
399
400Once created, symlink it and restart the web server.
401
402#+begin_src sh
403sudo ln -s /etc/nginx/sites-available/git.example.com /etc/nginx/sites-enabled/
404sudo systemctl restart nginx.service
405#+end_src
406
407As we can see below, my site at =git.example.com= is available and running:
408
409** Settings Up Git Details
410
411Once you have =cgit= running, you can add some small details, such as repository
412owners and descriptions by editing the following files within each repository.
413
414Alternatively, you can use the =cgitrc= file to edit these details if you only
415care to edit them for the purpose of seeing them on your website.
416
417The =description= file within the repository on your server will display the
418description online.
419
420#+begin_src sh
421cd /git/example.git
422nano description
423#+end_src
424
425You can add a =[gitweb]= block to the =config= file in order to display the
426owner of the repository.
427
428#+begin_src sh
429cd /git/example.git
430nano config
431#+end_src
432
433#+begin_src conf
434[gitweb]
435 owner = "YourName"
436#+end_src
437
438Note that you can ignore the configuration within each repository and simply set
439up this information in the =cgitrc= file, if you want to do it that way.
440
441** Editing =cgit=
442
443In order to edit certain items within =cgit=, you need to edit the =cgitrc=
444file.
445
446#+begin_src sh
447nano ~/cgit/cgitrc
448#+end_src
449
450Below is an example configuration for =cgitrc=. You can find all the
451configuration options within the [[https://git.zx2c4.com/cgit/plain/cgitrc.5.txt][configuration manual]].
452
453#+begin_src conf
454css=/cgit.css
455logo=/logo.png
456favicon=/favicon.png
457robots=noindex, nofollow
458
459enable-index-links=1
460enable-commit-graph=1
461enable-blame=1
462enable-log-filecount=1
463enable-log-linecount=1
464enable-git-config=1
465
466clone-url=git://git.example.com/$CGIT_REPO_URL ssh://git@git.example.com:/git/$CGIT_REPO_URL
467
468root-title=My Git Website
469root-desc=My personal git repositories.
470
471# Allow download of tar.gz, tar.bz2 and zip-files
472snapshots=tar.gz tar.bz2 zip
473
474##
475## List of common mimetypes
476##
477mimetype.gif=image/gif
478mimetype.html=text/html
479mimetype.jpg=image/jpeg
480mimetype.jpeg=image/jpeg
481mimetype.pdf=application/pdf
482mimetype.png=image/png
483mimetype.svg=image/svg+xml
484
485# Highlight source code
486# source-filter=/var/www/htdocs/cgit/filters/syntax-highlighting.sh
487source-filter=/var/www/htdocs/cgit/filters/syntax-highlighting.py
488
489# Format markdown, restructuredtext, manpages, text files, and html files
490# through the right converters
491about-filter=/var/www/htdocs/cgit/filters/about-formatting.sh
492
493##
494## Search for these files in the root of the default branch of repositories
495## for coming up with the about page:
496##
497readme=:README.md
498readme=:readme.md
499readme=:README.mkd
500readme=:readme.mkd
501readme=:README.rst
502readme=:readme.rst
503readme=:README.html
504readme=:readme.html
505readme=:README.htm
506readme=:readme.htm
507readme=:README.txt
508readme=:readme.txt
509readme=:README
510readme=:readme
511
512# Repositories
513
514# Uncomment the following line to scan a path instead of adding repositories manually
515# scan-path=/git
516
517## Test Section
518section=git/test-section
519
520repo.url=test.git
521repo.path=/git/test.git
522repo.readme=:README.md
523repo.owner=John Doe
524repo.desc=An example repository!
525#+end_src
526
527** Final Fixes: Syntax Highlighting & README Rendering
528
529After completing my initial install and playing around with it for a few days, I
530noticed two issues:
531
5321. Syntax highlighting did not work when viewing the source code within a file.
5332. The =about= tab within a repository was not rendered to HTML.
534
535The following process fixes these issues. To start, let's go to the =cgit=
536directory where we were editing our configuration file earlier.
537
538#+begin_src sh
539cd ~/cgit
540#+end_src
541
542In here, create two folders that will hold our syntax files:
543
544#+begin_src sh
545mkdir filters && mkdir filters/html-converters && cd filters
546#+end_src
547
548Next, download the default filters:
549
550#+begin_src sh
551curl https://git.zx2c4.com/cgit/plain/filters/about-formatting.sh > about-formatting.sh
552chmod 755 about-formatting.sh
553curl https://git.zx2c4.com/cgit/plain/filters/syntax-highlighting.py > syntax-highlighting.py
554chmod 755 syntax-highlighting.py
555#+end_src
556
557Finally, download the HTML conversion files you need. The example below
558downloads the Markdown converter:
559
560#+begin_src sh
561cd html-converters
562curl https://git.zx2c4.com/cgit/plain/filters/html-converters/md2html > md2html
563chmod 755 md2html
564#+end_src
565
566If you need other filters or html-converters found within [[https://git.zx2c4.com/cgit/tree/filters][the cgit project
567files]], repeat the =curl= and =chmod= process above for whichever files you need.
568
569However, formatting will not work quite yet since the Docker cgit container
570we're using doesn't have the formatting package installed. You can install this
571easily by install Python 3+ and the =pygments= package:
572
573#+begin_src sh
574# Enter the container's command line
575sudo docker exec -it cgit bash
576#+end_src
577
578#+begin_src sh
579# Install the necessary packages and then exit
580yum update -y && \
581yum upgrade -y && \
582yum install python3 python3-pip -y && \
583pip3 install markdown pygments && \
584exit
585#+end_src
586
587*You will need to enter the cgit docker container and re-run these =yum=
588commands every time you kill and restart the container!*
589
590If not done already, we need to add the following variables to our =cgitrc= file
591in order for =cgit= to know where our filtering files are:
592
593#+begin_src conf
594# Highlight source code with python pygments-based highlighter
595source-filter=/var/www/htdocs/cgit/filters/syntax-highlighting.py
596
597# Format markdown, restructuredtext, manpages, text files, and html files
598# through the right converters
599about-filter=/var/www/htdocs/cgit/filters/about-formatting.sh
600#+end_src
601
602Now you should see that syntax highlighting and README rendering to the =about=
603tab is fixed.
604
605** Theming
606
607I won't go into much detail in this section, but you can fully theme your
608installation of =cgit= since you have access to the =cgit.css= file in your web
609root. This is another file you can add as a volume to the =docker-compose.yml=
610file if you want to edit this without entering the container's command line.
611
612*** Remember to Back Up Your Data!
613
614The last thing to note is that running services on your own equipment means that
615you're assuming a level of risk that exists regarding data loss, catastrophes,
616etc. In order to reduce the impact of any such occurrence, I suggest backing up
617your data regularly.
618
619Backups can be automated via =cron=, by hooking your base directory up to a
620cloud provider, or even setting up hooks to push all repository info to git
621mirrors on other git hosts. Whatever the method, make sure that your data
622doesn't vanish in the event that your drives or servers fail.