content/blog/2022-07-01-git-server.org
639 lines · 18079 bytes
1#+date: <2022-07-01 Fri 00:00:00>
2#+title: Procedures for Establishing a Personal Git Repository Server
3#+description: Comprehensive instructions for secure and efficient setup, hosting, and management of personal Git repositories to facilitate collaborative development and remote access.
4#+slug: git-server
5#+filetags: :git:server:self-hosting:
6
7* My Approach to Self-Hosting Git
8
9I have often tried to self-host my Git repositories, but have always
10fallen short when I tried to find a suitable web interface to show on
11the front-end.
12
13After a few years, I have finally found a combination of methods that
14allow me to easily self-host my projects, view them on the web, and
15access them from anywhere.
16
17Before I dive into the details, I want to state a high-level summary of
18my self-hosted Git approach:
19
20- This method uses the =ssh://= (read & write) and =git://= (read-only)
21 protocols for push and pull access.
22 - For the =git://= protocol, I create a =git-daemon-export-ok= file in
23 any repository that I want to be cloneable by anyone.
24 - The web interface I am using (=cgit=) allows simple HTTP cloning by
25 default. I do not disable this setting as I want beginners to be
26 able to clone one of my repositories even if they don't know the
27 proper method.
28- I am not enabling Smart HTTPS for any repositories. Updates to
29 repositories must be pushed via SSH.
30- Beyond the actual repository management, I am using =cgit= for the
31 front-end web interface.
32 - If you use the =scan-path=<path>= configuration in the =cgitrc=
33 configuration file to automatically find repositories, you can't
34 exclude a repository from =cgit= if it's stored within the path that
35 =cgit= reads. To host private repositories, you'd need to set up
36 another directory that =cgit= can't read.
37
38* Assumptions
39
40For the purposes of this walkthrough, I am assuming you have a URL
41(=git.example.com=) or IP address (=207.84.26.991=) addressed to the
42server that you will be using to host your git repositories.
43
44* Adding a Git User
45
46In order to use the SSH method associated with git, we will need to add
47a user named =git=. If you have used the SSH method for other git
48hosting sites, you are probably used to the following syntax:
49
50#+begin_src sh
51git clone [user@]server:project.git
52#+end_src
53
54The syntax above is an =scp=-like syntax for using SSH on the =git= user
55on the server to access your repository.
56
57Let's delete any remnants of an old =git= user, if any, and create the
58new user account:
59
60#+begin_src sh
61sudo deluser --remove-home git
62sudo adduser git
63#+end_src
64
65** Import Your SSH Keys to the Git User
66
67Once the =git= user is created, you will need to copy your public SSH
68key on your local development machine to the =git= user on the server.
69
70If you don't have an SSH key yet, create one with this command:
71
72#+begin_src sh
73ssh-keygen
74#+end_src
75
76Once you create the key pair, the public should be saved to
77=~/.ssh/id_rsa.pub=.
78
79If your server still has password-based authentication available, you
80can copy it over to your user's home directory like this:
81
82#+begin_src sh
83ssh-copy-id git@server
84#+end_src
85
86Otherwise, copy it over to any user that you can access.
87
88#+begin_src sh
89scp ~/.ssh/id_rsa.pub your_user@your_server:
90#+end_src
91
92Once on the server, you will need to copy the contents into the =git=
93user's =authorized_keys= file:
94
95#+begin_src sh
96cat id_rsa.pub > /home/git/.ssh/authorized_keys
97#+end_src
98
99** (Optional) Disable Password-Based SSH
100
101If you want to lock down your server and ensure that no one can
102authenticate in via SSH with a password, you will need to edit your SSH
103configuration.
104
105#+begin_src sh
106sudo nano /etc/ssh/sshd_config
107#+end_src
108
109Within this file, find the following settings and set them to the values
110I am showing below:
111
112#+begin_src conf
113PermitRootLogin no
114PasswordAuthentication no
115AuthenticationMethods publickey
116#+end_src
117
118You may have other Authentication Methods required in your personal
119set-up, so the key here is just to ensure that =AuthenticationMethods=
120does not allow passwords.
121
122*** Setting up the Base Directory
123
124Now that we have set up a =git= user to handle all transport methods, we
125need to set up the directory that we will be using as our base of all
126repositories.
127
128In my case, I am using =/git= as my source folder. To create this folder
129and assign it to the user we created, execute the following commands:
130
131#+begin_src sh
132sudo mkdir /git
133sudo chown -R git:git /git
134#+end_src
135
136*** Creating a Test Repository
137
138On your server, switch over to the =git= user in order to start managing
139git files.
140
141#+begin_src sh
142su git
143#+end_src
144
145Once logged-in as the =git= user, go to your base directory and create a
146test repository.
147
148#+begin_src sh
149cd /git
150mkdir test.git && cd test.git
151git init --bare
152#+end_src
153
154If you want to make this repo viewable/cloneable to the public via the
155=git://= protocol, you need to create a =git-daemon-export-ok= file
156inside the repository.
157
158#+begin_src sh
159touch git-daemon-export-ok
160#+end_src
161
162* Change the Login Shell for =git=
163
164To make sure that the =git= user is only used for git operations and
165nothing else, you need to change the user's login shell. To do this,
166simply use the =chsh= command:
167
168#+begin_src sh
169sudo chsh git
170#+end_src
171
172The interactive prompt will ask which shell you want the =git= user to
173use. You must use the following value:
174
175#+begin_src sh
176/usr/bin/git-shell
177#+end_src
178
179Once done, no one will be able to SSH to the =git= user or execute
180commands other than the standard git commands.
181
182* Opening the Firewall
183
184Don't forget to open up ports on the device firewall and network
185firewall if you want to access these repositories publicly. If you're
186using default ports, forward ports =22= (ssh) and =9418= (git) from your
187router to your server's IP address.
188
189If your server also has a firewall, ensure that the firewall allows the
190same ports that are forwarded from the router. For example, if you use
191=ufw=:
192
193#+begin_src sh
194sudo ufw allow 22
195sudo ufw allow 9418
196#+end_src
197
198** Non-Standard SSH Ports
199
200If you use a non-standard port for SSH, such as =9876=, you will need to
201create an SSH configuration file on your local development machine in
202order to connect to your server's git repositories.
203
204To do this, you'll need to define your custom port on your client
205machine in your =~/.ssh/config= file:
206
207#+begin_src sh
208nano ~/.ssh/config
209#+end_src
210
211#+begin_src conf
212Host git.example.com
213 # HostName can be a URL or an IP address
214 HostName git.example.com
215 Port 9876
216 User git
217#+end_src
218
219** Testing SSH
220
221There are two main syntaxes you can use to manage git over SSH:
222
223- =git clone [user@]server:project.git=
224- =git clone ssh://[user@]server/project.git=
225
226I prefer the first, which is an =scp=-like syntax. To test it, try to
227clone the test repository you set up on the server:
228
229#+begin_src sh
230git clone git@git.example.com:/git/test.git
231#+end_src
232
233* Enabling Read-Only Access
234
235If you want people to be able to clone any repository where you've
236placed a =git-daemon-export-ok= file, you will need to start the git
237daemon.
238
239To do this on a system with =systemd=, create a service file:
240
241#+begin_src sh
242sudo nano /etc/systemd/system/git-daemon.service
243#+end_src
244
245Inside the =git-daemon.service= file, paste the following:
246
247#+begin_src conf
248[Unit]
249Description=Start Git Daemon
250
251[Service]
252ExecStart=/usr/bin/git daemon --reuseaddr --base-path=/git/ /git/
253
254Restart=always
255RestartSec=500ms
256
257StandardOutput=syslog
258StandardError=syslog
259SyslogIdentifier=git-daemon
260
261User=git
262Group=git
263
264[Install]
265WantedBy=multi-user.target
266#+end_src
267
268Once created, enable and start the service:
269
270#+begin_src sh
271sudo systemctl enable git-daemon.service
272sudo systemctl start git-daemon.service
273#+end_src
274
275To clone read-only via the =git://= protocol, you can use the following
276syntax:
277
278#+begin_src sh
279git clone git://git.example.com/test.git
280#+end_src
281
282* Migrating Repositories
283
284At this point, we have a working git server that works with both SSH and
285read-only access.
286
287For each of the repositories I had hosted a different provider, I
288executed the following commands in order to place a copy on my server as
289my new source of truth:
290
291Server:
292
293#+begin_src sh
294su git
295mkdir /git/<REPOSITORY_NAME>.git && cd /git/<REPOSITORY_NAME>.git
296git init --bare
297
298# If you want to make this repo viewable/cloneable to the public
299touch git-daemon-export-ok
300#+end_src
301
302Client:
303
304#+begin_src sh
305git clone git@<PREVIOUS_HOST>:<REPOSITORY_NAME>
306git remote set-url origin git@git.EXAMPLE.COM:/git/<REPOSITORY_NAME>.git
307git push
308#+end_src
309
310* Optional Web View: =cgit=
311
312If you want a web viewer for your repositories, you can use various
313tools, such as =gitweb=, =cgit=, or =klaus=. I chose =cgit= due to its
314simple interface and fairly easy set-up (compared to others). Not to
315mention that the [[https://git.kernel.org/][Linux kernel uses =cgit=]].
316
317** Docker Compose
318
319Instead of using my previous method of using a =docker run= command,
320I've updated this section to use =docker-compose= instead for an easier
321installation and simpler management and configuration.
322
323In order to use Docker Compose, you will set up a =docker-compose.yml=
324file to automatically connect resources like the repositories, =cgitrc=,
325and various files or folders to the =cgit= container you're creating:
326
327#+begin_src sh
328mkdir ~/cgit && cd ~/cgit
329nano docker-compose.yml
330#+end_src
331
332#+begin_src conf
333# docker-compose.yml
334version: '3'
335
336services:
337 cgit:
338 image: invokr/cgit
339 volumes:
340 - /git:/git
341 - ./cgitrc:/etc/cgitrc
342 - ./logo.png:/var/www/htdocs/cgit/logo.png
343 - ./favicon.png:/var/www/htdocs/cgit/favicon.png
344 - ./filters:/var/www/htdocs/cgit/filters
345 ports:
346 - "8763:80"
347 restart: always
348#+end_src
349
350Then, just start the container:
351
352#+begin_src sh
353sudo docker-compose up -d
354#+end_src
355
356Once it's finished installing, you can access the site at
357=<SERVER_IP>:8763= or use a reverse-proxy service to forward =cgit= to a
358URL, such as =git.example.com=. See the next section for more details on
359reverse proxying a URL to a local port.
360
361** Nginx Reverse Proxy
362
363I am using Nginx as my reverse proxy so that the =cgit= Docker container
364can use =git.example.com= as its URL. To do so, I simply created the
365following configuration file:
366
367#+begin_src sh
368sudo nano /etc/nginx/sites-available/git.example.com
369#+end_src
370
371#+begin_src conf
372server {
373 listen 80;
374 server_name git.example.com;
375
376 if ($host = git.example.com) {
377 return 301 https://$host$request_uri;
378 }
379
380 return 404;
381}
382
383server {
384 server_name git.example.com;
385 listen 443 ssl http2;
386
387 location / {
388 # The final `/` is important.
389 proxy_pass http://localhost:8763/;
390 add_header X-Frame-Options SAMEORIGIN;
391 add_header X-XSS-Protection "1; mode=block";
392 proxy_redirect off;
393 proxy_buffering off;
394 proxy_set_header Host $host;
395 proxy_set_header X-Real-IP $remote_addr;
396 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
397 proxy_set_header X-Forwarded-Proto $scheme;
398 proxy_set_header X-Forwarded-Port $server_port;
399 }
400
401 # INCLUDE ANY SSL CERTS HERE
402 include /etc/letsencrypt/options-ssl-nginx.conf;
403 ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
404}
405#+end_src
406
407Once created, symlink it and restart the web server.
408
409#+begin_src sh
410sudo ln -s /etc/nginx/sites-available/git.example.com /etc/nginx/sites-enabled/
411sudo systemctl restart nginx.service
412#+end_src
413
414As we can see below, my site at =git.example.com= is available and
415running:
416
417** Settings Up Git Details
418
419Once you have =cgit= running, you can add some small details, such as
420repository owners and descriptions by editing the following files within
421each repository.
422
423Alternatively, you can use the =cgitrc= file to edit these details if
424you only care to edit them for the purpose of seeing them on your
425website.
426
427The =description= file within the repository on your server will display
428the description online.
429
430#+begin_src sh
431cd /git/example.git
432nano description
433#+end_src
434
435You can add a =[gitweb]= block to the =config= file in order to display
436the owner of the repository.
437
438#+begin_src sh
439cd /git/example.git
440nano config
441#+end_src
442
443#+begin_src conf
444[gitweb]
445 owner = "YourName"
446#+end_src
447
448Note that you can ignore the configuration within each repository and
449simply set up this information in the =cgitrc= file, if you want to do
450it that way.
451
452** Editing =cgit=
453
454In order to edit certain items within =cgit=, you need to edit the
455=cgitrc= file.
456
457#+begin_src sh
458nano ~/cgit/cgitrc
459#+end_src
460
461Below is an example configuration for =cgitrc=. You can find all the
462configuration options within the [configuration manual]
463([[https://git.zx2c4.com/cgit/plain/cgitrc.5.txt]]).
464
465#+begin_src conf
466css=/cgit.css
467logo=/logo.png
468favicon=/favicon.png
469robots=noindex, nofollow
470
471enable-index-links=1
472enable-commit-graph=1
473enable-blame=1
474enable-log-filecount=1
475enable-log-linecount=1
476enable-git-config=1
477
478clone-url=git://git.example.com/$CGIT_REPO_URL ssh://git@git.example.com:/git/$CGIT_REPO_URL
479
480root-title=My Git Website
481root-desc=My personal git repositories.
482
483# Allow download of tar.gz, tar.bz2 and zip-files
484snapshots=tar.gz tar.bz2 zip
485
486##
487## List of common mimetypes
488##
489mimetype.gif=image/gif
490mimetype.html=text/html
491mimetype.jpg=image/jpeg
492mimetype.jpeg=image/jpeg
493mimetype.pdf=application/pdf
494mimetype.png=image/png
495mimetype.svg=image/svg+xml
496
497# Highlight source code
498# source-filter=/var/www/htdocs/cgit/filters/syntax-highlighting.sh
499source-filter=/var/www/htdocs/cgit/filters/syntax-highlighting.py
500
501# Format markdown, restructuredtext, manpages, text files, and html files
502# through the right converters
503about-filter=/var/www/htdocs/cgit/filters/about-formatting.sh
504
505##
506## Search for these files in the root of the default branch of repositories
507## for coming up with the about page:
508##
509readme=:README.md
510readme=:readme.md
511readme=:README.mkd
512readme=:readme.mkd
513readme=:README.rst
514readme=:readme.rst
515readme=:README.html
516readme=:readme.html
517readme=:README.htm
518readme=:readme.htm
519readme=:README.txt
520readme=:readme.txt
521readme=:README
522readme=:readme
523
524# Repositories
525
526# Uncomment the following line to scan a path instead of adding repositories manually
527# scan-path=/git
528
529## Test Section
530section=git/test-section
531
532repo.url=test.git
533repo.path=/git/test.git
534repo.readme=:README.md
535repo.owner=John Doe
536repo.desc=An example repository!
537#+end_src
538
539** Final Fixes: Syntax Highlighting & README Rendering
540
541After completing my initial install and playing around with it for a few
542days, I noticed two issues:
543
5441. Syntax highlighting did not work when viewing the source code within
545 a file.
5462. The =about= tab within a repository was not rendered to HTML.
547
548The following process fixes these issues. To start, let's go to the
549=cgit= directory where we were editing our configuration file earlier.
550
551#+begin_src sh
552cd ~/cgit
553#+end_src
554
555In here, create two folders that will hold our syntax files:
556
557#+begin_src sh
558mkdir filters && mkdir filters/html-converters && cd filters
559#+end_src
560
561Next, download the default filters:
562
563#+begin_src sh
564curl https://git.zx2c4.com/cgit/plain/filters/about-formatting.sh > about-formatting.sh
565chmod 755 about-formatting.sh
566curl https://git.zx2c4.com/cgit/plain/filters/syntax-highlighting.py > syntax-highlighting.py
567chmod 755 syntax-highlighting.py
568#+end_src
569
570Finally, download the HTML conversion files you need. The example below
571downloads the Markdown converter:
572
573#+begin_src sh
574cd html-converters
575curl https://git.zx2c4.com/cgit/plain/filters/html-converters/md2html > md2html
576chmod 755 md2html
577#+end_src
578
579If you need other filters or html-converters found within
580[[https://git.zx2c4.com/cgit/tree/filters][the cgit project files]],
581repeat the =curl= and =chmod= process above for whichever files you
582need.
583
584However, formatting will not work quite yet since the Docker cgit
585container we're using doesn't have the formatting package installed. You
586can install this easily by install Python 3+ and the =pygments= package:
587
588#+begin_src sh
589# Enter the container's command line
590sudo docker exec -it cgit bash
591#+end_src
592
593#+begin_src sh
594# Install the necessary packages and then exit
595yum update -y && \
596yum upgrade -y && \
597yum install python3 python3-pip -y && \
598pip3 install markdown pygments && \
599exit
600#+end_src
601
602*You will need to enter the cgit docker container and re-run these =yum=
603commands every time you kill and restart the container!*
604
605If not done already, we need to add the following variables to our
606=cgitrc= file in order for =cgit= to know where our filtering files are:
607
608#+begin_src conf
609# Highlight source code with python pygments-based highlighter
610source-filter=/var/www/htdocs/cgit/filters/syntax-highlighting.py
611
612# Format markdown, restructuredtext, manpages, text files, and html files
613# through the right converters
614about-filter=/var/www/htdocs/cgit/filters/about-formatting.sh
615#+end_src
616
617Now you should see that syntax highlighting and README rendering to the
618=about= tab is fixed.
619
620** Theming
621
622I won't go into much detail in this section, but you can fully theme
623your installation of =cgit= since you have access to the =cgit.css= file
624in your web root. This is another file you can add as a volume to the
625=docker-compose.yml= file if you want to edit this without entering the
626container's command line.
627
628*** :warning: Remember to Back Up Your Data!
629
630The last thing to note is that running services on your own equipment
631means that you're assuming a level of risk that exists regarding data
632loss, catastrophes, etc. In order to reduce the impact of any such
633occurrence, I suggest backing up your data regularly.
634
635Backups can be automated via =cron=, by hooking your base directory up
636to a cloud provider, or even setting up hooks to push all repository
637info to git mirrors on other git hosts. Whatever the method, make sure
638that your data doesn't vanish in the event that your drives or servers
639fail.