content/blog/2022-07-01-git-server.org
639 lines · 18079 bytes
Procedures for Establishing a Personal Git Repository Server
- My Approach to Self-Hosting Git
- Assumptions
- Adding a Git User
- Change the Login Shell for
git - Opening the Firewall
- Enabling Read-Only Access
- Migrating Repositories
- Optional Web View:
cgit
My Approach to Self-Hosting Git
I have often tried to self-host my Git repositories, but have always fallen short when I tried to find a suitable web interface to show on the front-end.
After a few years, I have finally found a combination of methods that allow me to easily self-host my projects, view them on the web, and access them from anywhere.
Before I dive into the details, I want to state a high-level summary of my self-hosted Git approach:
-
This method uses the
ssh://(read & write) andgit://(read-only) protocols for push and pull access.- For the
git://protocol, I create agit-daemon-export-okfile in any repository that I want to be cloneable by anyone. - The web interface I am using (
cgit) allows simple HTTP cloning by default. I do not disable this setting as I want beginners to be able to clone one of my repositories even if they don't know the proper method.
- For the
- I am not enabling Smart HTTPS for any repositories. Updates to repositories must be pushed via SSH.
-
Beyond the actual repository management, I am using
cgitfor the front-end web interface.- If you use the
scan-path=<path>configuration in thecgitrcconfiguration file to automatically find repositories, you can't exclude a repository fromcgitif it's stored within the path thatcgitreads. To host private repositories, you'd need to set up another directory thatcgitcan't read.
- If you use the
Assumptions
For the purposes of this walkthrough, I am assuming you have a URL
(git.example.com) or IP address (207.84.26.991) addressed to the
server that you will be using to host your git repositories.
Adding a Git User
In order to use the SSH method associated with git, we will need to add
a user named git. If you have used the SSH method for other git
hosting sites, you are probably used to the following syntax:
git clone [user@]server:project.git
The syntax above is an scp-like syntax for using SSH on the git user
on the server to access your repository.
Let's delete any remnants of an old git user, if any, and create the
new user account:
sudo deluser --remove-home git
sudo adduser git
Import Your SSH Keys to the Git User
Once the git user is created, you will need to copy your public SSH
key on your local development machine to the git user on the server.
If you don't have an SSH key yet, create one with this command:
ssh-keygen
Once you create the key pair, the public should be saved to
~/.ssh/id_rsa.pub.
If your server still has password-based authentication available, you can copy it over to your user's home directory like this:
ssh-copy-id git@server
Otherwise, copy it over to any user that you can access.
scp ~/.ssh/id_rsa.pub your_user@your_server:
Once on the server, you will need to copy the contents into the git
user's authorized_keys file:
cat id_rsa.pub > /home/git/.ssh/authorized_keys
(Optional) Disable Password-Based SSH
If you want to lock down your server and ensure that no one can authenticate in via SSH with a password, you will need to edit your SSH configuration.
sudo nano /etc/ssh/sshd_config
Within this file, find the following settings and set them to the values I am showing below:
PermitRootLogin no
PasswordAuthentication no
AuthenticationMethods publickey
You may have other Authentication Methods required in your personal
set-up, so the key here is just to ensure that AuthenticationMethods
does not allow passwords.
Setting up the Base Directory
Now that we have set up a git user to handle all transport methods, we
need to set up the directory that we will be using as our base of all
repositories.
In my case, I am using /git as my source folder. To create this folder
and assign it to the user we created, execute the following commands:
sudo mkdir /git
sudo chown -R git:git /git
Creating a Test Repository
On your server, switch over to the git user in order to start managing
git files.
su git
Once logged-in as the git user, go to your base directory and create a
test repository.
cd /git
mkdir test.git && cd test.git
git init --bare
If you want to make this repo viewable/cloneable to the public via the
git:// protocol, you need to create a git-daemon-export-ok file
inside the repository.
touch git-daemon-export-ok
Change the Login Shell for git
To make sure that the git user is only used for git operations and
nothing else, you need to change the user's login shell. To do this,
simply use the chsh command:
sudo chsh git
The interactive prompt will ask which shell you want the git user to
use. You must use the following value:
/usr/bin/git-shell
Once done, no one will be able to SSH to the git user or execute
commands other than the standard git commands.
Opening the Firewall
Don't forget to open up ports on the device firewall and network
firewall if you want to access these repositories publicly. If you're
using default ports, forward ports 22 (ssh) and 9418 (git) from your
router to your server's IP address.
If your server also has a firewall, ensure that the firewall allows the
same ports that are forwarded from the router. For example, if you use
ufw:
sudo ufw allow 22
sudo ufw allow 9418
Non-Standard SSH Ports
If you use a non-standard port for SSH, such as 9876, you will need to
create an SSH configuration file on your local development machine in
order to connect to your server's git repositories.
To do this, you'll need to define your custom port on your client
machine in your ~/.ssh/config file:
nano ~/.ssh/config
Host git.example.com
# HostName can be a URL or an IP address
HostName git.example.com
Port 9876
User git
Testing SSH
There are two main syntaxes you can use to manage git over SSH:
git clone [user@]server:project.gitgit clone ssh://[user@]server/project.git
I prefer the first, which is an scp-like syntax. To test it, try to
clone the test repository you set up on the server:
git clone git@git.example.com:/git/test.git
Enabling Read-Only Access
If you want people to be able to clone any repository where you've
placed a git-daemon-export-ok file, you will need to start the git
daemon.
To do this on a system with systemd, create a service file:
sudo nano /etc/systemd/system/git-daemon.service
Inside the git-daemon.service file, paste the following:
[Unit]
Description=Start Git Daemon
[Service]
ExecStart=/usr/bin/git daemon --reuseaddr --base-path=/git/ /git/
Restart=always
RestartSec=500ms
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=git-daemon
User=git
Group=git
[Install]
WantedBy=multi-user.target
Once created, enable and start the service:
sudo systemctl enable git-daemon.service
sudo systemctl start git-daemon.service
To clone read-only via the git:// protocol, you can use the following
syntax:
git clone git://git.example.com/test.git
Migrating Repositories
At this point, we have a working git server that works with both SSH and read-only access.
For each of the repositories I had hosted a different provider, I executed the following commands in order to place a copy on my server as my new source of truth:
Server:
su git
mkdir /git/<REPOSITORY_NAME>.git && cd /git/<REPOSITORY_NAME>.git
git init --bare
# If you want to make this repo viewable/cloneable to the public
touch git-daemon-export-ok
Client:
git clone git@<PREVIOUS_HOST>:<REPOSITORY_NAME>
git remote set-url origin git@git.EXAMPLE.COM:/git/<REPOSITORY_NAME>.git
git push
Optional Web View: cgit
If you want a web viewer for your repositories, you can use various
tools, such as gitweb, cgit, or klaus. I chose cgit due to its
simple interface and fairly easy set-up (compared to others). Not to
mention that the Linux kernel uses cgit.
Docker Compose
Instead of using my previous method of using a docker run command,
I've updated this section to use docker-compose instead for an easier
installation and simpler management and configuration.
In order to use Docker Compose, you will set up a docker-compose.yml
file to automatically connect resources like the repositories, cgitrc,
and various files or folders to the cgit container you're creating:
mkdir ~/cgit && cd ~/cgit
nano docker-compose.yml
# docker-compose.yml
version: '3'
services:
cgit:
image: invokr/cgit
volumes:
- /git:/git
- ./cgitrc:/etc/cgitrc
- ./logo.png:/var/www/htdocs/cgit/logo.png
- ./favicon.png:/var/www/htdocs/cgit/favicon.png
- ./filters:/var/www/htdocs/cgit/filters
ports:
- "8763:80"
restart: always
Then, just start the container:
sudo docker-compose up -d
Once it's finished installing, you can access the site at
<SERVER_IP>:8763 or use a reverse-proxy service to forward cgit to a
URL, such as git.example.com. See the next section for more details on
reverse proxying a URL to a local port.
Nginx Reverse Proxy
I am using Nginx as my reverse proxy so that the cgit Docker container
can use git.example.com as its URL. To do so, I simply created the
following configuration file:
sudo nano /etc/nginx/sites-available/git.example.com
server {
listen 80;
server_name git.example.com;
if ($host = git.example.com) {
return 301 https://$host$request_uri;
}
return 404;
}
server {
server_name git.example.com;
listen 443 ssl http2;
location / {
# The final `/` is important.
proxy_pass http://localhost:8763/;
add_header X-Frame-Options SAMEORIGIN;
add_header X-XSS-Protection "1; mode=block";
proxy_redirect off;
proxy_buffering off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Port $server_port;
}
# INCLUDE ANY SSL CERTS HERE
include /etc/letsencrypt/options-ssl-nginx.conf;
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
}
Once created, symlink it and restart the web server.
sudo ln -s /etc/nginx/sites-available/git.example.com /etc/nginx/sites-enabled/
sudo systemctl restart nginx.service
As we can see below, my site at git.example.com is available and
running:
Settings Up Git Details
Once you have cgit running, you can add some small details, such as
repository owners and descriptions by editing the following files within
each repository.
Alternatively, you can use the cgitrc file to edit these details if
you only care to edit them for the purpose of seeing them on your
website.
The description file within the repository on your server will display
the description online.
cd /git/example.git
nano description
You can add a [gitweb] block to the config file in order to display
the owner of the repository.
cd /git/example.git
nano config
[gitweb]
owner = "YourName"
Note that you can ignore the configuration within each repository and
simply set up this information in the cgitrc file, if you want to do
it that way.
Editing cgit
In order to edit certain items within cgit, you need to edit the
cgitrc file.
nano ~/cgit/cgitrc
Below is an example configuration for cgitrc. You can find all the
configuration options within the [configuration manual]
(https://git.zx2c4.com/cgit/plain/cgitrc.5.txt).
css=/cgit.css
logo=/logo.png
favicon=/favicon.png
robots=noindex, nofollow
enable-index-links=1
enable-commit-graph=1
enable-blame=1
enable-log-filecount=1
enable-log-linecount=1
enable-git-config=1
clone-url=git://git.example.com/$CGIT_REPO_URL ssh://git@git.example.com:/git/$CGIT_REPO_URL
root-title=My Git Website
root-desc=My personal git repositories.
# Allow download of tar.gz, tar.bz2 and zip-files
snapshots=tar.gz tar.bz2 zip
##
## List of common mimetypes
##
mimetype.gif=image/gif
mimetype.html=text/html
mimetype.jpg=image/jpeg
mimetype.jpeg=image/jpeg
mimetype.pdf=application/pdf
mimetype.png=image/png
mimetype.svg=image/svg+xml
# Highlight source code
# source-filter=/var/www/htdocs/cgit/filters/syntax-highlighting.sh
source-filter=/var/www/htdocs/cgit/filters/syntax-highlighting.py
# Format markdown, restructuredtext, manpages, text files, and html files
# through the right converters
about-filter=/var/www/htdocs/cgit/filters/about-formatting.sh
##
## Search for these files in the root of the default branch of repositories
## for coming up with the about page:
##
readme=:README.md
readme=:readme.md
readme=:README.mkd
readme=:readme.mkd
readme=:README.rst
readme=:readme.rst
readme=:README.html
readme=:readme.html
readme=:README.htm
readme=:readme.htm
readme=:README.txt
readme=:readme.txt
readme=:README
readme=:readme
# Repositories
# Uncomment the following line to scan a path instead of adding repositories manually
# scan-path=/git
## Test Section
section=git/test-section
repo.url=test.git
repo.path=/git/test.git
repo.readme=:README.md
repo.owner=John Doe
repo.desc=An example repository!
Final Fixes: Syntax Highlighting & README Rendering
After completing my initial install and playing around with it for a few days, I noticed two issues:
- Syntax highlighting did not work when viewing the source code within a file.
- The
abouttab within a repository was not rendered to HTML.
The following process fixes these issues. To start, let's go to the
cgit directory where we were editing our configuration file earlier.
cd ~/cgit
In here, create two folders that will hold our syntax files:
mkdir filters && mkdir filters/html-converters && cd filters
Next, download the default filters:
curl https://git.zx2c4.com/cgit/plain/filters/about-formatting.sh > about-formatting.sh
chmod 755 about-formatting.sh
curl https://git.zx2c4.com/cgit/plain/filters/syntax-highlighting.py > syntax-highlighting.py
chmod 755 syntax-highlighting.py
Finally, download the HTML conversion files you need. The example below downloads the Markdown converter:
cd html-converters
curl https://git.zx2c4.com/cgit/plain/filters/html-converters/md2html > md2html
chmod 755 md2html
If you need other filters or html-converters found within
the cgit project files,
repeat the curl and chmod process above for whichever files you
need.
However, formatting will not work quite yet since the Docker cgit
container we're using doesn't have the formatting package installed. You
can install this easily by install Python 3+ and the pygments package:
# Enter the container's command line
sudo docker exec -it cgit bash
# Install the necessary packages and then exit
yum update -y && \
yum upgrade -y && \
yum install python3 python3-pip -y && \
pip3 install markdown pygments && \
exit
You will need to enter the cgit docker container and re-run these yum
commands every time you kill and restart the container!
If not done already, we need to add the following variables to our
cgitrc file in order for cgit to know where our filtering files are:
# Highlight source code with python pygments-based highlighter
source-filter=/var/www/htdocs/cgit/filters/syntax-highlighting.py
# Format markdown, restructuredtext, manpages, text files, and html files
# through the right converters
about-filter=/var/www/htdocs/cgit/filters/about-formatting.sh
Now you should see that syntax highlighting and README rendering to the
about tab is fixed.
Theming
I won't go into much detail in this section, but you can fully theme
your installation of cgit since you have access to the cgit.css file
in your web root. This is another file you can add as a volume to the
docker-compose.yml file if you want to edit this without entering the
container's command line.
:warning: Remember to Back Up Your Data!
The last thing to note is that running services on your own equipment means that you're assuming a level of risk that exists regarding data loss, catastrophes, etc. In order to reduce the impact of any such occurrence, I suggest backing up your data regularly.
Backups can be automated via cron, by hooking your base directory up
to a cloud provider, or even setting up hooks to push all repository
info to git mirrors on other git hosts. Whatever the method, make sure
that your data doesn't vanish in the event that your drives or servers
fail.