content/blog/2021-03-19-clone-github-repos.org

  1#+date:        <2021-03-19 Fri 00:00:00>
  2#+title:       Automated Cloning Procedures for GitHub and Sourcehut Repositories
  3#+description: Script-based methodology for cloning multiple repositories from GitHub and Sourcehut accounts. Includes automation techniques to streamline repository management and backups.
  4#+slug:        clone-github-repos
  5#+filetags:    :git:cloning:scripting:
  6
  7* Cloning from GitHub
  8
  9If you're like me and use a lot of different devices (and sometimes
 10decide to just wipe your device and start with a new OS), you probably
 11know the pain of cloning all your old code repositories down to your
 12local file system.
 13
 14If you're using GitHub, you can easily clone all of your code back down
 15in just seconds. First, create a bash script. I do so by opening a new
 16file in =nano=, but you can use =gedit=, =vim=, or something else:
 17
 18#+begin_src sh
 19nano clone_github_repos.sh
 20#+end_src
 21
 22Next, paste in the following information. Note that you can replace the
 23word =users= in the first line with =orgs= and type an organization's
 24name instead of a user's name.
 25
 26#+begin_src sh
 27CNTX=users; NAME=YOUR-USERNAME; PAGE=1
 28curl "https://api.github.com/$CNTX/$NAME/repos?page=$PAGE&per_page=100" |
 29  grep -e 'git_url*' |
 30  cut -d " -f 4 |
 31  xargs -L1 git clone
 32#+end_src
 33
 34Finally, save the bash script and make it executable.
 35
 36#+begin_src sh
 37chmod a+x clone_github_repos.sh
 38#+end_src
 39
 40Now you can run the script and should see the cloning process begin.
 41
 42#+begin_src sh
 43./clone_github_repos.sh
 44#+end_src
 45
 46* Cloning from Sourcehut
 47
 48I haven't fully figured out how to directly incorporate Sourcehut's
 49GraphQL API into a bash script yet, so this one will take two steps.
 50
 51First, log-in to Sourcehut and go to their
 52[[https://git.sr.ht/graphql][GraphQL playground for Git]]. Next, paste
 53the following query into the left box:
 54
 55#+begin_src sh
 56query {
 57  me {
 58    canonicalName
 59    repositories() {
 60      cursor
 61      results {
 62        name
 63      }
 64    }
 65  }
 66}
 67#+end_src
 68
 69The output on the right side will give you an object of all your
 70repositories. Just grab that text and remove all the characters such as
 71quotation marks and curly brackets. You will need a single-line list of
 72space-separated values for the next step.
 73
 74Now let's create the bash script:
 75
 76#+begin_src sh
 77nano clone_sourcehut_repos.sh
 78#+end_src
 79
 80Next, paste the following bash script in with the list of repositories
 81you obtained above and replace =your-username= with your username.
 82
 83Note that this uses the SSH-based Git cloning method
 84(e.g. =git@git...=), so you'll need to ensure you have set up Sourcehut
 85with your SSH key.
 86
 87#+begin_src sh
 88repos=(repo1 repo2 repo3)
 89
 90# List all sub-directories in the current directory
 91for repo in "${repos[@]}"
 92do
 93    # Clone
 94    git clone git@git.sr.ht:~your-username/$repo
 95done
 96#+end_src
 97
 98Finally, save the bash script and make it executable.
 99
100#+begin_src sh
101chmod a+x clone_sourcehut_repos.sh
102#+end_src
103
104Now you can run the script and should see the cloning process begin.
105
106#+begin_src sh
107./clone_sourcehut_repos.sh
108#+end_src
109
110* Moving Repositories to a New Host
111
112Now that you have all of your code repositories cloned to your local
113computer, you may want to change the remote host (e.g., moving from
114GitHub to GitLab). To do this, let's create another bash script:
115
116#+begin_src sh
117nano change_remote_urls.sh
118#+end_src
119
120Past the following information and be sure to change the URL information
121to whichever host you are moving to. For this example, I am looping
122through all of my cloned GitHub directories and changing them to
123Sourcehut (e.g. =<YOUR_NEW_REMOTE_URL>= -> =git@git.sr.ht:~myusername=).
124
125#+begin_src sh
126# List all sub-directories in the current directory
127for dir in */
128do
129    # Remove the trailing "/"
130    dir=${dir%*/}
131    # Enter sub-directory
132    cd $dir
133    # Change remote Git URL
134    git remote set-url origin <YOUR_NEW_REMOTE_URL>/"${dir##*/}"
135    # Push code to new remote
136    git push
137    # Go back to main directory
138    cd ..
139done
140#+end_src
141
142Finally, save the bash script and make it executable.
143
144#+begin_src sh
145chmod a+x change_remote_urls.sh
146#+end_src
147
148Now you can run the script and should see the cloning process begin.
149
150#+begin_src sh
151./change_remote_urls.sh
152#+end_src