cmc/cleberg.net

  1#+date:        [2025-02-25 Mon 19:20:05]
  2#+title:       Email Migration: 5,000+ Emails, Proton to Migadu
  3#+description: Moving 5,000+ emails from Proton Mail to Migadu.
  4#+slug:        email-migration
  5#+filetags:    :privacy:
  6
  7* The Setup
  8
  9I recently migrated my emails from Proton Mail to Migadu after a failed attempt
 10to get myself into the Proton ecosystem and wanted to detail my process, as it
 11was far more painful than expected.
 12
 13To give some context: I had nearly 5000 messages stored, accounting for around
 142.5 GB of space.
 15
 16Overall, this process would have taken all day had I done it in one sitting, but
 17I decided to break it up and it lasted a couple days before I was able to say that
 18all my messages were stored in my new account.
 19
 20* Exporting Messages
 21
 22To start, I needed to export my messages from Proton Mail. As I am using macOS,
 23I was able to use the [[https://proton.me/support/proton-mail-export-tool][Proton Mail Export Tool]]. However, the downside is that
 24this dumps every single email on your account into a single folder in the =.eml=
 25format. They also export a JSON file for each message, in the case you're
 26importing back into Proton Mail.
 27
 28This means that anything in my Inbox, Sent, Archive, Trash, and user-created
 29folders were all dumped out into a single folder with incomprehensible names.
 30
 31Without a clear path to easily figure out how to re-organize my emails into a
 32new account, I was left a bit annoyed at Proton's export process.
 33
 34* Importing Messages
 35
 36Left with a pile of messages and no way to discern what they were without
 37opening each one, I decided to try and use Thunderbird to import messages into
 38my new Migadu IMAP account.
 39
 40This led to a dead end as my two methods failed:
 41
 421. [[https://addons.thunderbird.net/en-US/thunderbird/addon/importexporttools-ng/][ImportExportTools NG]] does not work with my version of Thunderbird (135).
 432. Manually dragging the =.eml= files onto a folder in Thunderbird worked for
 44   small batches of files, but seemed to lock up if I tried to import more than
 45   a few hundred at a time. It also seemed a bit buggy, as I ended up with many
 46   duplicate, and sometimes triplicate, messages.
 47
 48At this point, I decided to take a step back and use [[https://github.com/djcb/mu][mu]], a command-line utility
 49that would index my files and sync back and forth with Migadu for me.
 50
 51Using my blog post, [[https://cleberg.net/blog/mu4e.html][Email in Doom Emacs with Mu4e on macOS]], (and skipping the
 52mu4e parts) I was able to set up a minimal directory connected to my Migadu IMAP
 53account. Using my terminal, I simply moved all of my messages into the =mu=
 54directory and synchronized the account, and voila, my messages synchronized
 55successfully to the remote server and my other email clients.
 56
 57However, the remaining issue was that I now had all 5000 messages in the Archive
 58folder and needed to figure out how to organize them back into their proper
 59directories.
 60
 61* Organizing Messages Into Folders
 62
 63As with any problem, I used Python as my hammer to fix the problem. I started by
 64creating the directories required in Thunderbird, fetching them with =mbsync= so
 65that they appeared in my =mu= directory, and using Python to organize my
 66messages into the newly-created sub-folders.
 67
 68** Sent Messages
 69
 70I started by organizing my Sent messages. This required checking each file for
 71the =From= header and moving them to the Sent folder.
 72
 73#+begin_src shell
 74cd ~/.maildir/migadu/Archive/cur
 75nano _sent.py
 76#+end_src
 77
 78#+begin_src python
 79# _sent.py
 80import os
 81import glob
 82import shutil
 83
 84# Loop through all files in the current folder
 85for file in glob.glob("*.eml"):
 86	# Create boolean to check if we should move the file
 87	move = False
 88
 89	# Open the current file
 90		f = open(file, 'r')
 91
 92	# For each line in file, find the From header
 93	for line in f:
 94		if line.startswith("From:"):
 95			# If we find ourself, mark the message for move
 96			if "user@example.com" in line:
 97				move = True
 98
 99	# Close the file
100	f.close()
101
102	# Move the file, if marked for move
103	if move == True:
104		filepath = os.path.join("/Users/YOUR_USERNAME/.maildir/migadu/Archive/cur/", file)
105		new_filepath = os.path.join("/Users/YOUR_USERNAME/.maildir/migadu/Sent/cur/", file)
106		shutil.move(filepath, new_filepath)
107#+end_src
108
109#+begin_src python
110python3 _sent.py
111#+end_src
112
113The only downside to my current approach is that it was the quick and dirty
114option, so I re-ran it while editing the =user@example.com= string for each
115email I wanted to move. If I had wanted to create a more well-defined solution,
116I would have created an array of addresses to check for and have the =if=
117statement check against that array.
118
119Regardless, I was able to run this with the addresses I wanted to move to the
120Sent folder and was soon finished.
121
122** Archive Sub-Folders
123
124Next, I needed to move the remaining ~3000 messages from the Archive folder into
125dated sub-folders, organized as such:
126
127- Archive/2016
128- ...
129- Archive/2025
130
131
132To do this, I followed a similar approach as the method above but check for the
133=Date= header instead of the =From= header.
134
135#+begin_src shell
136cd ~/.maildir/migadu/Archive/cur
137nano _archive.py
138#+end_src
139
140This approach requires finding the =X-Pm-Date= header and splitting it by the
141spaces contained within. Once split into a list, we must select the fourth
142element, as that contains the year which will match the directory we should move
143it to.
144
145For example, the header =X-Pm-Date: Fri, 07 Feb 2025 16:12:08 +0000= will be
146split into a list as such:
147
148#+begin_src python
149[
150    'X-Pm-Date:', # 0
151    'Fri,',       # 1
152    '07',         # 2
153    'Feb',        # 3
154    '2025',       # 4
155    '16:12:08',   # 5
156    '+0000'       # 6
157]
158#+end_src
159
160From this list, we select the fourth element (=2025=) and use that to build the
161destination path.
162
163#+begin_src python
164# _archive.py
165import os
166import glob
167import shutil
168
169# Loop through all files in the sub-folders under Archive
170for file in glob.glob("*.eml"):
171	# Create boolean to check if we should move the file
172	move = False
173
174	# Open the current file
175	f = open(file, 'r')
176
177	# For each line in file, find the X-Pm-Date header
178	for line in f:
179		if line.startswith("X-Pm-Date"):
180		# Split the line into a list by spaces;
181		# Then select the item that contains the year
182			year = line.split(" ")[4]
183			move = True
184
185	# Close the file
186	f.close()
187
188	# Move the file, if marked for move
189	if move == True:
190		filepath = os.path.join("/Users/YOUR_USERNAME/.maildir/migadu/Archive/cur/", file)
191		new_filepath = os.path.join(f"/Users/YOUR_USERNAME/.maildir/migadu/Archive/{year}/cur/", file)
192		shutil.move(filepath, new_filepath)
193#+end_src
194
195#+begin_src python
196python3 _archive.py
197#+end_src
198
199At this point, we've now moved all Sent messages to the Sent box and organized
200all messages under the Archive folder into their correct sub-folders.
201
202If you exported other files, such as files from your Inbox, Trash, etc., you
203could follow a similar approach and determine the best header or attribute to
204identify them for further organization.
205
206** Synchronize the Results
207
208Before synchronizing the files in their new locations, I needed to remove the
209characters at the end of the file name since =mu= appends IDs to the end of file
210names.
211
212#+begin_src shell
213cd ~/.maildir/migadu/Archive
214nano _sync_prep.py
215#+end_src
216
217This script prepares the =Archive= sub-folders for synchronization, but the same
218concept applies to the Sent folder, except you'd replace =*/cur/*= with =*= if
219this script were inside the =Sent/cur= directory.
220
221#+begin_src python
222import glob
223import shutil
224
225# Loop through all files in the sub-folders under Archive
226for file in glob.glob("*/cur/*"):
227	# Remove the characters at the end of the file name created by =mu=
228	new_file = file.split(",U=",1)[0]
229
230	# Move the file to the new file name
231	shutil.move(file, new_file)
232#+end_src
233
234#+begin_src shell
235python3 _sync_prep.py
236#+end_src
237
238Finally, we can synchronize the results.
239
240#+begin_src shell
241mbsync -aV
242#+end_src
243
244* Removing Duplicates
245
246My only remaining issue at the time of writing is identifying and removing
247duplicate messages. I have toyed with simple Python and command-line solutions
248to identify duplicate files, but could not get them to effectively define all
249the duplicates found in any specific directory.
250
251I've even tried using the [[https://github.com/pkolaczk/fclones][fclones]] utility, to no avail. It seems that something
252in the Proton export, my manual Thunderbird method attempt, or possible sync
253issues between Thunderbird -> Migadu <-> mu caused duplicates where content
254within the message has been modified.
255
256Although I now seem to be wasting space and in need of a deduplication tool, I
257have all of my messages migrated to my new service.