Commit `66dea52d21`

66dea52d215b771c1d0004ab4e2f96714942ffc6

parent: b6da76df9f

Unsigned

cmc <hello@cleberg.net> · 2025-02-25 02:29 UTC

publish email-migration.org

Layout: unified · split

content/blog/2025-02-24-email-migration.org added +252

		@@ -0,0 +1,252 @@
	1	#+date: <2025-02-25 Mon 19:20:05>
	2	#+title: A Painful Email Migration
	3	#+description: Some notes on my painful experience migrating emails out of the Proton Mail platform.
	4	#+filetags: :email:
	5	#+slug: email-migration
	6
	7	* The Setup
	8
	9	I recently migrated my emails from Proton Mail to Migadu after a failed attempt
	10	to get myself into the Proton ecosystem and wanted to detail my process, as it
	11	was far more painful than expected.
	12
	13	To give some context: I had nearly 5000 messages stored, accounting for around
	14	2.5 GB of space.
	15
	16	Overall, this process would have taken all day had I done it in one sitting, but
	17	I decided to break it up and it lasted a couple days before I was able to say that
	18	all my messages were stored in my new account.
	19
	20	* Exporting Messages
	21
	22	To start, I needed to export my messages from Proton Mail. As I am using macOS,
	23	I was able to use the [[https://proton.me/support/proton-mail-export-tool][Proton Mail Export Tool]]. However, the downside is that
	24	this dumps every single email on your account into a single folder in the =.eml=
	25	format. They also export a JSON file for each message, in the case you're
	26	importing back into Proton Mail.
	27
	28	This means that anything in my Inbox, Sent, Archive, Trash, and user-created
	29	folders were all dumped out into a single folder with incomprehensible names.
	30
	31	Without a clear path to easily figure out how to re-organize my emails into a
	32	new account, I was left a bit annoyed at Proton's export process.
	33
	34	* Importing Messages
	35
	36	Left with a pile of messages and no way to discern what they were without
	37	opening each one, I decided to try and use Thunderbird to import messages into
	38	my new Migadu IMAP account.
	39
	40	This led to a dead end as my two methods failed:
	41
	42	1. [[https://addons.thunderbird.net/en-US/thunderbird/addon/importexporttools-ng/][ImportExportTools NG]] does not work with my version of Thunderbird (135).
	43	2. Manually dragging the =.eml= files onto a folder in Thunderbird worked for
	44	small batches of files, but seemed to lock up if I tried to import more than
	45	a few hundred at a time. It also seemed a bit buggy, as I ended up with many
	46	duplicate, and sometimes triplicate, messages.
	47
	48	At this point, I decided to take a step back and use [[https://github.com/djcb/mu][mu]], a command-line utility
	49	that would index my files and sync back and forth with Migadu for me.
	50
	51	Using my blog post, [[https://cleberg.net/blog/mu4e.html][Email in Doom Emacs with Mu4e on macOS]], (and skipping the
	52	mu4e parts) I was able to set up a minimal directory connected to my Migadu IMAP
	53	account. Using my terminal, I simply moved all of my messages into the =mu=
	54	directory and synchronized the account, and voila, my messages synchronized
	55	successfully to the remote server and my other email clients.
	56
	57	However, the remaining issue was that I now had all 5000 messages in the Archive
	58	folder and needed to figure out how to organize them back into their proper
	59	directories.
	60
	61	* Organizing Messages Into Folders
	62
	63	As with any problem, I used Python as my hammer to fix the problem. I started by
	64	creating the directories required in Thunderbird, fetching them with =mbsync= so
	65	that they appeared in my =mu= directory, and using Python to organize my
	66	messages into the newly-created sub-folders.
	67
	68	** Sent Messages
	69
	70	I started by organizing my Sent messages. This required checking each file for
	71	the =From= header and moving them to the Sent folder.
	72
	73	#+begin_src shell
	74	cd ~/.maildir/migadu/Archive/cur
	75	nano _sent.py
	76	#+end_src
	77
	78	#+begin_src python
	79	# _sent.py
	80	import os
	81	import glob
	82	import shutil
	83
	84	# Loop through all files in the current folder
	85	for file in glob.glob("*.eml"):
	86	# Create boolean to check if we should move the file
	87	move = False
	88
	89	# Open the current file
	90	f = open(file, 'r')
	91
	92	# For each line in file, find the From header
	93	for line in f:
	94	if line.startswith("From:"):
	95	# If we find ourself, mark the message for move
	96	if "user@example.com" in line:
	97	move = True
	98
	99	# Close the file
	100	f.close()
	101
	102	# Move the file, if marked for move
	103	if move == True:
	104	filepath = os.path.join("/Users/YOUR_USERNAME/.maildir/migadu/Archive/cur/", file)
	105	new_filepath = os.path.join("/Users/YOUR_USERNAME/.maildir/migadu/Sent/cur/", file)
	106	shutil.move(filepath, new_filepath)
	107	#+end_src
	108
	109	#+begin_src python
	110	python3 _sent.py
	111	#+end_src
	112
	113	The only downside to my current approach is that it was the quick and dirty
	114	option, so I re-ran it while editing the =user@example.com= string for each
	115	email I wanted to move. If I had wanted to create a more well-defined solution,
	116	I would have created an array of addresses to check for and have the =if=
	117	statement check against that array.
	118
	119	Regardless, I was able to run this with the addresses I wanted to move to the
	120	Sent folder and was soon finished.
	121
	122	** Archive Sub-Folders
	123
	124	Next, I needed to move the remaining ~3000 messages from the Archive folder into
	125	dated sub-folders, organized as such:
	126
	127	- Archive/2016
	128	- ...
	129	- Archive/2025
	130
	131
	132	To do this, I followed a similar approach as the method above but check for the
	133	=Date= header instead of the =From= header.
	134
	135	#+begin_src shell
	136	cd ~/.maildir/migadu/Archive/cur
	137	nano _archive.py
	138	#+end_src
	139
	140	This approach requires finding the =X-Pm-Date= header and splitting it by the
	141	spaces contained within. Once split into a list, we must select the fourth
	142	element, as that contains the year which will match the directory we should move
	143	it to.
	144
	145	For example, the header =X-Pm-Date: Fri, 07 Feb 2025 16:12:08 +0000= will be
	146	split into a list as such:
	147
	148	0. X-Pm-Date:
	149	1. Fri,
	150	2. 07
	151	3. Feb
	152	4. 2025
	153	5. 16:12:08
	154	6. +0000
	155
	156	From this list, we select the fourth element (=2025=) and use that to build the
	157	destination path.
	158
	159	#+begin_src python
	160	# _archive.py
	161	import os
	162	import glob
	163	import shutil
	164
	165	# Loop through all files in the sub-folders under Archive
	166	for file in glob.glob("*.eml"):
	167	# Create boolean to check if we should move the file
	168	move = False
	169
	170	# Open the current file
	171	f = open(file, 'r')
	172
	173	# For each line in file, find the X-Pm-Date header
	174	for line in f:
	175	if line.startswith("X-Pm-Date"):
	176	# Split the line into a list by spaces;
	177	# Then select the item that contains the year
	178	year = line.split(" ")[4]
	179	move = True
	180
	181	# Close the file
	182	f.close()
	183
	184	# Move the file, if marked for move
	185	if move == True:
	186	filepath = os.path.join("/Users/YOUR_USERNAME/.maildir/migadu/Archive/cur/", file)
	187	new_filepath = os.path.join(f"/Users/YOUR_USERNAME/.maildir/migadu/Archive/{year}/cur/", file)
	188	shutil.move(filepath, new_filepath)
	189	#+end_src
	190
	191	#+begin_src python
	192	python3 _archive.py
	193	#+end_src
	194
	195	At this point, we've now moved all Sent messages to the Sent box and organized
	196	all messages under the Archive folder into their correct sub-folders.
	197
	198	If you exported other files, such as files from your Inbox, Trash, etc., you
	199	could follow a similar approach and determine the best header or attribute to
	200	identify them for further organization.
	201
	202	** Synchronize the Results
	203
	204	Before synchronizing the files in their new locations, I needed to remove the
	205	characters at the end of the file name since =mu= appends IDs to the end of file
	206	names.
	207
	208	#+begin_src shell
	209	cd ~/.maildir/migadu/Archive
	210	nano _sync_prep.py
	211	#+end_src
	212
	213	This script prepares the =Archive= sub-folders for synchronization, but the same
	214	concept applies to the Sent folder, except you'd replace =/cur/= with =*= if
	215	this script were inside the =Sent/cur= directory.
	216
	217	#+begin_src python
	218	import glob
	219	import shutil
	220
	221	# Loop through all files in the sub-folders under Archive
	222	for file in glob.glob("/cur/"):
	223	# Remove the characters at the end of the file name created by =mu=
	224	new_file = file.split(",U=",1)[0]
	225	# Move the file to the new file name
	226	shutil.move(file, new_file)
	227	#+end_src
	228
	229	#+begin_src shell
	230	python3 _sync_prep.py
	231	#+end_src
	232
	233	Finally, we can synchronize the results.
	234
	235	#+begin_src shell
	236	mbsync -aV
	237	#+end_src
	238
	239	* Removing Duplicates
	240
	241	My only remaining issue at the time of writing is identifying and removing
	242	duplicate messages. I have toyed with simple Python and command-line solutions
	243	to identify duplicate files, but could not get them to effectively define all
	244	the duplicates found in any specific directory.
	245
	246	I've even tried using the [[https://github.com/pkolaczk/fclones][fclones]] utility, to no avail. It seems that something
	247	in the Proton export, my manual Thunderbird method attempt, or possible sync
	248	issues between Thunderbird -> Migadu <-> mu caused duplicates where content
	249	within the message has been modified.
	250
	251	Although I now seem to be wasting space and in need of a deduplication tool, I
	252	have all of my messages migrated to my new service.

theme/templates/index.html +5 −7

		@@ -8,6 +8,11 @@ uid [ultimate] Christian Cleberg <hello@cleberg.net></pre>
8	8	<section>
9	9	<h2>Recent Blog Posts</h2>
10	10
	11	<div class="post">
	12	<time datetime="2025-02-24 19:20:05">2025-02-24</time>
	13	<a href="/blog/email-migration.html" >A Painful Email Migration</a>
	14	</div>
	15
11	16	<div class="post">
12	17	<time datetime="2025-02-11 11:40:00">2025-02-11</time>
13	18	<a href="/blog/obscura-vpn.html">A Review of Obscura VPN: A Two-Party
		@@ -21,13 +26,6 @@ uid [ultimate] Christian Cleberg <hello@cleberg.net></pre>
21	26	>
22	27	</div>
23	28
24		<div class="post">
25		<time datetime="2024-12-29 17:45:00">2024-12-29</time>
26		<a href="/blog/self-hosting-the-lounge.html"
27		>Self-Hosting The Lounge: An IRC Web Client</a
28		>
29		</div>
30
31	29	<br />
32	30	<a href="/blog/">All Posts →</a>
33	31	</section>

Commit 66dea52d21

Commit `66dea52d21`