cmc/cleberg.net

  1#+date:        [2024-02-06 Tue 00:00:00]
  2#+title:       ZFS Storage Pools on Ubuntu
  3#+description: Setting up and managing ZFS storage pools on Ubuntu.
  4#+slug:        zfs
  5#+filetags:    :linux:
  6
  7This post details the process I used to create ZFS pools, datasets, and
  8snapshots on Ubuntu Server.
  9
 10I found the following pages very helpful while going through this process:
 11
 12- [[https://ubuntu.com/tutorials/setup-zfs-storage-pool][Setup a ZFS storage pool]]
 13- [[https://wiki.ubuntu.com/Kernel/Reference/ZFS][Kernel/Reference/ZFS]]
 14- [[https://blog.victormendonca.com/2020/11/03/zfs-for-dummies/][ZFS for Dummies]]
 15
 16* Installation
 17
 18To start, I installed the ZFS package with the following command:
 19
 20#+begin_src sh
 21sudo apt install zfsutils-linux
 22#+end_src
 23
 24Once installed, you can check the version to see if it installed correctly.
 25
 26#+begin_src sh
 27> zsf --version
 28
 29zfs-2.1.5-1ubuntu6~22.04.2
 30zfs-kmod-2.1.5-1ubuntu6~22.04.1
 31#+end_src
 32
 33* ZFS Configuration
 34
 35Now that ZFS is installed, we can create and configure the pool.
 36
 37You have various options for configuring ZFS pools that all come different pros
 38and cons. I suggest visiting the links at the top of this post or searching
 39online for the best configuration for your use-case.
 40
 41- Striped VDEVs (Raid0)
 42- Mirrored VDEVs (Raid1)
 43- Striped Mirrored VDEVs (Raid10)
 44- RAIDz (Raid5)
 45- RAIDz2 (Raidd6)
 46- RAIDz3
 47- Nested RAIDz (Raid50, Raid60)
 48
 49I will be using Raid10 in this guide. However, the majority of the steps are the
 50same regardless of your chosen pool configuration.
 51
 52** Creating the Pool
 53
 54To start, let's list the disks available to use. You can use =fdisk= command to
 55see all available disks.
 56
 57#+begin_src sh
 58sudo fdisk -l
 59#+end_src
 60
 61Or, if you currently have them mounted, you can use the =df= command to view
 62your disks.
 63
 64#+begin_src sh
 65> sudo df -h
 66
 67Filesystem      Size  Used Avail Use% Mounted on
 68...
 69/dev/sda1       7.3T   28K  6.9T   1% /mnt/red-01
 70/dev/sdb1       7.3T  144G  6.8T   3% /mnt/red-02
 71/dev/sdc1       7.3T  5.5T  1.9T  75% /mnt/white-02
 72/dev/sdd1       9.1T  8.7T  435G  96% /mnt/white-01
 73/dev/sde1       7.3T   28K  6.9T   1% /mnt/red-03
 74/dev/sdf1       7.3T   28K  6.9T   1% /mnt/red-04
 75#+end_src
 76
 77If you're going to use mounted disks, make sure to umount them before creating
 78the pool.
 79
 80#+begin_src sh
 81sudo umount /dev/sda1
 82sudo umount /dev/sdb1
 83#+end_src
 84
 85Now that I've identified the disks I want to use and have them unmounted, let's
 86create the pool. For this example, I will call it =tank=.
 87
 88#+begin_src sh
 89sudo zpool create -f -m /mnt/pool tank mirror /dev/sda /dev/sdb
 90#+end_src
 91
 92See below for the results of the new ZFS pool named =tank=, with a vdev
 93automatically named =mirror-0=.
 94
 95#+begin_src sh
 96> zfs list
 97
 98NAME   USED  AVAIL     REFER  MOUNTPOINT
 99tank   396K  7.14T       96K  /tank
100#+end_src
101
102#+begin_src sh
103> zpool status
104
105  pool: tank
106 state: ONLINE
107config:
108
109    NAME        STATE     READ WRITE CKSUM
110    tank        ONLINE       0     0     0
111      mirror-0  ONLINE       0     0     0
112        sda     ONLINE       0     0     0
113        sdb     ONLINE       0     0     0
114
115errors: No known data errors
116#+end_src
117
118We can also look at the mounted filesystem to see where the pool is mounted and
119some quick stats.
120
121#+begin_src sh
122> df -h
123
124Filesystem      Size  Used Avail Use% Mounted on
125...
126tank            7.2T  128K  7.2T   1% /tank
127#+end_src
128
129** Expanding the Pool
130
131If you want to expand this pool, you will need to add a new VDEV to the pool.
132Since I am using 2 disks per VDEV, I will need to add a new 2-disk VDEV to the
133existing =tank= pool.
134
135#+begin_src sh
136sudo zpool add tank mirror /dev/sdX /dev/sdY
137#+end_src
138
139If you're adding disks of different sizes, you'll need to use the =-f= flag.
140Keep in mind that the max size will be limited to the smallest disk added.
141
142#+begin_src sh
143sudo zpool add -f tank mirror /dev/sdX /dev/sdY
144#+end_src
145
146I added two 8TB hard drives and this process took around 10 seconds to complete.
147
148When viewing the pool again, you can see that the pool has now doubled in size.
149We have 14.3 TB usable space and the same space used for mirroring.
150
151#+begin_src sh
152> zfs list
153
154NAME         USED  AVAIL     REFER  MOUNTPOINT
155tank         145G  14.3T      104K  /tank
156tank/cloud   145G  14.3T      145G  /tank/cloud
157tank/media    96K  14.3T       96K  /tank/media
158#+end_src
159
160*** Converting Disks
161
162Some disks, such as NTFS-formatted drives, will need to be partitioned and
163formatted prior to being added to the pool.
164
165Start by identifying the disks you want to format and add to the pool.
166
167#+begin_src sh
168sudo fdisk -l | grep /dev
169#+end_src
170
171I am going to format my =/dev/sdc= and =/dev/sdd= disks with the =fdisk=
172command.
173
174See below for instructions on how to use =fdisk=. Here's what I did to create
175basic Linux formatted disks:
176
177- =g= : Create GPT partition table
178- =n= : Create a new partition, hit Enter for all default options
179- =t= : Change partition type to =20= for =Linux filesystem=
180- =w= : Write the changes to disk and exit
181
182I repeated this process for both disks.
183
184#+begin_src sh
185> sudo fdisk /dev/sdc
186
187Welcome to fdisk (util-linux 2.37.2). Changes will remain in memory only, until
188you decide to write them. Be careful before using the write command.
189
190This disk is currently in use - repartitioning is probably a bad idea. It's
191recommended to umount all file systems, and swapoff all swap partitions on this
192disk.
193
194
195Command (m for help): m
196
197Help:
198
199  GPT
200   M   enter protective/hybrid MBR
201
202  Generic
203   d   delete a partition
204   F   list free unpartitioned space
205   l   list known partition types
206   n   add a new partition
207   p   print the partition table
208   t   change a partition type
209   v   verify the partition table
210   i   print information about a partition
211
212  Misc
213   m   print this menu
214   x   extra functionality (experts only)
215
216  Script
217   I   load disk layout from sfdisk script file
218   O   dump disk layout to sfdisk script file
219
220  Save & Exit
221   w   write table to disk and exit
222   q   quit without saving changes
223
224  Create a new label
225   g   create a new empty GPT partition table
226   G   create a new empty SGI (IRIX) partition table
227   o   create a new empty DOS partition table
228   s   create a new empty Sun partition table
229#+end_src
230
231Once the drives are formatted, we can add these disks to the pool.
232
233#+begin_src sh
234sudo zpool add tank mirror /dev/sdc /dev/sdd
235#+end_src
236
237When we list the pool again, we can see that our size is now updated to
238approximately 22TB. This represents my hard drives totalling 45.6TB when shown
239with =fdisk -l=, with a Raid10 configuration using 22TB for mirroring and 22TB
240of usable space.
241
242#+begin_src sh
243> zfs list
244
245NAME         USED  AVAIL     REFER  MOUNTPOINT
246tank         145G  21.7T      104K  /tank
247tank/cloud   145G  21.7T      145G  /tank/cloud
248tank/media   145GT 21.7T       96K  /tank/media
249#+end_src
250
251** Creating Datasets
252
253According to [[https://docs.oracle.com/cd/E18752_01/html/819-5461/ftyue.html][ZFS Terminology]], a =dataset= can refer to “clones, file systems,
254snapshots, and volumes.
255
256For this guide, I will use the =dataset= term to refer to file systems created
257under a pool.
258
259Within my =tank= pool, I am going to create some datasets to help organize my
260files. This will give me location to store data rather than simply dumping
261everything at the =/tank/= location.
262
263#+begin_src sh
264sudo zfs create tank/cloud
265sudo zfs create tank/media
266#+end_src
267
268Once created, you can see these datasets in the output of your pool
269list:
270
271#+begin_src sh
272> zfs list
273NAME         USED  AVAIL     REFER  MOUNTPOINT
274tank         752K  7.14T      104K  /tank
275tank/cloud    96K  7.14T       96K  /tank/cloud
276tank/media    96K  7.14T       96K  /tank/media
277#+end_src
278
279** Creating Snapshots
280
281Next, let's create our first snapshot. We can do this by calling the =snapshot=
282command and give it an output name. I will be throwing the current date and time
283into my example.
284
285#+begin_src sh
286sudo zfs snapshot tank@$(date '+%Y-%m-%d_%H-%M')
287#+end_src
288
289We can list the snapshots in our pool with the following command:
290
291#+begin_src sh
292> zfs list -t snapshot
293NAME                    USED  AVAIL     REFER  MOUNTPOINT
294tank@2024-02-06_19-41     0B      -      104K  -
295#+end_src
296
297** Destroy Snapshots
298
299You can always destroy snapshots that are no longer needed:
300
301#+begin_src sh
302sudo zfs destroy tank@2024-02-06_19-41
303#+end_src
304
305Once deleted, they will no longer appear in the list:
306
307#+begin_src sh
308> zfs list -t snapshot
309no datasets available
310#+end_src
311
312* My Thoughts on ZFS So Far
313
314- I sacrificed 25TB to be able to mirror my data, but I feel more comfortable
315  with the potential to save my data by quickly replacing a disk if I need to.
316- The set-up was surprisingly easy and fast.
317- Disk I/O is fast as well. I was worried that the data transfer speeds would be
318  slower due to the RAID configuration.
319- Media streaming and transcoding has seen no noticeable drop in performance.
320- My only limitation really is the number of HDD bays in my server HDD cage.