Commit `f7e11cecff`

f7e11cecffa5fb13e3b73a471bacec2dd6aac2cc

parent: 2bf00f8311

Verified · cmc

cmc <hello@cleberg.net> · 2026-07-30 22:34 UTC

publish new post: 10gbps

Layout: unified · split

content/blog/2026-07-30-10gbps.org added +249

		@@ -0,0 +1,249 @@
	1	#+date: [2026-07-30 Thu 17:14:00]
	2	#+title: Upgrading My Homelab to 10 Gbps
	3	#+description: Moving my homelab off 1 Gbps and onto a 10 Gbps backbone.
	4	#+slug: 10gbps
	5	#+filetags: :self-hosting:
	6
	7	* Why 10 Gbps?
	8
	9	The short answer: fiber finally became available at my current residence, so I
	10	jumped on it immediately.
	11
	12	The slightly longer answer is that I'd previously enjoyed proper 1 Gbps fiber,
	13	but for a while now I'd been stuck limping along on "1 Gbps" cable from Cox
	14	that, in reality, often crawled along at 100 Mbps regardless of what I did to
	15	fix it. New modem, new cables, different ports, calls to support - nothing
	16	mattered. It ran at whatever speed it felt like that day.
	17
	18	So when [[https://fiber.google.com][Google Fiber]] showed up in my neighborhood, I ordered the 8 Gbps package
	19	immediately. I'll upgrade to their 10 Gbps package the moment it's released, but
	20	I'm not exactly suffering in the meantime.
	21
	22	A quick note on the title before we go further: my /internet/ is 8 Gbps for now,
	23	but this post is really about rebuilding my /internal/ network to 10 Gbps end to
	24	end. The WAN is just the excuse to upgrade everything.
	25
	26	#+caption: My WAN status over the past month.
	27	#+attr_html: :alt A chart of my WAN status and activity over the past month.
	28	https://img.cleberg.net/blog/20260730-10gbps/wan_history.webp
	29
	30	In the screenshot above, you can see that Google Fiber was installed on 7/20 and
	31	UniFi entered WAN failover mode (the blue bar) and then I unplugged Cox on 7/22,
	32	resulting in the green Google Fiber bar.
	33
	34	As you can also see, Cox was a /train wreck/ - cycling between yellow
	35	(disconnections, lag, etc.) and red (no internet connection) for the previous
	36	month.
	37
	38	* What I Was Running Before
	39
	40	Nothing exotic. My network was built entirely on 1 Gbps gear:
	41
	42	- A 24-port UniFi switch in the rack.
	43	- Two 8-port UniFi switches serving other rooms.
	44	- A Cox "1 Gbps" WAN that, as mentioned, frequently delivered closer to 100
	45	Mbps.
	46
	47	It worked. For years, it was more than enough - I've written before about how my
	48	[[https://cleberg.net/blog/homelab-maintenance.html][homelab basically maintains itself]], and the network was no exception. But once I
	49	had real bandwidth coming into the house, 1 Gbps internally became the
	50	bottleneck instead of the ISP.
	51
	52	As a result, I installed the hardware I'll list in the next section. You can see
	53	the 10G connections in blue and the 1G connections in green.
	54
	55	#+caption: My homelab infrastructure.
	56	#+attr_html: :alt A network diagram of my homelab infrastructure.
	57	https://img.cleberg.net/blog/20260730-10gbps/infrastructure.webp
	58
	59	* The Hardware
	60
	61	Everything here is UniFi. I'm already knee-deep in the ecosystem, and at this
	62	point fighting it would cost me more time than it's worth. Staying in one
	63	platform means everything shows up in a single controller and I don't have to
	64	think about it - which is exactly how I like my homelab.
	65
	66	#+caption: My homelab rack.
	67	#+attr_html: :alt A photo of my homelab rack of UniFi gear.
	68	https://img.cleberg.net/blog/20260730-10gbps/rack.webp
	69
	70	This rack shows:
	71	- UDM Pro router.
	72	- 10G 24-port switch.
	73	- Switch panel with keystone jacks.
	74	- 1G 24-port switch.
	75	- My server.
	76
	77	This is before the Google Fiber installation, so you can see that the WAN RJ45
	78	port is in use on the UDM Pro instead of the SFP+ port.
	79
	80	I also removed the old 1G switch after this picture and plan to sell it. I just
	81	didn't want to box it up quite yet when I took this photo.
	82
	83	** Switch
	84
	85	Two switches did the heavy lifting for the upgrade:
	86
	87	- [[https://store.ui.com/us/en/category/switching-professional-max-xg/products/usw-pro-hd-24-poe?variant=usw-pro-hd-24-poe][USW Pro HD 24 PoE]]: the new core switch in the rack, replacing my old 1 Gbps
	88	24-port unit.
	89	- [[https://store.ui.com/us/en/category/switching-utility/products/usw-pro-xg-8-poe?variant=usw-pro-xg-8-poe][USW Pro XG 8 PoE]]: a smaller 10G switch serving another room, replacing one of
	90	the old 8-port units.
	91
	92	** NICs
	93
	94	Neither of my two most important machines shipped with 10G, so both needed a new
	95	network card:
	96
	97	- Windows gaming PC: an Intel X520-DA1, a 10G SFP+ card.
	98	- Server: an Intel X710-DA2, a dual-port 10G SFP+ card.
	99
	100	Both are enterprise Intel cards you can find used for very little money, which
	101	is half the reason I went with them.
	102
	103	The Ubuntu server already had Intel drivers installed, so I wasn't required to
	104	install anything. I simply slotted the NIC into the PCIe port on the motherboard
	105	and it worked.
	106
	107	#+caption: The 10G NIC in my server.
	108	#+attr_html: :alt A photo of the 10G NIC installed in my server.
	109	https://img.cleberg.net/blog/20260730-10gbps/server_nic.webp
	110
	111	However, the Windows machine was a different story. It did come with an
	112	installation CD-ROM, but I of course ignored that since I have no disc drive in
	113	my PC. I tried to find the right drivers online, but Intel's websites are a vast
	114	wasteland of useless downloads and circular links.
	115
	116	Eventually, I pulled a USB-C disc reader from storage and used the CD, which
	117	magically installed on the first try and everything started working from there.
	118
	119	That'll teach me (not really).
	120
	121	#+caption: The 10G NIC in my Windows gaming PC.
	122	#+attr_html: :alt A photo of the 10G NIC installed in my Windows gaming PC.
	123	https://img.cleberg.net/blog/20260730-10gbps/desktop_nic.webp
	124
	125	** Cabling
	126
	127	This is where 10G gets fiddly, and where most of my shopping list ended up.
	128	Going 10G means leaving RJ45 behind for SFP+ almost everywhere:
	129
	130	- [[https://store.ui.com/us/en/category/accessories-modules-fiber/collections/accessories-pro-direct-attach-cables/products/uacc-cm-rj45-mg?variant=uacc-cm-rj45-mg][SFP+ to RJ45 Adapter]]: bridges the Google Fiber RJ45 handoff into the SFP+ port
	131	on my UDM Pro.
	132	- [[https://store.ui.com/us/en/category/accessories-cables-dacs/collections/accessories-pro-patch-cables/products/uacc-cable-patch-el?variant=uacc-cable-patch-el-0dot15m-w][UniFi Etherlighting Patch Cable]]: short in-rack runs between the switch, the
	133	other rack devices, and the fiber modem.
	134	- [[https://store.ui.com/us/en/category/accessories-modules-fiber/collections/accessories-pro-direct-attach-cables/products/10gbps-direct-attach-cable?variant=uacc-dac-sfp10-0dot5m][10G Direct Attach Cable (DAC)]]: the short SFP+ runs from the rack switch to the
	135	server, and from the second switch to the gaming PC.
	136	- [[https://store.ui.com/us/en/category/accessories-modules-fiber/collections/accessories-pro-direct-attach-cables/products/10-gbps-active-optical-cable?variant=uacc-aoc-sfp10-20m][10G Active Optical Cable (AOC)]]: the long run between the rack and the 8-port
	137	switch in the other room, where a copper DAC won't reach.
	138
	139	** Cost
	140
	141	None of this is cheap, so here's the full damage:
	142
	143	\| Item \| Qty \| Price \|
	144	\|---------------------------+-----+-----------\|
	145	\| USW Pro HD 24 PoE \| 1 \| $999.00 \|
	146	\| USW Pro XG 8 PoE \| 1 \| $499.00 \|
	147	\| SFP+ to RJ45 Adapter \| 1 \| $65.00 \|
	148	\| UniFi Etherlighting Patch \| 27 \| $112.97 \|
	149	\| Cat6A Keystone Coupler \| 2 \| $78.00 \|
	150	\| 10G Direct Attach Cable \| 3 \| $43.00 \|
	151	\| 10G Long-Range DAC (AOC) \| 1 \| $55.00 \|
	152	\| Intel X520-DA1 NIC \| 1 \| $45.00 \|
	153	\| Intel X710-DA2 NIC \| 1 \| $145.00 \|
	154	\|---------------------------+-----+-----------\|
	155	\| Total \| \| $2,041.97 \|
	156
	157	* The Upgrade
	158
	159	With the parts in hand, the actual topology is straightforward. Bandwidth flows
	160	in from the street and fans out through the rack:
	161
	162	#+begin_src
	163	Google Fiber
	164	│ (RJ45 → SFP+ adapter)
	165	▼
	166	UDM Pro
	167	│ (Etherlighting patch)
	168	▼
	169	USW Pro HD 24 (core switch)
	170	├─ DAC ──────────► Server
	171	├─ AOC ──────────► USW Pro XG 8 (other room)
	172	│ └─ DAC ──► Windows Gaming PC
	173	└─ ETH ──────────► USW 8-port (1G switch)
	174	#+end_src
	175
	176	The UDM Pro takes the Google Fiber handoff through the RJ45-to-SFP+ adapter,
	177	then hands everything down to the new 24-port switch over an Etherlighting patch
	178	cable. From there, the server hangs directly off the core switch on a short DAC,
	179	and a long active optical cable carries 10G across the house to the second
	180	switch, which finally feeds my gaming PC. My surviving 1 Gbps 8-port switch
	181	still hangs off the core switch too, handling everything that doesn't need 10G.
	182
	183	The physical installation was pretty easy. I needed to replace a drive in my
	184	server's ZFS pool, so I already needed to get inside the rack. Since I don't use
	185	rails, I had to unscrew and pull out each device by hand.
	186
	187	From there, the NIC (and HDD) installation in the server was seamless. Just pop
	188	out one of the metal slots on the back of the server, slot the NIC into the PCIe
	189	lane, and line up the port with the new opening in the back panel.
	190
	191	All in all, it was easy and simple. No physical snags, only one driver issue on
	192	the Windows machine (which was my fault for ignoring the provided CD), cabling
	193	runs were easy, and UniFi updates were minimal.
	194
	195	* Benchmarks
	196
	197	Let's nerd out on the numbers.
	198
	199	** Before
	200
	201	I don't actually have any speed tests from before the upgrade, but the Cox
	202	connection had degraded even further by the end - down to roughly 70 Mbps for
	203	the month prior to the upgrade.
	204
	205	Occasionally, I was able to get ~800 Mbps when the connection would work
	206	properly.
	207
	208	** After
	209
	210	So, here's the catch. I'm not getting the full 8 Gbps currently. When the Google
	211	tech installed the modem, we confirmed ~ 8500 Mbps together, which means the
	212	fiber is working through the ONT and the modem.
	213
	214	However, they would not connect directly to my router, which means the Google
	215	modem is in bridge mode for now.
	216
	217	The speed test below shows an average speed, where I'm getting 5 to 6 Gbps to my
	218	router. While not the full 8 Gbps, I'm not saturating it anyway, so I haven't
	219	dug into my UniFi settings to determine what's causing the difference.
	220
	221	#+caption: Speed test from the UniFi controller.
	222	#+attr_html: :alt A screenshot of a speed test result from the UniFi controller.
	223	https://img.cleberg.net/blog/20260730-10gbps/speed_unifi.webp
	224
	225	A supplementary test showing speeds directly to my server:
	226
	227	#+caption: Speed test from my server.
	228	#+attr_html: :alt A screenshot of a speed test result from my server.
	229	https://img.cleberg.net/blog/20260730-10gbps/speed_server.webp
	230
	231	* Was It Worth It?
	232
	233	Here's the honest part.
	234
	235	Do I /need/ 10 Gbps between my gaming PC and my server? Probably not. The number
	236	of times a day I move enough data to saturate even 1 Gbps is small, and for most
	237	of what I do, this changes nothing I'll consciously notice.
	238
	239	But that was never really the point. I finally have an internet connection that
	240	delivers what I pay for, and a network that won't be the bottleneck for years to
	241	come. After spending so long fighting a "1 Gbps" connection that never once hit
	242	1 Gbps, there's something deeply satisfying about watching a transfer actually
	243	move as fast as the hardware allows.
	244
	245	Instead of constantly monitoring speeds and pausing services so our
	246	work-from-home connections aren't impacted, I can let my server idle at 400 Mbps
	247	24/7, not to mention all the other devices in my home.
	248
	249	Was it worth it? For the peace of mind alone: yes.

Commit f7e11cecff

Commit `f7e11cecff`