I had a budget of around $3000 to build a new setup. I've just emigrated from the UK to the USA and as such needed to do everything. I ended up spending nearer $4k in the end, most of it on Black Friday on drives, but I won't need to spend any more on this server for at least 3-5 years and considering what it's capable of that's worth it to me.
Here are some pictures of the build.
I ended up finding serverbuilds.net and went all out on the components as this box is going to be pulling double duty as my main media server and a homelab. I wrote the Perfect Media Server guide last year and provide Plex to several family and friends. I also work for Red Hat and specialise in Openshift so building clusters is something I do for fun!
That means I need a lot of threads and a lot of memory and a lot of storage. I found the following spreadsheet extremely helpful when comparing CPUs.
Thanks to jdm_waaat and the serverbuilds.net site I was floored when I read about the Gigabyte motherboard he found. Dual LGA2011, supporting boatloads of RAM, built-in SAS, 10GBe?! For $180. Astonishing.
My final hardware ended up being:
- CPU: x2 E5-2690v2
- Memory: 128gb DDR3 ECC
- Motherboard: Gigabyte GA-7PESH2
- PSU: EVGA 850w
- Case: Rosewill RSV-L4500
- Fans: Replaced all stock fans with Noctua's
- Drives: Total raw space 120tb
Here are some pictures of the build.
Gotchas:
- When my board came out of the box IPMI was disabled
- The CMOS battery was flat, once I replaced it the 5 beeps on boot went away
- At first I thought it was a CPU error but 5 beeps on a Gigabyte board means CMOS error
- Just incase, buy a v1 Xeon CPU incase the motherboard needs a BIOS flash (I got one for $8 off ebay)
I'd intended to buy several hard drives on Black Friday (Best Buy easystores). Lo and behold Best Buy went and started stocking a 10tb variant. I snagged 8 on the day (luckily I live near several Best Buys) on top of the 5 other drives I'd already purchased. This gives me a total of 120tb raw which with dual parity gives me just a hair under 100TB of usable space.
The power cables for so many drives needed a bit of love. I bought these cables from Amazon and modified the plugs which are just push to fit so that they lined up properly and took care of the 3.3v rail (i.e. I didn't include it) at the same time which the WD easystores require.
To top it all off I finally made it IKEA this week and bought a IKEA lack coffee table to build the "IKEA lack rack enterprise edition". As you can see, it takes the rosewill case pretty much perfectly. Now, loaded with hard drives the Rosewill case weighs quite a bit so I think I might add a small support in the middle of the shelf but otherwise, it's a great solution for $30.
I'm currently evaluating hypervisors. I've run proxmox for a little while and have just switched to ESXI with vSphere and am trying to decide whether it's worth the VMUG annual price of $200 or not. I guess I could go with a fresh install every 2 months but that's a PITA. vSphere has great terraform support so I guess that'll probably win sigh.