swarm32: (Default)
After a bit of a hiatus due to my 5406zl chassis being full and acquiring a pallet of 5412zl and 5406zl switches over the holidays, I'm picking up the Cluster in a Switch Project again.

My end goal is to have at least a 3-node cluster to host the following
- 2x SAMBA 4 domain Controllers
- 1x Firewall VM (Ipfire )
- 1x Storage VM (TrueNAS or Open Media Vault)
- 1x Pi-hole VM

While I figure out the particulars ( ZFS or Ceph?, Some other abomination), I've been slowly picking up additional ZL compute modules to round out the cluster. While the Advanced Services V2 are natively more powerful with their i7 Quad-cores and 8GB of RAM, I haven't had much luck with the ones I've picked up actually working reliably. The Advanced Services V1 modules use a Core 2 Mobile CPU and cap out at 4GB of RAM from the factory, but upon further inspection they may have one big advantage: A Standard Com Express Type 2 module!

That's right, while the V2 Services modules use a pretty proprietary design, some of the V1 modules appear to be equipped with a Kontron ETXExpress-MC T7500 module adhering to the COM Express™ (by PICMG) “Basic” Form Factor Pin-out Type 2 standard. This means in theory, the module could be upgraded significantly, possibly all the way to a Xeon with DDR4 if I can acquite the right module.

While I'm intrigued by the prospect of a major upgrade to the Gen1 modules, I will first need to conquer them as they are. Firstly, I have backed up the factory loads of all the installed CF cards and HDDs. Second, I'll need to figure out how to get Proxmox or another OS loaded onto each module. Then, I'll need to load and configure the software stack on top of the 3 or 4 modules. It will be an interesting ride, at the end of which I'll hopefully retire a few older or more power hungry pieces of hardware out of my 24/7 home infrastructure.
swarm32: (Cauldron Born)
I recently replaced my venerable HP 2848 gigabit ethernet switch that operated as my primary LAN switch with an HP 5406zl chassis switch. I picked the 5400 series due to the availability of 10GigE and POE, plus the availability of services modules.

I ended up picking up a pair of HP J9857A Advanced services V2 modules. These modules feature a quad-core i7-3612QE, 16GB of DDR3 ECC, 2x SD card slots (USB 2.0), a 1TB SATA HDD, 2x 10GigE (internal), 2x 1GigE (1 internal, 1 external). My hope is to get Proxmox running on both of them in a cluster.

So far, I've found out a few interesting things:
- The HP Service OS on the included 4GB SD card is actually a Debian Linux live environment. The card is two partitions with a Primary and Secondary boot.
- You can boot a standard modern 64-bit Debian from an SD card by running the " services # boot service " command
- /dev/ttyS0 at 9600 connects to the serial port for the " services # serial " command

It looks like it might be a rather successful experiment eventually.
swarm32: (thalban)
Last night I decided to replace my venerable APC Smart UPS 1400 and a slightly newer Smart UPS 1500 that provide the main part of the lab with a much newer pair of Smart UPS SMX1500RM2U Units. That part of the evening maintenance went fairly well and I do like now being able to check how much wattage I'm pulling in my lab without having to guesstimate with a Kill-A-Watt.
I then decided to update the OS on both of my ZFS Storage nodes. That turned into an interesting experience that I am still in the process of sorting out the rest of the way.
First, the one that is primarily for virtual machines was encountering the "no hostid" bug still from when I did the upgrade from zfs 0.6.2 to 0.6.3 . Then, as there was a kernel update and a zfs update, neither one actually built the kernel modules for zfs for the correct kernel. I ended up employing the forced reinstall mentioned by deajan to rebuild the modules for the new kernel.
Still working on a networking issue on the file storage node, but with the new kernel modules things seem to be working okay now.
swarm32: (thalban)
So after building things out and getting a couple of power bills, decided I should refactor the lab a little bit. For starters, I sat down and actually put the single Xeon L5310 motherboard in the secondary storage node.
Next, I swapped out the virtualization node for one of the other Rackable Systems 2U machines that I had with a pair of Xeon L5420s in it. So now the Virtual Machine Host looks like this:
Rackable Systems 2U server
- 4 hot-swap bay 2U Short Depth Chassis
- 450W Rackables Power Supply
- Intel S5000PSL Motherboard
- 2x Intel Xeon L5420 2.5GHz Quad-Core
- 8x 4GB DDR2-ECC FB-DIMMS (32GB total)
- Intel Dual-port PCI-E NIC
- 1x 160GB Hitachi Desktar
- 1x 4GB USB Stick
With a Backup VM Host containing
Rackable Systems 2U server
- 4 hot-swap bay 2U Short Depth Chassis
- 450W Rackables Power Supply
- Intel S5000PSL Motherboard
- 2x Intel Xeon L5410 2.3GHz Quad-Core
- 6x 2GB DDR2-ECC FB-DIMMS (12GB total)
- 1x 160GB Hitachi Desktar
- 1x 4GB USB Stick
The backup node is currently off, as I'm not using enough RAM currently to justify it. I am hoping to later this year pick up one of the newer Rackable 2U servers that Supports PCIe passthrough in VMWare to be the primary node.
I have also replaced the HP Procurve 2824 with a 2848 as when I added in the extra server and all of the Ethernet connections from the BBUs I was a bit short.

The combined system changes reduced the power usage by 20-40%. I think I can reduce the power usage a bit more if I move my backups from a dedicated machine to off of my media center. I would connect then directly to the storage servers, but it is faster for me to connect the drives to another machine with a USB 3.0 controller and sync them over the 2x 1GigE links than to wait for the job to finish over USB 2.0

On top of that, I was able to mostly fix the ridiculous delays I was experiencing with Samba access from my Windows 7 machines by tweaking the Samba config on the primary storage node. The fix was the comment from the user unicolet on this github thread about socket_options.
swarm32: (thalban)
Since last summer I have been steadily working on rebuilding my lab while attempting some semblance of power efficiency.
While I would have loved to have built the storage server NAS Brian Moses did it's definitely out of my budget.

So what am I cobbling together instead?
Primary Storage Node - Media files, etc
Rackable Systems 3U server
- 12 how-swap bay 3U Short Depth Chassis
- 450W Rackables Power Supply
- Intel S5000PSL Motherboard
- 1x Intel Xeon L5310 1.6GHz Quad-Core
- 8x 1G DDR2-ECC FB-DIMMS
- LSI 8-port SAS 6 PCIE Controller
- 4x 2TB Hitachi 7200RPM SATA Drives
- 1x 4GB SATA Flash disk
- 1x 40GB 2.5" 5400RPM Sata Hard drive
Secondary Storage Node - Virtual Machine Storage
Rackable Systems 2U Server
- 4 hot-swap bay 2U Short Depth Chassis
- 450W Rackables Power Supply
- Intel S5000PSL Motherboard
- 1x Intel Xeon L5310 1.6GHz Quad-Core
- 8x 1G DDR2-ECC FB-DIMMS
- Intel Dual-port PCI-E NIC
- 4x 1TB Western Digital RE2 drives
- 1x 4GB SATA Flash disk
- 1x 40GB 2.5" 5400RPM Sata Hard drive
Virtual Machine Host
HP Proliant DL585 G2 (pdf)
- 4x Dual-Core AMD Operon 8220s
- 64GB DDR2 ECC RAM
- 2x Intel Dual-port gigabit PCI-X NICs
- Random 2GB Flash drive
- Redundant power Supplies
Networking
HP Procurve 2824

Software
Both storage nodes are currently running CentOS 7 combined with ZFS on linux. The Virtual host is running VMWare ESXi 5.5

So far other than swearing at firewalld, finding a bug and some update teething, things have not gone too bad so far. This is still a bit of a work in progress (rebuilding the VM storage node for more space, etc), and I figured for once I might post online how I am building things out, to either help others or serve as a warning to the rest of the internet

Profile

swarm32: (Default)
swarm32

August 2025

S M T W T F S
     12
3456789
10111213141516
1718192021 2223
24252627282930
31      

Syndicate

RSS Atom

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Oct. 5th, 2025 09:21 am
Powered by Dreamwidth Studios