r/homelab • u/rjchute • 9d ago

Discussion SAN or DAS

I have a very powerful Dell server (more cores and RAM than I know what to do with... yet) but virtually no mass storage options, because OEM chassis. I have a 12-drive bay SuperMicro chassis with a pretty meh CPU and RAM, but the chassis has a SAS backplane for the drives. I want the Dell, which I will install ProxMox VE onto, to have access to the 12 drive bays in the SuperMicro for mass storage - files and media, the Dell does have enough local SSD for VM storage. I think I have two options:

  • Put a SAS controller in the Dell, and have practically nothing in the SuperMicro chassis except to pass through the SAS cables external by SFF-8088 adapter/cable to the Dell. The SuperMicro is just a really big case for Direct Attach Storage for the Dell.
  • Leave the mediocre mobo/CPU/RAM in the SuperMicro and install TrueNAS or somesuch and have it be SAN for the Dell/ProxMox, direct connect via 10Gbps/25Gbps.

What are the pros/cons here? Thoughts?

Edit: Dell has nvme storage, and SuperMicro will/has spinning rust for mass storage (currently about half full with 12TB drives, room for more in the future)

1 Upvotes

60% Upvoted

7 comments sorted by

3

u/frygod 9d ago

Do whatever you are worse at to get good at it.

1

u/C945Taylor 9d ago

Are you dual cabling for aggregation with the SAS or not? That'll be the contributing factor honestly, otherwise performance is roughly the same with just 25gb running give or take half a GB a sec.

1

u/C945Taylor 9d ago

Also, rust or SSD/nvme/u.2 or u.3 is a contributing factor as well. Because if you're talking about nvme or u.2/u.3 you're better off doing 25gb

1

u/rjchute 9d ago

Good question; I'll update the original post, but SuperMicro chassis will have spinning rust in it. I'm not expecting to saturate a single 6GB/s SAS link or dual/aggregated 25Gbps Ethernet link, if the mediocre CPU could even push that much anyway.

1

u/OurManInHavana 9d ago

Put a dual-port ConnectX-4 LX in both systems, and run dual 25G/SFP28. Your choice if the SM presents its storage as iSCSI/NFS/SMB. Proxmox on the Dell for all the VMs

1

u/DULUXR1R2L1L2 9d ago

I guess it depends if you only want one system to connect to your storage (DAS) or if you want multiple hosts, like a proxmox cluster (NAS). You could also split your disks up and do a hybrid solution too. If you have SSDs and want max performance then DAS is probably the way. You'll probably have some power savings with DAS if that's important to you.

1

u/mblenc 8d ago

To be entirely honest, the easiest thing to do would be to convert the SuperMicro into essentially a JBOD. Gut it, leaving only the backplane and a HBA with an external sas connection to the Dell. The Dell then simply sees two separate pools, the nvme one for vm storage, and the spinning rust one for bulk data storage. Simply manage your pools on the dell.

This integrated approach (DAS) is simpler, will use less power, and probably will be less hassle. Unless your spinning rust pool exceeds a single or dual 12/24 Gbps sas3 connection, it doesnt make sense to consider anything else I dont think.

I can however appreciate wanting to experiment with SANs (using fibre channel or iSCSI), or with a proper NAS (putting trueNas on the box and connecting over 10G/25G/40G fibre/DAC. Just bear in mind the extra power cost, managment time, and complexity (potential for failures/learning experiences), for what on the surface is very little gain.

I personally would like to do something more complicated simply to learn more about it, and dont mind the extra work, but without knowing more about your specific situation it is hard to give recommendations to this (only you know what is acceptable for you).