For a few hundred TiB, LTO-9 magnetic tapes (18 TB per cartridge) become cheaper than hard disks, despite the huge cost of the tape drive (between $4500 and $5000, but in many places greater prices are shamelessly requested, which should not be accepted; besides a tape drive you need a SAS HBA card and appropriate cables; thus you need to amortize about $5000 through the price difference between HDDs and tapes, to reach the threshold where using tapes becomes cheaper), and they become cheaper and cheaper the greater is your total amount of data, when the cost of the tape drive is amortized over more tapes. One may choose tapes for better reliability even for amounts of data that are insufficient to amortize the initial cost for tape drive + SAS HBA card + SAS cables.

This is especially true when you take into account that regardless whether you use HDDs or tapes, you should better duplicate them and preferably not keep the copies in the same place.

The difference in cost between tapes and HDDs becomes significantly greater when you take into account that data stored on HDDs must be copied on new HDDs after a few years, due to the short lifetime of HDDs. The time after which you may need to move data on new tapes is not determined by the lifetime of tapes (guaranteed to be at least 30 years) but by the obsolescence of the tape drives for a given standard, and it should be after at least 10 to 15 years.

If you keep on a SSD/HDD a database of the content of the tapes, containing the metadata of the stored files and their location on tapes, the access time to archived data is composed of whatever time you need for taking the tape from a cabinet and inserting it into the drive, plus a seeking time of around 1 minute, on average.

Once the archived data is reached, the sequential transfer speed of tapes is greater than that of HDDs.

LTO-9 cartridges have a significantly lower volume and weight than 24-TB HDDs (for storing the same amount of data), which simplifies storage and transport.

You might want to look into using cephadm to setup CEPH. Use Erasure coding as data pool for very efficient data storage and protection (8+2). From that export large RBD to be used as zpool with dedup. Scales to Petabytes and has lots of failure protection options.

Not a pro data guy but someone running something like what you're talking about for many years. These days 200TiB is "normal storage server" territory, not anything exotic. You can just do the most boring thing and it will be fine. I'm just running 1, tho. The hard parts are having it be efficient, quiet and cheap which always feels like an impossible triangle.

Yeah, resilvers will take 24h if your pool is getting full but with RAIDZ2 it's not that scary.

I'm running TrueNAS scale. I used to just use Ubuntu (more flexible!) but over many years I had a some bad upgrades where kernel & zfs stopped being friends. My rack is pretty nearby so for me, a big 4U case with 120mm front fans was high priority, it has a good noise profile if you replace with Noctuas, you get a constant "whoosh" rather than a whine etc.

Running 8+2 with 24tb drives. I used to run with 20 slots full of old ex-cloud SAS drives but it's more heat / noise / power intensive. Also, you lose flexibility if you don't have free slots. So eventually ponied up for 24tb disks. It hurt my wallet but greatly reduced noise and power.

  Case: RM43-320-RS 4U

  CPU: Intel Xeon E3-1231 v3 @ 3.40GHz (4C/8T, 22nm, 80W TDP)
  RAM: 32GB DDR3 ECC
  Motherboard: Supermicro X10SL7-F (microATX, LGA1150 socket)
    - Onboard: Dual Intel I210 1GbE (unused)
    - Onboard: LSI SAS2308 8-port SAS2 controller (6Gbps, IT mode)
    - Onboard: Intel C220 chipset 6-port SATA controller

  Storage Controllers:
    - LSI SAS2308 (onboard) → Intel RES2SV240 backplane (SFF-8087 cables)
    - Intel C220 SATA (onboard) → boot SSD

  Backplane:
    - Intel RES2SV240 24-bay 2U/3U SAS2 Expander
    - 20× 3.5" hot-swap bays (10 populated, 10 empty)
    - Connects via Mini SAS HD SFF-8643 to Mini SAS SFF-8087 Cable, 0.8M x 5

  Boot/Cache:
    - Intel 120GB SSD SSDSC2CW120A3 (boot drive, SATA)
    - Intel Optane 280GB SSDPED1D280GA (ZFS SLOG device, NVMe)

  Network:
    - Intel 82599ES dual-port 10GbE SFP+ NIC (PCIe x8 add-in card)

I personally try to run with no more than 10 slots out of 20 used. This gives a bit of flexibility for expanding, auxiliary pools, etc etc. Often you find you need twice as much storage as you're planning on directly using. For upgrades, snapshots, transfers, ad-hoc stuff etc.

Re: dedup, I would personally look to dedup at the application layer rather than in the filesystem if I possibly could? If you are running custom archiving software then it's something you'd want to handle in the scope of that. Depends on the data obviously, but it's going to be more predictable, and you understand your data the best. I don't have zfs de-dup turned on but for a 200TiB pool with 128k blocks, the zfs DDT will want like 500GiB ram. Which is NOT cheap in 2026.

I also run a 7-node ceph cluster "for funsies". I love the flexibility of it... but I don't think ceph truly makes sense until you have multiple racks or you have hard 24/7 requirements.

a couple hundred TB arranged how? and for what purpose, generally? archival, warm, hot?

for the first two, depending on throughput desired, you can do with spinning rust. you pick your exposure, single platter or not, speed or not, and interface. And no fancy raid hardware needed.

I've had decent luck with 3+1 warm and 4+1 archival. if you don't need quick seeks but want streaming data to be nice, make sure your largest file fits on a single drive, and do two parity disks for archive, a single for warm. md + lvm; ext4 fs, too. my very biased opinion based on tried everything and am out of ideas, and i am tired, and that stuff just works. I am not quick to the point but you need to split your storage up. use 18+ SMR disks, shingled magnetic recording hard drives, for larger stuff that you don't need to transfer very fast. 4k video for consumption on a 4k televsion fits here. Use faster, more reliable disks for data used a lot, &c

Hot or fast seeks & transfers is different, but i didn't get the idea that's what you were after. Hadoop ought be used for hot data, imo. People may argue that zfs of xfs or jfs or ffs is better than ext4, but are they gunna jump in and fix it for free when something goes wrong for whatever reason?

sorry, this is confusing. Unsure how to fix that. i have files on this style system that have been in continuous readable condition since the mid 1990s. There's been some bumps as i tried every [sic] other system and method.

TL;dr to scale my 1/10th size up, i personally would just get a bigger box to put the disks in, and add an additional /volumeN/ mountpoint for each additional array i added. it goes without saying that under that directory i would CIFS/NFS share subdirectories that fit that array's specifications. again, i am just tired of all of this, i'm also all socialed out so, apologies.