32 replies to this topic

[40H3X]

logical

[WanWizard]

Busy with the new NAS cluster in our datcentre. I'm hitting 530MB/s in testing at the moment, not bad...

[40H3X]

amazing power

[WanWizard]

It truely is.

 

2 NetApp shelfs with 24 x 15k 600GB SAS disks. fronted by two SuperMicro Linux headends, running btrfs on drdb on dmadm, with NFS on top of btrfs subvolumes/quota's.

 

Amazing setup, drbd replication between the frontends peaks at about 4800Mbps, while still providing that throughput, and a full cluster failover takes 5 seconds tops.

 

Requesing a forum webpage and then trigger failuver is hardly noticable in the increase of page response.

 

Now hope it is more stable than the previous solution, which used other Netapp shelfs (which had hardware issues) and Zetavault frontends (runing ZFS, requiring shared storage) which worked fine, until it no longer worked fine...

[Erik Slagter]

BTW I'd prefer a filesystem that is used by "anybody" and is able to survive about 99 of 100 crashes (ext3 or ext4) to a filesystem that in theory can survice "any" crash and is only used by a limited group. As ext3 and ext4 are used by so many people, any vicious and significant bugs have been found and resolved by now. That might not be the case for other file systems. I have quite negative experiences in this regard with btrfs a few years ago. It was said to be "mature" by that time but only after the first reboot I had it completely foobar and no fsck to repair it (as they say btrfs doesn't need it, go figure...)

[40H3X]

The logic is sound and albeit that there is safety in numbers I take my change (in this case) with ZFS as the system in casu has ECC and thus will automatically reconstruct data after detecting an error, which in this case is not impossible as this system doesn't have an UPS and power surges can happen...

[WanWizard]

btrfs is a lot more mature now than it was years ago. Including the presence of repair tools. It is currently the filesystem of choice for Synology, so I guess it isn't that bad anymore. Having said that, I don't use btrfs for boot disks...

 

As to ZFS, the problem I had was that it simply isn't stable on Linux. At random moments it started to do funny things rendering the system unusable. ZFS on Solaris or FreeBSD (FreeNAS) on the other hand is in my experience stable as a rock.

[40H3X]

In this case the system in casu runs FreeBSD

Edited by 40H3X, 20 December 2019 - 16:33.

[catastrofus]

Alleen jammer dat de installed base van FreeBSD een 'beetje' achter loopt bij (welke variant van) Linux.dan ook vwb de toekomst van ZFS.

Oracle, als commercieel eigenaar van Sun en dus commercieel eigenaar van ZFS (niet van openzfs), heeft een aantal ZFS appliances in de aanbieding  (wij hebben zelf hier 2 x zs5-2 met elk 1 PB (bruto) in gebruik) maar buiten deze kom je maar bar weinig (professionele) ZFS-meuk tegen en zeker geen zfs op freebsd.

Hoewel ik zelf freebsd een geweldig os vind eigenlijk. Vroeger zelf ook veel gebruikt

[WanWizard]

Niet helemaal waar, er zijn zat commeriele offerings op basis van ZFS, hetzij op opvolgers van OpenSolaris (OpenIndiana/Illumos), hetzij op FreeBSD.

 

Er zijn ook maar zat organisaties die geen PB's nodig hebben, noch Oracle centen hebben.

[Erik Slagter]

Making use of extra ECC data is not the same as being able to repair a corrupted filesystem. The first is a workaround to the problem where a regular read or write operation passes undetected because the disk's own sector error detection is insufficient (which seems to be case in every something like 10^20 operations, so it's fairly academic but not completely impossible). The second is a workaround for the situation where the filesystem has gotten corrupt, either due to bad RAM in the host, a bug in the filesystem code, a failed write without a chance to correct it (e.g. when suddenly powering off) and journal was also corrupt (yes, indeed, also rare but not impossible) or a firmware bug in the disk itself (very rare).

[40H3X]

btrfs is a lot more mature now than it was years ago. Including the presence of repair tools. It is currently the filesystem of choice for Synology, so I guess it isn't that bad anymore. Having said that, I don't use btrfs for boot disks...
 
As to ZFS, the problem I had was that it simply isn't stable on Linux. At random moments it started to do funny things rendering the system unusable. ZFS on Solaris or FreeBSD (FreeNAS) on the other hand is in my experience stable as a rock.

Why don’t you use Btrfs for bootdisks?

[WanWizard]

For the same reason as Erik mentioned, I don't want any chance, no matter how small, that the system doesn't boot when there is a file system issue. As long as it boots, I can access it remotely.