I used to come down on the side of ECC being optional, but had a RAM stick fail in my parents server that would flip a certain bit so consistently that it not only caused scrubs to report errors, it actually caused ZFS to recalculate the parity data-- incorrectly-- and write it back to disk.
This caused permanent corruption of dozens of files before I realized what was happening, even with a RAIDZ2. Luckily I had backups of most of it, and only ended up losing 1 file that had not been backed up yet.
Just finished building them a new server last week, where I gladly paid the premium for ECC RAM. Not going to repeat that disaster if I can help it.