Windows ReFS

Christian,

There are some references to ReFS being very memory intensive and using block-writes for metadata and parity - see https://norcliffe.wordpress.com/2019/08/15/extremely-slow-windows-server-with-refs/. It looks like Microsoft have made some effort to address some of the issues in patches and by adding new registry tunables - see https://learn.microsoft.com/en-gb/troubleshoot/windows-server/backup-and-storage/fix-heavy-memory-usage-refs .

While not definitive, it suggests that a system using ReFS will need memory in proportion to the available disk churn and/or file volatility. With random-access files being generally heavily populated but updates sparsely distributed this suggests the memory load could be disproportionate compared to the volume of volatile data due to block caching in ReFS metadata and its extensive use of memory.

It should be noted as well that ReFS uses checkpointing and depending upon details of the the implementation and ReFS buffer volatility, pauses in updates and/or disk access may occur at checkpoint intervals for the duration of a checkpoint. While for BATCH and STREAMED updates this may give better overall performance, for random access/interactive updates under interactive load occasional pauses are likely to be less acceptable.

Additionally, while an independent flat multi-threaded I/O comparison of NTFS against ReFS gave a performance penalty of about 7% for ReFS. this figure should be treated with caution and may be a best case (registry tuning excepted). 

Microsoft guidelines recommend that under high workload in random-access environments for example a triple-mirror be utilised -see https://learn.microsoft.com/en-us/windows-server/storage/refs/refs-overview

 Circumstantially this advice would seem to try and make up for notably slower random access performance under ReFS and the additional hardware is an attempt to close the gap to some degree. 

Bottom line:

There are enough questions around ReFS mamory demands and metadata/buffer management that a full benchmark would seem warranted under interactive and batch workloads at peak-busy times with the ability to tune the Microsoft newly-added tunables to coax best performance. Examination by others of ReFS performance seems to fall in line with this as a general recommendation.

For reference, the registry entries are :

HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\FileSystem
Value Name: RefsEnableLargeWorkingSetTrim
Set RefsEnableLargeWorkingSetTrim = 1
Value Type: REG_DWORD

HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\FileSystem
Value Name: RefsNumberOfChunksToTrim
Value Type: REG_DWORD
DEFAULT (if not set or 0): 4

HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\FileSystem
Value Name: RefsEnableInlineTrim
Value Type: REG_DWORD
Set RefsEnableInlineTrim = 1

Benchmarking by others notes that ReFS with File Integrity ON changes the way that I/O requests are ordered - i.e. ReFS with FileIntegrity on changes the sequence of commands, changes requests and  I/O type, so it removes many random writes. ReFS in this case working similarly toa  Log Structured File System. As a result, there's a strong leap of IOPS, then IOPS fall, and the garbage collector is launched. Circumstantially the amount of residual free disk space  has been reported to be a singuler  issue in some tests. 

Not definitive in any way I know - it raises more questions though they are the rigfht type of question I think. While it is possible to adopt a 'suck it and see' approach, caution should be exercised as performancer could vary markedly between peak-busy interactive and batch workloads. Third-party test results to date indicate significantly more memory would be indicated for ReFS disk management and Microsoft (untested) guidelines recommend mirroring - possibly  triple-mrroring. 

Regards

JJ

------------------------------
John Jenkins
Thame, Oxfordshire
------------------------------