Memory Strategy and Predictable ZFS Performance

Based on a LinkedIn post originally published on 11 February 2026

On HG000009, the front-end data store node in my home lab, memory is treated as a first-class performance component.

It is not regarded as spare capacity waiting for an application to claim it.

Memory is an active part of the storage architecture. It affects read performance, metadata access, latency, recovery behaviour and the system’s ability to remain predictable under changing workloads.

In this design, ZFS acts as the central storage control plane. Back-end devices are aggregated through ZFS, which then provides controlled storage services to consumers through iSCSI targets and, where appropriate, SMB shares.

ZFS is used for more than filesystem semantics. It provides data integrity, caching, snapshots and an explicit operational model for the storage platform.

Memory plays a critical role in making those capabilities perform consistently.

Unused memory is not necessarily wasted memory

Traditional capacity thinking sometimes treats free memory as evidence that the system has sufficient headroom.

For a storage node, large amounts of permanently unused RAM may represent an opportunity that has not been used.

Data and metadata already read from slower storage can be retained in memory. A later request may then be served without waiting for another device or network operation.

This can reduce:

• physical-device reads;
• iSCSI traffic to back-end nodes;
• latency;
• contention;
• and pressure on slower storage paths.

The important distinction is between memory that is available for productive caching and memory that has been irreversibly consumed.

A well-designed cache should use available capacity aggressively while still responding safely when the operating system or applications require more memory.

The ZFS Adaptive Replacement Cache

ZFS uses the Adaptive Replacement Cache, normally referred to as the ARC.

The ARC keeps frequently or recently accessed data and metadata in main memory.

It is more sophisticated than a simple “most recently used” cache. It attempts to balance data accessed frequently with data accessed recently, adapting its behaviour according to the workload.

Application or storage client
             |
             v
        ZFS read request
             |
      +------+------+
      |             |
  ARC cache hit   ARC cache miss
      |             |
      v             v
 Data returned   Back-end storage read
 from RAM             |
                      v
                Data enters ARC

When the requested information is already present in the ARC, ZFS can return it from memory.

When it is absent, ZFS must retrieve it from the underlying storage. That operation may involve an iSCSI connection, a back-end storage node, an enclosure and a physical device.

A cache hit therefore avoids several layers of potential latency.

Why metadata matters

Storage performance is not determined only by the speed at which file contents can be read.

Before accessing data, the system may need to resolve:

• directory structures;
• dataset information;
• block pointers;
• permissions;
• object attributes;
• and other filesystem metadata.

If frequently used metadata remains in memory, many operations can be resolved without repeated reads from slower devices.

This becomes particularly important for workloads involving:

• large directory trees;
• many small files;
• snapshots;
• frequent file-status operations;
• or random access across a broad dataset.

In these cases, metadata caching may influence perceived performance as much as caching the user data itself.

Memory allocation on HG000009

On this host, most available RAM is deliberately made available to ZFS and the ARC.

The strategy prioritises:

• read performance;
• metadata caching;
• reduced dependency on slower back-end reads;
• and predictable latency.

This does not mean configuring ZFS to consume memory without limits or leaving the operating system unable to function.

The node still requires memory for:

• the Linux kernel;
• networking;
• iSCSI services;
• SMB or other storage services;
• monitoring;
• administration;
• and exceptional operating conditions.

The purpose is to provide ZFS with substantial caching capacity while preserving enough headroom for the complete node to remain healthy.

Total physical RAM
        |
        +-- Linux kernel and core services
        |
        +-- Networking and iSCSI
        |
        +-- Storage presentation services
        |
        +-- Monitoring and administration
        |
        +-- ZFS ARC
              |
              +-- Cached data
              +-- Cached metadata

ARC growth must remain adaptive

The ARC is designed to use memory dynamically.

It can grow when memory is available and release capacity under pressure. However, the interaction between ZFS, the operating system and other workloads must still be observed.

In some environments, explicit ARC limits may be appropriate.

A maximum can prevent the cache from competing excessively with applications or other essential services. A minimum can preserve a useful caching baseline.

Those limits should be based on the actual workload.

Setting a very small ARC simply to make a memory dashboard look comfortable can damage storage performance. Allowing the ARC to grow too aggressively on a multi-purpose host can create memory pressure elsewhere.

The correct balance is an engineering decision, not a universal number.

There is no universal memory formula

ZFS discussions frequently repeat simple rules about how much RAM is required for each terabyte of storage.

Such rules may provide a rough starting point in a specific context, but they are not universal architectural laws.

Memory requirements depend on factors including:

• working-set size;
• metadata volume;
• record size;
• deduplication use;
• number and type of consumers;
• read and write patterns;
• latency requirements;
• and the speed of the underlying devices.

A small pool serving a metadata-intensive workload may benefit from more memory than a much larger pool used for infrequent sequential access.

The system should be sized and tuned according to evidence from its intended workload.

Storage capacity alone does not determine useful ARC size. The working set and access pattern matter far more.

Predictability over benchmark peaks

The design is not intended to produce the highest possible benchmark result for a short test.

I prefer predictable behaviour under sustained and mixed workloads.

A system that produces exceptional throughput for thirty seconds and then collapses under cache pressure is less useful than one that maintains stable performance for hours.

Short benchmarks can be misleading because they may measure data already present in memory.

The result then represents RAM performance rather than the behaviour of the complete storage path.

A realistic test should distinguish between:

• warm-cache reads;
• cold-cache reads;
• sequential access;
• random access;
• small-block activity;
• large transfers;
• and sustained mixed workloads.

The objective is not to produce one impressive number.

It is to understand how performance changes as conditions change.

Benchmark-focused design:

Very high peak
      |
      v
Cache exhausted
      |
      v
Sharp performance collapse

Predictability-focused design:

Stable cache behaviour
      |
      v
Controlled back-end access
      |
      v
Consistent sustained latency

Latency is often more important than throughput

Throughput measures how much data can move during a period.

Latency measures how long an individual operation takes.

Many user-facing and transactional workloads are more sensitive to latency than to maximum aggregate throughput.

A storage system may deliver a high average transfer rate while still producing occasional long pauses.

Those pauses can affect:

• application response time;
• database transactions;
• timeouts;
• queue growth;
• and the stability of dependent services.

Memory helps reduce repeated dependence on slower storage paths, which can improve both average and tail latency for suitable workloads.

It cannot eliminate every delay.

Writes, cache misses, recovery operations and workloads larger than the available cache still reach the underlying storage.

The value lies in making behaviour more stable for the data and metadata used most often.

Caching during degraded conditions

When a back-end device or network path becomes slower, cached data may reduce the immediate effect on some read workloads.

If requested information is already present in the ARC, it can still be returned without waiting for the degraded component.

This can preserve useful performance while the problem is investigated.

However, the ARC should not be treated as a substitute for storage resilience.

It cannot guarantee that every required block is cached. It does not turn volatile memory into a durable copy. A restart removes its contents.

Memory may soften the impact of a degraded state, but the underlying fault still requires action.

ARC is primarily a read cache

The ARC is principally associated with caching reads.

Write behaviour involves additional ZFS mechanisms and should not be simplified into the claim that more ARC always makes writes faster.

Writes may involve:

• transaction groups;
• copy-on-write allocation;
• the ZFS Intent Log for synchronous semantics;
• underlying-device latency;
• and cache-flush behaviour across the complete storage path.

Memory remains important, but write performance and durability must be analysed according to the actual workload and required guarantees.

A storage design should never improve apparent write speed by silently weakening data safety.

L2ARC is not a replacement for RAM

ZFS can use a secondary cache device, known as L2ARC, to extend caching beyond main memory.

This can be useful when the working set is larger than available RAM and the secondary device is substantially faster than the main storage.

However, L2ARC is not equivalent to ARC.

RAM is faster, and managing the secondary cache also requires metadata in memory.

Adding an L2ARC device without understanding the workload may provide little benefit or even reduce the memory available for the primary cache.

The sensible order is:

1. measure the workload;
2. understand ARC effectiveness;
3. identify whether useful data is being evicted;
4. confirm that a secondary cache addresses the limitation;
5. and then test the result.

Installing a fast device labelled as cache is not, by itself, a performance strategy.

Observing cache behaviour

Memory strategy should be validated through evidence.

Relevant observations include:

• ARC size;
• cache hit and miss ratios;
• metadata effectiveness;
• eviction behaviour;
• memory pressure;
• swap activity;
• back-end read volume;
• and latency at both cache-hit and cache-miss paths.

A high overall hit ratio may still conceal poor performance for an important workload.

Similarly, a lower hit ratio is not automatically a problem if the workload is dominated by one-time sequential reads that would not benefit from caching.

Metrics require interpretation.

Question                         Evidence

Is ARC reducing device reads?    Cache hits vs back-end reads
Is memory pressure excessive?    Reclaim, swap and allocation state
Is metadata staying resident?    Metadata cache behaviour
Are misses causing long pauses?  Cache-miss latency
Is tuning helping?               Before-and-after workload tests

Avoiding memory exhaustion

A storage node must remain operable during abnormal conditions.

If memory is allocated too aggressively, pressure may affect:

• administrative access;
• monitoring;
• network services;
• recovery operations;
• and the stability of the host itself.

The worst time to discover that there is no operational headroom is during a storage incident.

The memory strategy must therefore account not only for normal workload but also for:

• scrubs;
• resilvering;
• replication;
• service restart;
• large administrative operations;
• and diagnostic activity.

Performance capacity and recovery capacity are not always the same thing.

Memory is part of the reliability model

Memory strategy affects more than speed.

Predictable caching can reduce pressure on devices and networks. Metadata availability can make administrative and recovery operations more responsive. Adequate headroom allows monitoring and control services to continue functioning under stress.

These characteristics contribute to operational safety.

At the same time, memory is volatile.

It does not replace:

• storage redundancy;
• snapshots;
• replication;
• backups;
• or tested recovery procedures.

Its role is to make the active storage system more efficient and predictable.

Memory is not unused headroom in this architecture. It is an active component of the storage system, directly influencing performance, latency and operational behaviour.

The production mindset

The design reflects a principle I consider important in production systems:

Stable behaviour under realistic pressure is more valuable than exceptional performance under ideal conditions.

Benchmarks are useful when they answer a defined question.

They become misleading when peak numbers are presented without explaining:

• cache state;
• test duration;
• working-set size;
• read and write mix;
• latency distribution;
• and behaviour after the system reaches steady state.

HG000009 is designed around sustained understandability rather than a momentary peak.

Allowing ZFS to use memory aggressively for data and metadata caching helps reduce repeated access to slower back-end devices. Preserving operating headroom keeps the complete node manageable. Monitoring validates whether the balance remains appropriate.

The objective is not simply to have a large ARC.

It is to use memory deliberately as part of a coherent storage architecture.

AI Assistance Disclosure

This article is based on my original ideas, experience, analysis and conclusions. Artificial intelligence tools were subsequently used as editorial and research assistants to review grammar and wording, improve structure and presentation, organise some arguments into clearer logical sections, and help review references to legal, regulatory and technical concepts.

Where relevant, factual and regulatory references were checked against the sources cited in the article. AI assistance does not replace professional legal, regulatory, financial or technical advice, and the final selection, interpretation, opinions and conclusions presented here remain my own.