Based on a LinkedIn post originally published on 8 April 2026
Performance in my home-lab platform is the result of architectural choices rather than one isolated optimisation.
The system does not depend on a single exceptionally fast disk or one configuration parameter intended to solve every workload problem.
Instead, the design combines:
- memory-backed read caching through ZFS ARC;
- storage distributed across several back-end nodes;
- a dedicated storage network;
- asynchronous handling where the required write semantics permit it;
- SSD-backed staging for temporary I/O pressure;
- and separation between client-facing and back-end responsibilities.
The objective is not maximum performance in one synthetic benchmark.
It is stable, explainable behaviour across mixed workloads and during ordinary background activity.
Peak throughput is useful. Predictable throughput under changing conditions is operationally more valuable.
The client sees one service, not the back-end complexity
HG000009 presents the storage service to client systems.
Behind that front end, data is distributed across separate storage nodes connected through a dedicated network.
Client systems
|
| Front-end storage access
v
+-------------------------------+
| HG000009 |
| ZFS, ARC and I/O coordination |
+-------------------------------+
|
| Dedicated storage network
|
+---+----------+----------+
| | |
v v v
Back-end Back-end Back-end
node A node B node CThe client does not need to understand which back-end node holds a physical device or which maintenance task is active there. Those details remain inside the storage architecture.
This separation allows the front-end system to present a coherent service while the back end provides capacity and parallelism.
ARC makes memory part of the storage path
ZFS uses the Adaptive Replacement Cache—ARC—to retain frequently useful data and metadata in system memory.
When a requested block is already present in ARC, it can be served without returning to the underlying disk path.
Client read request
|
v
Is the requested block in ARC?
|
+----+----+
| |
Yes No
| |
v v
Serve from Retrieve from back-end storage
memory and make eligible for cachingMemory latency is substantially lower than storage latency, so cache hits improve response time and reduce work imposed on the back-end devices.
Metadata caching is also important. Filesystem operations frequently require information about directories, blocks and object state before the application data itself can be returned.
Keeping frequently used metadata in memory reduces repeated device access and makes the complete read path more responsive.
ARC does not guarantee that every read is served from memory. Its benefit depends on workload locality, available RAM, cache pressure and the relationship between the active dataset and the cache size.
A cache hit also protects the back end
The immediate benefit of a cache hit is lower read latency.
The wider benefit is that the storage network and physical devices do not need to perform the request again.
This creates more capacity for:
- uncached reads;
- write activity;
- scrubbing and verification;
- recovery operations;
- and other background tasks.
Without useful caching: Client reads + writes + maintenance all compete for storage resources With useful caching: Repeated reads -> ARC New reads + writes + maintenance -> storage resources
The cache therefore changes not only the speed of repeated reads but also the contention experienced by the complete system.
Distributing storage creates parallelism
The underlying devices are attached through multiple back-end storage nodes.
This distributes I/O handling across separate systems and network paths rather than concentrating every operation on the same local host and controller.
Depending on the pool layout and workload, independent requests can progress against different back-end resources simultaneously.
Request stream
|
+---- I/O path A ---- Back-end node A ---- Devices A
|
+---- I/O path B ---- Back-end node B ---- Devices B
|
+---- I/O path C ---- Back-end node C ---- Devices CParallelism is not automatic merely because several nodes exist. The data layout, queueing, network design and workload must allow the resources to be used concurrently.
But distributing responsibility makes greater aggregate capability possible and reduces the amount of unrelated work competing inside one server.
The dedicated data network
Storage traffic uses a dedicated network rather than competing directly with management or general client traffic.
This separation provides a clearer capacity model.
If storage traffic shares links with backups, administration, monitoring and unrelated application flows, performance can change according to work that has nothing to do with the storage service itself.
SHARED NETWORK
Storage + management + backup + other traffic
|
v
Unrelated contention changes I/O latency
DEDICATED STORAGE NETWORK
Storage traffic only
|
v
Clearer ownership and more predictable capacityA dedicated network does not make bandwidth unlimited. It makes the source of saturation easier to understand and manage.
Asynchronous writes and responsiveness
Where the application’s required semantics permit asynchronous handling, write activity can be acknowledged and scheduled without forcing every client-facing operation to wait for the complete physical-device path.
This allows the system to group and order work more effectively while keeping the front end responsive during short bursts.
Asynchronous write path
Client write
|
Accepted into the controlled write path
|
Client continues
|
Back-end devices complete scheduled workThis model must not be confused with ignoring durability.
When an application requests synchronous write semantics, the system must honour the required persistence guarantee before reporting success. Performance choices cannot silently weaken the contract expected by the workload.
Asynchronous behaviour is an optimisation only when it remains consistent with the application’s correctness requirements.
SSD-backed staging for bursts
A dedicated SATA SSD is used as part of the ZFS I/O staging strategy for the storage pools.
Its role is to absorb short periods of elevated write activity and temporary I/O pressure so that slower back-end devices do not immediately transfer every burst into client-visible latency.
Short write burst
|
v
SSD-backed staging
|
| Absorb and smooth temporary pressure
v
Back-end storage pools
|
| Complete work at sustainable device rate
v
Stable client-facing behaviourThe SSD does not increase the sustained write capability of the back-end disks indefinitely.
If incoming work remains above the rate at which the pools can complete it, the staging capacity will eventually cease to hide the difference.
Its value is smoothing bursts, not violating the long-term throughput limit of the storage devices.
Burst performance and sustained performance
These are different properties and should be measured separately.
Burst workload: Temporary demand exceeds back-end rate Cache or staging absorbs the difference Sustained workload: Demand remains elevated Back-end completion rate becomes the limit
A benchmark that runs only long enough to remain inside cache or staging capacity may describe burst behaviour while being presented as storage performance.
A meaningful assessment observes what happens after the fast layers are full and the system reaches a steady state.
Why distributed storage can outperform local access
It may appear counterintuitive that a data path crossing a network and several systems can outperform direct access to local storage.
Network distance introduces overhead, but local storage is not automatically faster if it is constrained by one controller, one group of devices or contention from unrelated host activity.
The distributed design can gain from:
- ARC serving repeated reads from memory;
- parallel access to several back-end resources;
- dedicated network capacity;
- separation of front-end and back-end work;
- and reduced contention on the client-facing host.
LOCAL PATH
Application + filesystem + storage + other host work
all share one resource domain
DISTRIBUTED PATH
Front-end service
+
Memory-backed reads
+
Dedicated network
+
Parallel back-end resources
+
Separated operational responsibilitiesThe additional components create complexity, but they also create resource isolation and parallelism.
Performance during maintenance and recovery
A system should not be evaluated only when nothing else is happening.
Back-end nodes may perform:
- scrubbing;
- verification;
- resilvering or recovery;
- backup activity;
- or other maintenance operations.
These tasks consume I/O capacity.
The architecture aims to prevent every back-end activity from becoming immediate instability for client-facing workloads. Caching, staging, parallelism and isolation provide room for the service to continue operating while the infrastructure performs necessary work.
This does not make maintenance free. It makes the effect more gradual and manageable.
Measure latency distributions, not only averages
Average throughput can hide an unstable user experience.
A workload may achieve a respectable average while individual requests occasionally suffer severe delays during bursts or maintenance.
Average latency: 8 ms But: Most requests: 3 ms Some requests: 80 ms Rare requests: 800 ms The average alone does not describe predictability.
The design should therefore be evaluated using more than peak bandwidth:
- read and write latency distributions;
- cache-hit behaviour;
- queue depth;
- network utilisation;
- back-end device saturation;
- and client-visible behaviour during maintenance.
Clear resource ownership improves diagnosis
When storage responsibilities are isolated, abnormal behaviour is easier to investigate.
If the storage network is saturated, that observation has a narrower meaning because unrelated traffic is not expected there.
If ARC hit rates fall, the relationship between workload size and available memory can be examined. If staging pressure remains high, the sustained back-end completion rate becomes a likely constraint.
Symptom: Read latency rises Check: ARC effectiveness and back-end reads Symptom: Write bursts become visible Check: SSD staging pressure and pool completion rate Symptom: All I/O paths slow Check: Storage network and shared back-end resources Symptom: Degradation during recovery Check: Resource allocation between recovery and client work
Architecture that separates responsibilities also separates diagnostic questions.
Performance is an architectural property
No single element explains the final result.
ARC can reduce read latency but cannot help every uncached workload. An SSD can absorb bursts but cannot exceed the back-end’s sustainable throughput forever. Additional nodes create parallelism only when the data and request patterns use it. A dedicated network removes unrelated contention but still has a finite capacity.
The performance emerges from the relationship between the components:
Useful caching
+
Parallel back-end resources
+
Dedicated data paths
+
Controlled write behaviour
+
Burst absorption
+
Clear resource ownership
=
Predictable client-facing I/OPerformance is rarely created by one fast component. It emerges when contention is reduced, responsibilities are isolated and the complete data path remains understandable under load.
This article is based on my original ideas, experience, analysis and conclusions. Artificial intelligence tools were subsequently used as editorial and research assistants to review grammar and wording, improve structure and presentation, organise some arguments into clearer logical sections, and help review references to legal, regulatory and technical concepts.
Where relevant, factual and regulatory references were checked against the sources cited in the article. AI assistance does not replace professional legal, regulatory, financial or technical advice, and the final selection, interpretation, opinions and conclusions presented here remain my own.
Comments
Post a Comment