An Update on Zarr for Windows: So Many Files, So Little Time

In a previous post, I declared OME-Zarr the winner for modern microscopy — and I stand by that, if you are running a Linux cluster with a parallel file system like Lustre or GPFS.

But the reality for many of us in the lab is different. We are acquiring data on Windows workstations. And when you try to move a 4 TB OME-Zarr dataset on Windows, you discover a painful truth:

Windows hates multitudes of small files.

If you have ever tried to delete a Zarr dataset and watched Windows Explorer spend 20 minutes just "Discovering items," this post is for you. Here is why it happens and how to fix it without abandoning Zarr.

A standard OME-Zarr dataset splits your image into thousands (or millions) of small "chunks."

Linux handles this via "inodes" and usually manages fine until you hit the millions. Windows (NTFS), alternatively, manages file tabulation in the Master File Table (MFT).

When you copy a folder with 500,000 files on Windows:

• MFT Overload: Windows has to update the Master File Table for every single file. This is a synchronous, sequential operation.
• The "Defender" Tax: Real-time antivirus (like Windows Defender) intercepts every single file write to scan it. If you have 100,000 chunks, that is 100,000 interruptions.
• Explorer Panic: Windows Explorer tries to read metadata (Date Modified, Size) for every file to display the folder. This is why your "Right Click → Properties" takes forever to load.

Wait, I thought Zarr was "Cloud Native"?

It is — and if you are storing your data on AWS S3, Azure Blob, or Google Cloud, you are safe. Object storage systems don't have a file system hierarchy or a Master File Table. They are designed to handle billions of small objects without breaking a sweat.

The problem is that your local workstation is not S3. When you download that S3 bucket to your local D: drive, you are moving from a system designed for millions of objects (Cloud) to a system designed for large files (Windows NTFS). That is when the flood condition happens.

You don't need to switch back to TIFF. You just need to change how you package your Zarr.

The latest Zarr specification (v3 and OME-Zarr v0.5) supports Sharding. Instead of saving 1,000 chunks as 1,000 tiny files, Sharding groups them into one larger "Shard" file (e.g., 100 MB or 1 GB) while keeping an index of where the data lives.

A 100,000-file dataset becomes 100 files. Windows copies it instantly. Python reads it just as fast.

However, most tools still default to the older Zarr v2 standard. You must explicitly opt-in to Sharding or you will still get the file explosion.

If you aren't ready to upgrade your entire pipeline to Sharding, use Python's ZipStore. You write your Zarr data directly into an uncompressed Zip file. Windows sees one file — it moves, copies, and deletes instantly. Python (via zarr.ZipStore) can still read chunks randomly inside the Zip without unzipping the whole thing.

If you keep millions of files, go to Windows Security → Virus & threat protection → Exclusions and add your data drive (e.g., D:\Data) or your specific process (e.g., python.exe) to the exclusion list.

Zarr is still the future, but the default Zarr was designed for the cloud, not for your D: drive. By switching to Sharding or ZipStores, you can keep the speed of chunked analysis without the file-system overhead that freezes your workstation.

Need help tuning your pipeline? Contact us at [email protected].