Why do we aim for 1MiB (uncompressed) chunks when auto-sharding?

[ilan-gold]

Zarr version

v3.1.4

Numcodecs version

0.16.5

Python Version

3.12

Operating System

Mac

Installation

uv run

Description

I think this setting:

zarr-python/src/zarr/core/chunk_grids.py

Lines 258 to 269 in 94d543c

	if shard_shape is None:
	_shards_out: None | tuple[int, ...] = None
	if chunk_shape == "auto":
	_chunks_out = _guess_chunks(array_shape, item_size)
	else:
	_chunks_out = chunk_shape
	else:
	if chunk_shape == "auto":
	# aim for a 1MiB chunk
	_chunks_out = _guess_chunks(array_shape, item_size, max_bytes=1024)
	else:
	_chunks_out = chunk_shape

of 1MiB target is way too small. I don't really get why it should be different than the non-auto-sharding case

Steps to reproduce

# /// script
# requires-python = ">=3.11"
# dependencies = [
#   "zarr@git+https://github.com/zarr-developers/zarr-python.git@main",
# ]
# ///
#
# This script automatically imports the development branch of zarr to check for issues

import zarr
import numpy as np

with zarr.config.set({"array.target_shard_size_bytes": 10000000}):
    z = zarr.open("foo.zarr", mode="w")
    arr = z.create_array("arr", shards="auto", data=np.arange(100_000))
    print(arr.chunks) # tiny! 98! That's below a page size on most systems, even uncompressed!

Additional output

platform: macOS-15.1-arm64-arm-64bit
python: 3.12.3
zarr: 3.1.6.dev2+g94d543ccf

**Required dependencies:**
packaging: 25.0
numpy: 2.3.5
numcodecs: 0.16.5
typing_extensions: 4.15.0
donfig: 0.8.1.post1

**Optional dependencies:**
numcodecs: 0.16.5

[d-v-b]

1 MiB is a good target for data that needs to get visualized, and presumably the large number of chunks is offset by the IO ergonomics of sharding.

What problem are you having with the current settings?

[ilan-gold]

print(arr.chunks) # tiny! 98! That's below a page size on most systems, even uncompressed!

Maybe I should have made this clearer but for 1d data, this is the number that came out, which is very, very small (much smaller than 1MiB). It's possible this is just a bug in the 1D case but I also wondered why the chunk setting is so small only for sharding

I can understand that rationale. The issue is that at the moment, and maybe this is just a bug with 1d data, a size so small is

a) unwriteably slow
b) too small to be of any help on a file system where you're doing random reads given how page sizes work (I think!). For remote connections I can see the data transfer mattering but for local file systems, I'd think that with look ahead, you may as well write larger sizes and subset in memory.

Maybe I'm wrong, though. I'll look into if this is a 1d bug

[d-v-b]

yeah 2 questions here, is 1MiB a good target, and is our implementation actually working at getting us to that target. I think local file system page sizes are normally in the KiB range so 1MiB should be fine on that front.

[ilan-gold]

are normally in the KiB range so 1MiB should be fine on that front.

Yup, but the current behavior for 1D is no good there. So I'll do some digging. I think 1MiB is reasonable TBH it's just this case (and maybe others) that are broken, I think

[ilan-gold]

Once you said "sharding is when you have many small chunks," I got the 1MiB thing. I'm on board there.

[ilan-gold]

Simple enough! #3603