Skip to content

Avoid memory copy in local store write #2944

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

tomwhite
Copy link
Contributor

@tomwhite tomwhite commented Apr 2, 2025

This removes an unnecessary memory copy when writing to a local file, bringing the memory usage in line with Zarr Python v2 for this case (see https://github.com./tomwhite/memray-array).

TODO:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/user-guide/*.rst
  • Changes documented as a new file in changes/
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

@github-actions github-actions bot added the needs release notes Automatically applied to PRs which haven't added release notes label Apr 2, 2025
@d-v-b
Copy link
Contributor

d-v-b commented Apr 2, 2025

thanks! this looks great.

Copy link
Contributor

@dstansby dstansby left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As the pre-commit typing failure indicates, doesn't f.write() need bytes given instead of a numpy array here? Maybe it doesn't since the tests all seem to pass - if that's the case, could you add an appropriate # ignore comment to fix the mypy error, and add a comment in the code explaining why it's okay to hand a numpy array directly to write()?

@TomAugspurger
Copy link
Contributor

TomAugspurger commented Apr 2, 2025

write supports anything implementing the buffer protocol, which ndarrays do. Prior to 3.12 that wasn't available in Python (https://peps.python.org/pep-0688/).

Bumping the python version in our mypy config at

python_version = "3.11"
might let this work.

@dstansby
Copy link
Contributor

dstansby commented Apr 2, 2025

SPEC 0 has support Python 3.11 for a little while yet, so I don't think we should bump the Python version in the config. If it's all fine in practice, then just an ignore comment to make mypy happy should be fine.

Copy link
Contributor

@TomAugspurger TomAugspurger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also needs a release note: https://zarr.readthedocs.io/en/stable/developers/contributing.html#changelog

Looks good once those are addressed.

@github-actions github-actions bot removed the needs release notes Automatically applied to PRs which haven't added release notes label Apr 3, 2025
@TomAugspurger
Copy link
Contributor

The codecov failure seems to come from increasing the number of lines in an uncovered branch. #2859 is already tracking that, so I'll go ahead and merge this.

@TomAugspurger TomAugspurger merged commit 54b3d44 into zarr-developers:main Apr 3, 2025
29 of 30 checks passed
d-v-b pushed a commit to d-v-b/zarr-python that referenced this pull request Apr 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants