simplify upstream dependencies to avoid resolution errors (#1429)

* simplify upstream dependecies to avoid resolution errors

* move to use dependency groups - PEP 753

* more docs

* fix doc build

* fix docs build for real

* fix upstream not installing mypy

* upstream workflow will report failures to install

Author: ianhi

.github/setup-failure-template.md

@@ -0,0 +1,22 @@
+# Nightly Upstream Dependency Installation Failed
+
+**Workflow:** {{WORKFLOW}}
+**Run:** [{{RUN_ID}}]({{RUN_URL}})
+**Date:** {{DATE}}
+
+## Issue
+
+The upstream dependency installation step failed during the nightly CI run.
+
+This likely indicates an issue with:
+- The nightly wheels repository
+- Dependency resolution conflicts
+- Version incompatibilities between upstream packages
+
+## Setup Output
+
+```
+{{SETUP_OUTPUT}}
+```
+
+This issue was automatically generated from the nightly upstream dependency checks.

.github/workflows/python-check.yaml

@@ -99,7 +99,7 @@ jobs:
           python --version
           PYTHON_TAG="cp$(echo ${{ env.PYTHON_VERSION }} | tr -d '.')"
           WHEEL=$(ls dist/*-${PYTHON_TAG}-*.whl)
-          uv pip install "${WHEEL}[test]"
+          uv pip install "$WHEEL" --group test
           # Install specific versions based on matrix
           if [ "${{ matrix.deps-version.name }}" = "minimum" ]; then
             echo "Installing minimum versions:"
@@ -186,7 +186,7 @@ jobs:
           python --version
           PYTHON_TAG="cp$(echo ${{ env.PYTHON_VERSION }} | tr -d '.')"
           WHEEL=$(ls dist/*-${PYTHON_TAG}-*.whl)
-          uv pip install "${WHEEL}[test]"
+          uv pip install "$WHEEL" --group test
           uv pip install pytest-mypy-plugins
           # Install specific xarray version based on matrix
           if [ "${{ matrix.xarray-version.version }}" != "latest-release" ]; then

.github/workflows/python-upstream.yaml

@@ -72,19 +72,67 @@ jobs:
           # manylinux: ${{ matrix.platform.manylinux }} # https://github.com/PyO3/maturin-action/issues/245
 
       - name: setup
+        id: setup
         shell: bash
         working-directory: icechunk-python
         run: |
           set -e
           python3 -m venv .venv
           source .venv/bin/activate
           python --version
-          uv pip install --group upstream --pre
           PY_TAG="cp${PYTHON_VERSION//./}"
           WHEEL=$(ls dist/*-${PY_TAG}-*.whl)
-          uv pip install "$WHEEL" --no-deps --force-reinstall
+          # Install upstream dependencies from nightly wheels
+          export UV_INDEX="https://pypi.anaconda.org/scientific-python-nightly-wheels/simple/"
+          export UV_PRERELEASE=allow
+          uv pip install "$WHEEL" --group dev \
+            --resolution highest \
+            --index-strategy unsafe-best-match 2>&1 | tee setup-output.log
           uv pip list
 
+      - name: Create or update setup failure issue
+        if: |
+          always()
+          && steps.setup.outcome == 'failure'
+          && github.event_name == 'schedule'
+          && github.repository_owner == 'earth-mover'
+        shell: bash
+        working-directory: .
+        run: |
+          # Read the template and setup output
+          template=$(cat .github/setup-failure-template.md)
+
+          setup_output="No setup output captured"
+          if [ -f "icechunk-python/setup-output.log" ]; then
+            setup_output=$(cat icechunk-python/setup-output.log)
+          fi
+
+          # Replace placeholders
+          issue_body="${template//\{\{WORKFLOW\}\}/${{ github.workflow }}}"
+          issue_body="${issue_body//\{\{RUN_ID\}\}/${{ github.run_id }}}"
+          issue_body="${issue_body//\{\{RUN_URL\}\}/${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}}"
+          issue_body="${issue_body//\{\{DATE\}\}/$(date -u)}"
+          issue_body="${issue_body//\{\{SETUP_OUTPUT\}\}/$setup_output}"
+
+          # Check for existing open issue with same title
+          issue_title="Nightly upstream dependency installation failed"
+          existing_issue=$(gh issue list --state open --label "CI" --search "\"$issue_title\" in:title" --json number --jq '.[0].number // empty')
+
+          if [ -n "$existing_issue" ]; then
+            echo "Found existing open issue #$existing_issue, updating it..."
+            echo "$issue_body" | gh issue edit "$existing_issue" --body-file -
+            echo "Updated existing issue #$existing_issue"
+          else
+            echo "No existing open issue found, creating new one..."
+            echo "$issue_body" | gh issue create \
+              --title "$issue_title" \
+              --body-file - \
+              --label "CI"
+            echo "Created new issue"
+          fi
+        env:
+          GH_TOKEN: ${{ github.token }}
+
       - name: mypy
         id: mypy
         shell: bash
@@ -232,8 +280,12 @@ jobs:
           python --version
           PY_TAG="cp${PYTHON_VERSION//./}"
           WHEEL=$(ls dist/*-${PY_TAG}-*.whl)
-          uv pip install "$WHEEL" --group upstream
-          uv pip install pytest-mypy-plugins
+          # Install upstream dependencies from nightly wheels
+          export UV_INDEX="https://pypi.anaconda.org/scientific-python-nightly-wheels/simple/"
+          export UV_PRERELEASE=allow
+          uv pip install "$WHEEL" --group test pytest-mypy-plugins \
+            --resolution highest \
+            --index-strategy unsafe-best-match
           uv pip list
 
       - name: Stand up MinIO

docs/.readthedocs.yaml

@@ -6,13 +6,12 @@ build:
     python: "mambaforge-latest"
 
   jobs:
-     install:
+    install:
       - which mamba
-      - cd icechunk-python && maturin build && pip install "$(ls ../target/wheels/*.whl | head -n 1)[docs]" && cd ../docs
+      - cd icechunk-python && maturin build && pip install "$(ls ../target/wheels/*.whl | head -n 1)" && pip install --group docs && cd ../docs
       - mamba list
       # - cd icechunk-python && maturin develop && cd ../docs
 
-
 conda:
   environment: docs/doc-env.yml
 mkdocs:

docs/README.md

@@ -22,7 +22,7 @@ From the `icechunk-python` directory:
 
 ```bash
 # Install icechunk with docs dependencies
-uv sync --extra docs
+uv sync --group docs
 
 # Start the MkDocs development server
 cd ../docs
@@ -42,7 +42,6 @@ uv run mkdocs build
 
 Builds output to: `docs/.site` directory.
 
-
 ### Deploying
 
 Docs are automatically deployed upon commits to `main` branch via readthedocs

docs/docs/contributing.md

@@ -23,66 +23,53 @@ The Python code is developed in the `icechunk-python` subdirectory. To make chan
 cd icechunk-python
 ```
 
-Create / activate a virtual environment:
+#### Setting up your development environment
 
-=== "Venv"
+=== "uv (Recommended)"
 
-    ```bash
-    python3 -m venv .venv
-    source .venv/bin/activate
-    ```
-
-=== "Conda / Mamba"
-
-    ```bash
-    mamba create -n icechunk python=3.12 rust zarr
-    mamba activate icechunk
-    ```
-
-=== "uv"
+    The easiest way to get started is with [uv](https://docs.astral.sh/uv/), which handles virtual environments, dependencies, and building automatically:
 
     ```bash
+    # Install all development dependencies (includes test dependencies, mypy, ruff, maturin)
     uv sync
+
+    # Activate the virtual environment
     source .venv/bin/activate
-    ```
 
-Install `maturin`:
+    # Run tests
+    uv run pytest
 
-=== "Venv"
+    # Run type checking
+    uv run mypy python
 
-    ```bash
-    pip install maturin
+    # Run linting
+    uv run ruff check python
     ```
-
-    Build the project in dev mode:
-
-    ```bash
-    maturin develop
-
-    # or with the optional dependencies
-    maturin develop --extras=test,benchmark
-    ```
-
-    or build the project in editable mode:
+=== "Venv"
 
     ```bash
-    pip install -e .
-    ```
+    python3 -m venv .venv
+    source .venv/bin/activate
 
-=== "uv"
+    # Install maturin and dependencies
+    pip install maturin
+    pip install --group dev
 
-    uv manages rebuilding as needed, so it will run the Maturin build when using `uv run`.
+    # Build the Rust extension
+    maturin develop
 
-    To explicitly use Maturin, install it globally.
+=== "Conda / Mamba"
 
     ```bash
-    uv tool install maturin
-    ```
+    mamba create -n icechunk python=3.12 rust zarr
+    mamba activate icechunk
 
-    Maturin may need to know it should work with uv, so add `--uv` to the CLI.
+    # Install maturin and dependencies
+    pip install maturin
+    pip install --group dev
 
-    ```bash
-    maturin develop --uv --extras=test,benchmark
+    # Build the Rust extension
+    maturin develop
     ```
 
 #### Testing
@@ -96,6 +83,27 @@ They can be run from the root of the repo with `docker compose up` (`ctrl-c` the
     ```bash
     uv run pytest
     ```
+=== "Venv/Conda"
+
+    ```bash
+    pytest
+    ```
+
+#### Testing with Upstream Dependencies
+
+To test Icechunk against development versions of upstream packages (zarr, xarray, dask, distributed), use the nightly wheels from the scientific-python-nightly-wheels repository:
+
+```bash
+# Install with nightly wheels
+export UV_INDEX="https://pypi.anaconda.org/scientific-python-nightly-wheels/simple/"
+export UV_PRERELEASE=allow
+uv sync --group test \
+  --resolution highest \
+  --index-strategy unsafe-best-match
+
+# Run tests
+uv run pytest
+```
 
 #### Running Xarray Backend Tests
 
@@ -191,7 +199,7 @@ From the `icechunk-python` directory:
 
 ```bash
 # Install icechunk with docs dependencies
-uv sync --extra docs
+uv sync --group docs
 
 # Start the MkDocs development server
 cd ../docs

icechunk-python/README.md

@@ -4,32 +4,50 @@ Python library for Icechunk Zarr Stores
 
 ## Getting Started
 
-Activate the virtual environment:
+### For Development
+
+The recommended way to set up for development is with [uv](https://docs.astral.sh/uv/):
 
 ```bash
-python3 -m venv .venv
+# Install all development dependencies (test, mypy, ruff, maturin)
+uv sync
+
+# Activate the virtual environment
 source .venv/bin/activate
+
+# Run tests
+uv run pytest
+
+# Run type checking
+uv run mypy python
+
+# Run linting
+uv run ruff check python
 ```
 
-Install `maturin`:
+uv automatically rebuilds the Rust extension as needed when you run commands with `uv run`.
+
+### Alternative: Manual Setup with pip
+
+If not using uv:
 
 ```bash
+python3 -m venv .venv
+source .venv/bin/activate
+
+# Install maturin
 pip install maturin
-```
 
-Build the project in dev mode:
+# Install dev dependencies (includes test, mypy, ruff)
+pip install --group dev
 
-```bash
+# Build the Rust extension
 maturin develop
 ```
 
-or build the project in editable mode:
-
-```bash
-pip install -e .
-```
+Note: Modern pip (>=21.3) supports dependency groups. If you have an older pip, upgrade with `pip install --upgrade pip`. When the Rust code changes, re-run `maturin develop` to rebuild.
 
-**Note**: This only makes the python source code editable, the rust will need to be recompiled when it changes
+**Note**: When using editable mode, only the Python source code is editable; the Rust code will need to be recompiled when it changes.
 
 Now you can create or open an icechunk store for use with `zarr-python`: