Drop support for Python 3.9 and add Python 3.13. #1367
Conversation
This change does three things: - Drops support for Python 3.9 and removes it from our Github Actions tests. - Adds support for Python 3.13. This required updating a few dependencies (primarily scipy, matplotlib, and pandas). - Adds sycamore to the default ray runtime_env. I am not sure why this showed up for me with this change. Somehow upgrading from 2.43.0 to 2.47.1 (which I did not do explicitly, but must have come when I regenerated the poetry.lock) seems to expose a ModuleNotFound error on the workers. I believe this is the correct fix, based on https://docs.ray.io/en/latest/ray-core/handling-dependencies.html#library-development, but can't explain what changed.
There was a problem hiding this comment.
Pull Request Overview
This PR removes Python 3.9 support, adds Python 3.13 support, bumps several dependencies, and patches the default Ray runtime environment.
- Update Python version constraints and dependency versions in
pyproject.tomlfiles and CI workflows - Add
sycamoreto Ray’s defaultruntime_envand adjust the lock script - Regenerate and simplify lock files, commenting out legacy pins
Reviewed Changes
Copilot reviewed 12 out of 18 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| pyproject.toml | Tightened Python version range, bumped sycamore-ai |
| lib/sycamore/sycamore/executor.py | Imported sycamore, set default py_modules in runtime_env |
| lib/sycamore/pyproject.toml | Updated Python range, bumped pandas and matplotlib |
| lib/remote-processors/pyproject.toml | Updated Python range |
| lib/poetry-lock/pyproject.toml | Updated Python range, bumped pandas and matplotlib, commented legacy deps |
| lib/poetry-lock/poetry-lock-all.sh | Removed --no-update from poetry lock calls |
| apps/*/pyproject.toml | Updated Python range in each app |
| .github/workflows/*.yml | Removed 3.9 from test matrices, added 3.13 |
Comments suppressed due to low confidence (1)
pyproject.toml:19
- [nitpick] The comments above still reference specific 3.9.x exclusion ranges for
streamlitandcryptography. Since support for 3.9 is dropped, consider updating or removing these comments to avoid confusion.
python = ">=3.10,<3.14"
| from sycamore.data import Document | ||
| from sycamore.plan_nodes import Node | ||
|
|
||
| import sycamore |
There was a problem hiding this comment.
Overwriting runtime_env when provided will discard other settings. Consider merging py_modules into an existing runtime_env dict instead of replacing it entirely to preserve user-specified keys.
Copilot uses AI. Check for mistakes.
There was a problem hiding this comment.
It's only set when "runtime_env" not in ray_args:
| echo "--------------------- special casing in $i" | ||
| cd $(dirname "$i") | ||
| poetry lock --no-update || fail "broke on special case 'poetry lock --no-update' for $i" | ||
| poetry lock || fail "broke on special case 'poetry lock' for $i" |
There was a problem hiding this comment.
We should require poetry 2.0 here. 2.0 makes no-update the default, old poetry would change a lot more stuff.
lib/poetry-lock/pyproject.toml
Outdated
| guidance = "0.1.14" # They removed AzureOpenAIChat in https://github.com/guidance-ai/guidance/commit/a75896a851edf8b3cf15534af4fed8d15f277af1 | ||
| ray = ">=2.41.0" # Latest version supported on linux/aarch64 https://github.com/ray-project/ray/issues/46104 | ||
| pandas = "2.1.1" # Hardcoded into sycamore pyproject.toml. Unclear why. | ||
| pandas = "2.3.0" # Hardcoded into sycamore pyproject.toml. Unclear why. |
There was a problem hiding this comment.
Can we update the why? I"m assuming there's a reason since you changed it.
lib/sycamore/sycamore/executor.py
Outdated
| # logging.error("Spurious log 0: If you do not see spurious log 1 & 2, | ||
| # log messages are being dropped") | ||
| ray_args["runtime_env"]["worker_process_setup_hook"] = _ray_logging_setup | ||
| ray_args["runtime_env"]["worker_process_setup_hook"] = _ray_logging_setup # type: ignore |
There was a problem hiding this comment.
it's not an issue that you needed the type ignore?
There was a problem hiding this comment.
Responded offline. The typechecker was inferringray_args["runtime_env"] from the previous assignment. There is probably someway to specify it with an assertion, but it doesn't seem super critical and I was having trouble getting the syntax right.
This change does three things:
Drops support for Python 3.9 and removes it from our Github Actions tests.
Adds support for Python 3.13. This required updating a few dependencies (primarily scipy, matplotlib, and pandas).
Adds sycamore to the default ray runtime_env. I am not sure why this showed up for me with this change. Somehow upgrading from 2.43.0 to 2.47.1 (which I did not do explicitly, but must have come when I regenerated the poetry.lock) seems to expose a ModuleNotFound error on the workers. I believe this is the correct fix, based on
https://docs.ray.io/en/latest/ray-core/handling-dependencies.html#library-development, but can't explain what changed.