+
Skip to content

Conversation

eric-anderson
Copy link
Collaborator

  • Add script to run tests that will
    • optionally run in both local and ray mode
    • split between fast and slow tests
    • verify all notebooks are accounted
  • Fix file scans so they can run with invalid AWS credentials in environment, same trick as for materialize, try being anonymous if default doesn't work.
  • Fix ndd_example notebook to not specify parallelism.
  • Fix tutorial notebook to use materialize to avoid some re-execution

* Add script to run tests that will
  * optionally run in both local and ray mode
  * split between fast and slow tests
  * verify all notebooks are accounted
* Fix file scans so they can run with invalid AWS credentials in environment, same trick
  as for materialize, try being anonymous if default doesn't work.
* Fix ndd_example notebook to not specify parallelism.
* Fix tutorial notebook to use materialize to avoid some re-execution
@eric-anderson eric-anderson requested a review from bsowell October 16, 2024 21:38
Copy link
Contributor

@bsowell bsowell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fine with this as long as you coordinate with Bohou on the file scan stuff.

else:
return all(path.startswith("s3:") for path in self._paths)

def _try_infer_fs(self):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems related/overlapping with #925. Are they going to conflict?

@eric-anderson eric-anderson merged commit f0a358c into main Oct 18, 2024
12 of 13 checks passed
@eric-anderson eric-anderson deleted the eric-localmode branch October 18, 2024 17:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

点击 这是indexloc提供的php浏览器服务,不要输入任何密码和下载