+
Skip to content

Clean Up Dead Code #1132

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Jan 31, 2025
Merged

Clean Up Dead Code #1132

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
d6e4034
delete
karanataryn Jan 27, 2025
8b44b5a
remove rps unit tests
karanataryn Jan 27, 2025
da94283
deletion
karanataryn Jan 29, 2025
44b24ec
query-ui change
karanataryn Jan 29, 2025
415d355
comments and add caching
karanataryn Jan 31, 2025
1e76c5c
merge main
karanataryn Jan 31, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions .github/workflows/testing.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ jobs:
- name: Install poetry
run: pipx install poetry
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
uses: useblacksmith/setup-python@v6
with:
python-version: ${{ matrix.python-version }}
cache: "poetry"
Expand Down Expand Up @@ -101,7 +101,7 @@ jobs:
- name: Install poetry
run: pipx install poetry
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
uses: useblacksmith/setup-python@v6
with:
python-version: ${{ matrix.python-version }}
cache: "poetry"
Expand Down Expand Up @@ -170,7 +170,7 @@ jobs:
- name: Install poetry
run: pipx install poetry
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
uses: useblacksmith/setup-python@v6
with:
python-version: ${{ matrix.python-version }}
cache: "poetry"
Expand Down Expand Up @@ -211,7 +211,7 @@ jobs:
- name: Install poetry
run: pipx install poetry
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
uses: useblacksmith/setup-python@v6
with:
python-version: ${{ matrix.python-version }}
cache: "poetry"
Expand Down Expand Up @@ -250,7 +250,7 @@ jobs:
- name: Install poetry
run: pipx install poetry
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
uses: useblacksmith/setup-python@v6
with:
python-version: ${{ matrix.python-version }}
cache: "poetry"
Expand Down
1 change: 0 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,6 @@ lib/sycamore/dist/
luna_traces/
traces/
apps/jupyter/bind_dir/poetry_cache
apps/query-ui/cache_dir
docs/build
cache
notebooks/default-prep-data
Expand Down
15 changes: 1 addition & 14 deletions .pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,25 +31,12 @@ repos:
# same args as used in .github/workflows/linting.yml
args: [--install-types, --explicit-package-bases, --non-interactive]
exclude: docs/

- repo: local
hooks:
- id: pytest-check
name: pytest-check
# Needed because otherwise it seems to always run pytest in the git root.
entry: poetry run pytest lib/sycamore/sycamore/tests/unit apps/crawler/crawler/s3/tests/unit
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like it removes the sycamore unit tests as well? Do you not just want to remove the crawler part?

I don't use pre-commit any more, so not sure if this works at all, but if you are keeping it it seems like it makes sense to run the sycamore unit tests.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, changed.

# lib/remote-processors/remote_processors/test/unit
# not run here but via github actions because it requires a make step to happen first
# and it requires a poetry env setup also.
language: system
pass_filenames: false
always_run: true

- repo: local
hooks:
- id: http-integration-test
name: http-integration-test
entry: bash apps/crawler/crawler/http/tests/http_crawler_integration.sh
entry: poetry run pytest lib/sycamore/sycamore/tests/unit
language: system
pass_filenames: false
always_run: true
Expand Down
37 changes: 0 additions & 37 deletions apps/crawler/README.md
View file
Open in desktop

This file was deleted.

39 changes: 0 additions & 39 deletions apps/crawler/crawler/http/Dockerfile
View file
Open in desktop

This file was deleted.

Empty file removed apps/crawler/crawler/http/crawler/__init__.py
View file
Open in desktop
Empty file.
12 changes: 0 additions & 12 deletions apps/crawler/crawler/http/crawler/items.py
View file
Open in desktop

This file was deleted.

103 changes: 0 additions & 103 deletions apps/crawler/crawler/http/crawler/middlewares.py
View file
Open in desktop

This file was deleted.

13 changes: 0 additions & 13 deletions apps/crawler/crawler/http/crawler/pipelines.py
View file
Open in desktop

This file was deleted.

36 changes: 0 additions & 36 deletions apps/crawler/crawler/http/crawler/settings.py
View file
Open in desktop

This file was deleted.

4 changes: 0 additions & 4 deletions apps/crawler/crawler/http/crawler/spiders/__init__.py
View file
Open in desktop

This file was deleted.

Loading
Loading
点击 这是indexloc提供的php浏览器服务，不要输入任何密码和下载