这是indexloc提供的服务，不要输入任何密码
Skip to content

bugfix: use correct page count, feature: fileworker integration with Trieve #2770

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 28 commits into from
Closed

bugfix: use correct page count, feature: fileworker integration with Trieve #2770

wants to merge 28 commits into from

Conversation

@cdxker
Copy link
Member

@cdxker cdxker commented Nov 19, 2024

  • feature: add file chunker microservice
  • feature: add logging and write to clickhouse
  • feature: added api key and file chunking status
  • feature: add CLI to test pdf chunking
  • cleanup: add .env.dist + use env's for model in pdf_chunk.rs
  • feat: use pdla boxes for chunking
  • cleanup: clippy
  • feat: add docker-compose.yml for local testing
  • feature: pdf to md :)
  • feature: add page right after its been processed
  • refactor: complete rename from file-chunker to pdf2md
  • bugfix: include snippet to handle loading env's at runtime instead of compile
  • feature: docker setup for pdf2md service
  • feature: add CI workflow to push images for pdf2md service
  • feature: enable chunk bulk delete as an async operation where chunks matching delete filter can be inserted safely after resp
  • feature: skeleton for pdf2md demo page
  • feature: working CSS for demo-ui on pdf2md
  • feature: finished navbar for pdf2md
  • feature: skeleton upload form for pdf2md
  • feature: add health check + fix redoc
  • feature: add skeleton template others can slot into
  • feature: add api key functionality to demo-ui app
  • feature: working API req on file form add
  • feature: add advanced options and usage metrics to each chunk
  • cleanup: remove creating a file on the file_handler
  • feature: setup fileworker to communicate with pdf2md
  • bugfix: correct page count using redis
  • feature: update to new route name

Please indicate what issue this PR is related to and @ any maintainers who are relevant

densumesh and others added 28 commits November 11, 2024 18:34
@densumesh @drew-harris
feature: add file chunker microservice
858880a
@densumesh @drew-harris
feature: add logging and write to clickhouse
c826a85
@densumesh @drew-harris
feature: added api key and file chunking status
7b79067
@densumesh @drew-harris
feature: add CLI to test pdf chunking
c105be6
@skeptrunedev
cleanup: add .env.dist + use env's for model in pdf_chunk.rs
9ecffbe
@drew-harris @skeptrunedev
feat: use pdla boxes for chunking
6352b4e 
cleanup: move test program

feat: different prompt

feat: build metadata strings

fix: pdla server returns floats

feat: fix image cropping

feat: move pdla types

feat: new system prompt

feat: include tables

cleanup: remove test binary

fix: handle empty text better

fix: issues after rebase

fix: keep llm model in env
@drew-harris @skeptrunedev
cleanup: clippy
f6e9633
@drew-harris @skeptrunedev
feat: add docker-compose.yml for local testing
72ca0ba
@densumesh @skeptrunedev
feature: pdf to md :)
d717c1e
@densumesh
feature: add page right after its been processed
052ccc9
@skeptrunedev
refactor: complete rename from file-chunker to pdf2md
a2a8a97
@skeptrunedev
bugfix: include snippet to handle loading env's at runtime instead of…
1c89f89 
… compile
@skeptrunedev
feature: docker setup for pdf2md service
b34e5ba
@skeptrunedev
feature: add CI workflow to push images for pdf2md service
fdb9298
@skeptrunedev
feature: enable chunk bulk delete as an async operation where chunks …
9e2a55c 
…matching delete filter can be inserted safely after resp
@skeptrunedev
feature: skeleton for pdf2md demo page
aed7531
@skeptrunedev
feature: working CSS for demo-ui on pdf2md
4f3f464
@skeptrunedev
feature: finished navbar for pdf2md
ac7c7fd
@skeptrunedev
feature: skeleton upload form for pdf2md
b5d7aa3
@skeptrunedev
feature: add health check + fix redoc
a1d8bda
@skeptrunedev
feature: add skeleton template others can slot into
97b5f88
@skeptrunedev
feature: add api key functionality to demo-ui app
c4bf0b6
@skeptrunedev
feature: working API req on file form add
2888eb0
@densumesh @skeptrunedev
feature: add advanced options and usage metrics to each chunk
41a51a5
@cdxker
cleanup: remove creating a file on the file_handler
4e5fe5f
@cdxker
feature: setup fileworker to communicate with pdf2md
4e9ae19
@cdxker
bugfix: correct page count using redis
17caccc
@cdxker
feature: update to new route name
ee5e88a
@cdxker
Copy link
Member Author

cdxker commented Nov 19, 2024

#2763

@cdxker cdxker closed this Nov 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@cdxker @densumesh @skeptrunedev @drew-harris