Patch lanceDB not deleting vectors from workspace #655
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
documentVectors self-sanitize on delete of parent document
server/utils/vectorDbProviders/lance/index.jsIt's a good practice to provide more context in error messages. This will help in debugging and understanding the error better. Also, consider using a logging library instead of console.error for better logging management. console.error(`Error in addDocumentToNamespace: ${e.message}`);server/models/documents.jsInstead of deleting document vectors one by one, consider deleting them in bulk. This will reduce the number of database calls and improve performance. await prisma.document_vectors.deleteMany({
where: { docId: { in: removals.map(removal => removal.docId) } },
}); |
timothycarambat
commented
Jan 29, 2024
|
|
||
| vectors.push(vectorRecord); | ||
| submissions.push({ | ||
| ...vectorRecord.metadata, |
Member
Author
There was a problem hiding this comment.
spread first so that our values overwrite whatever might be colliding in metadata - namely vector and id key. This way what we define as a property will carry throughout the creation and upsert of the vectors.
cabwds
pushed a commit
to cabwds/anything-llm
that referenced
this pull request
Jul 3, 2025
patch lanceDB not deleting vectors from workspace documentVectors self-sanitize on delete of parent document
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request Type
Relevant Issues
resolves #594
What is in this change?
RCA:
When adding a new document to a workspace for lanceDB that also created the table, the
submissionsitem would have a differentidgiven to it by thecollectorthat would overwrite the known document id that was stored in the database - making deletion of it impossible.This only impacted vectors that were totally new to the system (uncached) as cached documents would be able to be properly tracked.
If users have this issue, the easiest full solve is just to delete the workspace and it will fully remove the vectors. They will persist in the DB, but be orphaned which is a negligible impact.
Developer Validations
yarn lintfrom the root of the repo & committed changes