[lancedb] SQLITE_BUSY when embedding many documents #44
Description
When embedding a larger number of docs (> 100 (>100k tokens)), the DB seems to lock itself during the process (using lancedb as vector db):
INSERT INTO document_vectors (docId, vectorId) VALUES ('f03e0ec4-1307-4928-b982-91a8a2e34563', '5a791ec8-f1be-4276-8928-5e129accc0d6')
{ result: 0 }
node:internal/process/promises:288
triggerUncaughtException(err, true /* fromPromise */);
^
Error: SQLITE_BUSY: database is locked
--> in Statement#run([
'4f09f8e7-b1a8-4c0b-82e0-bbde80473612',
'05-26-2023-db4e845d-9c45-4d80-9581-73c53844132c.json',
'custom-documents/05-26-2023-db4e845d-9c45-4d80-9581-73c53844132c.json',
2,
'{"id":"db4e845d-9c45-4d80-9581-73c53844132c","url":"file:///home/<username>/git/anything-llm/collector/hotdir/processed/05-26-2023.md","title":"05-26-2023.md","description":"a custom file uploaded by the user.","published":"2023-06-12 09:26:29","wordCount":12758,"token_count_estimate":2849}'
], [Function (anonymous)])
at /home/<username>/git/anything-llm/server/node_modules/sqlite/build/Statement.js:80:23
at new Promise (<anonymous>)
at Statement.run (/home/<username>/git/anything-llm/server/node_modules/sqlite/build/Statement.js:78:16)
at Object.addDocuments (/home/<username>/git/anything-llm/server/models/documents.js:89:12)
at async /home/<username>/git/anything-llm/server/endpoints/workspaces.js:35:7 {
errno: 5,
code: 'SQLITE_BUSY',
__augmented: true
}
Node.js v18.16.0
[nodemon] app crashed - waiting for file changes before starting...
Cannot tell if this is caused by the larger number of docs or if the issue is caused by token count (as in, the issue would be caused even with low doc but high token count).