θΏ™ζ˜―indexlocζδΎ›ηš„ζœεŠ‘οΌŒδΈθ¦θΎ“ε…₯任何密码
Skip to content

3416 file picker load times #3833

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
May 15, 2025
Merged

3416 file picker load times #3833

merged 3 commits into from
May 15, 2025

Conversation

timothycarambat
Copy link
Member

@timothycarambat timothycarambat commented May 15, 2025 •
edited
Loading

Pull Request Type

  • ✨ feat
  • πŸ› fix
  • ♻️ refactor
  • πŸ’„ style
  • πŸ”¨ chore
  • πŸ“ docs

Relevant Issues

closes #3497
closes #3416
connect #3135

What is in this change?

  • Implements a conditional readStreaming for rendering of the local file picker.
  • The current file size limit is 150mb, as most files are less than this value.
  • Most users will not notice a delay or change in the load times since this path is unlikely to be hit often.
  • Will run concurrent IO for streaming files while also pruning the pageContent to reduce memory overhead.
  • While streaming a file does take longer if hit, it is more optimized now to not load the whole file into memory at once and the page content will be dumped at the end.

Additional Information

  • we did try a streaming JSON parser, however this impacted the performance even worse (+400% time) and that tradeoff is simply not worth it at this time.
  • The inclusion of lazy loading folders will allow us to raise the threshold and then we get the best of both.

Developer Validations

  • I ran yarn lint from the root of the repo & committed changes
  • Relevant documentation has been updated
  • I have tested my code functionality
  • Docker build succeeds locally

@timothycarambat timothycarambat added the PR:needs review Needs review by core team label May 15, 2025
shatfield4
shatfield4 approved these changes May 15, 2025
View reviewed changes
Copy link
Collaborator

@shatfield4 shatfield4 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@timothycarambat timothycarambat merged commit 3bbceb8 into master May 15, 2025
@timothycarambat timothycarambat deleted the 3416-file-picker-load-times branch May 15, 2025 20:54
timothycarambat added a commit that referenced this pull request May 16, 2025
@timothycarambat
patch Cached tag in UI not showing from #3833
6c36443
lohawk-azalea pushed a commit to azalea-gograbcode/anything-llm that referenced this pull request May 25, 2025
@ariyanki @KPCOFGS
@timothycarambat @shatfield4 @17ColinMiPerry @maoguy @omahs
Sync fork 202505 v2 (#60)
7341d14 
* Updated README.md and README.zh_CN.md (Mintplex-Labs#3775)

* Update README.zh-CN.md

* Update README.zh-CN.md

* Update README.zh-CN.md

* Update README.md

* Merge commit from fork

* linting

* match openai versions across collector/backend

* Updated README.md and README.zh-CN.md (Mintplex-Labs#3776)

* Update README.zh-CN.md

* Update README.md

* Update README.md

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>

* Enable workflow rule for package verification (Mintplex-Labs#3778)

enable workflow rule

* Update README.zh-CN.md (Mintplex-Labs#3780)

* Update README.zh-CN.md

* Update README.zh-CN.md

* Chat prompt history menu (Mintplex-Labs#3770)

* wip system prompt history sidebar ui

* lint

* backend/frontend implementation for prompt history wip

* lint

* rework ui

* add delete menu and delete chat history by id

* lint

* ref for menu button

* reorganize components + light mode styles

* lint

* UI refactor

* backend refactor

* remove unused import

* add border-none to all buttons

* fix spacing on dots 3 icon button

* add window to confirm

* add english translations

* normalize translations

* lin

* patch store logic

* sticky header

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* fix null entry on new workspace

* minor UI patches (Mintplex-Labs#3784)

* Updated zh locale (Mintplex-Labs#3786)

* Update common.js

* linting

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* locale change (Mintplex-Labs#3787)

* Update common.js

* Update common.js

* lint

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* Add Disable Spellcheck Option (Mintplex-Labs#3795)

* Implemented spellcheck toggle in Settings

* ran yarn lint

* reduce read complexity for appearance spellcheck
lint

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* Update:WorkspaceChat-ChatContainer-PromptInput (Mintplex-Labs#3792)

update:WorkspaceChat ChatContainer PromptInput

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>

* updated locale (Mintplex-Labs#3796)

* Update README.zh-CN.md

* Update common.js

* PGvector vector database support (Mintplex-Labs#3788)

* PGVector support for vector db storage

* forgot files

* comments

* dev build

* Add ENV connection and table schema validations for vector table
add .reset call to drop embedding table when changing the AnythingLLM embedder
update instrutions
Add preCheck error reporting in UpdateENV
add timeout to pg connection

* update setup

* update README

* update doc

* add error fix for pgvector

* Fix MCP Start/Stop Toast Text (Mintplex-Labs#3803)

* corrected toggle server toast text

* reordered code for readability

* reordered code for readability

* ran yarn lint

* reordered code for readability

* ran yarn lint

* Fixed grammar and typos (Mintplex-Labs#3802)

* Update common.js

* Update README.md

* Update common.js

* Update README.md

* linting

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* patch Azure image reading regressions
resolves Mintplex-Labs#3811

* Obsidian data connector (Mintplex-Labs#3798)

* add obsidian vault data connector

* lint

* add english translations

* normalize translations

* improve file parser and reader

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* Sponsors Showcase (Mintplex-Labs#3816)

* Fix typos (Mintplex-Labs#3826)

* Fix typos

* language

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* Update light mode designs for system prompt history (Mintplex-Labs#3818)

* update light mode designs for system prompt history

* lint

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>

* Warn bad docker command (Mintplex-Labs#3827)

* Add context window finder from litellm maintained list
apply to all cloud providers, have client cache for 3 days

* docker container bootup warning

* update invalid ENV warning

* rebased with current HEAD

* update newline printing

* Automatic Context window detection (Mintplex-Labs#3817)

* Add context window finder from litellm maintained list
apply to all cloud providers, have client cache for 3 days

* linting

* Improve UX for API keys and invitations (Mintplex-Labs#3828)

* fix ux for create api key and create invite

* lint

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>

* Support Dell Pro AI Studio provider (Mintplex-Labs#3829)

* forgot files for DPAIS

* patch api key bad ref.current on add post removal

* hoisting cleanup for format var

* Optional summarization for web-scraping flow block (Mintplex-Labs#3496)

* remove summarization from web-scraping flow block

* add option to enable/disable summarization in web-scraping block

* change layout
summarize default to true (was the normal prior)

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>

* 3416 file picker load times (Mintplex-Labs#3833)

* wip

* implment conditional streaming

* no bench

* patch Cached tag in UI not showing from Mintplex-Labs#3833

* Respect Max Chunk Len set - Generic OpenAI Embedder connection (Mintplex-Labs#3838)

Respect Max Chunk Len set
resolves Mintplex-Labs#3834

* Disable message send while content is embedding (Mintplex-Labs#3832)

* Disable message send while content is embedding

* opacity

* normalize langs

* remove console log

---------

Co-authored-by: shatfield4 <seanhatfield5@gmail.com>

* fix folders accumulating many files

* refactor citation parser

* Updated Chinese locale (Mintplex-Labs#3852)

* Update README.zh-CN.md

* Update common.js

* Update common.js

* remove comment in file util

* zh lint

* Filter malformed post-processed docs from file picker (Mintplex-Labs#3853)

Filter malformed post-processed files from UI

---------

Co-authored-by: Shixian Sheng <shixian_sheng-2@protonmail.com>
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
Co-authored-by: Sean Hatfield <seanhatfield5@gmail.com>
Co-authored-by: Colin Perry <55003831+17ColinMiPerry@users.noreply.github.com>
Co-authored-by: Lao <laodanfeng@hotmail.com>
Co-authored-by: omahs <73983677+omahs@users.noreply.github.com>
cabwds pushed a commit to cabwds/anything-llm that referenced this pull request Jul 3, 2025
@timothycarambat
3416 file picker load times (Mintplex-Labs#3833)
c289610 
* wip

* implment conditional streaming

* no bench
cabwds pushed a commit to cabwds/anything-llm that referenced this pull request Jul 3, 2025
@timothycarambat
patch Cached tag in UI not showing from Mintplex-Labs#3833
c5ab33c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shatfield4 shatfield4 shatfield4 approved these changes

Assignees

@shatfield4 shatfield4

Labels
PR:needs review Needs review by core team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@timothycarambat @shatfield4