resolves #1240

Pull Request Type

• ✨ feat
• 🐛 fix
• ♻️ refactor
• 💄 style
• 🔨 chore
• 📝 docs

In an effort to improve RAG results without adding unwarranted or "conditional" middleware like ReRankers, there is a lot of room for improvement of the traditional RAG pipeline AnythingLLM uses. This PR includes the ability to backfill sources during a chat session.

What is backfilling?

Currently, AnythingLLM will append Context snippets to the system prompt on each chat. These contexts come from the attached VectorStores .similaritySearch method. While this works for direct questions this does not scale well as chat's become higher context or more vague. This leads to a bad RAG experience in a traditional chat, but is far worse in query where a follow-up query, that is contextually relevant, is deemed irrelevant.

Example

prompt 1: "What is anythingllm?"

• possibly get 4 good sources and get a good LLM response as it is operating with high context.

prompt 2: "Tell me some features"

• This question is appropriate, topical, and a decent inquiry but standalone is quite vague when viewed outside of the context of the chat.
• Possible get 0 - 1 maybe relevant sources
• We now must rely on the maybe existent snippet for source and hope the previous LLM response has something worth looking at

introduced in this pr

prompt 1: "What is anythingllm?"

• possibly get 4 good sources and get a good LLM response as it is operating with high context.

prompt 2: "Tell me some features"

• Possible get 0 - 1 maybe relevant sources
• We now can backfill the context window in this order of priority
  • Pinned documents
  • This queries search results
  • Work backwards in chat history citations to backfill the source window to the workspace's assigned snippet window value.

With this, we can now ensure that follow-up questions will be more relevant and contextually pertinent as a conversation continues.

"What if the contextual topic changes"?

• This is handled as we prioritize sources from search results first, then backfill in by more recent chats with historical references
• If you have a relevant contextual change in the chat, those sources will take precedence over the historical ones and will become the new historical reference