Using OpenAI API locally #335
Conversation
server/utils/helpers/index.jsConsider using a more efficient way to handle the switch cases in the getLLMProvider function. You can create a map of the LLM providers and their corresponding classes, and then use this map to instantiate the appropriate class. This approach is more scalable and maintainable as the number of LLM providers grows. const LLMProviderMap = {
"openai": OpenAiLLM,
"azure": AzureOpenAiLLM,
"anthropic": AnthropicLLM,
"localai": LocalAiLLM
};
function getLLMProvider() {
const vectorSelection = process.env.LLM_PROVIDER || "openai";
const LLMClass = LLMProviderMap[vectorSelection];
if (!LLMClass) {
throw new Error(`ENV: No LLM_PROVIDER value found in environment!`);
}
let embedder = null;
if (vectorSelection === "anthropic" || vectorSelection === "localai") {
embedder = getEmbeddingEngineSelection();
}
return new LLMClass(embedder);
}Consider using destructuring to import the required classes from the AiProviders module. This will make your code cleaner and easier to read. const { OpenAiLLM, AzureOpenAiLLM, AnthropicLLM, LocalAiLLM } = require("../AiProviders");server/utils/helpers/updateENV.jsConsider validating the LocalAiBasePath input more thoroughly. The current validation only checks if the input is a valid URL and contains "v1". You might want to add more specific checks, such as checking if the URL uses the http or https protocol, or if it points to a trusted domain. function validLocalAiBasePath(input = "") {
try {
const url = new URL(input);
if (!url.protocol.startsWith("http")) return "URL must use http or https protocol";
if (!input.includes("v1")) return "URL must include v1";
// Add more checks as needed
return null;
} catch {
return "Not a valid URL";
}
} |
* WIP on continuous prompt window summary * wip * Move chat out of VDB simplify chat interface normalize LLM model interface have compression abstraction Cleanup compressor TODO: Anthropic stuff * Implement compression for Anythropic Fix lancedb sources * cleanup vectorDBs and check that lance, chroma, and pinecone are returning valid metadata sources * Resolve Weaviate citation sources not working with schema * comment cleanup
* disable import on hosted instances * Update UI on disabled import/export --------- Co-authored-by: timothycarambat <rambat1010@gmail.com>
resolves #336 Add support for gpt-4-turbo 128K model
* settings for similarity score threshold and prisma schema updated * prisma schema migration for adding similarityScore setting * WIP * Min score default change * added similarityThreshold checking for all vectordb providers * linting --------- Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
|
@franzbischoff I'm really looking forward to this release. How can I use LocaAI with this branch? |
|
@franzbischoff Can you please add support LocalAI for embeddings also? It would be very nice. |
|
Hello, @franzbischoff ! Thank you for quick reply. I try to use master branch at actual top commit (6e72519) but still don't see ability to use LocalAI for Embeddings. Only OpenAI and Azuer OpenAI are available. Can you please add it? |
* Using OpenAI API locally * Infinite prompt input and compression implementation (Mintplex-Labs#332) * WIP on continuous prompt window summary * wip * Move chat out of VDB simplify chat interface normalize LLM model interface have compression abstraction Cleanup compressor TODO: Anthropic stuff * Implement compression for Anythropic Fix lancedb sources * cleanup vectorDBs and check that lance, chroma, and pinecone are returning valid metadata sources * Resolve Weaviate citation sources not working with schema * comment cleanup * disable import on hosted instances (Mintplex-Labs#339) * disable import on hosted instances * Update UI on disabled import/export --------- Co-authored-by: timothycarambat <rambat1010@gmail.com> * Add support for gpt-4-turbo 128K model (Mintplex-Labs#340) resolves Mintplex-Labs#336 Add support for gpt-4-turbo 128K model * 315 show citations based on relevancy score (Mintplex-Labs#316) * settings for similarity score threshold and prisma schema updated * prisma schema migration for adding similarityScore setting * WIP * Min score default change * added similarityThreshold checking for all vectordb providers * linting --------- Co-authored-by: shatfield4 <seanhatfield5@gmail.com> * rename localai to lmstudio * forgot files that were renamed * normalize model interface * add model and context window limits * update LMStudio tagline * Fully working LMStudio integration --------- Co-authored-by: Francisco Bischoff <984592+franzbischoff@users.noreply.github.com> Co-authored-by: Timothy Carambat <rambat1010@gmail.com> Co-authored-by: Sean Hatfield <seanhatfield5@gmail.com>
This PR implements the basics for using the OpenAI API for connecting to a locally compatible server like gpt4all or lmstudio.ai while keeping separate configurations for the embedding model.
the apps mentioned above are just those I used by chance. Both work very well, but lmstudio.ai allows you to use your GPU along with the CPU. It has a nice browser look that allows you to discover models published on Huggingface.
Refer to issue #313