I think it's a good idea to integrate with the Litellm as it provides an easy to integrate with 100+ LLMs.

https://github.com/BerriAI/litellm

LLM as a drop-in replacement for GPT. Use Azure, OpenAI, Cohere, Anthropic, Ollama, VLLM, Sagemaker, HuggingFace, Replicate (100+ LLMs)

cc @ishaan-jaff