DESCRIPTION:

The main focus of this project is to create a private network of devices (laptops, phones, desktops...) where every user can access an API with the ability to communicate with various locally hosted LLMs.

The biggest advantage this solution has is it's privacy, as the AIs can't directly access the network, leaving the user's data completely private and transfered through secure protocols.

➡️ Localchat AI ⬅️

TODO:

• LLM selection on client.
• API creation and hosting on the server.
  • JSON Conversation exporting
  • Message history
• Private port tunneling through Cloudflare to access the API.
• Authentication to be able to use the API.
  • Secure DB to manage credentials and OAuth (Supabase).
• Secure DB connection and realtime events subscriptions.
• Client interface.
  • Message bubbles w/ smooth animations.
  • AI response formatting (Markdown and code).
  • Local storage for past messages on the conversation.
• Web search feature to allow AI to access realtime data.
• Use .env credentials
• Multiple LLMs running simultaneously.
• API rate limiting
• Multimodal LLM (with images and audio).
• Admin app to remotely manage the server.
• Multiple conversations

NOTES:

CMD:

API: uvicorn api.api:app --host localhost --port 8080 --reload

CLOUDFLARE TUNNEL: cloudflared tunnel --url http://localhost:8080

AI:

Ollama:

• Llama-3.2 3B Tools - 2GB
  
• Deepseek-coder 1.3B - 776MB
  
• Deepseek-R1 1.5B - 1.1GB
  
• Deepseek-llm 7B - 4GB
  
• Qwen-2.5 3B Tools - 1.9B
  
• Qwen-2.5-coder 3B Tools - 1.9B
  
• + Gemma-2 2B - 1.6GB
  
• + Phi-3.5 3.8B - 2.2GB
  
• + Llava-phi-3 3.8B Vision - 2.3GB
  

            Total storage: 11.4GB

TOOLS:

FastAPI: https://github.com/fastapi/fastapi

FastAPI is a modern, high performance, fast API framework, written in Python.

Cloudflared: https://github.com/cloudflare/cloudflared

Creates a secure tunnel between your local device and a remote server, effectively bypassing network restrictions such as firewalls.

DuckDNS https://www.duckdns.org

Service to point a DuckDNS Subdomain to my IP

Let's Encrypt https://letsencrypt.org/es

Generate a third-party valid certificate for your own domains

[OBSOLETE] Telebit: https://github.com/fastapi/fastapi

Telebit operates by establishing a secure tunnel between your local device and a remote server, effectively bypassing network restrictions such as firewalls.

Local Setup: Telebit Remote client on a local device.

Relay Server: Telebit relies on a relay server (telebit.cloud) to facilitate the connection. This server acts as an intermediary, allowing your local device to communicate with external clients.