- Docker
- GPU: https://huggingface.co/docs/text-generation-inference/en/quicktour#supported-hardware
- 80GB of disk storage for the model and docker image
- Start the service.
docker compose up
- TGI is deployed as a server that implements the OpenAI API protocol.
The server can be queried via http://localhost:8080 in the same format as the OpenAI API. For example:
curl http://localhost:8080/v1/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Llama-SEA-LION-v3-8B-IT", "prompt": "Artificial Intelligence is", "max_tokens": 50, "temperature": 0.8, "repetition_penalty": 1.2 }'