PIPELINE is an image annotation service that leverages FastAPI as a modern, async REST API backend and Celery for robust background task processing. Its primary goal is to allow users to upload images, automatically annotate them using advanced vision-language models (e.g., moondream:v2 via Ollama), and retrieve the results efficiently, with notification support via email.

• FastAPI powers the synchronous and asynchronous REST endpoints, serving both HTML pages and API responses.
• Celery manages long-running or resource-intensive tasks (such as image uploads, annotation, and notifications) using a distributed task queue backed by Redis.
• The system is designed for extensibility, ease of deployment (with Docker/Docker Compose), and developer friendliness.

• FastAPI – Web API framework
• Celery – Distributed task queue for background jobs
• Redis – Broker for Celery and caching
• PostgreSQL – Primary database
• SQLModel/SQLAlchemy – ORM for database models
• Cloudinary – Media storage for uploaded images
• Ollama – Vision-language model integration (LangChain)
• Docker / Docker Compose – Deployment and service orchestration
• Jinja2 – HTML templating
• Emails – For sending annotation result notifications

• Python >= 3.12
• Docker & Docker Compose (recommended for local development)
• Make sure your machine has sufficient RAM, especially for running Ollama models.

1. Clone the repository:

   git clone https://github.com/al-chris/PIPELINE.git
   cd PIPELINE

2. Create a .env file: Copy .env.example to .env and fill in your secrets (Postgres, Redis, Cloudinary, Ollama, etc.).

3. Start all services with Docker Compose:

   docker-compose up --build

   This will launch:

   • FastAPI backend (localhost:8000)
   • Celery worker
   • Redis (and Redis Commander UI at localhost:8081)
   • PostgreSQL

4. (Optional) Manual Setup If you prefer, set up a Python virtual environment and install dependencies:

   python -m venv venv
   source venv/bin/activate
   pip install -e .

5. Database Initialization: The database tables are automatically created on FastAPI startup.

See .env.example for all options. Key variables:

• DATABASE_URL
• REDIS_URL
• CLOUDINARY_*
• OLLAMA_BASE_URL
• FRONTEND_HOST
• POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_DB, etc.

• With Docker Compose:
  The backend is accessible at http://localhost:8000
• Manual:

  uvicorn app.main:app --reload

• With Docker Compose:
  The worker starts automatically.
• Manual:

  celery -A app.tasks.celery worker --loglevel=info

• POST /annotate
  • Body: file (UploadFile), email (string)
  • Response: { "message": "Annotation in progress", "id": "<task_id>" }

• GET /api/results/{id}
  • Response: { "annotation": "...", "file_url": "..." } if found

• GET /

  • Upload page (HTML)

• GET /results/{id}

  • Results page (HTML)

1. User uploads an image via the web or API.
2. FastAPI triggers the Celery task chain:
   • Upload to Cloudinary
   • Commit DB record
   • Call LLM for annotation (via Ollama)
   • Update DB with result
   • Send notification email
3. User retrieves annotation via API or web.

PIPELINE/
├── app/
│   ├── main.py           # FastAPI entrypoint & routes
│   ├── tasks.py          # Celery tasks and workflow logic
│   ├── db.py             # Database models and initialization
│   ├── file.py           # File handling (Cloudinary integration)
│   ├── email.py          # Email utilities
│   ├── config.py         # Configuration and settings
│   ├── logging.py        # Logging setup
│   ├── static/           # Static assets (CSS, JS, images)
│   └── templates/        # Jinja2 HTML templates
├── docker-compose.yml    # Multi-service orchestration
├── dockerfile            # Backend image definition
├── pyproject.toml        # Python package metadata
├── uv.lock               # Dependency lockfile
└── .env.example          # Example environment variables

• User uploads an image via the frontend or API.
• FastAPI receives the request and triggers a Celery workflow.
• Celery tasks:
  1. Upload image to Cloudinary.
  2. Store metadata in PostgreSQL.
  3. Invoke Ollama model for annotation.
  4. Update DB with results.
  5. Send email notification (with result link).
• Redis is used for Celery's broker and backend.
• PostgreSQL stores file and annotation records.
• Cloudinary is the media storage backend.

We welcome contributions!

• Fork the repo and create your feature branch (git checkout -b feature/my-feature)
• Commit your changes (git commit -am 'Add new feature')
• Push to the branch (git push origin feature/my-feature)
• Open a Pull Request

• Use PEP8 compliant code.
• Add docstrings and type annotations.
• Maintain clear commit history.

• Use main for stable releases.
• Use feature branches (feature/, bugfix/, etc.) for development.

• Use GitHub Issues for bug reports and feature requests.

• License: MIT (see LICENSE file)
• Third-party libraries:
  FastAPI, Celery, Redis, PostgreSQL, SQLModel, Cloudinary, LangChain, Ollama, and others (see pyproject.toml).
• Credits:
  Contributions by al-chris.