Fine‑tune the multimodal Pixtral‑12B model (Mistral × PixArt) on custom vision‑language instruction datasets using LoRA adapters and Hugging Face's 🤗 ecosystem.

• 🧠 Lightweight LoRA tuning (~3% trainable params)
• 🎯 Supports multimodal JSON with [IMG] token injection
• 📦 Self-contained train.py script powered by 🤗 PEFT + Trainer
• 🚀 Compatible with Flash-Attn 2 for faster training (optional)
• 🧩 Easily pluggable into Hugging Face Hub

You can install all dependencies using the provided environment.yml file (recommended for conda users):

# Step 1: Create conda environment from YAML
conda env create -f environment.yml

# Step 2: Activate the environment
conda activate pixtral-ft

If you prefer pip, use the requirements.txt instead:

pip install -r requirements.txt

Each file is a list of conversations. Every message can contain text and/or image parts:

{
  "messages": [
    {
      "role": "user",
      "content": [
        { "type": "text",  "text": "What’s in this image?" },
        { "type": "image", "image_path": "img/apple.jpg" }
      ]
    },
    {
      "role": "assistant",
      "content": [
        { "type": "text",  "text": "A red apple on a wooden table." }
      ]
    }
  ]
}

Place your JSON under data/ and point --train_json / --eval_json to the files.

# clone repo
git clone https://github.com/<your‑handle>/pixtral-finetune.git
cd pixtral-finetune

conda env create -f environment.yml   # full spec
conda activate pixtral-ft
#  └─ or:  pip install -r requirements.txt  # minimal spec

python scripts/train.py \
  --model_id mistral-community/pixtral-12b \
  --train_json data/train.json \
  --eval_json  data/val.json \
  --output_dir out/pixtral-ft

Run python scripts/train.py --help to see all flags.

Here are the most commonly used arguments in train.py:

To see the full list of arguments at any time:

python scripts/train.py --help

A runnable demo lives in examples/inference.py:

python examples/inference.py \
  --adapter_path out/pixtral-ft \
  --image demo.jpg \
  --prompt "Describe this image."

PRs & issues welcome! 🎉 Please follow these steps to contribute:

1. Fork the repository.
2. Create a new branch (git checkout -b feature-branch).
3. Make your changes.
4. Commit your changes (git commit -m 'Add some feature').
5. Push to the branch (git push origin feature-branch).
6. Open a pull request.

Created by Ojasva Goyal - feel free to contact me at ojasvagoyal9@gmail.com for any questions or feedback.