-
-
-至此，`MiniMind-V`的所有细节已经呈现完毕。
-`MiniMind-V`的模型子类完全继承自`MiniMind`，
-仅基于后者做**最小**变更而产生，
-其核心算法改动`< 50行`，迁移难度极低。
-因此可能和`LlAVA`等模型细节可能存在区别，但思路完全统一。
-
-# 📌 Experiment
-
-## Ⅰ 数据集
-
-来源：[Chinese-LLaVA-Vision](https://huggingface.co/datasets/LinkSoul/Chinese-LLaVA-Vision-Instructions)
-包含约57万张预训练图像，来自CC-3M和COCO 2014；
-[llava-en-zh-300k](https://huggingface.co/datasets/BUAADreamer/llava-en-zh-300k)
-包含300k条指令微调数据和15万张图像。
-问答内容经过翻译，
-对中文支持更友好，进一步经过整理并`resize`。
-
-(pretrain_vlm_data.jsonl) 预训练数据集格式：
-
-```json lines
-{
- "conversations": [
- {
- "role": "user",
- "content": "提供给定图像的简要描述。\n视频理解的拓展思路
- -write by [@xinyanghuang7](https://github.com/xinyanghuang7) - -对于多模态大模型的视频理解能力，一个可行的思路是参考现有MiniCPM-V 2.6 进行视频理解的Python示例。 -主要思想是通过提取视频关键帧，而后进行多图推理。 -因此，如果希望在MiniMind-V中添加视频理解能力，可以在现有多图训练的基础上，参考此python脚本中对于关键帧的提取方法，而后加大训练文件中支持图片的数量。 -所支持的MAX_NUM_FRAMES越多，所消耗的显存越大。 - -```text -import torch -from PIL import Image -from transformers import AutoModel, AutoTokenizer -from decord import VideoReader, cpu # pip install decord - -model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True, - attn_implementation='sdpa', - torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager -model = model.eval().cuda() -tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True) - -MAX_NUM_FRAMES = 64 # if cuda OOM set a smaller number - - -def encode_video(video_path): - def uniform_sample(l, n): - gap = len(l) / n - idxs = [int(i * gap + gap / 2) for i in range(n)] - return [l[i] for i in idxs] - - vr = VideoReader(video_path, ctx=cpu(0)) - sample_fps = round(vr.get_avg_fps() / 1) # FPS - frame_idx = [i for i in range(0, len(vr), sample_fps)] - if len(frame_idx) > MAX_NUM_FRAMES: - frame_idx = uniform_sample(frame_idx, MAX_NUM_FRAMES) - frames = vr.get_batch(frame_idx).asnumpy() - frames = [Image.fromarray(v.astype('uint8')) for v in frames] - print('num frames:', len(frames)) - return frames - - -video_path = "video_test.mp4" -frames = encode_video(video_path) -question = "Describe the video" -msgs = [ - {'role': 'user', 'content': frames + [question]}, -] - -# Set decode params for video -params = {} -params["use_image_id"] = False -params["max_slice_nums"] = 2 # 如果cuda OOM且视频分辨率大于448*448可设为1 - -answer = model.chat( - image=None, - msgs=msgs, - tokenizer=tokenizer, - **params -) -print(answer) -``` - -
-
-
-数据集下载地址：([ModelScope](https://www.modelscope.cn/datasets/gongjy/minimind-v_dataset) | [HuggingFace](https://huggingface.co/datasets/jingyaogong/minimind-v_dataset))
-
-## Ⅱ 训练
-
-> train_pretrain_vlm
-
-预训练从595K条数据集中学习图片的通用知识，比如鹿是鹿，狗是狗。
-
-> train_sft_vlm
-
-指令微调从300K条真实对话数据集中学习对图片提问的真实问答格式，更符合与人类的交流习惯。
-
-> train_sft_vlm
-
-多图微调提供demo：鸟类对比数据集，长度为13.6k的真实问答格式。
-
-训练时均冻结visual encoder也就是clip模型梯度，
-只训练Projection和LLM两部分。
-预训练中，只设置Projection和LLM的最后一层参数可学习。
-指令微调中，设置Projection和LLM的全部参数可学习。
-
-> 训练时间和Loss走势（仅供参考）
-
-Pretrain [512+8] & [768+16]
-
-
-SFT [512+8] & [768+16]
-
-
-## Ⅲ 模型权重
-
-(原生PyTorch`*.pth`权重文件) 下载地址：
-([ModelScope](https://www.modelscope.cn/models/gongjy/MiniMind2-V-PyTorch) | [HuggingFace](https://huggingface.co/jingyaogong/MiniMind2-V-PyTorch))
-
-(`Transformers`格式模型)
-下载地址：
-([ModelScope](https://www.modelscope.cn/profile/gongjy) | [HuggingFace](https://huggingface.co/collections/jingyaogong/minimind-v-67000833fb60b3a2e1f3597d))
-
-> 注：Transformers版本均为单图指令微调后的`MiniMind-V`模型
-
-# 📌 Test
-
-### 效果测试
-
-#### 单图对话
-
-数据说明
- -* 多图数据集规模相对较小且为英文对话，数据集仅包含两图对比的场景，因此微调效果有限，这里只提供一种参考思路。 - - -* `jsonl`均为文本指令，`images.zip`均为配套的图像数据（下载后需要解压） - -| 图片 | -MiniMind2-V | -MiniMind2-V-Small | -
|---|---|---|
|
- |
- 图中是一个繁忙的城市街道，一条长长的街道两旁都是高楼大厦。这条街上挤满了汽车、卡车和公共汽车，还有许多其他车辆在路上行驶。在街道上，可以看到许多汽车，有的在高速行驶，而其他的则停在街道一侧。此外还有一辆公交车也停在街道的右侧。街道上可以看到交通灯，表明这是一个繁忙的城市环境。 | -图中是一个繁忙的城市景象，有几辆汽车和一辆卡车行驶在城市街道上。可以看到许多交通信号灯，其中一些位于街道左侧，另一些则在右侧。可以看到有几个人在街上行走，其中一些人站得离街道更近一些，而另一些则距离较远。还有一个停车标志位于画面的左侧，暗示着城市环境。可以看到街道上有两辆汽车，一辆在右边，另一辆在左边，还有一辆在左边。这幅图像捕捉到了都市环境中典型的一天。 | -
|
- |
- 图片显示了一个宇航员的宇航员身穿宇航服，坐在一架大型航天飞机上。他们似乎正在进行一次宇航员登机或下机的旅程。在宇航员的身后，有一个火箭发射架，可能是用来支撑宇航员在旅程中的任务。此外，还有一架飞机停在机库附近，进一步表明这是一次航空展。在飞机的周围，还有一些人，但他们看起来离飞机很近。可以看到一个人站在飞机附近，可能正在观察或等待航天飞机准备起飞。 | -场景中，一名士兵戴着头盔站在一架大型飞机上。这架飞机似乎是一架军用军用飞机，似乎正准备登上一架飞机。另一个人则站在前面，可能正在观察飞行过程。在飞机周围，有几个人，其中一些站在左侧，另一些则站在右侧。他们似乎正在观看飞行员的表现。此外，还有一辆卡车停在靠近左侧的位置，可能是为了更具体地观察飞行过程。 | -
|
- |
- 图片中，一个女人坐在沙滩上，手里拿着一只白色的狗。她看起来像是个女人，坐在沙地上，看着她。一只狗也坐在她旁边，看起来很放松和舒适。海滩上散布着其他沙滩游客，有些人坐着，而另一些人则坐在更远的地方。背景中可以看到一艘船，这表明这是一个受欢迎的海滩旅游目的地。 | -两个人坐在海滩上，一边懒洋洋地躺在沙滩上，另一边则坐着。他们似乎正在享受海边时光。海滩上有几把椅子，其中一把靠近沙滩的左侧，另一把在中间。此外，还有一只狗躺在沙地上，为这个场景增添了一种放松的气氛。 | -
|
- |
- 照片捕捉到一幅美丽如画的大自然场景，背景是高山峦崖。在水边，一座巨大的喷泉横跨着水面，吸引着许多游客。水面上有几个人，他们或站或坐在喷泉周围，或站或坐。有些人可以看到他们在水中行走，而其他人则站在水边。总体而言，这幅画描绘的是一个美丽而宁静的环境，在那里人们可以欣赏到如画般的美景。 | -在一个美丽的蓝色天空下，一座巨大而巨大的白色瀑布上方悬挂着一只巨大的湿流水。这只瀑布位于一座山上，为整个场景增添了一种迷人而又宁静的气氛。在这幅图像的背景中，可以看到几艘船，其中一些靠近水边，其他的则离得较远。这些船只似乎正在为风景或户外活动做准备。 | -
|
- |
- 图中，一个男人坐在公园的长椅上，旁边是一把绿色椅子。他身边有一本打开的书，上面写着"读书"一句话，暗示他可能正在阅读。公园里有一张长椅和一张公园长椅，为周围的环境增添了几分生气。在公园的周围，有几辆汽车和一辆卡车，表明这是一个公共区域。此外，还可以看到一个人站在公园的不同位置上，可能是等着上路或过马路。 | -一个穿着短裤的老人坐在公园长椅上，周围是树木。他似乎正在读一本书，可能是在读书。背景中有一座长凳，为这个场景提供了充足的座位。在背景中，可以看到一把椅子和一张餐桌，这说明这个场景可能是在一个户外座位区，那里有椅子供人们坐下来放松。 | -
|
- |
- 图中，一只白色的棕熊坐在草地上，旁边是一只长着棕色斑点的大熊。这只熊看起来很害羞或顽皮，因为它正躺在草地上休息，看上去很放松。 | -在这幅图像中，一只棕色的熊正在草地上漫步。这只熊被放置在草地上，占据了画面的大部分空间。它似乎正在自然环境中行走，可能是在草地上。在背景中，有几棵树，为画面增添了自然元素。一只鸟在场景的中间附近飞翔，为画面增添了生气勃勃的气氛。 | -
|
- |
- 图片展示了一个漂亮的花瓶，里面插满了五颜六色的鲜花和花束。这些花束散落在整个花瓶中，给人一种赏心悦目的感觉。花瓶里插着五颜六色鲜花，创造出一种令人赏心悦目的景象。这些鲜花被摆放在一张桌子上，很可能是为了展示它们的美丽而摆放的。 | -场景中，一辆绿色和紫色相间的自行车停在一栋建筑旁边，它被放置在一棵大树旁。这辆自行车被摆放在附近，为这个场景增添了几分色彩。除了自行车外，还有一些其他的自行车，包括两个位于前景中的一个和位于背景中靠近中心位置的另一个。自行车的存在表明它可能是停在那里的。 | -
|
- |
- 图片中的女人穿着一件白色连衣裙，脚上还系着一条黑色网球裙。她正在表演一个网球比赛，很可能是在比赛中。在背景中可以看到几把椅子，可能是为了观众或其他观众的座位安排而摆放的。此外，还有一个长凳放在场景左侧，为人们提供了一个休息的地方。 | -一名身穿白色衣服的女子站在舞台上，手里拿着一只手拿着白色飞盘。她似乎正在参加一个舞台舞会或比赛。场景中还有其他几个人，其中一个站在舞台左侧，另一个站在右侧，第三个人则站在场地右侧。舞台上有几个观众，有的站着，有的坐着，还有一些站着。这看起来像是一场欢乐的节日或活动。 | -
| 图片1 | -图片2 | -512_sft_multi | -768_sft_multi | -
|---|---|---|---|
| 这幅图像显示了一种鸟簸戮的场景：一个女人站在红绿相间的红绿相间的紫色鸟簸戴在她身上。女人站在红色的鸟簸戴在她身上，而她的翻领上的那只红鸟则站在她身后。 | -这两只鸟在同一片树林中飞翔，有的位于画面中心，而另一些则较小，形成了鲜明对比。这种鸟类的出现突出了其飞行能力和适应性，因为它们能够在树林中快速迅速移动。此外，两只鸟的位置不同，一个在图像的左边，另一个在右边，这表明它们在同一片树林中移动得很近。这种鸟类的自然行为也有助于区分这两种鸟类物种。 | -
-> 水平有限难免存在未知的纰漏，欢迎所有人在Issues交流指正或提交PR改进项目
-> 您的支持就是持续改进项目的动力，谢谢！ - -## 🤝[贡献者](https://github.com/jingyaogong/minimind/graphs/contributors) - - -
-
-
-## 🫶支持者
-
-
- 参考链接 & 感谢以下优秀的论文或项目
- -- 排名不分任何先后顺序 -- [LlaVA](https://arxiv.org/pdf/2304.08485) -- [LlaVA-VL](https://arxiv.org/pdf/2310.03744) -- [Chinese-LLaVA-Vision-Instructions](https://huggingface.co/datasets/LinkSoul/Chinese-LLaVA-Vision-Instructions) - -
-
-
-
-
-
-
-
-
-[](https://github.com/jingyaogong/minimind-v/stargazers)
-[](LICENSE)
-[](https://github.com/jingyaogong/minimind-v/commits/master)
-[](https://github.com/jingyaogong/minimind-v/pulls)
-[](https://huggingface.co/collections/jingyaogong/minimind-v-67000833fb60b3a2e1f3597d)
-
-
-
-
-
-
-
-
-
-
-
-
-
-"The Greatest Path is the Simplest"
-
-
-[中文](./README.md) | English
-
-
-
-* This project aims to train a super-small multimodal vision-language model, **MiniMind-V**, with just a cost of 1.3 RMB
- and 1 hours of work, starting from scratch!
-* The smallest version of **MiniMind-V** is only about $\frac{1}{7000}$ the size of GPT-3, designed to enable fast
- inference and even training on personal GPUs.
-* **MiniMind-V** is an extension of the visual capabilities of the [MiniMind](https://github.com/jingyaogong/minimind)
- pure language model.
-* The project includes full code for the minimalist structure of large VLM models, dataset cleaning, pretraining, and
- supervised fine-tuning (SFT).
-* This is not only the smallest implementation of an open-source VLM model but also a concise tutorial for beginners in
- vision-language models.
-* The hope is that this project can provide a useful example to inspire others and share the joy of creation, helping to
- drive progress in the wider AI community!
-
-> To avoid misunderstandings, the "1 hours" is based on testing (`1 epoch`) with an NVIDIA 3090 hardware device (single GPU), and
-> the "1.3 RMB" refers to GPU server rental costs.
-
-
-
-
-
-[🔗🤖 Online Experience](https://www.modelscope.cn/studios/gongjy/MiniMind-V) | [🔗🎞️ Video Introduction](https://www.bilibili.com/video/BV1Sh1vYBEzY)
-
-
-
-# 📌 Introduction
-
-“Building a plane with Legos is much more exciting than flying in first class!”
-Is it really as complex as imagined to build a VLM-based multimodal large model? How is the code implementation done?
-Is the training process difficult? Now, let's explore the answers and feel the joy of creation together!
-
-> [!TIP]
-> (As of 2025-02-20) The MiniMind-V series has completed the training of the following model versions, with the smallest
-> requiring only 26M (0.026B) parameters, capable of both image recognition and conversation!
-
-| Model (Size) | Inference Memory | Release |
-|---------------------------|------------------|------------|
-| MiniMind2-V (104M) | 0.6 GB | 2025.02.20 |
-| MiniMind2-Small-V (26M) | 1.1 GB | 2025.02.20 |
-| minimind-v-v1-small (27M) | 0.6 GB | 2024.10.04 |
-| minimind-v-v1 (109M) | 1.1 GB | 2024.10.04 |
-
-### 👉**Recent Updates**
-
-
-
-
-2025-10-24
- -- Bug fix: model weights mismatch -- Adapted to ["minimind-1024 update"](https://github.com/jingyaogong/minimind) -- Code refactoring: training and evaluation scripts standardized -- Added complete checkpoint resumption support - -
-
-
-2025-04-27
- -- Compatibility updates -- Adapted to the new feature in the "minimind" repository -- Standardized parts of the code - -
-
-
-2025-02-20
- -- MiniMind2-V updated alongside MiniMind2 -- Significant reduction of all redundant code, standardized code format -- Major simplification of the model's redundant structure -- Updated dataset format, expanded with new SFT datasets -- Better performance than the previous VLM version! - -
-
-
-
-# 📌 Quick Start
-
-More...
- -**2024-10-05** - -- MiniMind-V released on schedule, first open-source release - -
-
-
-### Step 0
-
-```bash
-# Clone the code repository
-git clone https://github.com/jingyaogong/minimind-v
-```
-
-```bash
-# Download the clip model to the ./model/vision_model directory
-git clone https://huggingface.co/openai/clip-vit-base-patch16
-# or
-git clone https://www.modelscope.cn/models/openai-mirror/clip-vit-base-patch16
-```
-
-```bash
-# Download the minimind language model to the ./out directory (as the base language model for training VLM):
-# HuggingFace
-https://huggingface.co/jingyaogong/MiniMind2-V-PyTorch/blob/main/llm_512.pth # or llm_768.pth
-# Domestic source
-https://modelscope.cn/models/gongjy/MiniMind2-V-PyTorch/resolve/master/llm_512.pth # or llm_768.pth
-```
-
-
-## Ⅰ Test an existing model's performance
-
-### 1. Environment Preparation
-
-```bash
-pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple
-```
-
-### 2. Download the model
-
-```bash
-git clone https://huggingface.co/jingyaogong/MiniMind2-V
-```
-
-### 3. Command-line Q&A
-
-```bash
-# load_from='model': load native PyTorch weights, load_from='other path': load transformers format
-python eval_vlm.py --load_from model --weight sft_vlm
-
-# Or use transformers format model
-python eval_vlm.py --load_from MiniMind2-V
-```
-
-### 4. Or start the WebUI
-
-```bash
-python web_demo_vlm.py
-```
-
-## Ⅱ Train from scratch
-
-### 1. Environment Preparation
-
-```bash
-pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple
-```
-
-Sharing my hardware and software configuration (for reference only)
- -* CPU: Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz -* RAM: 128 GB -* GPU: NVIDIA GeForce RTX 3090(24GB) * 8 -* Ubuntu==20.04 -* CUDA==12.2 -* Python==3.10.16 -* [requirements.txt](./requirements.txt) - -
-
-
-### 2. Download Data
-
-Download the required content from the [dataset link](https://huggingface.co/datasets/jingyaogong/minimind-v_dataset)
-and place it under `./dataset`.
-
-Note: Test if Torch can use CUDA
- -```bash -import torch -print(torch.cuda.is_available()) -``` - -If unavailable, download the whl file from [torch_stable](https://download.pytorch.org/whl/torch_stable.html) for -installation. Refer -to [this link](https://blog.csdn.net/weixin_45456738/article/details/141029610?ops_request_misc=&request_id=&biz_id=102&utm_term=%E5%AE%89%E8%A3%85torch&utm_medium=distribute.pc_search_result.none-task-blog-2~all~sobaiduweb~default-2-141029610.nonecase&spm=1018.2226.3001.4187) -for help. - -
-
-
-### 3. Start Training
-
-**3.1 Pretraining (Learning image description)**
-
-```bash
-# Basic training command (start from LLM weights, train vision_proj only)
-python train_pretrain_vlm.py --epochs 4 --from_weight llm
-```
-
-> Run pretraining to get `pretrain_vlm_*.pth` as the pretrained model's output weights (* represents the model
-> dimension, default is 512).
-
-**3.2 Supervised Fine-Tuning (Learning image-caption dialogue style)**
-
-```bash
-# Basic training command (start from pretrain weights, full parameter fine-tuning)
-python train_sft_vlm.py --epochs 2 --from_weight pretrain_vlm
-```
-
-> Perform supervised fine-tuning to get `sft_vlm_*.pth` as the output weights for the fine-tuned model.
-
-Note: Dataset Details
- -Pretrain data: -```bash -wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/pretrain_data.jsonl -wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/pretrain_images.zip -unzip pretrain_images.zip && rm pretrain_images.zip -``` - -SFT data: -```bash -wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/sft_data.jsonl -wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/sft_images.zip -unzip sft_images.zip && rm sft_images.zip -``` - -`*.jsonl` contains Q&A text, and `*images` are the accompanying image data. After downloading, decompress the image data. - -Please reserve about 5GB of space for the dataset. If there is insufficient space for pretrain data, you can try skipping the pretrain training step and proceed directly to SFT training. - -
-
-
----
-
-### 4. Test the Model's Performance
-
-Ensure that the model `*.pth` file you want to test is located in the `./out/` directory.
-You can also directly download the pre-trained `*.pth` file
-from [here](https://huggingface.co/jingyaogong/MiniMind2-V-PyTorch).
-
-```bash
-# Test SFT model (default)
-python eval_vlm.py --weight sft_vlm
-
-# Test Pretrain model
-python eval_vlm.py --weight pretrain_vlm
-```
-
----
-
-> [!TIP]
-> The training scripts are based on PyTorch's native framework and support multi-card acceleration. If your device has
-> N (N>1) GPUs:
-
-Single-machine N-card training method (DDP, supports multi-machine multi-card cluster)
-
-```bash
-torchrun --nproc_per_node N train_xxx.py
-```
-
-Note: Training Details
- -**Training Features:** -- Support checkpoint resumption: add `--from_resume 1` parameter to continue from last interruption -- Support GPU count changes: automatically convert steps when GPU count changes during resumption -- Atomic saving: use temporary file + replacement mechanism to prevent weight corruption from interruption -- Each save generates `out/**.pth` (model weights) and `checkpoints/**_resume.pth` (training state) files - -```bash -# To resume training after interruption, use the same command and add --from_resume 1 -python train_sft_vlm.py --epochs 4 --from_resume 1 -``` - -**Parameter Description:** -- `--from_weight`: base weight name (llm, pretrain_vlm, none, etc.) -- `--save_weight`: save weight prefix name -- `--from_resume`: whether to resume training (0=start from scratch, 1=continue from checkpoint) -- `--freeze_llm`: whether to freeze LLM parameters (pretrain use only) -- More details can be found in the code - -
-
-
-# 📌 VLM Detail
-
-The base language model of MiniMind-V (VLM), MiniMind (LLM), comes from the twin
-project [minimind](https://github.com/jingyaogong/minimind). For detailed information on the model structure, training
-specifics, principles, and testing results, please refer to the [minimind](https://github.com/jingyaogong/minimind)
-project. To reduce redundancy, the discussion on LLM-related topics is omitted here, assuming you have a basic
-understanding of MiniMind (LLM).
-
-> Even if you are not very familiar with the details of LLMs, you can still follow the "Quick Start" guide to train a
-> MiniMind-V, as it remains unaffected and the repository focuses on the lowest cost for out-of-the-box use!
-
-MiniMind-V's structure adds two submodules, a Visual Encoder and a feature projection, with a modality-mixing branch to
-support inputs from multiple modalities:
-
-
-
-
-Note: Other Details
- -Single-machine N-card training (DeepSpeed) - -```bash -deepspeed --master_port 29500 --num_gpus=N train_xxx.py -``` - -You can enable wandb logging during training: - -```bash -# You need to log in: wandb login -torchrun --nproc_per_node N train_xxx.py --use_wandb -# and -python train_xxx.py --use_wandb -``` - -By adding the `--use_wandb` parameter, you can log the training process, and after training is complete, you can view -the process on the wandb website. You can specify the project name and run name by modifying the `wandb_project` -and `wandb_run_name` parameters. - -[Note]: After June 2025, the domestic network environment cannot directly connect to WandB. The MiniMind project by default switches to using [SwanLab](https://swanlab.cn/) as the training visualization tool (fully compatible with WandB API), that is, just change `import wandb` to `import swanlab as wandb`, no other changes are needed. - -
-
-
-
----
-
-Next, let's briefly discuss the changes in the external input and output of MiniMind-V.
-
-The input to the VLM is still a segment of text containing special `[Important] Some Interesting Thoughts
- -Let's take a moment to think about two questions: - -* What is a **Large Language Model (LLM)**? -* What is a multimodal model? - -[This article](https://www.jiqizhixin.com/articles/2024-09-15-3) perfectly aligns with my thoughts: -Although the name "large language model" (LLM) contains the word "language," they are actually not closely related to -language; this is just a historical issue. A more accurate name would be self-regressive Transformer or something else. -LLMs are more of a general statistical modeling technology, mainly using a self-regressive Transformer to simulate token -flows. These tokens can represent text, images, audio, action choices, and even molecules—anything, really. -Therefore, as long as the problem can be converted into a process of simulating a series of discrete tokens, LLM can -theoretically solve it. In fact, with the increasing maturity of large language model technologies, we may see more and -more problems falling under this modeling paradigm. In other words, the problem is fixed in using LLM to "predict the -next token," but the role and meaning of the tokens differ in each domain. - -[ZJU-LiXi](https://person.zju.edu.cn/xilics#694283) has also mentioned a similar viewpoint (roughly stated below): -Text, video, audio, actions, etc., are considered "multimodal" signals in human perception, but the term "modality" is -essentially just a classification concept based on how humans store information. Just like `.txt` and `.png` files, -though they differ in visual presentation and higher-level forms, they are fundamentally the same. The concept of " -multimodal" arose simply because humans need to categorize these signals based on different sensory dimensions. -However, for machines, regardless of the signal's "modality," they are ultimately presented as a sequence of binary " -monomodal" numbers. Machines do not differentiate the origin of these signals; they just process and analyze the -information contained within these sequences. - -Personally, I think **Generative Pretrained Transformer (GPT)** is a more fitting term than **Large Language Model (LLM) -**, and I prefer to use "GPT" to represent models in the LLM/VLM/GPT-like architecture series rather than to ride on -OpenAI's coattails. - -To summarize what GPTs do in one sentence: - -A GPT model predicts the next, next-next, next-next-next token, etc., based on the current token... until the model -outputs the end token; here, the "token" doesn’t necessarily have to be text! - -```text -> For an LLM model, if we need to understand an "image," we just treat the "image" as a special "foreign language" that has never been encountered before, and translate it into the "LLM language" via a "foreign language dictionary." -> For an LLM model, if we need to understand "audio," we just treat "audio" as a special "foreign language" that has never been encountered before, and translate it into the "LLM language" via a "foreign language dictionary." -> ... -``` - -**To obtain MiniMind-V, we only need to do these 2 things:** - -1. Use the **"foreign language dictionary"** that is good at translating images, to translate the image from the **" - foreign language"** into a model-understandable **"LLM language."** -2. Fine-tune the LLM so that it and the **"foreign language dictionary"** go through a period of adaptation, thereby - better understanding images. - -The "foreign language dictionary" is referred to as the Visual Encoder model. -Like LlaVA, Qwen-VL, and other visual language models, MiniMind-V also uses the open-source Clip series models as the -Visual Encoder. -Specifically, we use [clip-vit-base-patch16](https://huggingface.co/openai/clip-vit-base-patch16), a classic Visual -Encoder based on the ViT-B/16 architecture for describing image-text information. -The input image size is 224x224, and because the Patch size is 16×16, it generates 14*14=196 tokens as the input to the -encoder layer, which produces a 1×768 dimensional embedding vector for calculating error with the text. -We don’t need the final embedding representation, so we only take the output from the encoder layer, which is the output -feature from the core ViT backbone. -It receives the feature of size 196×768 from the previous layer, which we use as 196 visual tokens to input into -MiniMind-V. -After obtaining the image encoder features, the integration with the LLM requires aligning the 768-dimensional visual -tokens with the LLM's text tokens, and mapping the image features into the same space as text embeddings. In other -words, the image features and native visual tokens cannot be directly treated the same; they require cross-modal feature -alignment. -[LlaVA-1](https://arxiv.org/pdf/2304.08485) uses a simple unbiased linear transformation to achieve this, with great -success, and MiniMind-V does the same. - - - -With that, the internal structural changes of MiniMind-V are now fully presented. - -
-
-
-At this point, all the details of `MiniMind-V` have been presented.
-The `MiniMind-V` model subclass completely inherits from `MiniMind`,
-and is generated with **minimal** changes based on the latter,
-with core algorithm modifications `< 50 lines`, making the migration difficulty very low.
-Therefore, there may be differences with models like `LlAVA`, but the overall idea remains consistent.
-
-# 📌 Experiment
-
-## Ⅰ Dataset
-
-Source: [Chinese-LLaVA-Vision](https://huggingface.co/datasets/LinkSoul/Chinese-LLaVA-Vision-Instructions)
-Contains approximately 570,000 pre-trained images from CC-3M and COCO 2014;
-[llava-en-zh-300k](https://huggingface.co/datasets/BUAADreamer/llava-en-zh-300k)
-Contains 300k instruction fine-tuning data and 150k images.
-The Q&A content has been translated, with better support for Chinese, further organized and resized.
-
-(pretrain_vlm_data.jsonl) Pre-training dataset format:
-
-```json lines
-{
- "conversations": [
- {
- "role": "user",
- "content": "Provide a brief description of the given image.\nExpansion Ideas for Video Understanding
- -written by [@xinyanghuang7](https://github.com/xinyanghuang7) - -For the video understanding capabilities of multimodal large models, one feasible approach is to refer to the existing -MiniCPM-V 2.6 Python example for video understanding. -The main idea is to extract key frames from the video and then perform multi-image inference. -Therefore, if you want to add video understanding capabilities to MiniMind-V, you can base it on the existing -multi-image training, refer to the key frame extraction method in this Python script, and increase the number of images -supported in the training files. -The more MAX_NUM_FRAMES supported, the more GPU memory it will consume. - -```text -import torch -from PIL import Image -from transformers import AutoModel, AutoTokenizer -from decord import VideoReader, cpu # pip install decord - -model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True, - attn_implementation='sdpa', - torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager -model = model.eval().cuda() -tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True) - -MAX_NUM_FRAMES = 64 # if cuda OOM set a smaller number - - -def encode_video(video_path): - def uniform_sample(l, n): - gap = len(l) / n - idxs = [int(i * gap + gap / 2) for i in range(n)] - return [l[i] for i in idxs] - - vr = VideoReader(video_path, ctx=cpu(0)) - sample_fps = round(vr.get_avg_fps() / 1) # FPS - frame_idx = [i for i in range(0, len(vr), sample_fps)] - if len(frame_idx) > MAX_NUM_FRAMES: - frame_idx = uniform_sample(frame_idx, MAX_NUM_FRAMES) - frames = vr.get_batch(frame_idx).asnumpy() - frames = [Image.fromarray(v.astype('uint8')) for v in frames] - print('num frames:', len(frames)) - return frames - - -video_path = "video_test.mp4" -frames = encode_video(video_path) -question = "Describe the video" -msgs = [ - {'role': 'user', 'content': frames + [question]}, -] - -# Set decode params for video -params = {} -params["use_image_id"] = False -params["max_slice_nums"] = 2 # If cuda OOM and video resolution is greater than 448*448, set to 1 - -answer = model.chat( - image=None, - msgs=msgs, - tokenizer=tokenizer, - **params -) -print(answer) -``` - -
-
-
-Dataset download
-link: ([ModelScope](https://www.modelscope.cn/datasets/gongjy/minimind-v_dataset) | [HuggingFace](https://huggingface.co/datasets/jingyaogong/minimind-v_dataset))
-
-## Ⅱ Training
-
-> train_pretrain_vlm
-
-Pre-training learns general image knowledge from a dataset of 595K samples, such as a deer is a deer, a dog is a dog.
-
-> train_sft_vlm
-
-Instruction fine-tuning learns the real Q&A format for image-related questions from a dataset of 300K real
-conversations, which better aligns with human communication habits.
-
-> train_sft_vlm
-
-Multi-image fine-tuning provides a demo: a bird comparison dataset with 13.6k real Q&A formats.
-
-During training, the visual encoder, i.e., the CLIP model's gradients, are frozen, and only the Projection and LLM parts
-are trained.
-In pre-training, only the last layer parameters of Projection and LLM are learnable.
-In instruction fine-tuning, all parameters of Projection and LLM are learnable.
-
-> Training Time and Loss Trend (for reference only)
-
-Pretrain [512+8] & [768+16]
-
-
-SFT [512+8] & [768+16]
-
-
-## Ⅲ Model Weights
-
-(Native PyTorch `*.pth` weight files) Download link:
-([ModelScope](https://www.modelscope.cn/models/gongjy/MiniMind2-V-PyTorch) | [HuggingFace](https://huggingface.co/jingyaogong/MiniMind2-V-PyTorch))
-
-(`Transformers` format models)
-Download link:
-([ModelScope](https://www.modelscope.cn/profile/gongjy) | [HuggingFace](https://huggingface.co/collections/jingyaogong/minimind-v-67000833fb60b3a2e1f3597d))
-
-> Note: The Transformers version is the `MiniMind-V` model after single-image instruction fine-tuning
-
-# 📌 Test
-
-### Effect Test
-
-#### Single Image Dialogue
-
-Data Description
- -* The multi-image dataset is relatively small and contains English conversations, focusing only on scenes with two image - comparisons. Therefore, the fine-tuning effect is limited, and this is just one reference approach. - -* `jsonl` contains textual instructions, and `images.zip` contains the corresponding image data (to be unzipped after - download). - -| Image | -MiniMind2-V | -MiniMind2-V-Small | -
|---|---|---|
|
- |
- The image shows a busy city street with tall buildings on both sides of a long road. The street is packed with cars, trucks, and buses, along with many other vehicles driving on the road. There are many cars visible on the street, some driving at high speed, while others are parked at the side. Additionally, there is a bus parked on the right side of the street. Traffic lights are visible on the street, indicating a busy city environment. | -The image depicts a bustling urban scene with several cars and a truck driving down the city street. There are many traffic lights visible, some on the left side of the street, and others on the right. Several pedestrians can be seen on the street, some standing closer to the road while others are farther away. A parking sign is located on the left side of the image, hinting at the city environment. Two cars are visible on the street, one on the right and the other on the left, with another one also on the left side. This image captures a typical day in a city environment. | -
|
- |
- The image shows an astronaut in a spacesuit sitting in a large spacecraft. They appear to be embarking on or disembarking from a space mission. Behind the astronaut, there is a rocket launch platform, likely to support the astronaut's mission during the journey. Additionally, an aircraft is parked near a hangar, further suggesting this is an aviation event. There are a few people around the aircraft, some standing close, possibly observing or waiting for the spacecraft to prepare for takeoff. | -The scene shows a soldier wearing a helmet standing on a large airplane. This aircraft appears to be a military one, likely preparing to board another plane. Another person stands in front, possibly observing the flight process. There are several people around the airplane, some standing on the left side, others on the right. They seem to be watching the pilot's performance. Additionally, a truck is parked near the left side, likely to observe the flight process more closely. | -
|
- |
- The image shows a woman sitting on the beach, holding a white dog in her hands. She appears to be a woman sitting on the sand, looking at her. A dog is also sitting beside her, looking relaxed and comfortable. Other beachgoers are scattered across the beach, some sitting, while others are farther away. A boat can be seen in the background, indicating that this is a popular beach destination for tourists. | -Two people are sitting on the beach, one lounging lazily on the sand, while the other is sitting. They seem to be enjoying their time by the sea. There are a few beach chairs, one close to the left side of the sand, another in the middle. Additionally, a dog is lying on the sand, adding to the relaxed atmosphere of the scene. | -
|
- |
- The photo captures a beautiful natural scene with high mountains in the background. By the water, a large fountain spans across the surface, attracting many visitors. There are several people on the water's surface, some standing or sitting around the fountain, while others walk in the water. Overall, this image portrays a beautiful and peaceful environment where people can enjoy scenic views of nature. | -Under a beautiful blue sky, a massive white waterfall hangs above with a huge stream of wet water. This waterfall is located on a mountain, adding a charming and serene atmosphere to the whole scene. In the background of this image, several boats can be seen, some near the water's edge, others farther away. These boats seem to be preparing for scenic or outdoor activities. | -
|
- |
- The image shows a man sitting on a park bench next to a green chair. There is an open book beside him with the words "reading" written on it, suggesting he may be reading. The park has a bench and a park seat, adding life to the surrounding environment. There are several cars and a truck nearby, indicating this is a public space. Additionally, a person can be seen standing at different locations in the park, possibly waiting to cross the road or walk further. | -An elderly person wearing shorts sits on a park bench surrounded by trees. He seems to be reading a book, possibly engaged in reading. In the background, there is another bench providing ample seating for the scene. You can also see a chair and a table in the background, suggesting this may be an outdoor seating area where people can relax. | -
|
- |
- The image shows a white brown bear sitting on the grass, next to a large bear with brown spots. This bear seems shy or playful as it lies on the grass, resting and looking relaxed. | -In this image, a brown bear is strolling on the grass. The bear occupies much of the frame, seemingly walking in its natural environment on the grass. In the background, there are several trees, adding natural elements to the scene. A bird is flying near the middle of the scene, bringing a lively atmosphere to the image. | -
|
- |
- The image shows a beautiful vase filled with colorful flowers and bouquets. The bouquets are scattered throughout the vase, creating a visually pleasant scene. The colorful flowers in the vase create a delightful visual. These flowers are placed on a table, likely displayed to showcase their beauty. | -The scene shows a green and purple bicycle parked next to a building, placed near a large tree. This bicycle is situated nearby, adding some color to the scene. In addition to the bicycle, there are other bicycles, including one in the foreground and another near the center of the background. The presence of the bicycles suggests they may be parked there. | -
|
- |
- The woman in the image is wearing a white dress with a black tennis skirt. She is performing a tennis match, likely part of the competition. Several chairs are visible in the background, possibly set up for the audience or other spectators. Additionally, a bench is placed on the left side of the scene, providing a place for people to rest. | -A woman in white clothing stands on a stage holding a white frisbee. She seems to be participating in a stage dance or competition. Several other people are present in the scene, one standing on the left side of the stage, another on the right, and a third person standing near the right of the venue. The stage has several spectators, some standing, others sitting, with some remaining standing. This appears to be a joyful festival or event. | -
| Image1 | -Image2 | -512_sft_multi | -768_sft_multi | -
|---|---|---|---|
| This image displays a bird scenario: a woman standing with a red and green mixed purple bird perched on her. The woman stands with the bird on her shoulders, while the red bird on her collar stands behind her. | -The two birds are flying in the same forest, some are in the center of the image, while others are smaller, creating a contrast. The birds’ presence highlights their flight ability and adaptability as they swiftly move through the woods. Additionally, the birds’ positions vary, with one on the left and the other on the right, indicating they are moving close within the same forest. Their natural behavior helps distinguish the two bird species. | -
-> Given the limited expertise, there may be unknown issues, and we welcome everyone to discuss, correct, or submit PRs -> to improve the project in Issues.
-> Your support is the driving force behind continuous improvements to the project. Thank you! - -## 🤝 [Contributors](https://github.com/jingyaogong/minimind/graphs/contributors) - - -
-
-
-## 🫶Supporter
-
-
- Reference Links & Thanks to the following excellent papers or projects
- -- No particular order -- [LlaVA](https://arxiv.org/pdf/2304.08485) -- [LlaVA-VL](https://arxiv.org/pdf/2310.03744) -- [Chinese-LLaVA-Vision-Instructions](https://huggingface.co/datasets/LinkSoul/Chinese-LLaVA-Vision-Instructions) - -
- 
- Hi, I'm MiniMind2-V
-
- """)
-
- with gr.Row():
- with gr.Column(scale=3):
- def get_current_image_path(image):
- global current_image_path
- if image is None:
- current_image_path = ''
- return
- current_image_path = image
- return current_image_path
-
- with gr.Blocks() as iface:
- with gr.Row():
- image_input = gr.Image(type="filepath", label="选择图片", height=650)
- image_input.change(fn=get_current_image_path, inputs=image_input)
-
- def update_parameters(temperature_, top_p_):
- global temperature, top_p
- temperature = float(temperature_)
- top_p = float(top_p_)
- return temperature, top_p
-
- with gr.Blocks() as iface_param:
- with gr.Row():
- temperature_slider = gr.Slider(label="Temperature", minimum=0.5, maximum=1.1, value=0.65)
- top_p_slider = gr.Slider(label="Top-P", minimum=0.7, maximum=0.95, value=0.85)
-
- temperature_slider.change(fn=update_parameters, inputs=[temperature_slider, top_p_slider])
- top_p_slider.change(fn=update_parameters, inputs=[temperature_slider, top_p_slider])
-
- with gr.Column(scale=6):
- def chat_with_vlm(message, history):
- if not message:
- yield history + [("错误", "错误：提问不能为空。")]
- return
- if not current_image_path:
- yield history + [("错误", "错误：图片不能为空。")]
- return
-
- image_html = f'