本人使用记录一下训练过程中的心得和bug
1.数据集准备
数据集的标签形式见官方readme,如下:
[{"id": "identity_0","conversations": [{"from": "user","value": "你好"},{"from": "assistant","value": "我是Qwen-VL,一个支持视觉输入的大模型。"}]},{"id": "identity_1","conversations": [{"from": "user","value": "Picture 1: <img>https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg</img>\n图中的狗是什么品种？"},{"from": "assistant","value": "图中是一只拉布拉多犬。"},{"from": "user","value": "框出图中的格子衬衫"},{"from": "assistant","value": "<ref>格子衬衫</ref><box>(588,499),(725,789)</box>"}]},{ "id": "identity_2","conversations": [{"from": "user","value": "Picture 1: <img>assets/mm_tutorial/Chongqing.jpeg</img>\nPicture 2: <img>assets/mm_tutorial/Beijing.jpeg</img>\n图中都是哪"},{"from": "assistant","value": "第一张图片是重庆的城市天际线，第二张图片是北京的天际线。"}]}
]可以训练纯文本,文本+图,文本+多图(中英文皆可,路径最好绝对路径).将该文件存为label.json.之后进入finetune_qlora_single_gpu.sh将最上面的DATA设置为json文件的路径:
DATA="./label.json"
之后运行finetune_qlora_single_gpu.sh即可开始训练.可以在finetune_qlora_single_gpu.sh中修改epoch参数.3090只能使用qlora微调,多卡训练还没有弄明白.
2.运行报错:“erfinv_vml_cpu” not implemented for ‘Half’
见:https://github.com/QwenLM/Qwen-VL/issues/76#issuecomment-1731020993
3.mpi4py安装失败
见:https://blog.csdn.net/weixin_43255962/article/details/103681400
4.auto-gptq安装失败
从源码安装,即:
pip install auto-gptq --extra-index-url https://huggingface.github.io/autogptq-index/whl/cu118/
其他问题会继续更新,总的来看Qwen-VL比VisualGLM更强.
