模型下载
https://huggingface.co/baichuan-inc
微调框架
https://github.com/hiyouga/LLaMA-Factory
环境搭建
git clone https://github.com/hiyouga/LLaMA-Factory.git
conda create -n llama_factory python=3.10
conda activate llama_factory
cd LLaMA-Factory
pip install -r requirements.txt
#注意Baichuan2-13B-Chat模型的transformers==4.33.3 ,trl==0.7.2高版本跟模型不兼容
数据集
仿照格式，增加自己的数据集
https://github.com/hiyouga/LLaMA-Factory/blob/main/data/dataset_info.json
微调命令
这里要注意，github里的命令要将template改为baichuan2，lora_target改为W_pack
CUDA_VISIBLE_DEVICES=0 python src/train_bash.py \--stage sft \--model_name_or_path path_to_llama_model \--do_train \--dataset alpaca_gpt4_zh \--template baichuan2\--finetuning_type lora \--lora_target W_pack\--output_dir path_to_sft_checkpoint \--overwrite_cache \--per_device_train_batch_size 4 \--gradient_accumulation_steps 4 \--lr_scheduler_type cosine \--logging_steps 10 \--save_steps 1000 \--learning_rate 5e-5 \--num_train_epochs 3.0 \--plot_loss \--fp16
多GPU训练
accelerate config
accelerate launch src/train_bash.py \--stage sft \--model_name_or_path path_to_llama_model \--do_train \--dataset alpaca_gpt4_zh \--template baichuan2\--finetuning_type lora \--lora_target W_pack\--output_dir path_to_sft_checkpoint \--overwrite_cache \--per_device_train_batch_size 4 \--gradient_accumulation_steps 4 \--lr_scheduler_type cosine \--logging_steps 10 \--save_steps 1000 \--learning_rate 5e-5 \--num_train_epochs 3.0 \--plot_loss \--fp16
)
