一、大模型推理

https://github.com/hiyouga/LLaMA-Factory/blob/main/README_zh.md
https://github.com/hiyouga/LLaMA-Factory/blob/main/examples/README_zh.md

安装 v7.1

https://github.com/hiyouga/LLaMA-Factory/releases/tag/v0.7.1

git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e .[torch,metrics]

推理
- 以 Llama-3-8B-Instruct 为例，这里全部用 vllm 作为后端，不写就是 transformers 后端
使用命令行接口
- llama3_vllm.yaml

model_name_or_path: eta-Llama-3-8B-Instruct
template: llama3
infer_backend: vllm
vllm_enforce_eager: true

CUDA_VISIBLE_DEVICES=0 llamafactory-cli chat examples/inference/llama3_vllm.yaml

使用浏览器界面

CUDA_VISIBLE_DEVICES=0 llamafactory-cli webchat examples/inference/llama3_vllm.yaml

启动 OpenAI 风格 API

CUDA_VISIBLE_DEVICES=0 llamafactory-cli api examples/inference/llama3_vllm.yaml

模拟发起请求

curl http://0.0.0.0:8000/v1/chat/completions \-H "Content-Type: application/json" \-H "Authorization: Bearer " \-d '{"model": "Meta-Llama-3-8B-Instruct","messages": [{"role": "system","content": "You are a helpful assistant."},{"role": "user","content": "Who won the world series in 2020?"},{"role": "assistant","content": "The Los Angeles Dodgers won the World Series in 2020."},{"role": "user","content": "Where was it played?"}]}'