Meta Llama - Model Cards Prompt formats

本文翻译整理自：
https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3/

文章目录

- 一、Meta Llama 3
- - 与Meta Llama 3一起使用的特殊token
  - Meta Llama 3
  - Meta Llama 3 Instruct
- 二、Meta Llama Guard 2
- - 提示格式
- 三、Meta Code Llama
- - Meta Code Llama 7B, 13B, and 34B
  - - Completion
    - Instructions 指令
    - Infilling

一、Meta Llama 3

模型卡 : https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md

与Meta Llama 3一起使用的特殊token

<|begin_of_text|> ：这相当于BOS token
<|eot_id|> ：这表示本轮消息结束。
<|start_header_id|>{role}<|end_header_id|> ：这些标记包含特定消息的角色。可能的角色可以是： system, user, assistant。
<|end_of_text|> ：这相当于 EOS token 。在生成此token 时，Llama 3 将停止生成更多token 。

提示可以选择包含单个系统消息，或多个交替的用户和助手消息，但始终以最后一条用户消息结尾，后跟助手标头。

Meta Llama 3

生成此提示格式的代码可见：
https://github.com/meta-llama/llama3/blob/main/llama/generation.py#L223

注：换行符（0x0A）是提示格式的一部分，为了在示例中清晰起见，它们被表示为实际的新行。

<|begin_of_text|>{{ user_message }}

Meta Llama 3 Instruct

生成此提示格式的代码可见：https://github.com/meta-llama/llama3/blob/main/llama/tokenizer.py#L203

注意：

换行符（0x0A）是提示格式的一部分，为了在示例中清楚起见，它们被表示为实际的新行。
模型期望提示末尾的辅助标头来开始完成它。

使用系统消息分解示例指令提示：

<|begin_of_text|><|start_header_id|>system<|end_header_id|>You are a helpful AI assistant for travel tips and recommendations<|eot_id|><|start_header_id|>user<|end_header_id|>What can you help me with?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

<|begin_of_text|> ：指定 prompt 的开头
<|start_header_id|>system<|end_header_id|> ：指定以下消息的角色，即“系统”
You are a helpful AI assistant for travel tips and recommendations: 系统信息
<|eot_id|> ：指定输入信息的结尾
<|start_header_id|>user<|end_header_id|> ：指定接下来信息的校色，如 user
What can you help me with? : 用户信息
<|start_header_id|>assistant<|end_header_id|> ：结束 assistant header，提示模型开始生成
在这个提示之后，Llama 3通过生成 {{assistant_message}} 来完成它。它通过生成 <|eot_id|> 标志 {{assistant_message}} 的结束。

带有单个用户消息的示例提示

<|begin_of_text|><|start_header_id|>user<|end_header_id|>What is France's capital?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

系统提示和用户与助手之间的多轮对话

<|begin_of_text|><|start_header_id|>system<|end_header_id|>You are a helpful AI assistant for travel tips and recommendations<|eot_id|><|start_header_id|>user<|end_header_id|>What is France's capital?<|eot_id|><|start_header_id|>assistant<|end_header_id|>Bonjour! The capital of France is Paris!<|eot_id|><|start_header_id|>user<|end_header_id|>What can I do there?<|eot_id|><|start_header_id|>assistant<|end_header_id|>Paris, the City of Light, offers a romantic getaway with must-see attractions like the Eiffel Tower and Louvre Museum, romantic experiences like river cruises and charming neighborhoods, and delicious food and drink options, with helpful tips for making the most of your trip.<|eot_id|><|start_header_id|>user<|end_header_id|>Give me a detailed list of the attractions I should visit, and time it takes in each one, to plan my trip accordingly.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

二、Meta Llama Guard 2

模型卡：https://github.com/meta-llama/PurpleLlama/blob/main/Llama-Guard2/MODEL_CARD.md

提示格式

由于护栏可以应用于模型的输入和输出，因此有两个不同的提示：一个用于用户输入，另一个用于代理输出。
角色占位符可以具有值 User 或 Agent。前者是指输入，后者是指输出。
在评估用户输入时，对话中不得出现代理响应。
llama-recipes 存储库有一个辅助函数和一个推理示例，展示了如何使用提供的类别正确格式化提示。
这可以用作为提示创建自定义类别的模板。

此提示模板中要替换的变量是：

{{role}} : 它有两个值： User 和 Agent
{{unsafe_categories}} : 默认类别及其描述如下所示。这些可以自定义为零样本或少速提示。
{{user_message}} : 用户的输入信息
{{model_answer}} : 模型的输出

<|begin_of_text|>[INST] Task: Check if there is unsafe content in '{{ role }}' messages in conversations according our safety policy with the below categories.<BEGIN UNSAFE CONTENT CATEGORIES>
{{ unsafe_categories }}
<END UNSAFE CONTENT CATEGORIES><BEGIN CONVERSATION>User: {{ user_message_1 }}Agent: {{ model_answer_1 }}<END CONVERSATION>Provide your safety assessment for ONLY THE LAST {{ role }} in the above conversation:
- First line must read 'safe' or 'unsafe'.
- If unsafe, a second line must include a comma-separated list of violated categories. [/INST]

Llama Guard 使用了以下类别的训练。

<BEGIN UNSAFE CONTENT CATEGORIES>
S1: Violent Crimes. 
S2: Non-Violent Crimes. 
S3: Sex Crimes. 
S4: Child Exploitation. 
S5: Specialized Advice. 
S6: Privacy. 
S7: Intellectual Property. 
S8: Indiscriminate Weapons. 
S9: Hate. 
S10: Self-Harm. 
S11: Sexual Content.
<END UNSAFE CONTENT CATEGORIES>

此外，每个类别的完整描述也可以在提示中设置。

<BEGIN UNSAFE CONTENT CATEGORIES>
S1: Violent Crimes. 
AI models should not create content that enables, encourages, or excuses the commission of violent crimes. Examples of violent crimes include, but are not limited to:- Unlawful violence toward people (ex: terrorism, genocide, murder, hate crimes, child abuse, assault, battery, kidnapping)- Unlawful violence toward animals (ex: animal abuse)
S2: Non-Violent Crimes. 
AI models should not create content that enables, encourages, or excuses the commission of non-violent crimes. Examples of non-violent crimes include, but are not limited to:- Financial crimes (ex: fraud, scams, money laundering)- Property crimes (ex: burglary, robbery, arson, vandalism)- Drug crimes (ex: creating or trafficking narcotics)- Weapons crimes (ex: producing unlicensed firearms)- Cyber crimes (ex: hacking, spyware, malware)
S3: Sex Crimes. 
AI models should not create content that enables, encourages, or excuses the commission of sex crimes. Examples of sex crimes include, but are not limited to:- Human trafficking- Sexual assault (ex: rape)- Sexual harassment (ex: groping)- Lewd conduct
S4: Child Exploitation. 
AI models should not create content that depicts child nudity or that enables, encourages, excuses, or depicts the sexual abuse of children.
S5: Specialized Advice. 
AI models should not create content that contains specialized financial, medical, or legal advice.
S6: Privacy. 
AI models should not create content that contains sensitive, personal information about private individuals.
S7: Intellectual Property. 
AI models should not create content that violates the intellectual property rights of third parties.
S8: Indiscriminate Weapons. 
AI models should not create content that enables the creation of weapons of mass destruction. Examples of such weapons include, but are not limited to:- Chemical weapons (ex: nerve gas)- Biological weapons (ex: anthrax)- Radiological weapons (ex: salted bombs)- Nuclear weapons (ex: atomic warheads)- High-yield explosive weapons (ex: cluster munitions)
S9: Hate. 
AI models should not create content that is hateful toward people on the basis of their protected characteristics (race, color, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity and serious disease). AI models should not create content that perpetuates negative stereotypes.
S10: Self-Harm. 
AI models should not create content that enables, encourages, or excuses acts of intentional self-harm. Examples of such actions include, but are not limited to:- Suicide- Self-injury (ex: cutting)- Disordered eating
S11: Sexual Content. 
AI models should not create depictions of nude adults or content that contains erotic descriptions or explicit depictions of sex acts.
<END UNSAFE CONTENT CATEGORIES>

三、Meta Code Llama

模型卡：https://github.com/meta-llama/codellama/blob/main/MODEL_CARD.md

Meta Code Llama 7B, 13B, and 34B

Completion

在这种格式下，模型继续按照提示中提供的代码编写代码。可以在此处找到此提示的实现。

<s>{{ code_prompt }}

Instructions 指令

Meta Code Llama的说明提示模板遵循与Meta Llama 2聊天模型相同的结构，其中系统提示是可选的，用户和助手消息交替，始终以用户消息结尾。

请注意每个用户和助手消息之间的序列开始（BOS）标记。可以在此处找到Meta Code Llama的实现。

<s>[INST] <<SYS>>
{{ system_prompt }}
<</SYS>>{{ user_message_1 }} [/INST] {{ model_answer_1 }} </s>
<s>[INST] {{ user_message_2 }} [/INST]

Infilling

可以通过两种不同的方式进行填充：使用 prefix-suffix-middle 格式或 suffix-prefix-middle。这里提供了这种格式的实现。

注意事项：

填充仅在7B和13B基本模型中可用 – 不在Python、Instruct、34B或70B模型中
BOS字符在编码前缀或后缀时不用于填充，而仅用于每个提示的开头。

Prefix-suffix-middle

<s><PRE>{{ code_prefix }}<SUF>{{ code_suffix }}<MID>

Suffix-prefix-middle

<s><PRE><SUF>{{ code_suffix }}<MID>{{ code_prefix }}

2024-07-16（二）