随着人工智能技术的快速发展， 大规模预训练自然语言模型成为了研究热点和关注焦点。OpenAI于2018年提出了第一代GPT模型，开辟了自然语言模型生成式预训练的路线。沿着这条路线，随后又陆续发布了GPT-2和GPT-3模型。与此同时，谷歌也探索了不同的大规模预训练模型方案，例如如T5, Flan等。OpenAI在2022年11月发布ChatGPT，展示了强大的问答能力，逻辑推理能力和内容创作能力，将模型提升到了实用水平，改变人们对大模型能力的认知。在2023年4月，OpenAI发布了新升级的GPT-4模型，通过引入多模态能力，进一步拓展了大语言模型的能力边界，朝着通用人工智能更进一步。ChatGPT和GPT-4推出之后，微软凭借强大的产品化能力迅速将其集成进搜索引擎和Office办公套件中，形成了New Bing和 Office Copilot等产品。谷歌也迅速上线了基于自家大语言模型PaLM和PaLM-2的Bard，与OpenAI和微软展开正面竞争。国内的多家企业和研究机构也在开展大模型的技术研发，百度，阿里，华为，商汤，讯飞等都发布了各自的国产语言大模型，清华，复旦等高校也相继发布了GLM, MOSS等模型。
为了准确和公正地评估大模型的能力，国内外机构在大模型评测上开展了大量的尝试和探索。斯坦福大学提出了较为系统的评测框架HELM，从准确性，安全性，鲁棒性和公平性等维度开展模型评测。纽约大学联合谷歌和Meta提出了SuperGLUE评测集，从推理能力，常识理解，问答能力等方面入手，构建了包括8个子任务的大语言模型评测数据集。加州大学伯克利分校提出了MMLU测试集，构建了涵盖高中和大学的多项考试，来评估模型的知识能力和推理能力。谷歌也提出了包含数理科学，编程代码，阅读理解，逻辑推理等子任务的评测集Big-Bench，涵盖200多个子任务，对模型能力进行系统化的评估。在中文评测方面，国内的学术机构也提出了如CLUE,CUGE等评测数据集，从文本分类，阅读理解，逻辑推理等方面评测语言模型的中文能力。
随着大模型的蓬勃发展，如何全面系统地评估大模型的各项能力成为了亟待解决的问题。由于大语言模型和多模态模型的能力强大，应用场景广泛，目前学术界和工业界的评测方案往往只关注模型的部分能力维度，缺少系统化的能力维度框架与评测方案。OpenCompass提供设计一套全面、高效、可拓展的大模型评测方案，对模型能力、性能、安全性等进行全方位的评估。OpenCompass提供分布式自动化的评测系统，支持对(语言/多模态)大模型开展全面系统的能力评估。
OpenCompass介绍
评测对象
本算法库的主要评测对象为语言大模型与多模态大模型。我们以语言大模型为例介绍评测的具体模型类型。
-
基座模型：一般是经过海量的文本数据以自监督学习的方式进行训练获得的模型（如OpenAI的GPT-3，Meta的LLaMA），往往具有强大的文字续写能力。
-
对话模型：一般是在的基座模型的基础上，经过指令微调或人类偏好对齐获得的模型（如OpenAI的ChatGPT、上海人工智能实验室的书生·浦语），能理解人类指令，具有较强的对话能力。
工具架构
- 模型层：大模型评测所涉及的主要模型种类，OpenCompass以基座模型和对话模型作为重点评测对象。
- 能力层：OpenCompass从本方案从通用能力和特色能力两个方面来进行评测维度设计。在模型通用能力方面，从语言、知识、理解、推理、安全等多个能力维度进行评测。在特色能力方面，从长文本、代码、工具、知识增强等维度进行评测。
- 方法层：OpenCompass采用客观评测与主观评测两种评测方式。客观评测能便捷地评估模型在具有确定答案（如选择，填空，封闭式问答等）的任务上的能力，主观评测能评估用户对模型回复的真实满意度，OpenCompass采用基于模型辅助的主观评测和基于人类反馈的主观评测两种方式。
- 工具层：OpenCompass提供丰富的功能支持自动化地开展大语言模型的高效评测。包括分布式评测技术，提示词工程，对接评测数据库，评测榜单发布，评测报告生成等诸多功能。
能力维度
评测方法
OpenCompass采取客观评测与主观评测相结合的方法。针对具有确定性答案的能力维度和场景，通过构造丰富完善的评测集，对模型能力进行综合评价。针对体现模型能力的开放式或半开放式的问题、模型安全问题等，采用主客观相结合的评测方式。
客观评测
针对具有标准答案的客观问题，我们可以我们可以通过使用定量指标比较模型的输出与标准答案的差异，并根据结果衡量模型的性能。同时，由于大语言模型输出自由度较高，在评测阶段，我们需要对其输入和输出作一定的规范和设计，尽可能减少噪声输出在评测阶段的影响，才能对模型的能力有更加完整和客观的评价。
为了更好地激发出模型在题目测试领域的能力，并引导模型按照一定的模板输出答案，OpenCompass采用提示词工程 （prompt engineering）和语境学习（in-context learning）进行客观评测。
在客观评测的具体实践中，我们通常采用下列两种方式进行模型输出结果的评测：
- 判别式评测：该评测方式基于将问题与候选答案组合在一起，计算模型在所有组合上的困惑度（perplexity），并选择困惑度最小的答案作为模型的最终输出。例如，若模型在 问题? 答案1 上的困惑度为 0.1，在 问题? 答案2 上的困惑度为 0.2，最终我们会选择 答案1 作为模型的输出。
- 生成式评测：该评测方式主要用于生成类任务，如语言翻译、程序生成、逻辑分析题等。具体实践时，使用问题作为模型的原始输入，并留白答案区域待模型进行后续补全。我们通常还需要对其输出进行后处理，以保证输出满足数据集的要求。
主观评测
语言表达生动精彩，变化丰富，大量的场景和能力无法凭借客观指标进行评测。针对如模型安全和模型语言能力的评测，以人的主观感受为主的评测更能体现模型的真实能力，并更符合大模型的实际使用场景。
OpenCompass采取的主观评测方案是指借助受试者的主观判断对具有对话能力的大语言模型进行能力评测。在具体实践中，我们提前基于模型的能力维度构建主观测试问题集合，并将不同模型对于同一问题的不同回复展现给受试者，收集受试者基于主观感受的评分。由于主观测试成本高昂，本方案同时也采用使用性能优异的大语言模拟人类进行主观打分。在实际评测中，本文将采用真实人类专家的主观评测与基于模型打分的主观评测相结合的方式开展模型能力评估。
在具体开展主观评测时，OpenComapss采用单模型回复满意度统计和多模型满意度比较两种方式开展具体的评测工作。
实践
安装
conda create --name opencompass --clone=/root/share/conda_envs/internlm-base
conda activate opencompass
git clone https://github.com/open-compass/opencompass
cd opencompass
pip install -e .
数据准备
cp /share/temp/datasets/OpenCompassData-core-20231110.zip /root/opencompass/
unzip OpenCompassData-core-20231110.zip
查看支持的数据集和模型
# 列出所有跟 internlm 及 ceval 相关的配置
python tools/list_configs.py internlm ceval
复制模型
cp -r /root/share/model_repos/internlm2-chat-7b /root/model
启动评测
python run.py --datasets ceval_gen --hf-path /root/model/internlm2-chat-7b/ --tokenizer-path /root/model/internlm2-chat-7b/ --tokenizer-kwargs padding_side='left' truncation='left' trust_remote_code=True --model-kwargs trust_remote_code=True device_map='auto' --max-seq-len 2048 --max-out-len 16 --batch-size 4 --num-gpus 1 --debug
命令解析
--datasets ceval_gen \
--hf-path /share/temp/model_repos/internlm-chat-7b/ \ # HuggingFace 模型路径
--tokenizer-path /share/temp/model_repos/internlm-chat-7b/ \ # HuggingFace tokenizer 路径（如果与模型路径相同，可以省略）
--tokenizer-kwargs padding_side='left' truncation='left' trust_remote_code=True \ # 构建 tokenizer 的参数
--model-kwargs device_map='auto' trust_remote_code=True \ # 构建模型的参数
--max-seq-len 2048 \ # 模型可以接受的最大序列长度
--max-out-len 16 \ # 生成的最大 token 数
--batch-size 2 \ # 批量大小
--num-gpus 1 # 运行模型所需的 GPU 数量
--debug
评测结果
20240301_214622
tabulate format
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dataset version metric mode opencompass.models.huggingface.HuggingFace_model_internlm2-chat-7b
---------------------------------------------- --------- ------------- ------ --------------------------------------------------------------------
ceval-computer_network db9ce2 accuracy gen 47.37
ceval-operating_system 1c2571 accuracy gen 57.89
ceval-computer_architecture a74dad accuracy gen 42.86
ceval-college_programming 4ca32a accuracy gen 51.35
ceval-college_physics 963fa8 accuracy gen 36.84
ceval-college_chemistry e78857 accuracy gen 33.33
ceval-advanced_mathematics ce03e2 accuracy gen 15.79
ceval-probability_and_statistics 65e812 accuracy gen 27.78
ceval-discrete_mathematics e894ae accuracy gen 18.75
ceval-electrical_engineer ae42b9 accuracy gen 40.54
ceval-metrology_engineer ee34ea accuracy gen 58.33
ceval-high_school_mathematics 1dc5bf accuracy gen 44.44
ceval-high_school_physics adf25f accuracy gen 47.37
ceval-high_school_chemistry 2ed27f accuracy gen 52.63
ceval-high_school_biology 8e2b9a accuracy gen 26.32
ceval-middle_school_mathematics bee8d5 accuracy gen 26.32
ceval-middle_school_biology 86817c accuracy gen 66.67
ceval-middle_school_physics 8accf6 accuracy gen 57.89
ceval-middle_school_chemistry 167a15 accuracy gen 95
ceval-veterinary_medicine b4e08d accuracy gen 39.13
ceval-college_economics f3f4e6 accuracy gen 47.27
ceval-business_administration c1614e accuracy gen 51.52
ceval-marxism cf874c accuracy gen 84.21
ceval-mao_zedong_thought 51c7a4 accuracy gen 70.83
ceval-education_science 591fee accuracy gen 72.41
ceval-teacher_qualification 4e4ced accuracy gen 79.55
ceval-high_school_politics 5c0de2 accuracy gen 21.05
ceval-high_school_geography 865461 accuracy gen 47.37
ceval-middle_school_politics 5be3e7 accuracy gen 42.86
ceval-middle_school_geography 8a63be accuracy gen 58.33
ceval-modern_chinese_history fc01af accuracy gen 65.22
ceval-ideological_and_moral_cultivation a2aa4a accuracy gen 89.47
ceval-logic f5b022 accuracy gen 54.55
ceval-law a110a1 accuracy gen 41.67
ceval-chinese_language_and_literature 0f8b68 accuracy gen 56.52
ceval-art_studies 2a1300 accuracy gen 69.7
ceval-professional_tour_guide 4e673e accuracy gen 86.21
ceval-legal_professional ce8787 accuracy gen 43.48
ceval-high_school_chinese 315705 accuracy gen 68.42
ceval-high_school_history 7eb30a accuracy gen 75
ceval-middle_school_history 48ab4a accuracy gen 68.18
ceval-civil_servant 87d061 accuracy gen 55.32
ceval-sports_science 70f27b accuracy gen 73.68
ceval-plant_protection 8941f9 accuracy gen 77.27
ceval-basic_medicine c409d6 accuracy gen 63.16
ceval-clinical_medicine 49e82d accuracy gen 45.45
ceval-urban_and_rural_planner 95b885 accuracy gen 58.7
ceval-accountant 002837 accuracy gen 44.9
ceval-fire_engineer bc23f5 accuracy gen 38.71
ceval-environmental_impact_assessment_engineer c64e2d accuracy gen 45.16
ceval-tax_accountant 3a5e3c accuracy gen 51.02
ceval-physician 6e277d accuracy gen 51.02
ceval-stem - naive_average gen 44.33
ceval-social-science - naive_average gen 57.54
ceval-humanities - naive_average gen 65.31
ceval-other - naive_average gen 54.94
ceval-hard - naive_average gen 34.62
ceval - naive_average gen 53.55
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$-------------------------------------------------------------------------------------------------------------------------------- THIS IS A DIVIDER --------------------------------------------------------------------------------------------------------------------------------csv format
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dataset,version,metric,mode,opencompass.models.huggingface.HuggingFace_model_internlm2-chat-7b
ceval-computer_network,db9ce2,accuracy,gen,47.37
ceval-operating_system,1c2571,accuracy,gen,57.89
ceval-computer_architecture,a74dad,accuracy,gen,42.86
ceval-college_programming,4ca32a,accuracy,gen,51.35
ceval-college_physics,963fa8,accuracy,gen,36.84
ceval-college_chemistry,e78857,accuracy,gen,33.33
ceval-advanced_mathematics,ce03e2,accuracy,gen,15.79
ceval-probability_and_statistics,65e812,accuracy,gen,27.78
ceval-discrete_mathematics,e894ae,accuracy,gen,18.75
ceval-electrical_engineer,ae42b9,accuracy,gen,40.54
ceval-metrology_engineer,ee34ea,accuracy,gen,58.33
ceval-high_school_mathematics,1dc5bf,accuracy,gen,44.44
ceval-high_school_physics,adf25f,accuracy,gen,47.37
ceval-high_school_chemistry,2ed27f,accuracy,gen,52.63
ceval-high_school_biology,8e2b9a,accuracy,gen,26.32
ceval-middle_school_mathematics,bee8d5,accuracy,gen,26.32
ceval-middle_school_biology,86817c,accuracy,gen,66.67
ceval-middle_school_physics,8accf6,accuracy,gen,57.89
ceval-middle_school_chemistry,167a15,accuracy,gen,95.00
ceval-veterinary_medicine,b4e08d,accuracy,gen,39.13
ceval-college_economics,f3f4e6,accuracy,gen,47.27
ceval-business_administration,c1614e,accuracy,gen,51.52
ceval-marxism,cf874c,accuracy,gen,84.21
ceval-mao_zedong_thought,51c7a4,accuracy,gen,70.83
ceval-education_science,591fee,accuracy,gen,72.41
ceval-teacher_qualification,4e4ced,accuracy,gen,79.55
ceval-high_school_politics,5c0de2,accuracy,gen,21.05
ceval-high_school_geography,865461,accuracy,gen,47.37
ceval-middle_school_politics,5be3e7,accuracy,gen,42.86
ceval-middle_school_geography,8a63be,accuracy,gen,58.33
ceval-modern_chinese_history,fc01af,accuracy,gen,65.22
ceval-ideological_and_moral_cultivation,a2aa4a,accuracy,gen,89.47
ceval-logic,f5b022,accuracy,gen,54.55
ceval-law,a110a1,accuracy,gen,41.67
ceval-chinese_language_and_literature,0f8b68,accuracy,gen,56.52
ceval-art_studies,2a1300,accuracy,gen,69.70
ceval-professional_tour_guide,4e673e,accuracy,gen,86.21
ceval-legal_professional,ce8787,accuracy,gen,43.48
ceval-high_school_chinese,315705,accuracy,gen,68.42
ceval-high_school_history,7eb30a,accuracy,gen,75.00
ceval-middle_school_history,48ab4a,accuracy,gen,68.18
ceval-civil_servant,87d061,accuracy,gen,55.32
ceval-sports_science,70f27b,accuracy,gen,73.68
ceval-plant_protection,8941f9,accuracy,gen,77.27
ceval-basic_medicine,c409d6,accuracy,gen,63.16
ceval-clinical_medicine,49e82d,accuracy,gen,45.45
ceval-urban_and_rural_planner,95b885,accuracy,gen,58.70
ceval-accountant,002837,accuracy,gen,44.90
ceval-fire_engineer,bc23f5,accuracy,gen,38.71
ceval-environmental_impact_assessment_engineer,c64e2d,accuracy,gen,45.16
ceval-tax_accountant,3a5e3c,accuracy,gen,51.02
ceval-physician,6e277d,accuracy,gen,51.02
ceval-stem,-,naive_average,gen,44.33
ceval-social-science,-,naive_average,gen,57.54
ceval-humanities,-,naive_average,gen,65.31
ceval-other,-,naive_average,gen,54.94
ceval-hard,-,naive_average,gen,34.62
ceval,-,naive_average,gen,53.55
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$-------------------------------------------------------------------------------------------------------------------------------- THIS IS A DIVIDER --------------------------------------------------------------------------------------------------------------------------------raw format
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-------------------------------
Model: opencompass.models.huggingface.HuggingFace_model_internlm2-chat-7b
ceval-computer_network: {'accuracy': 47.368421052631575}
ceval-operating_system: {'accuracy': 57.89473684210527}
ceval-computer_architecture: {'accuracy': 42.857142857142854}
ceval-college_programming: {'accuracy': 51.35135135135135}
ceval-college_physics: {'accuracy': 36.84210526315789}
ceval-college_chemistry: {'accuracy': 33.33333333333333}
ceval-advanced_mathematics: {'accuracy': 15.789473684210526}
ceval-probability_and_statistics: {'accuracy': 27.77777777777778}
ceval-discrete_mathematics: {'accuracy': 18.75}
ceval-electrical_engineer: {'accuracy': 40.54054054054054}
ceval-metrology_engineer: {'accuracy': 58.333333333333336}
ceval-high_school_mathematics: {'accuracy': 44.44444444444444}
ceval-high_school_physics: {'accuracy': 47.368421052631575}
ceval-high_school_chemistry: {'accuracy': 52.63157894736842}
ceval-high_school_biology: {'accuracy': 26.31578947368421}
ceval-middle_school_mathematics: {'accuracy': 26.31578947368421}
ceval-middle_school_biology: {'accuracy': 66.66666666666666}
ceval-middle_school_physics: {'accuracy': 57.89473684210527}
ceval-middle_school_chemistry: {'accuracy': 95.0}
ceval-veterinary_medicine: {'accuracy': 39.130434782608695}
ceval-college_economics: {'accuracy': 47.27272727272727}
ceval-business_administration: {'accuracy': 51.515151515151516}
ceval-marxism: {'accuracy': 84.21052631578947}
ceval-mao_zedong_thought: {'accuracy': 70.83333333333334}
ceval-education_science: {'accuracy': 72.41379310344827}
ceval-teacher_qualification: {'accuracy': 79.54545454545455}
ceval-high_school_politics: {'accuracy': 21.052631578947366}
ceval-high_school_geography: {'accuracy': 47.368421052631575}
ceval-middle_school_politics: {'accuracy': 42.857142857142854}
ceval-middle_school_geography: {'accuracy': 58.333333333333336}
ceval-modern_chinese_history: {'accuracy': 65.21739130434783}
ceval-ideological_and_moral_cultivation: {'accuracy': 89.47368421052632}
ceval-logic: {'accuracy': 54.54545454545454}
ceval-law: {'accuracy': 41.66666666666667}
ceval-chinese_language_and_literature: {'accuracy': 56.52173913043478}
ceval-art_studies: {'accuracy': 69.6969696969697}
ceval-professional_tour_guide: {'accuracy': 86.20689655172413}
ceval-legal_professional: {'accuracy': 43.47826086956522}
ceval-high_school_chinese: {'accuracy': 68.42105263157895}
ceval-high_school_history: {'accuracy': 75.0}
ceval-middle_school_history: {'accuracy': 68.18181818181817}
ceval-civil_servant: {'accuracy': 55.319148936170215}
ceval-sports_science: {'accuracy': 73.68421052631578}
ceval-plant_protection: {'accuracy': 77.27272727272727}
ceval-basic_medicine: {'accuracy': 63.1578947368421}
ceval-clinical_medicine: {'accuracy': 45.45454545454545}
ceval-urban_and_rural_planner: {'accuracy': 58.69565217391305}
ceval-accountant: {'accuracy': 44.89795918367347}
ceval-fire_engineer: {'accuracy': 38.70967741935484}
ceval-environmental_impact_assessment_engineer: {'accuracy': 45.16129032258064}
ceval-tax_accountant: {'accuracy': 51.02040816326531}
ceval-physician: {'accuracy': 51.02040816326531}
ceval-stem: {'naive_average': 44.330303885938896}
ceval-social-science: {'naive_average': 57.54025149079596}
ceval-humanities: {'naive_average': 65.30999398082602}
ceval-other: {'naive_average': 54.944902032059396}
ceval-hard: {'naive_average': 34.6171418128655}
ceval: {'naive_average': 53.554085553239936}
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
