返回列表
LLMxAICC每日性能基线
2023-10-08
浏览量：12733
# 写在前面
update@2024.1.16:
对于目前的社区顶流MoE模型的推理，已经基于AICC完成推理的开箱适配（以DeepSeek-MoE-16B为例），并在下文中提供性能参考。
update@2023.10.18:
对于当前的64GB/32GBA芯片，推荐使用CANN7.0+MindSpore2.2/PyTorch2.1的镜像（已经全量上架AICC镜像列表），较老版本镜像可能存在性能与精度问题。
update@2023.10.17:
64GB+CANN7.0+PyTorch2.1+Transformer4.32.0已经支持LLM的开箱即用（如qwen等）。
first@2023.10.9:
本文提供武汉AICC运营团队基于各架构硬件进行性能实测的结果，作为参考
* 文章中性能结果为AICC运营团队复现的结果，*不代表该模型在芯片上的最佳性能表现，最佳性能表现与并行策略、版本等相关，以官方发布数据为准*
* 部分性能结果还在施工中
* 未明确指定镜像的模型，请在镜像列表选取最新的对应MindSpore/PyTorch镜像
* 未提供相关的模型信息可以联系武汉AICC运营团队人员进行反馈
# 参考资料一览
* [镜像仓库](http://mirrors.cn-central-221.ovaijisuan.com)：获取最新的配套镜像
* [MindFormers仓库](https://gitee.com/mindspore/mindformers/tree/dev/research/baichuan2)：MindSpore大模型开发的套件
* [MindFormers手册](https://mindformers.readthedocs.io/zh\_CN/latest/)：MindFormer相关手册、接口
* [AscendPyTorch](https://gitee.com/ascend/pytorch)：Ascend+PyTorch代码仓库
* [AscendSpeed](https://gitee.com/ascend/AscendSpeed)：Ascend+PyTorch大模型开发套件
* [Ascend ModelZoo](https://www.hiascend.com/software/modelzoo/big-models?activeTab=computer-vision)：Ascend已适配模型仓库
# 训练场景
| 模型 | 芯片型号 | 状态 | 框架 | 卡数 | 性能 | 环境 | 环境镜像 | 参考文档 |
| ------- | -------- | -------- | --------- | ---- | -------- | ---- | -------- | -------- |
| ChatGLM-6B | 32GB | OK | MindSpore | 1*8 | - | MA/BMS | - | - |
| ChatGLM-6B | 64GB | OK | PyTorch | 1*8 | - | BMS | [HERE](/detail/104.html) | [HERE](/detail/104.html#Chatglm-6B) |
| Baichuan2-7B | 32GB | OK | MindSpore | 1*8 | - | MA/BMS | - | - |
| Baichuan2-13B | 64GB | OK | MindSpore | 1*8 | 563.2tokens/s/p | MA/BMS | [HERE](detail/107.html) | [HERE](https://gitee.com/mindspore/mindformers/blob/dev/research/baichuan2/baichuan2.md#64GB-2) |
| Baichuan2-7B | 64GB | OK | PyTorch | 1*8 | - | BMS | [HERE](/detail/104.html) | [HERE](/detail/104.html) |
| Baichuan2-13B | 64GB | OK | PyTorch | 1*8 | 1124.6tokens/s/p | BMS | [HERE](/detail/105.html) | [HERE](https://gitee.com/ascend/AscendSpeed/blob/master/examples/baichuan/README.md#accuracy-of-the-loss-1) |
# 推理场景
| 模型 | 芯片型号 | 状态 | 框架 | 卡数 | 性能 | 环境 | 环境镜像 | 参考文档 |
| ------- | -------- | -------- | --------- | ---- | -------- | ---- | -------- | -------- |
| ChatGLM-6B | 32GB | OK | MindSpore | 1*1 | 22.4tokens/s/p | MA/BMS | - | - |
| ChatGLM-6B | 32GB | OK | PyTorch | 1*1 | 0.24tokens/s/p | MA/BMS | [HERE](/detail/69.html) | - |
| ChatGLM2-6B | 64GB | OK | MindSpore | 1*1 | 24.1tokens/s/p | MA | [HERE](/detail/103.html) | - |
|ChatGLM3-6B | 32GB | OK | MindSpore | 1*1 | 28.86tokens/s/p | MA | - | - |
| Baichuan2-7B | 32GB | OK | MindSpore | 1*1 | - | MA/BMS | [HERE](/detail/103.html) | - |
| Baichuan2-7B | 64GB | OK | MindSpore| 1*1 | - | MA/BMS | [HERE](/detail/103.html) | - |
| Baichuan2-7B | 64GB | OK | PyTorch| 1*1 | - | MA/BMS | [HERE](/detail/104.html) | [HERE](/detail/104.html#ChatGLM-6B) |
| Baichuan2-13B | 64GB | OK | MindSpore | 1*1 | 14.1tokens/s/p | MA | [HERE](/detail/107.html) | [HERE](https://gitee.com/mindspore/mindformers/blob/dev/research/baichuan2/baichuan2.md#64GB-3) |
| Baichuan2-13B | 64GB | OK | PyTorch | 1*1 | - | BMS | - | - |
| QWEN-14B | 64GB | OK | PyTorch | 1*1 | - | BMS | [HERE](/detail/106.html) | - |
| QWEN-7B | 64GB | OK | PyTorch | 1*1 | - | BMS | [HERE](/detail/106.html) | - |
| ChatGLM3-6B | 64GB | OK | PyTorch | 1*1 | 16.48tokens/s/p | BMS | - | - |
| Yi-6B-200K | 64GB | OK | PyTorch | 1*1 | 14.67tokens/s/p | BMS | [HERE](/detail/106.html) | - |
| DeepSeek-MoE-16B | 64GB | OK | PyTorch | 1*1 | 4.01tokens/s/p | BMS | [HERE](/detail/106.html) | - |
# LoRA
| 模型 | 芯片型号 | 状态 | 框架 | 卡数 | 性能 | 环境 | 环境镜像 | 参考文档 |
| ------- | -------- | -------- | --------- | ---- | -------- | ---- | -------- | -------- |
| ChatGLM-6B | 32GB | OK | MindSpore | 1*1 | - | NA/BMS | - | - |
| ChatGLM-6B | 32GB | OK | PyTorch | 1*1 | - | MA/BMS | [HERE](/detail/81.html) | - |
| ChatGLM-6B | 64GB | OK | PyTorch | 1*1 | - | BMS | [HERE](/detail/104.html) | [HERE](/detail/104.html#ChatGLM-6B) |
| Baichuan2-7B | 32GB | OK | MindSpore | 1*1 | - | MA/BMS | - | - |
# 模型镜像
因为众所周知的原因，在国内通过HuggingFace直接拉取模型可能会因为一系列魔法而导致失败，根据我们实际的测试情况，可以基于ModelScope等第三方平台拉取模型，这里在武汉AICC上也提供了常见模型的内部镜像源，可以通过[obsutils](https://support.huaweicloud.com/utiltg-obs/obs\_11\_0003.html)组件直接拉取。
| 模型 | 来源 | 版本 | 推荐框架 | 模型镜像地址 |
| ------- | -------- | -------- | --------- | ---- |
| ChatGLM-6B | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/huggingface/ChatGLM/ChatGLM-6B/ |
| ChatGLM2-6B | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/huggingface/ChatGLM/ChatGLM2-6B/ |
| ChatGLM3-6B | ModelScope | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/huggingface/ChatGLM/ChatGLM3-6B/ |
| Yi-6B-200K | ModelScope | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/huggingface/Yi-6B-200K/ |
| baichuan-7B | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/baichuan-7B/ |
| Baichuan-13B-Chat | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/huggingface/Baichuan/Baichuan-13B-Chat/ |
| Baichuan-13B-Base | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/huggingface/Baichuan/Baichuan-13B-Base/ |
| Baichuan2-7B-Base | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/Baichuan2-7B-Base/ |
| Baichuan2-13B-Base | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/Baichuan2-13B-Base/ |
| LLaMA-7B | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/llama-7b-hf/ |
| LLaMA-13B | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/llama-13b-hf/ |
| LLaMA-33B | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/llama-33b-hf/ |
| LLaMA-65B | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/llama-65b-hf/ |
| LLaMA2-7B | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/llama-2-7b-hf/ |
| LLaMA2-13B | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/Llama-2-13b-hf/ |
| LLaMA2-70B-Chat | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/Llama-2-70b-chat-ms/ |
| 鹏城.脑海-7B | OpenI | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/huggingface/PengCheng-7B/ |
| SAM | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/huggingface/SAM/ |
| Qwen-72B | ModelScope | 1.0.4 | PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/Qwen-72B/ |
| QWEN-14B-Chat | ModelScope | 1.0.4 | PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/Qwen-14B/ |
| QWEN-7B-Chat | ModelScope | 1.0.4 | PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/Qwen-7B/|
| Qwen1.5-72B | ModelScope | 1.0.4 | PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/Qwen1.5-72B/ |
| Vicuna-7B | Huggingface | 1.5 | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/huggingface/vicuna/ |
| Aquila-7B | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/Aquila-7B/ |
| Mixtral-8x7B | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/Mixtral-8x7B-v0.1/ |
| Bloom-7B | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/bloom-7b1/ |
| Internlm-7B | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/internlm-7b/ |
| Meta-Llama-3-8B-Instruct | ModelScope | - | PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/Meta-Llama-3-8B-Instruct/ |
| Meta-Llama-3-70B-Instruct | ModelScope | - | PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/Meta-Llama-3-70B-Instruct/ |
|Qwen-VL-Chat | Huggingface | - | PyTorch | obs://obs-whaicc-fae-public/checkpoint/huggingface/Qwen/Qwen-VL-Chat/|
|Qwen1.5-0.5B | Huggingface | - | PyTorch | obs://obs-whaicc-fae-public/checkpoint/huggingface/Qwen/Qwen1.5-0.5B/|
|Qwen1.5-7B | Huggingface | - | PyTorch | obs://obs-whaicc-fae-public/checkpoint/huggingface/Qwen/Qwen1.5-7B/|
|gpt2-xl | Huggingface | - | MindSpore/yTorch | obs://obs-whaicc-fae-public/checkpoint/huggingface/gpt/gpt2-xl/|
|gpt2 | Huggingface | - |MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/huggingface/gpt/gpt2/|
|CodeFuse-DeepSeek-33B | ModelScope | - | PyTorch | obs://obs-whaicc-fae-public/checkpoint/huggingface/CodeFuse-DeepSeek-33B/|
# 测试数据集
```
```