返回列表
LLMxAICC每日性能基线
2023-10-08
浏览量:9997
# 写在前面 update@2024.1.16: 对于目前的社区顶流MoE模型的推理,已经基于AICC完成推理的开箱适配(以DeepSeek-MoE-16B为例),并在下文中提供性能参考。 update@2023.10.18: 对于当前的64GB/32GBA芯片,推荐使用CANN7.0+MindSpore2.2/PyTorch2.1的镜像(已经全量上架AICC镜像列表),较老版本镜像可能存在性能与精度问题。 update@2023.10.17: 64GB+CANN7.0+PyTorch2.1+Transformer4.32.0已经支持LLM的开箱即用(如qwen等)。 first@2023.10.9: 本文提供武汉AICC运营团队基于各架构硬件进行性能实测的结果,作为参考 * 文章中性能结果为AICC运营团队复现的结果,*不代表该模型在芯片上的最佳性能表现,最佳性能表现与并行策略、版本等相关,以官方发布数据为准* * 部分性能结果还在施工中 * 未明确指定镜像的模型,请在镜像列表选取最新的对应MindSpore/PyTorch镜像 * 未提供相关的模型信息可以联系武汉AICC运营团队人员进行反馈 # 参考资料一览 * [镜像仓库](http://mirrors.cn-central-221.ovaijisuan.com):获取最新的配套镜像 * [MindFormers仓库](https://gitee.com/mindspore/mindformers/tree/dev/research/baichuan2):MindSpore大模型开发的套件 * [MindFormers手册](https://mindformers.readthedocs.io/zh\_CN/latest/):MindFormer相关手册、接口 * [AscendPyTorch](https://gitee.com/ascend/pytorch):Ascend+PyTorch代码仓库 * [AscendSpeed](https://gitee.com/ascend/AscendSpeed):Ascend+PyTorch大模型开发套件 * [Ascend ModelZoo](https://www.hiascend.com/software/modelzoo/big-models?activeTab=computer-vision):Ascend已适配模型仓库 # 训练场景 | 模型 | 芯片型号 | 状态 | 框架 | 卡数 | 性能 | 环境 | 环境镜像 | 参考文档 | | ------- | -------- | -------- | --------- | ---- | -------- | ---- | -------- | -------- | | ChatGLM-6B | 32GB | OK | MindSpore | 1*8 | - | MA/BMS | - | - | | ChatGLM-6B | 64GB | OK | PyTorch | 1*8 | - | BMS | [HERE](/detail/104.html) | [HERE](/detail/104.html#Chatglm-6B) | | Baichuan2-7B | 32GB | OK | MindSpore | 1*8 | - | MA/BMS | - | - | | Baichuan2-13B | 64GB | OK | MindSpore | 1*8 | 563.2tokens/s/p | MA/BMS | [HERE](detail/107.html) | [HERE](https://gitee.com/mindspore/mindformers/blob/dev/research/baichuan2/baichuan2.md#64GB-2) | | Baichuan2-7B | 64GB | OK | PyTorch | 1*8 | - | BMS | [HERE](/detail/104.html) | [HERE](/detail/104.html) | | Baichuan2-13B | 64GB | OK | PyTorch | 1*8 | 1124.6tokens/s/p | BMS | [HERE](/detail/105.html) | [HERE](https://gitee.com/ascend/AscendSpeed/blob/master/examples/baichuan/README.md#accuracy-of-the-loss-1) | # 推理场景 | 模型 | 芯片型号 | 状态 | 框架 | 卡数 | 性能 | 环境 | 环境镜像 | 参考文档 | | ------- | -------- | -------- | --------- | ---- | -------- | ---- | -------- | -------- | | ChatGLM-6B | 32GB | OK | MindSpore | 1*1 | 22.4tokens/s/p | MA/BMS | - | - | | ChatGLM-6B | 32GB | OK | PyTorch | 1*1 | 0.24tokens/s/p | MA/BMS | [HERE](/detail/69.html) | - | | ChatGLM2-6B | 64GB | OK | MindSpore | 1*1 | 24.1tokens/s/p | MA | [HERE](/detail/103.html) | - | |ChatGLM3-6B | 32GB | OK | MindSpore | 1*1 | 28.86tokens/s/p | MA | - | - | | Baichuan2-7B | 32GB | OK | MindSpore | 1*1 | - | MA/BMS | [HERE](/detail/103.html) | - | | Baichuan2-7B | 64GB | OK | MindSpore| 1*1 | - | MA/BMS | [HERE](/detail/103.html) | - | | Baichuan2-7B | 64GB | OK | PyTorch| 1*1 | - | MA/BMS | [HERE](/detail/104.html) | [HERE](/detail/104.html#ChatGLM-6B) | | Baichuan2-13B | 64GB | OK | MindSpore | 1*1 | 14.1tokens/s/p | MA | [HERE](/detail/107.html) | [HERE](https://gitee.com/mindspore/mindformers/blob/dev/research/baichuan2/baichuan2.md#64GB-3) | | Baichuan2-13B | 64GB | OK | PyTorch | 1*1 | - | BMS | - | - | | QWEN-14B | 64GB | OK | PyTorch | 1*1 | - | BMS | [HERE](/detail/106.html) | - | | QWEN-7B | 64GB | OK | PyTorch | 1*1 | - | BMS | [HERE](/detail/106.html) | - | | ChatGLM3-6B | 64GB | OK | PyTorch | 1*1 | 16.48tokens/s/p | BMS | - | - | | Yi-6B-200K | 64GB | OK | PyTorch | 1*1 | 14.67tokens/s/p | BMS | [HERE](/detail/106.html) | - | | DeepSeek-MoE-16B | 64GB | OK | PyTorch | 1*1 | 4.01tokens/s/p | BMS | [HERE](/detail/106.html) | - | # LoRA | 模型 | 芯片型号 | 状态 | 框架 | 卡数 | 性能 | 环境 | 环境镜像 | 参考文档 | | ------- | -------- | -------- | --------- | ---- | -------- | ---- | -------- | -------- | | ChatGLM-6B | 32GB | OK | MindSpore | 1*1 | - | NA/BMS | - | - | | ChatGLM-6B | 32GB | OK | PyTorch | 1*1 | - | MA/BMS | [HERE](/detail/81.html) | - | | ChatGLM-6B | 64GB | OK | PyTorch | 1*1 | - | BMS | [HERE](/detail/104.html) | [HERE](/detail/104.html#ChatGLM-6B) | | Baichuan2-7B | 32GB | OK | MindSpore | 1*1 | - | MA/BMS | - | - | # 模型镜像 因为众所周知的原因,在国内通过HuggingFace直接拉取模型可能会因为一系列魔法而导致失败,根据我们实际的测试情况,可以基于ModelScope等第三方平台拉取模型,这里在武汉AICC上也提供了常见模型的内部镜像源,可以通过[obsutils](https://support.huaweicloud.com/utiltg-obs/obs\_11\_0003.html)组件直接拉取。 | 模型 | 来源 | 版本 | 推荐框架 | 模型镜像地址 | | ------- | -------- | -------- | --------- | ---- | | ChatGLM-6B | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/huggingface/ChatGLM/ChatGLM-6B/ | | ChatGLM2-6B | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/huggingface/ChatGLM/ChatGLM2-6B/ | | ChatGLM3-6B | ModelScope | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/huggingface/ChatGLM/ChatGLM3-6B/ | | Yi-6B-200K | ModelScope | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/huggingface/Yi-6B-200K/ | | baichuan-7B | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/baichuan-7B/ | | Baichuan-13B-Chat | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/huggingface/Baichuan/Baichuan-13B-Chat/ | | Baichuan-13B-Base | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/huggingface/Baichuan/Baichuan-13B-Base/ | | Baichuan2-7B-Base | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/Baichuan2-7B-Base/ | | Baichuan2-13B-Base | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/Baichuan2-13B-Base/ | | LLaMA-7B | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/llama-7b-hf/ | | LLaMA-13B | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/llama-13b-hf/ | | LLaMA-33B | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/llama-33b-hf/ | | LLaMA-65B | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/llama-65b-hf/ | | LLaMA2-7B | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/llama-2-7b-hf/ | | LLaMA2-13B | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/Llama-2-13b-hf/ | | LLaMA2-70B-Chat | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/Llama-2-70b-chat-ms/ | | 鹏城.脑海-7B | OpenI | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/huggingface/PengCheng-7B/ | | SAM | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/huggingface/SAM/ | | Qwen-72B | ModelScope | 1.0.4 | PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/Qwen-72B/ | | QWEN-14B-Chat | ModelScope | 1.0.4 | PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/Qwen-14B/ | | QWEN-7B-Chat | ModelScope | 1.0.4 | PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/Qwen-7B/| | Qwen1.5-72B | ModelScope | 1.0.4 | PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/Qwen1.5-72B/ | | Vicuna-7B | Huggingface | 1.5 | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/huggingface/vicuna/ | | Aquila-7B | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/Aquila-7B/ | | Mixtral-8x7B | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/Mixtral-8x7B-v0.1/ | | Bloom-7B | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/bloom-7b1/ | | Internlm-7B | Huggingface | - | MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/internlm-7b/ | | Meta-Llama-3-8B-Instruct | ModelScope | - | PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/Meta-Llama-3-8B-Instruct/ | | Meta-Llama-3-70B-Instruct | ModelScope | - | PyTorch | obs://obs-whaicc-fae-public/checkpoint/ModelLink/Meta-Llama-3-70B-Instruct/ | |Qwen-VL-Chat | Huggingface | - | PyTorch | obs://obs-whaicc-fae-public/checkpoint/huggingface/Qwen/Qwen-VL-Chat/| |Qwen1.5-0.5B | Huggingface | - | PyTorch | obs://obs-whaicc-fae-public/checkpoint/huggingface/Qwen/Qwen1.5-0.5B/| |Qwen1.5-7B | Huggingface | - | PyTorch | obs://obs-whaicc-fae-public/checkpoint/huggingface/Qwen/Qwen1.5-7B/| |gpt2-xl | Huggingface | - | MindSpore/yTorch | obs://obs-whaicc-fae-public/checkpoint/huggingface/gpt/gpt2-xl/| |gpt2 | Huggingface | - |MindSpore/PyTorch | obs://obs-whaicc-fae-public/checkpoint/huggingface/gpt/gpt2/| |CodeFuse-DeepSeek-33B | ModelScope | - | PyTorch | obs://obs-whaicc-fae-public/checkpoint/huggingface/CodeFuse-DeepSeek-33B/| # 测试数据集 ``` ```