# MindSpore大模型报错解决地图

-   [MindSpore常见大模型问题和解决方案](MindSpore常见大模型问题和解决方案.md)
-   [环境问题案例](环境问题案例.md)
    -   [MindSpore和Flash attention特性的适配问题](MindSpore和Flash-attention特性的适配问题.md)
    -   [MindSpore保存模型出现维度不对应问题](MindSpore保存模型出现维度不对应问题.md)
    -   [MindSpore开启profiler功能报错IndexError:list index out of range](MindSpore开启profiler功能报错IndexError-list-index-out-of-range.md)
    -   [MindSpore多机运行Profiler报错ValueError: not enough values to unpack \(expected 4, got 0\)](MindSpore多机运行Profiler报错ValueError-not-enough-values-to-unpack-(expected-4-got-0).md)
    -   [MindSpore开启summary功能失效](MindSpore开启summary功能失效.md)
    -   [MindSpore模型权重功能无法保存更新后的权重](MindSpore模型权重功能无法保存更新后的权重.md)
    -   [MindSpore开启MS\_DISABLE\_REF\_MODE导致报错The device address type is wrong:type name in address:CPU,type name in context:Ascend](MindSpore开启MS_DISABLE_REF_MODE导致报错The-device-address-type-is-wrong-type-name-in-address-CPU-type-nam.md)
    -   [MindSpore和tbe环境相关报错](MindSpore和tbe环境相关报错.md)
    -   [Ascend上构建MindSpore报头文件缺失](Ascend上构建MindSpore报头文件缺失.md)
    -   [MindSpore开启profile，使用并行策略报错ValueError: When dıstrıbuted loads are slıced we1ghts, sınk mode must be set True.](MindSpore开启profile-使用并行策略报错ValueError-When-d-str-buted-loads-are-sl-ced-we1ghts-s-nk-mode-must-be-se.md)
    -   [模型启动时报Malloc device memory failed](模型启动时报Malloc-device-memory-failed.md)
    -   [MindSpore盘古模型报错Failed to allocate memory.Possible Cause: Available memory is insufficient.](MindSpore盘古模型报错Failed-to-allocate-memory-Possible-Cause-Available-memory-is-insufficient.md)
    -   [Ascend910环境分离部署时请求超时](Ascend910环境分离部署时请求超时.md)

-   [配置问题案例](配置问题案例.md)
    -   [MindSpore报错ImportError cannot import name "build dataset loader' from 'mindformers.dataset. dataloader'](MindSpore报错ImportError-cannot-import-name-build-dataset-loader-from-mindformers-dataset-dataloader.md)
    -   [MindSpore大模型报错xxx is not a supported default model or a valid path to checkpoint.](MindSpore大模型报错xxx-is-not-a-supported-default-model-or-a-valid-path-to-checkpoint.md)
    -   [llama2模型转换报错ImportError: cannot import name 'swap\_cache' from 'mindspore.\_c\_expression'](llama2模型转换报错ImportError-cannot-import-name-swap_cache-from-mindspore-_c_expression.md)
    -   [MindSpore报错报错BrokenPipeError: \[Errno 32\] Broken pipe](MindSpore报错报错BrokenPipeError-Errno-32-Broken-pipe.md)
    -   [昇腾910上CodeLlama报错Session options is not equal in diff config infos when models' weights are shared, last session options](昇腾910上CodeLlama报错Session-options-is-not-equal-in-diff-config-infos-when-models-weights-are-shared-la.md)
    -   [集成通信库初始化init\(\)报错要求使用mpirun启动多卡训练](集成通信库初始化init()报错要求使用mpirun启动多卡训练.md)
    -   [INFNAN模式溢出问题](INFNAN模式溢出问题.md)

-   [数据处理案例](数据处理案例.md)
    -   [Mindrecoder 格式转换报错ValueError: For 'Mul'. x.shape and y.shape need to broadcast. The value of x. shapel-2\] or y.shape\[-2\] must be 1 or -1 when they a are not the same, but got x.shape = \[2, 2047,2047\] and y.shape = \[1. 2048, 2048\]](Mindrecoder-格式转换报错ValueError-For-Mul-x-shape-and-y-shape-need-to-broadcast-The-value-of-x-shapel-2-o.md)
    -   [数据处理成Mindrecord数据时出现ImportError: cannot import name 'tik' from 'te'](数据处理成Mindrecord数据时出现ImportError-cannot-import-name-tik-from-te.md)
    -   [MindSpore报错Failed to combine the net and the parameters for param backbone.embedding.word\_embedding.embedding\_table.](MindSpore报错Failed-to-combine-the-net-and-the-parameters-for-param-backbone-embedding-word_embedding.md)
    -   [MindSpore训练报错TypeError: Invalid Kernel Build Info! Kernel type: AICPU\_KERNEL, node: Default/Concat-op1](MindSpore训练报错TypeError-Invalid-Kernel-Build-Info-Kernel-type-AICPU_KERNEL-node-Default-Concat-op1.md)
    -   [MindSpore模型正向报错Sync stream failed:Ascend\_0，plog显示Gather算子越界](MindSpore模型正向报错Sync-stream-failed-Ascend_0-plog显示Gather算子越界.md)
    -   [model.train报错Exception in training: The input value must be int and must \> 0, but got '0' with type 'int'.](model-train报错Exception-in-training-The-input-value-must-be-int-and-must-0-but-got-0-with-type-int.md)
    -   [数据集异常导致编译\(model.build\)或者训练\(model.train\)卡住](数据集异常导致编译(model-build)或者训练(model-train)卡住.md)
    -   [baichaun2-13b 在Ascend910上持续溢出](baichaun2-13b-在Ascend910上持续溢出.md)
    -   [MTP数据集分布式读写锁死，Failed to execute the sql \[SELECT NAME from SHARD NAME;\] while verifying meta file, database is locked\]](MTP数据集分布式读写锁死-Failed-to-execute-the-sql-SELECT-NAME-from-SHARD-NAME-while-verifying-meta-file-databa.md)
    -   [MTP任务卡死，平台报错信息\['ROOT\_CLUSTER'\] job failed.](MTP任务卡死-平台报错信息-ROOT_CLUSTER-job-failed.md)
    -   [大模型获取性能数据方法](大模型获取性能数据方法.md)
    -   [模型解析性能数据](模型解析性能数据.md)
    -   [数据集处理结果精细对比](数据集处理结果精细对比.md)
    -   [通过优化数据来加速训练速度](通过优化数据来加速训练速度.md)

-   [并行问题案例](并行问题案例.md)
    -   [MindSpore分布式模型并行报错：operator Mul init failed或者CheckStrategy failed.](MindSpore分布式模型并行报错-operator-Mul-init-failed或者CheckStrategy-failed.md)
    -   [MindSpore跑模型并行报错ValueError: array split does not result in an equal division](MindSpore跑模型并行报错ValueError-array-split-does-not-result-in-an-equal-division.md)
    -   [MindSpore训练大模型报错：BrokenPipeError: \[Errno 32\] Broken pipe, EOFError](MindSpore训练大模型报错-BrokenPipeError-Errno-32-Broken-pipe-EOFError.md)
    -   [MindSpore大模型并行需要在对应的yaml里面做哪些配置](MindSpore大模型并行需要在对应的yaml里面做哪些配置.md)
    -   [模型并行显示内存溢出](模型并行显示内存溢出.md)
    -   [MindSpore模型Pipeline并行发现有些卡的log中loss为0](MindSpore模型Pipeline并行发现有些卡的log中loss为0.md)
    -   [MindSpore8卡报Socket times out问题](MindSpore8卡报Socket-times-out问题.md)
    -   [MindSpore模型Pipeline并行训练报错RuntimeError: Stage 0 should has at least 1 parameter. but got none.](MindSpore模型Pipeline并行训练报错RuntimeError-Stage-0-should-has-at-least-1-parameter-but-got-none.md)
    -   [并行策略为8:1:1时报错RuntimeError: May you need to check if the batch size etc. in your 'net' and 'parameter dict' are same.](并行策略为8-1-1时报错RuntimeError-May-you-need-to-check-if-the-batch-size-etc-in-your-net-and-parameter-dict.md)
    -   [模型并行策略为 1:1:8 时报错RuntimeError: Stage num is 8 is not equal to stage used: 5](模型并行策略为-1-1-8-时报错RuntimeError-Stage-num-is-8-is-not-equal-to-stage-used-5.md)
    -   [MindSpore报错：wq.weight in the argument 'net' should have the same shape as wq.weight in the argument 'parameter\_dict'.](MindSpore报错-wq-weight-in-the-argument-net-should-have-the-same-shape-as-wq-weight-in-the-argument-pa.md)
    -   [MindSpore跑分布式报错TypeError: The parameters number of the function is 636, but the number of provided arguments is 635.](MindSpore跑分布式报错TypeError-The-parameters-number-of-the-function-is-636-but-the-number-of-provided-arg.md)
    -   [MindSpore数据并行报错Call GE RunGraphWithStreamAsync Failed，EL0004: Failed to allocate memory.](MindSpore数据并行报错Call-GE-RunGraphWithStreamAsync-Failed-EL0004-Failed-to-allocate-memory.md)
    -   [MindSpore分布式8节点报错Call GE RunGraphWithStreamAsync Failed, ret is: 4294967295](MindSpore分布式8节点报错Call-GE-RunGraphWithStreamAsync-Failed-ret-is-4294967295.md)
    -   [MindFormers进行单机八卡调用时报错No parameter is entered. Notice that the program will run on default 8 cards.](MindFormers进行单机八卡调用时报错No-parameter-is-entered-Notice-that-the-program-will-run-on-default-8-cards.md)
    -   [MindSpore分布式并行报错The strategy is XXX, shape XXX cannot be divisible by strategy value XXX](MindSpore分布式并行报错The-strategy-is-XXX-shape-XXX-cannot-be-divisible-by-strategy-value-XXX.md)
    -   [MTP使用多进程生成mindrecord，报错RuntimeError: Unexpected error. \[Internal ERROR\] Failed to write mindrecord meta files.](MTP使用多进程生成mindrecord-报错RuntimeError-Unexpected-error-Internal-ERROR-Failed-to-write-mindrecord-meta.md)
    -   [流水线并行报错Reshape op can't be a border.](流水线并行报错Reshape-op-can-t-be-a-border.md)
    -   [增加数据并行数之后模型占用显存增加](增加数据并行数之后模型占用显存增加.md)
    -   [MindSpore分布式ckpt权重A转换为其他策略的分布式权重B](MindSpore分布式ckpt权重A转换为其他策略的分布式权重B.md)
    -   [流水线并行没开Cell共享导致编译时间很长](流水线并行没开Cell共享导致编译时间很长.md)
    -   [多机训练报错：import torch\_npu.\_C ImportError: libascend\_hal.so: cannot open shared object file: No such file or directory](多机训练报错-import-torch_npu-_C-ImportError-libascend_hal-so-cannot-open-shared-object-file-No-such-file.md)
    -   [docker执行报错：RuntimeError: Maybe you are trying to call 'mindspore.communication.init\(\)' without using 'mpirun'](docker执行报错-RuntimeError-Maybe-you-are-trying-to-call-mindspore-communication-init()-without-using-mp.md)

-   [训练推理问题案例](训练推理问题案例.md)
    -   [MindSpore报错：TypeError: Multiply values for specific argument: query\_embeds](MindSpore报错-TypeError-Multiply-values-for-specific-argument-query_embeds.md)
    -   [Tokenizer指向报错TypeError GPT2Tokenizer: \_\_init\_\_ \(\) missing 2 required positional arguments: 'vocab\_file' and "merges\_file'.](Tokenizer指向报错TypeError-GPT2Tokenizer-__init__-()-missing-2-required-positional-arguments-vocab_file.md)
    -   [Tokenizer文件缺失报错TypeError:\_\_init\_\(\) missing 2 required positional arguments: 'vocab\_file' and 'merge\_file'](Tokenizer文件缺失报错TypeError-__init_()-missing-2-required-positional-arguments-vocab_file-and-merge_file.md)
    -   [模型推理时验证model.generate\(\)功能报错RuntimeError A model class needs to define a \`prepare inputs fordgeneration\` method in order to use .generate\(\)\`](模型推理时验证model-generate()功能报错RuntimeError-A-model-class-needs-to-define-a-prepare-inputs-fordgeneratio.md)
    -   [MindSpore模型推理报错：memory isn't enough and alloc failed, kernel name: kernel\_graph\_@ HostDSActor, alloc size: 8192B](MindSpore模型推理报错-memory-isn-t-enough-and-alloc-failed-kernel-name-kernel_graph_-HostDSActor-alloc-siz.md)
    -   [MindSpore模型推理报错TypeError: cell\_reuse\(\) takes 0 positional arguments but 1 was given](MindSpore模型推理报错TypeError-cell_reuse()-takes-0-positional-arguments-but-1-was-given.md)
    -   [运行wizardcoder迁移代码报错broken pipe](运行wizardcoder迁移代码报错broken-pipe.md)
    -   [MindSpore Lite模型加载报错RuntimeError: build from file failed! Error is Common error code.](MindSpore-Lite模型加载报错RuntimeError-build-from-file-failed-Error-is-Common-error-code.md)
    -   [Mindspore在自回归推理时的精度对齐设置](Mindspore在自回归推理时的精度对齐设置.md)
    -   [MindSpore大模型打开pp并行或者梯度累积之后loss不溢出也不收敛](MindSpore大模型打开pp并行或者梯度累积之后loss不溢出也不收敛.md)
    -   [Ascend上用ADGEN数据集评估时报错not support in PyNative RunOp!](Ascend上用ADGEN数据集评估时报错not-support-in-PyNative-RunOp.md)
    -   [qwen1.5\_1.8B推理出现回答混乱问题及解决](qwen1-5_1-8B推理出现回答混乱问题及解决.md)
    -   [单机4卡分布式推理失报错RuntimeError: Ascend kernel runtime initialization failed. The details refer to 'Ascend Error Message'.](单机4卡分布式推理失报错RuntimeError-Ascend-kernel-runtime-initialization-failed-The-details-refer-to-Ascend-Err.md)
    -   [使用单卡Ascend910进行LLaMA2-7B推理,速度缓慢](使用单卡Ascend910进行LLaMA2-7B推理-速度缓慢.md)
    -   [MindSpore大模型微调时报溢出及解决](MindSpore大模型微调时报溢出及解决.md)
    -   [MindSpore大模型在线推理速度慢及解决方案](MindSpore大模型在线推理速度慢及解决方案.md)
    -   [Llama推理报参数校验错误TypeError: The input value must be int. but got 'NoneType.](Llama推理报参数校验错误TypeError-The-input-value-must-be-int-but-got-NoneType.md)
    -   [MindSpore模型报错Reason: Memory resources are exhausted.](MindSpore模型报错Reason-Memory-resources-are-exhausted.md)
    -   [MindSpore2.2.10 ge图模式报错: Current execute mode is KernelByKernel, the processes must be launched with OpenMPI or ...](MindSpore2-2-10-ge图模式报错-Current-execute-mode-is-KernelByKernel-the-processes-must-be-launched-with-O.md)
    -   [baichuan2-13b算子溢出 loss跑飞问题和定位](baichuan2-13b算子溢出-loss跑飞问题和定位.md)
    -   [MindSpore训练异常中止：Try to send request before Open\(\)、Try to get response before Open\(\)、Response is empty](MindSpore训练异常中止-Try-to-send-request-before-Open()-Try-to-get-response-before-Open()-Response-is-empt.md)
    -   [Ascend 910训练脚本刚运行就报错：RuntimeError: Initialize GE failed!](Ascend-910训练脚本刚运行就报错-RuntimeError-Initialize-GE-failed.md)
    -   [昇腾910上算子溢出问题分析](昇腾910上算子溢出问题分析.md)
    -   [使用ops.nonzero算子报错TypeError](使用ops-nonzero算子报错TypeError.md)
    -   [训练过程中评测超时导致训练过程发生中断](训练过程中评测超时导致训练过程发生中断.md)
    -   [昇腾910上CodeLlama推理报错get fail deviceLogicId\[0\]](昇腾910上CodeLlama推理报错get-fail-deviceLogicId-0.md)
    -   [昇腾910上CodeLlama导出mindir模型报错rankTablePath is invalid](昇腾910上CodeLlama导出mindir模型报错rankTablePath-is-invalid.md)
    -   [权重文件被异常修改导致加载权重提示Failed to read the checkpoint file](权重文件被异常修改导致加载权重提示Failed-to-read-the-checkpoint-file.md)
    -   [Mindformers模型启动时因为host侧OOM导致任务被kill](Mindformers模型启动时因为host侧OOM导致任务被kill.md)
    -   [昇腾910FlashAttention适配alibi问题](昇腾910FlashAttention适配alibi问题.md)
    -   [llama3.1-8b的lora微调，不开启权重转换会导致维度不匹配，开启了之后会报错找不到rank1的ckpt，但是strategy目录里面是全的](llama3-1-8b的lora微调-不开启权重转换会导致维度不匹配-开启了之后会报错找不到rank1的ckpt-但是strategy目录里面是全的.md)
    -   [Mindspore+MIndformer训练plog报错halMemAlloc failed，drvRetCode=6](Mindspore+MIndformer训练plog报错halMemAlloc-failed-drvRetCode-6.md)
    -   [MindSpore的离线权重转换接口说明及转换过程](MindSpore的离线权重转换接口说明及转换过程.md)
    -   [MindSpore的ckpt格式完整权重和分布式权重互转](MindSpore的ckpt格式完整权重和分布式权重互转.md)
    -   [MindSpore+MindFormer-r.1.2.0微调qwen1.5 报错](MindSpore+MindFormer-r-1-2-0微调qwen1-5-报错.md)
    -   [Mindformer训练plog中算子GatherV2\_xxx\_high\_precision\_xx报错](Mindformer训练plog中算子GatherV2_xxx_high_precision_xx报错.md)
    -   [mindformers进行Lora微调后的权重合并](mindformers进行Lora微调后的权重合并.md)
    -   [pangu-100b 2k集群线性度问题定位](pangu-100b-2k集群线性度问题定位.md)
    -   [Mixtral 8\*7B 大模型精度问题总结](Mixtral-8-7B-大模型精度问题总结.md)
    -   [大模型内存占用调优](大模型内存占用调优.md)
    -   [大模型网络算法参数和输出对比](大模型网络算法参数和输出对比.md)
    -   [使用jit编译加速时编译流程中的常见if控制流问题](使用jit编译加速时编译流程中的常见if控制流问题.md)
    -   [jit编译加速功能开启时，如何避免多次重新编译](jit编译加速功能开启时-如何避免多次重新编译.md)
    -   [编译时报错ValueError: Please set a unique name for the parameter.](编译时报错ValueError-Please-set-a-unique-name-for-the-parameter.md)
    -   [大模型动态图训练内存优化](大模型动态图训练内存优化.md)
    -   [大模型精度收敛分析和调优](大模型精度收敛分析和调优.md)
    -   [Dump工具应用—算子执行报错，输入数据值越界](Dump工具应用-算子执行报错-输入数据值越界.md)
    -   [Dump工具应用—网络训练溢出](Dump工具应用-网络训练溢出.md)
    -   [模型训练长稳性能抖动或劣化问题经验总结](模型训练长稳性能抖动或劣化问题经验总结.md)
    -   [msprobe工具应用–网络训练溢出](msprobe工具应用-网络训练溢出.md)
    -   [msprobe精度定位工具常见问题整理](msprobe精度定位工具常见问题整理.md)
    -   [模型编译的性能优化总结](模型编译的性能优化总结.md)
    -   [大模型动态图训练性能调优指南](大模型动态图训练性能调优指南.md)
    -   [大模型迭代拖尾和其他性能优化](大模型迭代拖尾和其他性能优化.md)
    -   [大模型前反向计算的性能优化](大模型前反向计算的性能优化.md)
    -   [大模型迭代间隙的性能优化](大模型迭代间隙的性能优化.md)

-   [模型切分问题案例](模型切分问题案例.md)
    -   [MindSpore权重自动切分后报错ValueError: Failed to read the checkpoint. please check the correct of the file.](MindSpore权重自动切分后报错ValueError-Failed-to-read-the-checkpoint-please-check-the-correct-of-the-file.md)
    -   [MindSpore2机自动切分模型权重报错NotADirectoryError: ./output/transformed chec kpoint/rank\_15 is not a real directory](MindSpore2机自动切分模型权重报错NotADirectoryError-output-transformed-chec-kpoint-rank_15-is-not-a-real-directo.md)
    -   [MindSpore并行策略下hccl\_tools工具使用报错](MindSpore并行策略下hccl_tools工具使用报错.md)
    -   [MindSpore大模型报错: Inner Error! EZ9999 \[InferShape\] The k-axis of a\(131072\) and b\(16384\) tensors must be the same.](MindSpore大模型报错-Inner-Error-EZ9999-InferShape-The-k-axis-of-a(131072)-and-b(16384)-tensors-must-be-th.md)
    -   [Transformers报错google.protobuf.message.DecodeError: Wrong wire type in tag.](Transformers报错google-protobuf-message-DecodeError-Wrong-wire-type-in-tag.md)

-   [其他类型问题案例](其他类型问题案例.md)
    -   [报错日志不完整](报错日志不完整.md)
    -   [日志显示没有成功加载预训练模型：model built, but weights is unloaded, since the config has no attribute or is None.](日志显示没有成功加载预训练模型-model-built-but-weights-is-unloaded-since-the-config-has-no-attribute-or-is-None.md)
    -   [工作目录问题：'from mindformers import Trainer'报错ModuleNotFoundError:No module named ' mindformers'](工作目录问题-from-mindformers-import-Trainer-报错ModuleNotFoundError-No-module-named-mindformers.md)
    -   [NFS上生成mindrecord报错Failed to write mindrecord meta files](NFS上生成mindrecord报错Failed-to-write-mindrecord-meta-files.md)
    -   [盘古-智子38B在昇腾910上，greedy模式下无法固定输出](盘古-智子38B在昇腾910上-greedy模式下无法固定输出.md)
    -   [昇腾上moxing拷贝超时导致Notify超时](昇腾上moxing拷贝超时导致Notify超时.md)
    -   [MTP Ascend910切换不同型号设备报错：KeyError：‘group\_list’](MTP-Ascend910切换不同型号设备报错-KeyError-group_list.md)

