# MindSpore报错Failed to combine the net and the parameters for param backbone.embedding.word\_embedding.embedding\_table.

**1. 系统环境**

硬件环境\(Ascend/GPU/CPU\): Ascend

软件环境: MindSpore 版本: 2.2

执行模式： 静态图

**2. 报错信息**

2.1 问题描述

context设置enable\_alltoall=True出现AllToAllv算子报错

2.2 报错信息

16机128卡

```
- Framework Error Message: (For framework developers)----------------------------------------------------Distribute Task Failed, error msg: davinci_model : load task fail, return ret: 1343225860

----------------------------------------------------- C++ Call Stack: (For framework developers)----------------------------------------------------mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_kernel_executor.cc:241 PreprocessBeforeRunGraphmindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:587 LoadTaskmindspore/ccsrc/plugin/device/ascend/hal/device/ge_runtime/task/hccl_task.cc:111 Distribute

time stamp 2023.10.25-11:16:10 before loading ckpt[CRITICAL] ME(1094:281473579920368,MainProcess):2023-10-25-11:16:10.522.225 [mindspore/train/serialization.py:116] Failed to combine the net and the parameters for param backbone.embedding.word_embedding.embedding_table.
```

单机8卡：

```
- Framework Error Message: (For framework developers)----------------------------------------------------Distribute Task Failed, error msg: davinci_model : load task fail, return ret: 1343225860, the task is Default/AllToAllv-op16385_1532618

----------------------------------------------------- C++ Call Stack: (For framework developers)----------------------------------------------------mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_kernel_executor.cc:198 PreprocessBeforeRunGraphmindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:587 LoadTaskmindspore/ccsrc/plugin/device/ascend/hal/device/ge_runtime/task/hccl_task.cc:104 Distribute
```

**3. 根因分析**

报错出现_Distribute Task Failed，_如果报错日志信息没有其他的ERROR或CRITICAL报错，需要export GLOG\_v=1开启info日志，看看具体是哪个算子下发失败。

AlltoAllv的算子报错：

1、查看脚本中enable\_alltoall是否为True，如果是，可改成False看一下是否还有这个报错；

2、如果改成False，不报错，那就是通信方式修改导致的问题，可先改成False规避，但性能会很差；

3、dump图，分析为什么会插alltoall算子（查看hwopt\_d\_end\_graph\_\*.ir图）（mtp平台保存图，需要创建输出（如graphs）；并在任务创建输出引用（如名字为output，关联创建的输出名graphs）；修改脚本路径为/opt/huawei/quoteModel/output/\*\*\*）

分析示例如下：

```
%1214 (eguiv[CNode]23017) A11ToA11v(%1212, %1213) primitive attrs: (IsFeatureMapInputList: (0, 1),comm reuse: true, group:2-16453000547691086251,pri format: NC1HWCO, recv_rank ids: (0, 1), send rar
         : (<Tensor[Float16], (l, 4096, l, 8792)>, <Tensor[Float16], (1, 4096, 1,8792)>) -> (<Tuple[Tensor[Float16]*2], Tupleshape((1, 4096, 1, 8792), (1,4096, 1, 8792))>)LF
         : (<Float16xDefaultFormat[const vector][1, 4096, 1, 8792]>, <Float16xDefaultFormat[const vector][1, 4096, 1, 8792]>) -> (<Float16xDefaultFormat[const vector] [1, 4096, 1, 8792]>, <Float16xDefaultFor
         : (Tensor, Tensor) ->(TupleUnfold) LF
         # fullname_with_scope:(Default/A11ToA1lv-op12859) LF
```

_查看alltoallv的输入_，往上追溯到非框架插的第一个算子，记住alltoallv的数据切分策略。

```
%1206 (equivlogits) - MatMul(%1198, %1205) (instance name:matmul} primitive_att rs: {IsFeatureMapInputList: (0), input names: fx1, x21, pri format: FRACTAL NZ, out_strategy: None, transpose x2: true, transpose b: true, visited: true, custaicpu: mindspore cpukernels, tbe fusiontype: Matmul, is dynamic len: false, in strategy: ((1, 1). (8, 1)). output_names: [output], transpose a: false, IsFeatureMapOutput: true,is kernel dynamie impl: false, transpose xl: false) cnode primal attrs: (unique id: 905614}LF
         :  (<Tensor[Float16], (8192, 5120)>, <Tensor[Float16], (8792, 5120)>) -> (<Tensor[Float16], (8192, 8792)>)LF
         :  (<Float16xFRACTAL NZ[const vector][320, 512, 16, 16]>,<Float16xFRACTAL NZ[const vector][320, 550, 16, 16]>) -> (<Float16xFRACTAL NZ[const vector][550, 512, 16, 16]>)LF
         :  (Tensor, Tensor) ->(Tensor)LF
         # fullname with scope:
         (Default/network-PanguAlphaTrainOneStepWithLossScaleCe11WithLossMask/network-_VirtualDatasetCel1/_backbone-MicroBatchInterleaved/network-RrhfWithLoss/network-PanguAlphaModel/head-PanGuHead/MatMul-op6643)LF
```

_查看alltoall的输出_，往下追溯到非框架插的第一个算子，记住alltoallv的数据切分策略。

```
$1228 (equivl0gprobs) 1 LogSoftmaxV2 (%1227) (instance name: loa softmaxl nrimit ve attrs: (IsFeatureMapInputList: (0), pri_format: DefaultFormat,tbe_fusion_type: Opaque, me op name: LogSoftmax, axis: -1, out strategy: None, is dynamıc ten: false, in strategy: ((1, 1, 1)), visited: true, IsFeatureMapOutput: true, is kerneldynamic_impl: true, op_adaptation_processed: true)cnode_primal_attrs:(unique_id: 905407}LF
```

查看，两个算子的切分策略是否符合预期。一般能找到错误的切分策略，对应代码行，进行修改即可。

**4. 解决方案**

本次报错引入原因在于，alltoall的输出算子切分策略为（1,1,1），不符合预期。对应代码，发现是切分没有指定策略，是默认按数据dp切分，修改即可

```
logprobs = self.log softmax(logits)/LF
logits = self.logprobs_from lables(logits[ : ,  : ,  : ] input_ids[ : , 1: ]) #:
```

