# LoadTask Distribute Task Failed 报错解决

**1. 系统环境**

硬件环境\(Ascend/GPU/CPU\): Ascend

软件环境: MindSpore 版本: 1.6.1

执行模式（静态图/动态图）： 不限

Python 版本: 3.7.6

操作系统平台: linux

**2. 报错信息**

2.1 问题描述

在分布式运行环境中，需要bash启动脚本和python文件，分别如下：

将下述的bash脚本另存为 run.sh

```
#!/bin/bash
set -e
EXEC_PATH=$(pwd)
export HCCL_CONNECT_TIMEOUT=120 # 避免复现需要很长时间，设置超时为120s
export RANK_SIZE=8
export RANK_TABLE_FILE=${EXEC_PATH}/rank_table_8pcs.json

for((i=0;i&lt;4;i++))
do
    rm -rf device$i
    mkdir device$i
    cp ./train.py ./device$i
    cd ./device$i
    export DEVICE_ID=$i
    export RANK_ID=$i
    echo &quot;start training for device $i&quot;
    env &gt; env$i.log
    python ./train.py &gt; train.log$i 2&gt;&amp;1 &amp;
    cd ../
done
echo &quot;The program launch succeed, the log is under device0/train.log0.&quot;
```

将下述的脚本命名为run.py

```
&quot;&quot;&quot;Operator Parallel Example&quot;&quot;&quot;
import numpy as np

from mindspore import context, Parameter
from mindspore.nn import Cell, Momentum
from mindspore.ops import operations as P
from mindspore.train import Model
from mindspore.nn.loss import SoftmaxCrossEntropyWithLogits
import mindspore.dataset as ds
import mindspore.communication.management as D
from mindspore.train.callback import LossMonitor
from mindspore.train.callback import ModelCheckpoint
from mindspore.common.initializer import initializer

step_per_epoch = 4

def get_dataset(*inputs):
    def generate():
        for _ in range(step_per_epoch):
            yield inputs
    return generate


class Net(Cell):
    &quot;&quot;&quot;define net&quot;&quot;&quot;
    def __init__(self):
        super().__init__()
        self.matmul = P.MatMul().shard(((2, 4), (4, 1)))
        self.weight = Parameter(initializer(&quot;normal&quot;, [32, 16]), &quot;w1&quot;)
        self.relu = P.ReLU().shard(((8, 1),))

    def construct(self, x):
        out = self.matmul(x, self.weight)
        out = self.relu(out)
        return out


if __name__ == &quot;__main__&quot;:
    context.set_context(mode=context.GRAPH_MODE, device_target=&quot;Ascend&quot;, save_graphs=True)
    D.init()
    rank = D.get_rank()
    context.set_auto_parallel_context(parallel_mode=&quot;semi_auto_parallel&quot;, device_num=8, full_batch=True)

    np.random.seed(1)
    input_data = np.random.rand(16, 32).astype(np.float32)
    label_data = np.random.rand(16, 16).astype(np.float32)
    fake_dataset = get_dataset(input_data, label_data)

    net = Net()

    callback = [LossMonitor(), ModelCheckpoint(directory=&quot;{}&quot;.format(rank))]
    dataset = ds.GeneratorDataset(fake_dataset, [&quot;input&quot;, &quot;label&quot;])
    loss = SoftmaxCrossEntropyWithLogits()

    learning_rate = 0.001
    momentum = 0.1
    epoch_size = 1
    opt = Momentum(net.trainable_params(), learning_rate, momentum)

    model = Model(net, loss_fn=loss, optimizer=opt)
    model.train(epoch_size, dataset, callbacks=callback, dataset_sink_mode=False)
```

对应的rank\_table如下，命名为rank\_table\_8pcs.json

```
{
    &quot;version&quot;: &quot;1.0&quot;,
    &quot;server_count&quot;: &quot;1&quot;,
    &quot;server_list&quot;: [
        {
            &quot;server_id&quot;: &quot;10.90.41.205&quot;,
            &quot;device&quot;: [
                {
                    &quot;device_id&quot;: &quot;0&quot;,
                    &quot;device_ip&quot;: &quot;192.98.92.107&quot;,
                    &quot;rank_id&quot;: &quot;0&quot;
                },
                {
                    &quot;device_id&quot;: &quot;1&quot;,
                    &quot;device_ip&quot;: &quot;192.98.93.107&quot;,
                    &quot;rank_id&quot;: &quot;1&quot;
                },
                {
                    &quot;device_id&quot;: &quot;2&quot;,
                    &quot;device_ip&quot;: &quot;192.98.94.107&quot;,
                    &quot;rank_id&quot;: &quot;2&quot;
                },
                {
                    &quot;device_id&quot;: &quot;3&quot;,
                    &quot;device_ip&quot;: &quot;192.98.95.107&quot;,
                    &quot;rank_id&quot;: &quot;3&quot;
                },
                {
                    &quot;device_id&quot;: &quot;4&quot;,
                    &quot;device_ip&quot;: &quot;192.98.92.108&quot;,
                    &quot;rank_id&quot;: &quot;4&quot;
                },
                {
                    &quot;device_id&quot;: &quot;5&quot;,
                    &quot;device_ip&quot;: &quot;192.98.93.108&quot;,
                    &quot;rank_id&quot;: &quot;5&quot;
                },
                {
                    &quot;device_id&quot;: &quot;6&quot;,
                    &quot;device_ip&quot;: &quot;192.98.94.108&quot;,
                    &quot;rank_id&quot;: &quot;6&quot;
                },
                {
                    &quot;device_id&quot;: &quot;7&quot;,
                    &quot;device_ip&quot;: &quot;192.98.95.108&quot;,
                    &quot;rank_id&quot;: &quot;7&quot;
                }
            ],
            &quot;host_nic_ip&quot;: &quot;reserve&quot;
        }
    ],
    &quot;status&quot;: &quot;completed&quot;
}
```

假设本地拥有8个Ascend 910设备，那么可以设置启动命令如下

```
bash run.sh 8
```

日志文件输出在当前执行目录下：device0/train.log0

2.2 报错信息

```
[CRITICAL] GE(88778,7f440899a740,python):2022-03-10-18:22:59.846.306 [mindspore/ccsrc/plugin/device/ascend/hal/device/ge_runtime/task/hccl_task.cc:100] Distribute] davinci_model : load task fail, return ret: 1343225860
[CRITICAL] DEVICE(88778,7f440899a740,python):2022-03-10-18:22:59.846.792 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:560] LoadTask] Distribute Task Failed,
error msg: mindspore/ccsrc/plugin/device/ascend/hal/device/ge_runtime/task/hccl_task.cc:100 Distribute] davinci_model : load task fail, return ret: 1343225860
[ERROR] DEVICE(88778,7f440899a740,python):2022-03-10-18:22:59.846.859 [mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_device_context.cc:641] ReportErrorMessage] Ascend error occurred, error message:
EI9999: Inner Error!
EI9999  connected p2p timeout, timeout:120 s.local logicDevid:0,remote physic id:4 The possible causes are as follows:1.the connectionbetween this device and the target device is abnormal 2.an exception occurredat the target devices 3.The ranktable is not matched.[FUNC:WaitP2PConnected][FILE:p2p_mgmt.cc][LINE:228]

[CRITICAL] DEVICE(88778,7f440899a740,python):2022-03-10-18:22:59.846.878 [mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_device_context.cc:413] PreprocessBeforeRunGraph] Preprocess failed before run graph 1,
error msg: mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:560 LoadTask] Distribute Task Failed,
error msg: mindspore/ccsrc/plugin/device/ascend/hal/device/ge_runtime/task/hccl_task.cc:100 Distribute] davinci_model : load task fail, return ret: 1343225860

Traceback (most recent call last):
  File &quot;/rootmodels/test_matmul_net.py&quot;, line 91, in &lt;module&gt;
    run_net(args.shape)
  File &quot;/rootmodels/test_matmul_net.py&quot;, line 81, in run_net
    model.train(1, ds_train2, callbacks=[LossMonitor()], dataset_sink_mode=True)
  File &quot;/rootmindspore/build/package/mindspore/train/model.py&quot;, line 777, in train
    sink_size=sink_size)
  File &quot;/rootmindspore/build/package/mindspore/train/model.py&quot;, line 86, in wrapper
    func(self, *args, **kwargs)
  File &quot;/rootmindspore/build/package/mindspore/train/model.py&quot;, line 543, in _train
    self._train_dataset_sink_process(epoch, train_dataset, list_callback, cb_params, sink_size)
  File &quot;/rootmindspore/build/package/mindspore/train/model.py&quot;, line 611, in _train_dataset_sink_process
    outputs = train_network(*inputs)
  File &quot;/rootmindspore/build/package/mindspore/nn/cell.py&quot;, line 542, in __call__
    out = self.compile_and_run(*args)
  File &quot;/rootmindspore/build/package/mindspore/nn/cell.py&quot;, line 872, in compile_and_run
    self.compile(*inputs)
  File &quot;/rootmindspore/build/package/mindspore/nn/cell.py&quot;, line 857, in compile
    _cell_graph_executor.compile(self, *inputs, phase=self.phase, auto_parallel_mode=self._auto_parallel_mode)
  File &quot;/rootmindspore/build/package/mindspore/common/api.py&quot;, line 712, in compile
    result = self._graph_executor.compile(obj, args_list, phase, self._use_vm_mode())
RuntimeError: mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_device_context.cc:413 PreprocessBeforeRunGraph] Preprocess failed before run graph 1,
error msg: mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:560 LoadTask] Distribute Task Failed,
error msg: mindspore/ccsrc/plugin/device/ascend/hal/device/ge_runtime/task/hccl_task.cc:100 Distribute] davinci_model : load task fail, return ret: 1343225860
```

**3. 解决方案**

从上往下分别是MindSpore报错信息为LoadTask的日志，以及打印的ReportErrorMessage中具体的报错信息，以及MindSpore报错的具体信息。 其中关键信息如下。

EI9999  connected p2p timeout, timeout:120 s.local logicDevid:0,remote physic id:4 The possible causes are as follows:1.the connectionbetween this device and the target device is abnormal 2.an exception occurredat the target devices 3.The ranktable is not matched.\[FUNC:WaitP2PConnected\]\[FILE:p2p\_mgmt.cc\]\[LINE:228\]

这个日志信息表示p2p连接超过了120s。本地是卡0\(local logicDevid:0\)，想要连接的目标是卡4\(remote physic id:4\)。因此可以判断两卡之间的网络连接是否正常，卡4的程序是否正常。

回过来我们想要查看卡4的日志，却发现没有日志。进而判断是否是程序没有成功启动？

![](figures/zh-cn_image_0000002099303822.png)看下启动脚本，发现里面只运行了4个进程，但是我们配置的rank\_table却是8卡的。因此，只有03卡启动了程序，48卡并没有进程在启动。卡0无法连接卡4，因此造成上述的报错，解决方法：将bash启动脚本中的4改成8，如下所示：

```
#!/bin/bash
set -e
EXEC_PATH=$(pwd)

export RANK_SIZE=8

test_dist_${RANK_SIZE}pcs

for((i=0;i&lt;8;i++))
do
    rm -rf device$i
    mkdir device$i
    cp ./train.py ./device$i
    cd ./device$i
    export DEVICE_ID=$i
    export RANK_ID=$i
    echo &quot;start training for device $i&quot;
    env &gt; env$i.log
    python ./train.py &gt; train.log$i 2&gt;&amp;1 &amp;
    cd ../
done
echo &quot;The program launch succeed, the log is under device0/train.log0.&quot;
```

