# 新节点24使用piecewise\_constant\_lr造成梯度异常

**1. 系统环境**

硬件环境\(Ascend/GPU/CPU\): Ascend/GPU/CPU

软件环境: MindSpore 版本: 2.1.1

执行模式（静态图/动态图）： 不限

Python 版本: 3.9

操作系统平台: linux

**2. 报错信息**

2.1 问题描述

使用官网入门实例https://gitee.com/mindspore/docs/blob/r2.1/tutorials/source\_zh\_cn/beginner/quick\_start.ipynb。然后对学习率进行了修改，使用了动态学习率出现梯度异常问题。最后一行注释的代码是官方原始代码，学习率为固定常量 lr=0.1。

2.2 报错信息

原官方代码运行结果

```
Epoch 1
loss: 2.315734 [  0/9381]
loss: 0.335083 [100/9381]
loss: 8.289867 [200/9381]
loss: 0.320835 [300/9381]
loss: 0.218015 [400/9381]
loss: 0.288610 [500/9381]
loss: 0.149030 [600/9381]
loss: 0.081295 [700/9381]
loss: 0.140247 [800/9381]
loss: 0.175849 [900/9381]
```

新增动态学习率后的运行结果

```
Epoch 1
loss: 2.309528 [  0/9381]
loss: 2.198998 [100/9381]
loss: 2.167274 [200/9381]
loss: 2.155498 [300/9381]
loss: 2.148228 [400/9381]
loss: 2.139027 [500/9381]
loss: 2.106419 [600/9381]
loss: 2.123494 [700/9381]
loss: 2.160592 [800/9381]
loss: 2.184410 [900/9381]
```

2.3 脚本信息

```
class Network(nn.Cell):
    def __init__(self):
        super().__init__()
        self.flatten = nn.Flatten()
        self.dense_relu_sequential = nn.SequentialCell(
            nn.Dense(28*28, 512),
            nn.ReLU(),
            nn.Dense(512, 512),
            nn.ReLU(),
            nn.Dense(512, 10)
        )

    def construct(self, x):
        x = self.flatten(x)
        logits = self.dense_relu_sequential(x)
        return logits

model = Network()
print(model)

milestone = [1,3,10,100]
learning_rates = [0.1,0.05,0.01,0.001]
lr = nn.piecewise_constant_lr(milestone,learning_rates)

loss_fn = nn.CrossEntropyLoss()
optimizer = nn.SGD(model.trainable_params(),learning_rate=lr)
# optimizer = nn.SGD(model.trainable_params(),learning_rate=0.1)
```

**3. 根因分析**

```
piecewise_constant_lr
milestone = [1,3,10,100]
learning_rates = [0.1,0.05,0.01,0.001]
step 1-3 学习率 0.1
step 3-10 学习率 0.05
step 10 -100 学习率0.01
step 100之后学习率0.001
```

看loss变化情况, 这个lr的太小了,会导致无法收敛

可以修改milestone或learning\_rates的值

```
milestone = [1,100,500,700]
```

或

```
learning_rates = [1.0,0.5,0.2,0.1]
```

**4. 解决方案**

```
loss_fn = nn.CrossEntropyLoss()
milestone = [1,100,500,700]
learning_rates = [0.1,0.05,0.01,0.001]
lr = nn.piecewise_constant_lr(milestone,learning_rates)

optimizer = nn.SGD(model.trainable_params(), lr)
```

或者

```
loss_fn = nn.CrossEntropyLoss()
milestone = [1,3,10,100]
learning_rates = [1.0,0.5,0.2,0.1]
lr = nn.piecewise_constant_lr(milestone,learning_rates)

optimizer = nn.SGD(model.trainable_params(), lr)
```

