# baichaun2-13b 在Ascend910上持续溢出

**1. 系统环境**

硬件环境\(Ascend/GPU/CPU\): Ascend910

软件环境: MindSpore 版本: 2.2.10

执行模式： 不限

Python 版本: 3.8.15

操作系统平台: linux

**2. 报错信息**

1、baichaun2-13b 在Ascend910上持续溢出；

2、baichaun2-13b 加载belle数据集溢出wiki数据集不溢出。

**3. 根因分析**

1、根因是baichaun2-13b对attention\_mask进行了pad，attention mask存在全0的无效行，pad\_token没有任何可注意对象，Ascend910在不同核的机器上处理模板不一致，导致flash\_attention出现了溢出。

```
class CausalMask(nn.Cell):
    r""" Get the Lower triangular matrix from the input_ids."""
    @_LogActiononce(m_logger=logger,  key='AttentionMask',no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,))
    def __init__(self, seq_length, compute -type=mstype.float16, is_dynamic=False, pad_token_id=0, use_flash_attent ion=False):

    def construct(self, tokens=None, masks=None):
        """Forward process of the CausalMask"
            if tokens is not None:
                bs = self.shape(tokens)[O]
                seq_len = self.shape(tokens)[1]
                input_mask = self.cast(self. not_equal(tokens, self .pad_token_id), self.dtype)
            else:
                bs = self.shape(masks)[o]
                seq_len = self.shape(masks)[1]
                input_mask = self.cast(masks,  self.dtype)

            shape_right = (bs, 1, seq_len)
            shape_left = (bs, seq_len, 1)
            # Mask the padded inputs
            mask_left = self.reshape(input_mask, shape_left)
            mask_right = self.re shape(input_mask, shape_right)
            attention_mask = self.bmm(mask_left, mask_right)
            if not self.is_dynamic:
                lower_traiangle = self. expand_dim(self . lower_triangle_mask,	0)
            else:
                lower_triangle_mask = self.slice( self.lower_triangle_mask, (0, 0), (seq_len, seq_len), (1, 1))
                lower_traiangle = self.expand_dim(lower- triangle_mask, 0)
            # the returned shape is [bs, seq_length, seq_length]
            attention_mask = self.mul(at tention_mask, lower_traiangle)
            return attention_mask
```

2、belle数据集处理生成过程因为文本比较短，对sql\_length进行了pad，wiki数据集采用的拼接超过sql\_length进行截断，不会pad，在生成attention\_mask上出现了全0行，非严格下三角。

**4. 解决方案**

将attention\_mask改成严格下三角问题解决：

```
class CausalMask(nn.Cell):
    r""" Get the Lower triangular matrix from the input_ids."""
    @_LogActiononce(m_logger=logger,  key='AttentionMask',no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,))
    def __init__(self, seq_length, compute -type=mstype.float16, is_dynamic=False, pad_token_id=0, use_flash_attent ion=False):

    def construct(self, tokens=None, masks=None):
        """Forward process of the CausalMask"
            if tokens is not None:
                bs = self.shape(tokens)[O]
                seq_len = self.shape(tokens)[1]
                input_mask = self.cast(self. not_equal(tokens, self .pad_token_id), self.dtype)
            else:
                bs = self.shape(masks)[o]
                seq_len = self.shape(masks)[1]
                input_mask = self.cast(masks,  self.dtype)

            shape_right = (bs, 1, seq_len)
            # shape_left = (bs, seq_len, 1)
            # Mask the padded inputs
            mask_left = self.reshape(input_mask, shape_left)
            # mask_right = self.re shape(input_mask, shape_right)
            # attention_mask = self.bmm(mask_left, mask_right)
            if not self.is_dynamic:
                lower_traiangle = self. expand_dim(self . lower_triangle_mask,	0)
            else:
                lower_triangle_mask = self.slice( self.lower_triangle_mask, (0, 0), (seq_len, seq_len), (1, 1))
                lower_traiangle = self.expand_dim(lower- triangle_mask, 0)
            # the returned shape is [bs, seq_length, seq_length]
            attention_mask = self.mul(at tention_mask, lower_traiangle)
            return attention_mask
```

