# CSRTensor 矩阵乘法计算出错RuntimeError:CUDA Error: cudaMemcpy failed.|Error Number: 700 an illegal memory access was encountered

**1. 系统环境**

硬件环境\(Ascend/GPU/CPU\): GPU

软件环境: MindSpore 版本: 2.0.0

执行模式（静态图/动态图）： 动态图

Python 版本: 3.7.6

操作系统平台: linux

**2. 报错信息**

2.1 问题描述

问题描述 CSRTensor 矩阵乘法在GPU下计算报错。

2.2 报错信息

```
RuntimeError:CUDA Error: cudaMemcpy failed. | Error Number: 700 an illegal memory access was encountered
```

2.3 脚本信息

```
import mindspore as ms

a = ms.numpy.ones((7600, 7600), ms.float32).to_csr()
b = ms.numpy.ones((7600, 7600), ms.float32).to_csr()
a.mm(b)
```

**3. 根因分析**

3.1根据报错信息，可以确定该问题GPU算子执行时，出现显存访问越界，问题缩小到如下：

```
void SparseMatrixSparseMatMulGpuKernelMod::Compute(cusparseHandle_t handle, cusparseMatDescr_t desc,                                                    csrgemm2Info_t info, const int m, const int n, const int k,                                                    float *A_val, const int *const A_colind, const int *const A_rowptr,                                                    const int A_nnz, float *B_val, const int *const B_colind,                                                    const int *const B_rowptr, const int B_nnz, float **C_val,                                                    int **const C_colind, int **const C_rowptr, int *const C_nnz) {   float alpha = 1.0;   size_t buffer_size;   cusparseScsrgemm2_bufferSizeExt(handle, m, n, k, &alpha, desc, A_nnz, A_rowptr, A_colind, desc, B_nnz, B_rowptr,                                   B_colind, NULL, desc, B_nnz, B_rowptr, B_colind, info, &buffer_size);   void *buffer = NULL;   buffer = allocator.AllocTensorMem(buffer_size);   cusparseXcsrgemm2Nnz(handle, m, n, k, desc, A_nnz, A_rowptr, A_colind, desc, B_nnz, B_rowptr, B_colind, desc, B_nnz,                        B_rowptr, B_colind, desc, *C_rowptr, C_nnz, info, buffer);   cusparseScsrgemm2(handle, m, n, k, &alpha, desc, A_nnz, A_val, A_rowptr, A_colind, desc, B_nnz, B_val, B_rowptr,                     B_colind, NULL, desc, B_nnz, B_val, B_rowptr, B_colind, desc, *C_val, *C_rowptr, *C_colind, info,                     buffer);   allocator.FreeTensorMem(buffer); } 
```

3.2 修改算子输入Tensor维度，可以看到当维度较小时结果正常，使用高维度报错，怀疑显存分配问题导致，结合代码怀疑和\*C\_val, \*C\_rowptr, \*C\_colind相关

3.3 分析代码，当前算子维度推导逻辑中，支持最大值为100000，在修改MAX\_LENGTH = 7600 \* 7600后用例可以正常执行

```
namespace mindspore { namespace ops { namespace { const int MAX_LENGTH = 100000; ...  abstract::TupleShapePtr SparseMatrixSparseMatMulInferShape(const PrimitivePtr &primitive,                                                            const std::vector<AbstractBasePtr> &input_args) {   ...   ShapeVector col_shape = {abstract::Shape::kShapeDimAny};   ShapeVector values_shape = {abstract::Shape::kShapeDimAny};   ShapeVector infer_shape_max = {MAX_LENGTH};   y_col_shape = std::make_shared<abstract::Shape>(col_shape, infer_shape_max);   y_values_shape = std::make_shared<abstract::Shape>(values_shape, infer_shape_max);   ... 
```

**综上：**SparseMatrixSparseMatMul算子在维度推导时设置了最大支持的维度，算子实际输入的维度超过了支持的范围，并没有做合适的保护，导致内存访问越界。

**4. 解决方案**

4.1 修改算子推导代码，使用环境变量控制SparseMatrixSparseMatMul支持维度

```
diff --git a/mindspore/core/ops/sparse_matrix_sparse_mat_mul.cc b/mindspore/core/ops/sparse_matrix_sparse_mat_mul.cc index 5cd42fb3be9..2ca80a7643f 100644--- a/mindspore/core/ops/sparse_matrix_sparse_mat_mul.cc +++ b/mindspore/core/ops/sparse_matrix_sparse_mat_mul.cc @@ -122,7 +122,12 @@ abstract::TupleShapePtr SparseMatrixSparseMatMulInferShape(const PrimitivePtr &p    ShapeVector col_shape = {abstract::Shape::kShapeDimAny};    ShapeVector values_shape = {abstract::Shape::kShapeDimAny};    ShapeVector infer_shape_min = {1}; -  ShapeVector infer_shape_max = {MAX_LENGTH}; +  int64_t max_length = MAX_LENGTH; +  const char *max_length_char = std::getenv("MAX_SPARSE_MM_LENGTH"); +  if (max_length_char != nullptr) { +    max_length = atol(max_length_char); +  } +  ShapeVector infer_shape_max = {max_length};    y_col_shape = std::make_shared<abstract::Shape>(col_shape, infer_shape_min, infer_shape_max);    y_values_shape = std::make_shared<abstract::Shape>(values_shape, infer_shape_min, infer_shape_max); 
```

4.2 导出环境变量，后再次执行用例，结果正常

```
(py37) ➜  temp export MAX_SPARSE_MM_LENGTH=231040000(py37) ➜  temp cat sparse_matmul.py import mindspore as ms  a = ms.numpy.ones((7600, 7600), ms.float32).to_csr() b = ms.numpy.ones((7600, 7600), ms.float32).to_csr() a.mm(b)   (py37) ➜  temp python sparse_matmul.py [WARNING] ME(89158:139873739707328,MainProcess):2023-04-23-12:05:46.164.508 [mindspore/run_check/_check_version.py:102] MindSpore version 2.0.0a0 and cuda version 11.7.60 does not match, CUDA version [['10.1', '11.1', '11.6']] are supported by MindSpore officially. Please refer to the installation guide for version matching information: https://www.mindspore.cn/install. (py37) ➜  temp
```

