{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![下载Notebook](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.9.0/resource/_static/logo_notebook.svg)](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/r2.9.0/tutorials/zh_cn/beginner/mindspore_save_load.ipynb) \n",
    "[![下载样例代码](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.9.0/resource/_static/logo_download_code.svg)](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/r2.9.0/tutorials/zh_cn/beginner/mindspore_save_load.py) \n",
    "[![查看源文件](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.9.0/resource/_static/logo_source.svg)](https://atomgit.com/mindspore/docs/blob/r2.9.0/tutorials/source_zh_cn/beginner/save_load.ipynb)\n",
    "\n",
    "[基本介绍](https://www.mindspore.cn/tutorials/zh-CN/r2.9.0/beginner/introduction.html) || [快速入门](https://www.mindspore.cn/tutorials/zh-CN/r2.9.0/beginner/quick_start.html) || [张量 Tensor](https://www.mindspore.cn/tutorials/zh-CN/r2.9.0/beginner/tensor.html) || [数据加载与处理](https://www.mindspore.cn/tutorials/zh-CN/r2.9.0/beginner/dataset.html) || [网络构建](https://www.mindspore.cn/tutorials/zh-CN/r2.9.0/beginner/model.html) || [函数式自动微分](https://www.mindspore.cn/tutorials/zh-CN/r2.9.0/beginner/autograd.html) || [模型训练](https://www.mindspore.cn/tutorials/zh-CN/r2.9.0/beginner/train.html) || **保存与加载** || [Graph Mode加速](https://www.mindspore.cn/tutorials/zh-CN/r2.9.0/beginner/accelerate_with_static_graph.html) ||"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# 保存与加载\n",
    "\n",
    "上一章节主要介绍了如何调整超参数，并进行网络模型训练。在训练网络模型的过程中，通常希望保存中间和最后的结果，用于微调（fine-tune）和后续的模型推理与部署，本章节我们将介绍如何保存与加载模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import mindspore\n",
    "from mindspore import nn\n",
    "from mindspore import Tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def network():\n",
    "    model = nn.SequentialCell(\n",
    "                nn.Flatten(),\n",
    "                nn.Dense(28*28, 512),\n",
    "                nn.ReLU(),\n",
    "                nn.Dense(512, 512),\n",
    "                nn.ReLU(),\n",
    "                nn.Dense(512, 10))\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## 保存和加载模型权重\n",
    "\n",
    "保存模型使用[mindspore.save_checkpoint](https://www.mindspore.cn/docs/zh-CN/r2.9.0/api_python/mindspore/mindspore.save_checkpoint.html)接口，传入网络和指定的保存路径："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "model = network()\n",
    "mindspore.save_checkpoint(model, \"model.ckpt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "为了加载模型权重，需要先创建相同模型的实例，然后使用`load_checkpoint`和`load_param_into_net`方法加载参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = network()\n",
    "param_dict = mindspore.load_checkpoint(\"model.ckpt\")\n",
    "param_not_load, _ = mindspore.load_param_into_net(model, param_dict)\n",
    "print(param_not_load)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "> - `param_not_load`是未被加载的参数列表，为空时代表所有参数均加载成功。\n",
    "> - 当环境中安装有MindX DL（昇腾深度学习组件）6.0及以上版本时，默认启动MindIO加速CheckPoint功能，详情查看[MindIO介绍](https://www.hiascend.com/document/detail/zh/mindcluster/70rc1/clustersched/dlug/mindioacp001.html)。MindX DL在[此处](https://www.hiascend.com/developer/download/community/result?module=dl+cann)下载。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 保存和加载MindIR\n",
    "\n",
    "除Checkpoint外，MindSpore提供了云侧（训练）和端侧（推理）统一的中间表示（Intermediate Representation，IR）。可使用`export`接口直接将模型保存为MindIR（当前仅支持严格图模式）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "mindspore.set_context(mode=mindspore.GRAPH_MODE, jit_syntax_level=mindspore.STRICT)\n",
    "model = network()\n",
    "inputs = Tensor(np.ones([1, 1, 28, 28]).astype(np.float32))\n",
    "mindspore.export(model, inputs, file_name=\"model\", file_format=\"MINDIR\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> MindIR同时保存了Checkpoint和模型结构，因此需要定义输入Tensor来获取输入shape。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "已有的MindIR模型可以方便地通过`load`接口加载，传入[mindspore.nn.GraphCell](https://www.mindspore.cn/docs/zh-CN/r2.9.0/api_python/nn/mindspore.nn.GraphCell.html)即可进行推理。\n",
    "\n",
    "> `nn.GraphCell`仅支持图模式。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 10)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "graph = mindspore.load(\"model.mindir\")\n",
    "model = nn.GraphCell(graph)\n",
    "outputs = model(inputs)\n",
    "print(outputs.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 语法支持范围\n",
    "\n",
    "并不是所有的 Python 语法和数据类型都支持 MindIR 导出，若不在支持范围内，导出时会报错。\n",
    "\n",
    "1. MindIR导出仅支持**STRICT级别的基础语法**，详细的支持范围，可参考[静态图语法支持](https://www.mindspore.cn/tutorials/zh-CN/r2.9.0/compile/static_graph.html)。\n",
    "\n",
    "2. 返回值的数据类型只支持：\n",
    "\n",
    "    - Python 内置类型：`int`、`float`、`bool`、`str`、`tuple`、`list`。\n",
    "    - MindSpore 框架内置类型：[Tensor](https://www.mindspore.cn/docs/zh-CN/r2.9.0/api_python/mindspore/mindspore.Tensor.html)、[Parameter](https://www.mindspore.cn/docs/zh-CN/r2.9.0/api_python/mindspore/mindspore.Parameter.html)、[COOTensor](https://www.mindspore.cn/docs/zh-CN/r2.9.0/api_python/mindspore/mindspore.COOTensor.html)、[CSRTensor](https://www.mindspore.cn/docs/zh-CN/r2.9.0/api_python/mindspore/mindspore.CSRTensor.html)。\n",
    "\n",
    "    例如下面的程序，返回值类型是 [mindspore.dtype](https://www.mindspore.cn/docs/zh-CN/r2.9.0/api_python/mindspore/mindspore.dtype.html)，不在支持范围内，MindIR 导出的时候就会报错。\n",
    "\n",
    "    ```python\n",
    "    import mindspore\n",
    "    from mindspore import nn, Tensor\n",
    "\n",
    "    class Model(nn.Cell):\n",
    "\n",
    "        def construct(self, x: Tensor) -> mindspore.dtype:\n",
    "            return x.dtype\n",
    "   ```\n",
    "\n",
    "3. `nn.Cell`的`construct()`方法中，不支持使用 [mindspore.mint](https://www.mindspore.cn/docs/zh-CN/r2.9.0/api_python/mindspore.mint.html) 包下的随机数生成接口，如`mint.rand`、`mint.randn`、`mint.randint`、`mint.randperm`。（建议改为使用 [mindspore.ops](https://www.mindspore.cn/docs/zh-CN/r2.9.0/api_python/mindspore.ops.html) 包下的随机数生成接口）。\n",
    "\n",
    "4. `Parameter`对象只能定义在`nn.Cell`的`__init__()`方法中或者作为函数的输入参数，否则 MindIR 不支持导出该`Parameter`。例如下面的程序，有一个`Parameter`是全局变量，导出时会报错不支持。\n",
    "\n",
    "    ```python\n",
    "    import mindspore\n",
    "    from mindspore import Parameter, nn\n",
    "\n",
    "    # Parameter在nn.Cell外创建，并作为全局变量被Model使用。\n",
    "    global_param = Parameter([1, 2, 3], name='global_param')\n",
    "\n",
    "    class Model(nn.Cell):\n",
    "\n",
    "        def __init__(self):\n",
    "            super().__init__()\n",
    "            # Parameter定义在nn.Cell的__init__()方法中，支持导出。\n",
    "            self.bias = Parameter([0, 1, -1])\n",
    "\n",
    "        def construct(self, x: Parameter):  # Parameter是函数的入参，支持导出。\n",
    "            # global_param是全局变量，导出时会报错。\n",
    "            return x + global_param + self.bias\n",
    "\n",
    "    model = Model()\n",
    "    param = Parameter([1, 2, 3], name='input_param')\n",
    "    mindspore.export(model, param, file_name=\"model\", file_format=\"MINDIR\")\n",
    "    ```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}