1. MindNLP ChatGLM-6B StreamChat

本案例基于MindNLP和ChatGLM-6B实现一个聊天应用。

ChatGLM-6B应该是国内第一个发布的可以在消费级显卡上进行推理部署的国产开源大模型，2023年3月就发布了。我在23年6月份的时候就在自己的笔记本电脑上部署测试过，当时的1代6B模型已经能解鸡兔同笼的数学问题，感觉上是真正“理解”了人类语言的语义。我认为和chatgpt相比，也并没有非常明显的差距。当然存在的问题也有不少。

1.1 环境配置

• 安装mindnlp

!pip install mindnlp 

安装过程:

Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting mindnlpDownloading https://pypi.tuna.tsinghua.edu.cn/packages/72/37/ef313c23fd587c3d1f46b0741c98235aecdfd93b4d6d446376f3db6a552c/mindnlp-0.3.1-py3-none-any.whl (5.7 MB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.7/5.7 MB 16.7 MB/s eta 0:00:00a 0:00:01
Requirement already satisfied: mindspore in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from mindnlp) (2.2.14)
Requirement already satisfied: tqdm in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from mindnlp) (4.66.4)
Requirement already satisfied: requests in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from mindnlp) (2.32.3)
Collecting datasets (from mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/60/2d/963b266bb8f88492d5ab4232d74292af8beb5b6fdae97902df9e284d4c32/datasets-2.20.0-py3-none-any.whl (547 kB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 547.8/547.8 kB 16.4 MB/s eta 0:00:00
Collecting evaluate (from mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c2/d6/ff9baefc8fc679dcd9eb21b29da3ef10c81aa36be630a7ae78e4611588e1/evaluate-0.4.2-py3-none-any.whl (84 kB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 84.1/84.1 kB 25.4 MB/s eta 0:00:00
Collecting tokenizers (from mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ba/26/139bd2371228a0e203da7b3e3eddcb02f45b2b7edd91df00e342e4b55e13/tokenizers-0.19.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.6 MB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/3.6 MB 19.6 MB/s eta 0:00:0000:0100:01
Collecting safetensors (from mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c6/02/28e6280ed0f1bde89eed644b80f2ece4e5ae212dc9ee70d7f56fadc93602/safetensors-0.4.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.2 MB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 20.2 MB/s eta 0:00:00a 0:00:01
Collecting sentencepiece (from mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a3/69/e96ef68261fa5b82379fdedb325ceaf1d353c6e839ec346d8244e0da5f2f/sentencepiece-0.2.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.3 MB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 15.5 MB/s eta 0:00:00a 0:00:01
Collecting regex (from mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/70/70/fea4865c89a841432497d1abbfd53878513b55c6543245fabe31cf8df0b8/regex-2024.5.15-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (774 kB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 774.7/774.7 kB 17.0 MB/s eta 0:00:00a 0:00:01
Collecting addict (from mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/6a/00/b08f23b7d7e1e14ce01419a467b583edbb93c6cdb8654e54a9cc579cd61f/addict-2.4.0-py3-none-any.whl (3.8 kB)
Collecting ml-dtypes (from mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/50/96/13d7c3cc82d5ef597279216cf56ff461f8b57e7096a3ef10246a83ca80c0/ml_dtypes-0.4.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.2 MB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/2.2 MB 12.6 MB/s eta 0:00:00a 0:00:01
Collecting pyctcdecode (from mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a5/8a/93e2118411ae5e861d4f4ce65578c62e85d0f1d9cb389bd63bd57130604e/pyctcdecode-0.5.0-py2.py3-none-any.whl (39 kB)
Collecting jieba (from mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c6/cb/18eeb235f833b726522d7ebed54f2278ce28ba9438e3135ab0278d9792a2/jieba-0.42.1.tar.gz (19.2 MB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.2/19.2 MB 19.3 MB/s eta 0:00:0000:0100:01Preparing metadata (setup.py) ... done
Collecting pytest==7.2.0 (from mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/67/68/a5eb36c3a8540594b6035e6cdae40c1ef1b6a2bfacbecc3d1a544583c078/pytest-7.2.0-py3-none-any.whl (316 kB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 316.8/316.8 kB 17.8 MB/s eta 0:00:00
Requirement already satisfied: attrs>=19.2.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pytest==7.2.0->mindnlp) (23.2.0)
Requirement already satisfied: iniconfig in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pytest==7.2.0->mindnlp) (2.0.0)
Requirement already satisfied: packaging in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pytest==7.2.0->mindnlp) (23.2)
Requirement already satisfied: pluggy<2.0,>=0.12 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pytest==7.2.0->mindnlp) (1.5.0)
Requirement already satisfied: exceptiongroup>=1.0.0rc8 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pytest==7.2.0->mindnlp) (1.2.0)
Requirement already satisfied: tomli>=1.0.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pytest==7.2.0->mindnlp) (2.0.1)
Requirement already satisfied: filelock in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from datasets->mindnlp) (3.15.3)
Requirement already satisfied: numpy>=1.17 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from datasets->mindnlp) (1.26.4)
Collecting pyarrow>=15.0.0 (from datasets->mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/87/60/cc0645eb4ef73f88847e40a7f9d238bae6b7409d6c1f6a5d200d8ade1f09/pyarrow-16.1.0-cp39-cp39-manylinux_2_28_aarch64.whl (38.1 MB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 38.1/38.1 MB 18.3 MB/s eta 0:00:0000:0100:01
Collecting pyarrow-hotfix (from datasets->mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/e4/f4/9ec2222f5f5f8ea04f66f184caafd991a39c8782e31f5b0266f101cb68ca/pyarrow_hotfix-0.6-py3-none-any.whl (7.9 kB)
Requirement already satisfied: dill<0.3.9,>=0.3.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from datasets->mindnlp) (0.3.8)
Requirement already satisfied: pandas in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from datasets->mindnlp) (2.2.2)
Collecting xxhash (from datasets->mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/7c/b9/93f860969093d5d1c4fa60c75ca351b212560de68f33dc0da04c89b7dc1b/xxhash-3.4.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (220 kB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 220.6/220.6 kB 17.8 MB/s eta 0:00:00
Collecting multiprocess (from datasets->mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/da/d9/f7f9379981e39b8c2511c9e0326d212accacb82f12fbfdc1aa2ce2a7b2b6/multiprocess-0.70.16-py39-none-any.whl (133 kB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.4/133.4 kB 17.5 MB/s eta 0:00:00
Collecting fsspec<=2024.5.0,>=2023.1.0 (from fsspec[http]<=2024.5.0,>=2023.1.0->datasets->mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ba/a3/16e9fe32187e9c8bc7f9b7bcd9728529faa725231a0c96f2f98714ff2fc5/fsspec-2024.5.0-py3-none-any.whl (316 kB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 316.1/316.1 kB 19.4 MB/s eta 0:00:00
Collecting aiohttp (from datasets->mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/eb/45/eebe8d2215328434f33ccb44a05d2741ff7ed4b96b56ca507e2ecf598b73/aiohttp-3.9.5-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.2 MB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 18.2 MB/s eta 0:00:00a 0:00:01
Requirement already satisfied: huggingface-hub>=0.21.2 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from datasets->mindnlp) (0.23.4)
Requirement already satisfied: pyyaml>=5.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from datasets->mindnlp) (6.0.1)
Requirement already satisfied: charset-normalizer<4,>=2 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from requests->mindnlp) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from requests->mindnlp) (3.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from requests->mindnlp) (2.2.2)
Requirement already satisfied: certifi>=2017.4.17 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from requests->mindnlp) (2024.6.2)
Requirement already satisfied: protobuf>=3.13.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from mindspore->mindnlp) (5.27.1)
Requirement already satisfied: asttokens>=2.0.4 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from mindspore->mindnlp) (2.0.5)
Requirement already satisfied: pillow>=6.2.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from mindspore->mindnlp) (10.3.0)
Requirement already satisfied: scipy>=1.5.4 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from mindspore->mindnlp) (1.13.1)
Requirement already satisfied: psutil>=5.6.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from mindspore->mindnlp) (5.9.0)
Requirement already satisfied: astunparse>=1.6.3 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from mindspore->mindnlp) (1.6.3)
Collecting pygtrie<3.0,>=2.1 (from pyctcdecode->mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ec/cd/bd196b2cf014afb1009de8b0f05ecd54011d881944e62763f3c1b1e8ef37/pygtrie-2.5.0-py3-none-any.whl (25 kB)
Collecting hypothesis<7,>=6.14 (from pyctcdecode->mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/58/14/a4c621cb713f6053f37afa78ab3809f9d879182422071ca9d4af61c6d1d9/hypothesis-6.105.0-py3-none-any.whl (462 kB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 462.2/462.2 kB 21.2 MB/s eta 0:00:00
Requirement already satisfied: six in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from asttokens>=2.0.4->mindspore->mindnlp) (1.16.0)
Requirement already satisfied: wheel<1.0,>=0.23.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from astunparse>=1.6.3->mindspore->mindnlp) (0.43.0)
Collecting aiosignal>=1.1.2 (from aiohttp->datasets->mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/76/ac/a7305707cb852b7e16ff80eaf5692309bde30e2b1100a1fcacdc8f731d97/aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
Collecting frozenlist>=1.1.1 (from aiohttp->datasets->mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/57/15/172af60c7e150a1d88ecc832f2590721166ae41eab582172fe1e9844eab4/frozenlist-1.4.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (239 kB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 239.4/239.4 kB 19.6 MB/s eta 0:00:00
Collecting multidict<7.0,>=4.5 (from aiohttp->datasets->mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/d0/10/2ff646c471e84af25fe8111985ffb8ec85a3f6e1ade8643bfcfcc0f4d2b1/multidict-6.0.5-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (125 kB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 125.9/125.9 kB 16.9 MB/s eta 0:00:00
Collecting yarl<2.0,>=1.0 (from aiohttp->datasets->mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c6/d6/5b30ae1d8a13104ee2ceb649f28f2db5ad42afbd5697fd0fc61528bb112c/yarl-1.9.4-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (300 kB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 300.9/300.9 kB 14.5 MB/s eta 0:00:00
Collecting async-timeout<5.0,>=4.0 (from aiohttp->datasets->mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a7/fa/e01228c2938de91d47b307831c62ab9e4001e747789d0b05baf779a6488c/async_timeout-4.0.3-py3-none-any.whl (5.7 kB)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from huggingface-hub>=0.21.2->datasets->mindnlp) (4.11.0)
Collecting sortedcontainers<3.0.0,>=2.1.0 (from hypothesis<7,>=6.14->pyctcdecode->mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB)
Requirement already satisfied: python-dateutil>=2.8.2 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pandas->datasets->mindnlp) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pandas->datasets->mindnlp) (2024.1)
Requirement already satisfied: tzdata>=2022.7 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pandas->datasets->mindnlp) (2024.1)
Building wheels for collected packages: jiebaBuilding wheel for jieba (setup.py) ... doneCreated wheel for jieba: filename=jieba-0.42.1-py3-none-any.whl size=19314459 sha256=b34d50dd74d300723e8f42cfea23a680ef40525b2f94488be07da842e838a41cStored in directory: /home/nginx/.cache/pip/wheels/1a/76/68/b6d79c4db704bb18d54f6a73ab551185f4711f9730c0c15d97
Successfully built jieba
Installing collected packages: sortedcontainers, sentencepiece, pygtrie, jieba, addict, xxhash, safetensors, regex, pytest, pyarrow-hotfix, pyarrow, multiprocess, multidict, ml-dtypes, hypothesis, fsspec, frozenlist, async-timeout, yarl, pyctcdecode, aiosignal, tokenizers, aiohttp, datasets, evaluate, mindnlpAttempting uninstall: pytestFound existing installation: pytest 8.0.0Uninstalling pytest-8.0.0:Successfully uninstalled pytest-8.0.0Attempting uninstall: fsspecFound existing installation: fsspec 2024.6.0Uninstalling fsspec-2024.6.0:Successfully uninstalled fsspec-2024.6.0
Successfully installed addict-2.4.0 aiohttp-3.9.5 aiosignal-1.3.1 async-timeout-4.0.3 datasets-2.20.0 evaluate-0.4.2 frozenlist-1.4.1 fsspec-2024.5.0 hypothesis-6.105.0 jieba-0.42.1 mindnlp-0.3.1 ml-dtypes-0.4.0 multidict-6.0.5 multiprocess-0.70.16 pyarrow-16.1.0 pyarrow-hotfix-0.6 pyctcdecode-0.5.0 pygtrie-2.5.0 pytest-7.2.0 regex-2024.5.15 safetensors-0.4.3 sentencepiece-0.2.0 sortedcontainers-2.4.0 tokenizers-0.19.1 xxhash-3.4.1 yarl-1.9.4[notice] A new release of pip is available: 24.1 -> 24.1.1
[notice] To update, run: python -m pip install --upgrade pip

• 安装mdtex2html

!pip install mdtex2html

Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting mdtex2htmlDownloading https://pypi.tuna.tsinghua.edu.cn/packages/ff/e8/c5fab9aa5d9254ad7c7e37d33a3c32fd49d82b4c6b54da337bbca378eb5c/mdtex2html-1.3.0-py3-none-any.whl (13 kB)
Requirement already satisfied: gradio in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (4.26.0)
Collecting markdown (from mdtex2html)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/fc/b3/0c0c994fe49cd661084f8d5dc06562af53818cc0abefaca35bdc894577c3/Markdown-3.6-py3-none-any.whl (105 kB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 105.4/105.4 kB 11.6 MB/s eta 0:00:00
Collecting latex2mathml (from mdtex2html)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/f2/0a/181ed55562ce90179aedf33b09fcd79db31c868a5d480f3cb71a31d19692/latex2mathml-3.77.0-py3-none-any.whl (73 kB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 73.7/73.7 kB 22.7 MB/s eta 0:00:00
Requirement already satisfied: aiofiles<24.0,>=22.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (22.1.0)
Requirement already satisfied: altair<6.0,>=4.2.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (5.3.0)
Requirement already satisfied: fastapi in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (0.111.0)
Requirement already satisfied: ffmpy in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (0.3.2)
Requirement already satisfied: gradio-client==0.15.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (0.15.1)
Requirement already satisfied: httpx>=0.24.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (0.27.0)
Requirement already satisfied: huggingface-hub>=0.19.3 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (0.23.4)
Requirement already satisfied: importlib-resources<7.0,>=1.3 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (6.4.0)
Requirement already satisfied: jinja2<4.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (3.1.4)
Requirement already satisfied: markupsafe~=2.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (2.1.5)
Requirement already satisfied: matplotlib~=3.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (3.9.0)
Requirement already satisfied: numpy~=1.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (1.26.4)
Requirement already satisfied: orjson~=3.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (3.10.5)
Requirement already satisfied: packaging in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (23.2)
Requirement already satisfied: pandas<3.0,>=1.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (2.2.2)
Requirement already satisfied: pillow<11.0,>=8.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (10.3.0)
Requirement already satisfied: pydantic>=2.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (2.7.4)
Requirement already satisfied: pydub in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (0.25.1)
Requirement already satisfied: python-multipart>=0.0.9 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (0.0.9)
Requirement already satisfied: pyyaml<7.0,>=5.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (6.0.1)
Requirement already satisfied: ruff>=0.2.2 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (0.4.10)
Requirement already satisfied: semantic-version~=2.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (2.10.0)
Requirement already satisfied: tomlkit==0.12.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (0.12.0)
Requirement already satisfied: typer<1.0,>=0.9 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from typer[all]<1.0,>=0.9; sys_platform != "emscripten"->gradio) (0.12.3)
Requirement already satisfied: typing-extensions~=4.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (4.11.0)
Requirement already satisfied: uvicorn>=0.14.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (0.30.1)
Requirement already satisfied: fsspec in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio-client==0.15.1->gradio) (2024.5.0)
Requirement already satisfied: websockets<12.0,>=10.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio-client==0.15.1->gradio) (11.0.3)
Requirement already satisfied: jsonschema>=3.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from altair<6.0,>=4.2.0->gradio) (4.22.0)
Requirement already satisfied: toolz in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from altair<6.0,>=4.2.0->gradio) (0.12.1)
Requirement already satisfied: anyio in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from httpx>=0.24.1->gradio) (4.4.0)
Requirement already satisfied: certifi in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from httpx>=0.24.1->gradio) (2024.6.2)
Requirement already satisfied: httpcore==1.* in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from httpx>=0.24.1->gradio) (1.0.5)
Requirement already satisfied: idna in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from httpx>=0.24.1->gradio) (3.7)
Requirement already satisfied: sniffio in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from httpx>=0.24.1->gradio) (1.3.1)
Requirement already satisfied: h11<0.15,>=0.13 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from httpcore==1.*->httpx>=0.24.1->gradio) (0.14.0)
Requirement already satisfied: filelock in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from huggingface-hub>=0.19.3->gradio) (3.15.3)
Requirement already satisfied: requests in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from huggingface-hub>=0.19.3->gradio) (2.32.3)
Requirement already satisfied: tqdm>=4.42.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from huggingface-hub>=0.19.3->gradio) (4.66.4)
Requirement already satisfied: zipp>=3.1.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from importlib-resources<7.0,>=1.3->gradio) (3.17.0)
Requirement already satisfied: contourpy>=1.0.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from matplotlib~=3.0->gradio) (1.2.1)
Requirement already satisfied: cycler>=0.10 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from matplotlib~=3.0->gradio) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from matplotlib~=3.0->gradio) (4.53.0)
Requirement already satisfied: kiwisolver>=1.3.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from matplotlib~=3.0->gradio) (1.4.5)
Requirement already satisfied: pyparsing>=2.3.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from matplotlib~=3.0->gradio) (3.1.2)
Requirement already satisfied: python-dateutil>=2.7 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from matplotlib~=3.0->gradio) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pandas<3.0,>=1.0->gradio) (2024.1)
Requirement already satisfied: tzdata>=2022.7 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pandas<3.0,>=1.0->gradio) (2024.1)
Requirement already satisfied: annotated-types>=0.4.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pydantic>=2.0->gradio) (0.7.0)
Requirement already satisfied: pydantic-core==2.18.4 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pydantic>=2.0->gradio) (2.18.4)
Requirement already satisfied: click>=8.0.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from typer<1.0,>=0.9->typer[all]<1.0,>=0.9; sys_platform != "emscripten"->gradio) (8.1.7)
Requirement already satisfied: shellingham>=1.3.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from typer<1.0,>=0.9->typer[all]<1.0,>=0.9; sys_platform != "emscripten"->gradio) (1.5.4)
Requirement already satisfied: rich>=10.11.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from typer<1.0,>=0.9->typer[all]<1.0,>=0.9; sys_platform != "emscripten"->gradio) (13.7.1)
WARNING: typer 0.12.3 does not provide the extra 'all'
Requirement already satisfied: starlette<0.38.0,>=0.37.2 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from fastapi->gradio) (0.37.2)
Requirement already satisfied: fastapi-cli>=0.0.2 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from fastapi->gradio) (0.0.4)
Requirement already satisfied: ujson!=4.0.2,!=4.1.0,!=4.2.0,!=4.3.0,!=5.0.0,!=5.1.0,>=4.0.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from fastapi->gradio) (5.10.0)
Requirement already satisfied: email_validator>=2.0.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from fastapi->gradio) (2.2.0)
Requirement already satisfied: importlib-metadata>=4.4 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from markdown->mdtex2html) (7.0.1)
Requirement already satisfied: dnspython>=2.0.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from email_validator>=2.0.0->fastapi->gradio) (2.6.1)
Requirement already satisfied: attrs>=22.2.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio) (23.2.0)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio) (2023.12.1)
Requirement already satisfied: referencing>=0.28.4 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio) (0.35.1)
Requirement already satisfied: rpds-py>=0.7.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio) (0.18.1)
Requirement already satisfied: six>=1.5 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from python-dateutil>=2.7->matplotlib~=3.0->gradio) (1.16.0)
Requirement already satisfied: markdown-it-py>=2.2.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from rich>=10.11.0->typer<1.0,>=0.9->typer[all]<1.0,>=0.9; sys_platform != "emscripten"->gradio) (3.0.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from rich>=10.11.0->typer<1.0,>=0.9->typer[all]<1.0,>=0.9; sys_platform != "emscripten"->gradio) (2.15.1)
Requirement already satisfied: exceptiongroup>=1.0.2 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from anyio->httpx>=0.24.1->gradio) (1.2.0)
Requirement already satisfied: httptools>=0.5.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from uvicorn[standard]>=0.12.0->fastapi->gradio) (0.6.1)
Requirement already satisfied: python-dotenv>=0.13 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from uvicorn[standard]>=0.12.0->fastapi->gradio) (1.0.1)
Requirement already satisfied: uvloop!=0.15.0,!=0.15.1,>=0.14.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from uvicorn[standard]>=0.12.0->fastapi->gradio) (0.19.0)
Requirement already satisfied: watchfiles>=0.13 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from uvicorn[standard]>=0.12.0->fastapi->gradio) (0.22.0)
Requirement already satisfied: charset-normalizer<4,>=2 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from requests->huggingface-hub>=0.19.3->gradio) (3.3.2)
Requirement already satisfied: urllib3<3,>=1.21.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from requests->huggingface-hub>=0.19.3->gradio) (2.2.2)
Requirement already satisfied: mdurl~=0.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from markdown-it-py>=2.2.0->rich>=10.11.0->typer<1.0,>=0.9->typer[all]<1.0,>=0.9; sys_platform != "emscripten"->gradio) (0.1.2)
Installing collected packages: latex2mathml, markdown, mdtex2html
Successfully installed latex2mathml-3.77.0 markdown-3.6 mdtex2html-1.3.0[notice] A new release of pip is available: 24.1 -> 24.1.1
[notice] To update, run: python -m pip install --upgrade pip

• 安装gradio

!pip install gradio

• 配置环境变量

# 设置环境变量 HF_ENDPOINT，其值为 https://hf-mirror.com
# 这个环境变量通常用于指定 Hugging Face Transformers 模型位于国内的镜像站点
# 以便从该镜像站点下载模型和相关资源，提高下载速度和稳定性,不需要代理
export HF_ENDPOINT=https://hf-mirror.com

1.2 代码开发

模型参数量为6B, 磁盘空间大小占用约12G, 下载权重产加载大约需要20分钟

# 导入 MindNLP 库中的 AutoModelForSeq2SeqLM 类和 AutoTokenizer 类
# MindNLP 是一个基于 PyTorch 的自然语言处理库，提供了许多预训练模型和工具
from mindnlp.transformers import AutoModelForSeq2SeqLM, AutoTokenizer# 导入 Gradio 库，用于创建一个交互式的 Web 界面
import gradio as gr# 导入 mdtex2html 库，用于将 Markdown 格式的文本转换为 HTML 格式
import mdtex2html# 使用 AutoModelForSeq2SeqLM 类从预训练模型 'ZhipuAI/ChatGLM-6B' 创建一个模型实例
# 'ZhipuAI/ChatGLM-6B' 是一个序列到序列的语言模型，用于文本生成任务
# mirror="modelscope" 参数指定了模型的镜像站点为 "modelscope"，以提高下载速度和稳定性.half()是将float32转为float16.
model = AutoModelForSeq2SeqLM.from_pretrained('ZhipuAI/ChatGLM-6B', mirror="modelscope").half()# 将模型设置为评估模式，即不进行训练
model.set_train(False)# 使用 AutoTokenizer 类从预训练模型 'ZhipuAI/ChatGLM-6B' 创建一个分词器实例
# 分词器用于将输入文本转换为模型可以理解的 tokens
tokenizer = AutoTokenizer.from_pretrained('ZhipuAI/ChatGLM-6B', mirror="modelscope")

输出:

100%773/773 [00:00<00:00, 50.6kB/s]
100%32.6k/32.6k [00:00<00:00, 2.78MB/s]
Downloading shards: 100%8/8 [16:43<00:00, 107.69s/it]
100%1.62G/1.62G [02:14<00:00, 14.2MB/s]
100%1.75G/1.75G [02:19<00:00, 16.8MB/s]
100%1.84G/1.84G [02:27<00:00, 17.9MB/s]
100%1.78G/1.78G [02:21<00:00, 17.5MB/s]
100%1.75G/1.75G [02:20<00:00, 14.6MB/s]
100%1.75G/1.75G [02:20<00:00, 18.7MB/s]
100%1.00G/1.00G [01:18<00:00, 16.7MB/s]
100%1.00G/1.00G [01:19<00:00, 8.88MB/s]
Loading checkpoint shards: 100%8/8 [00:51<00:00,  5.84s/it]
100%441/441 [00:00<00:00, 38.4kB/s]
100%2.58M/2.58M [00:00<00:00, 6.99MB/s]

1.3 进行推理

# 定义一个字符串变量 prompt，内容为 "你好"，表示用户的提问或对话输入
prompt = '你好'# 定义一个列表变量 history，用于存储对话历史
# 在这个例子中，对话历史为空，因为这是新的对话
history = []# 调用 model 的 chat 方法，传入 tokenizer、prompt、history 和 max_length 参数
# chat 方法是模型的一个函数，用于生成对话响应
# tokenizer 是之前加载的分词器，用于处理输入文本
# prompt 是用户的输入文本
# history 是对话历史，用于提供上下文信息
# max_length 是生成响应的最大长度
response, _ = model.chat(tokenizer, prompt, history=history, max_length=20)# 打印出模型的对话响应
print(response)

输出:

\
The dtype of attention mask (Float32) is not bool
|
'你好👋！我是人工智能助手 ChatGLM-6B'

响应非常地慢, 不知是什么原因.
根据npu监控信息来看, 一开始完全没有利用到ai core, 而是一直在跑内存. 直到开始调用ai core之后,很快就给出了响应.

prompt = '房间里有7只鸡和兔子,共20只脚.问有几只鸡,几只兔子?'
history = []
response, _ = model.chat(tokenizer, prompt, history=history, max_length=512)
response

输出:

可以看到,他有正确的解题思路. 虽然数学稀烂,二元一次方程组都能解错.
耗时约200s, 输出250多个字符.如果算token的话,可能就是每秒1个token, 这个性能是真的拉垮…
当然,这是免费的虚拟资源，并且很大概率是共享的，可能并不能和实体的昇腾芯片相提并论。

2. 小结

本文主要介绍了使用mindnlp下载chatglm-6B预训练模型，并基于此模型，通过输入提示词完成回答文本生成的模型推理任务，从而实现了一个简单的聊天应用。