写在前方：ubuntu系统，显卡重启后驱动失效，显卡不可用。网上冲浪之后得以有效解决，以下是解决方案

• 查看显卡nvidia-smi；驱动失效消息：

(base) root@node:~# nvidia-smi 
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

• 驱动失效原因：
  　　系统内核升级，与原驱动信息不匹配

• 解决办法：
  　　不建议重新安装驱动，可通过DKMS(Dynamic Kernel Module Support)修复，它能够维护内核外的驱动程序，并且在内核版本变化后自动生成新的模块。

１、下载dkms，apt-get install dkms：

(base) root@node:~# apt-get install dkms

２、查看驱动版本信息ls /usr/src |grep nvidia：

(base) root@node:~# ls /usr/src |grep nvidia
nvidia-550.90.07

３、使用dkms修复：

(base) root@node:~# dkms install -m nvidia -v 550.90.07

４、检查驱动是否可用：nvidia-smi

(base) root@node:~# nvidia-smi 
Fri Jul 12 06:00:52 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.90.07              Driver Version: 550.90.07      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A800 80GB PCIe          Off |   00000000:4B:00.0 Off |                    0 |
| N/A   41C    P0             68W /  300W |       1MiB /  81920MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA A800 80GB PCIe          Off |   00000000:65:00.0 Off |                    0 |
| N/A   43C    P0             68W /  300W |       1MiB /  81920MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA A800 80GB PCIe          Off |   00000000:B1:00.0 Off |                    0 |
| N/A   42C    P0             71W /  300W |       1MiB /  81920MiB |      3%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA A800 80GB PCIe          Off |   00000000:E3:00.0 Off |                    0 |
| N/A   48C    P0             74W /  300W |       1MiB /  81920MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------++-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

参考资料：
https://blog.csdn.net/trainingVIP/article/details/137789875