一、Esxcli
使用 Esxcli 命令可获取有关 vSAN 的信息,以及对您的 vSAN 环境进行故障排除。
可用命令如下:
命令 | 描述 |
---|---|
esxcli vsan network list | 确认哪些 VMkernel 适配器可用于 vSAN 通信。 |
esxcli vsan storage list | 列出由 vSAN 声明的存储磁盘。 |
esxcli vsan cluster get | 获取 vSAN 群集信息。 |
esxcli vsan health | 获取 vSAN 群集运行状况。 |
esxcli vsan debug | 获取 vSAN 群集调试信息。 |
使用 esxcli vsan debug 命令可调试 vSAN 群集并对群集进行故障排除,尤其是 vCenter Server 不可用时。
使用:esxcli vsan debug {cmd} [cmd options]
调试命令:
命令 | 描述 |
---|---|
esxcli vsan debug disk | 调试 vSAN 物理磁盘。 |
esxcli vsan debug object | 调试 vSAN 对象。 |
esxcli vsan debug resync | 调试 vSAN 重新同步对象。 |
esxcli vsan debug controller | 调试 vSAN 磁盘控制器。 |
esxcli vsan debug limit | 调试 vSAN 限制。 |
esxcli vsan debug vmdk | 调试 vSAN VMDK。 |
示例:查看磁盘物理槽位
[root@ESXi-003:~] esxcli storage core device list
......
[root@ESXi-003:~] esxcli storage core device physical get -d naa.5000039858390775
Physical Location: enclosure 2, slot 4
移出vSAN磁盘组中故障磁盘
esxcli vsan storage remove -u <UUID>
二、MegaCli
登录 ESXi 或 VCenter,开启相应的 ESXi 的 SSH 服务,登录到主机并上传MegaCli的vib文件
安装MegaCli-8.04.07
[root@ESXi-003:/tmp] esxcli software vib install -v /tmp/vmware-esx-MegaCli-8.04.07.vib --no-sig-check
Installation Result
Message: Operation finished successfully.
Reboot Required: false
VIBs Installed: LSI_bootbank_vmware-esx-MegaCli-8.04.07_8.04.07-01
VIBs Removed:
VIBs Skipped:
[root@ESXi-003:/tmp]
查看版本,检查MegaCLI命令是否可用
[root@IDC02-Prod-ESXi-003:/tmp] /opt/lsi/MegaCLI/MegaCli -v
./libstorelib.so: cannot open shared object file: No such file or directory
MegaCLI SAS RAID Management Tool Ver 8.04.07 May 28, 2012
(c)Copyright 2011, LSI Corporation, All Rights Reserved.
Exit Code: 0x00
[root@IDC02-Prod-ESXi-003:/tmp]
查看磁盘状态
[root@IESXi-003:/opt/lsi/storcli64] cd /opt/lsi/MegaCLI/
[root@003:/opt/lsi/MegaCLI] ls
MegaCli MegaSAS.log libstorelib.so
[root@ESXi-003:/opt/lsi/MegaCLI] ./MegaCli -PDList -aALL
常用命令:
./MegaCli -LDinfo -Lall -aAll 查看raid状态
./MegaCli -PDList -aAll 查看每块物理磁盘信息(可以看到具体哪块磁盘出问题)
./MegaCli -PDList -aAll | grep "Firmware state:"
./MegaCli -PDList -aAll | grep "Raw Size:"
./MegaCli -PDList -aAll | grep "Slot Number:"
./MegaCli -AdpAllInfo -aALL 查raid卡信息
./MegaCli -AdpBbuCmd -aAll 查看电池信息
./MegaCli -FwTermLog -Dsply -aALL 查看raid卡日志
./MegaCli -adpCount 显示适配器个数
./MegaCli -AdpGetTime –aALL 显示适配器时间
./MegaCli -AdpAllInfo -aAll 显示所有适配器信息
./MegaCli -AdpBbuCmd -GetBbuStatus -aALL |grep ‘Charger Status’ 查看充电状态
./MegaCli -AdpBbuCmd -GetBbuStatus -aALL 显示BBU状态信息
./MegaCli -AdpBbuCmd -GetBbuCapacityInfo -aALL 显示BBU容量信息
./MegaCli -AdpBbuCmd -GetBbuDesignInfo -aALL 显示BBU设计参数
./MegaCli -AdpBbuCmd -GetBbuProperties -aALL 显示当前BBU属性
./MegaCli -cfgdsply -aALL 显示Raid卡型号,Raid设置,Disk相关信息
部分情况下MegaCli输入命令无法查看磁盘状态返回错误Exit Code: 0x01,可参考使用其他工具
[root@ESXi-003:/opt/lsi/MegaCLI] ./MegaCli -PDList -aALL
Exit Code: 0x01
三、smartctl
登录ESXI主机web界面修改或者SSH登录主机输入命令修改设置主机软件安装等级修改为“社区支持”
esxcli software acceptance set --level=CommunitySupported
安装smartctl
[root@ESXi-003:/tmp] esxcli software vib install -v /tmp/smartctl-6.6-4433.x86_64.vib
Installation Result
Message: Operation finished successfully.
Reboot Required: false
VIBs Installed: smartmontools_bootbank_smartctl_6.6-4433
VIBs Removed:
VIBs Skipped:
[root@ESXi-003:/tmp] cd /opt/smartmontools/
[root@IDC02-Prod-ESXi-003:/opt/smartmontools] ls
smartctl
[root@ESXi-003:/opt/smartmontools]
查看命令:
/opt/smartmontools/smartctl -d sat -a /dev/disk/<disk>
四、STORCLI
安装STORCLI
[root@ESXi-003:/tmp] esxcli software vib install -v /tmp/vmware-storcli64.vib --no-sig-check
Installation Result
Message: Operation finished successfully.
Reboot Required: false
VIBs Installed: Broadcom_bootbank_vmware-storcli64_007.1504.0000.0000-01
VIBs Removed:
VIBs Skipped:
[root@ESXi-003:/tmp] cd /opt/lsi/storcli64/
设置链接
# ln -s /opt/lsi/storcli/storcli /bin/storcli
查看控制器状态
[root@IESXi-003:/opt/lsi/storcli64] ./storcli64 show all //查看控制器
CLI Version = 007.1504.0000.0000 June 22, 2020
Operating system = VMkernel 7.0.2
Status Code = 0
Status = Success
Description = None
Number of Controllers = 1
Host Name = IDC02-Prod-ESXi-003.tlmvm.local
Operating System = VMkernel 7.0.2
StoreLib IT Version = 07.1503.0200.0000
StoreLib IR3 Version = 16.12-0
System Overview :
===============
--------------------------------------------------------------------------
Ctl Model Ports PDs DGs DNOpt VDs VNOpt BBU sPR DS EHS ASOs Hlth
--------------------------------------------------------------------------
0 ServeRAIDM5210 8 22 0 0 0 0 N/A On - Y 0 Opt
--------------------------------------------------------------------------
Ctl=Controller Index|DGs=Drive groups|VDs=Virtual drives|Fld=Failed
PDs=Physical drives|DNOpt=Array NotOptimal|VNOpt=LD NotOptimal|Opt=Optimal
Msng=Missing|Dgd=Degraded|NdAtn=Need Attention|Unkwn=Unknown
sPR=Scheduled Patrol Read|DS=DimmerSwitch|EHS=Emergency Spare Drive
Y=Yes|N=No|ASOs=Advanced Software Options|BBU=Battery backup unit/CV
Hlth=Health|Safe=Safe-mode boot|CertProv-Certificate Provision mode
ASO :
===
----------------------------------------------------
Ctl Cl SAS MD R6 WC R5 SS FP Re CR RF CO CW HA SSHA
----------------------------------------------------
0 X U X X U X X X X X X X X X X
----------------------------------------------------
Ctl=Controller Index|Cl=Cluster|MD=Max Disks|WC=Wide Cache|SS=Safe Store|FP=Fast Path
Re=Recovery|CR=CacheCade(Read)|RF=Reduced Feature Set|CO=Cache Offload
CW=CacheCade(Read/Write)|X=Not Available/Not Installed|U=Unlimited|T=Trial
|HA=High Availability |SSHA=Single server High Availability
[root@IDC02-Prod-ESXi-003:/opt/lsi/storcli64] ./storcli /c0 show 查看对应磁盘
-sh: ./storcli: not found
[root@IDC02-Prod-ESXi-003:/opt/lsi/storcli64] ./
libstorelib.so storcli.log storcli64
[root@IDC02-Prod-ESXi-003:/opt/lsi/storcli64] ./storcli64 /c0 show
Generating detailed summary of the adapter, it may take a while to complete.
CLI Version = 007.1504.0000.0000 June 22, 2020
Operating system = VMkernel 7.0.2
Controller = 0
Status = Success
Description = None
Product Name = ServeRAID M5210
Serial Number = SK74880985
SAS Address = 500605b00d98fb40
PCI Address = 00:10:00:00
System Time = 03/13/2023 05:16:39
Mfg. Date = 12/04/17
Controller Time = 03/13/2023 05:16:33
FW Package Build = 24.21.0-0143
BIOS Version = 6.36.00.3_4.19.08.00_0x06180204
FW Version = 4.680.01-8544
Driver Name = lsi_mr3
Driver Version = 7.716.03.00
Vendor Id = 0x1000
Device Id = 0x5D
SubVendor Id = 0x1014
SubDevice Id = 0x454
Host Interface = PCI-E
Device Interface = SAS-12G
Bus Number = 16
Device Number = 0
Function Number = 0
Domain ID = 0
Security Protocol = None
JBOD Drives = 22
JBOD LIST :
=========
-----------------------------------------------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type
-----------------------------------------------------------------------------------------------------
12:0 18 JBOD - 745.211 GB SATA SSD N N 512B SSDSC2BB800G7N 01KR467 01KR470LEN U -
12:1 19 JBOD - 745.211 GB SATA SSD N N 512B SSDSC2BB800G7N 01KR467 01KR470LEN U -
12:2 20 JBOD - 1.090 TB SAS HDD N N 512B ST1200MM0088 U -
12:3 14 JBOD - 1.090 TB SAS HDD N N 512B AL14SEB120N U -
12:18 23 JBOD - 1.090 TB SAS HDD N N 512B ST1200MM0018 U -
12:19 22 JBOD - 1.090 TB SAS HDD N N 512B ST1200MM0018 U -
12:20 21 JBOD - 1.090 TB SAS HDD N N 512B ST1200MM0018 U -
12:21 38 JBOD - 1.090 TB SAS HDD N N 512B ST1200MM0018 U -
12:22 37 JBOD - 1.090 TB SAS HDD N N 512B ST1200MM0018 U -
12:23 39 JBOD - 1.090 TB SAS HDD N N 512B ST1200MM0018 U -
15:4 10 JBOD - 1.090 TB SAS HDD N N 512B AL14SEB120N U -
15:5 13 JBOD - 1.090 TB SAS HDD N N 512B AL14SEB120N U -
15:6 8 JBOD - 1.090 TB SAS HDD N N 512B AL14SEB120N U -
15:7 9 JBOD - 1.090 TB SAS HDD N N 512B AL14SEB120N U -
15:8 40 JBOD - 558.911 GB SAS HDD N Y 512B AL13SEB600 U -
15:9 30 JBOD - 745.211 GB SATA SSD N N 512B INTEL SSDSC2BB800G4R U -
15:10 24 JBOD - 1.090 TB SAS HDD N N 512B HUC101812CSS200 U -
15:11 25 JBOD - 1.090 TB SAS HDD N N 512B HUC101812CSS200 U -
15:12 26 JBOD - 1.090 TB SAS HDD N N 512B HUC101812CSS200 U -
15:13 27 JBOD - 1.090 TB SAS HDD N N 512B HUC101812CSS200 U -
15:14 28 JBOD - 1.090 TB SAS HDD N N 512B HUC101812CSS200 U -
15:15 29 JBOD - 1.090 TB SAS HDD N N 512B HUC101812CSS200 U -
-----------------------------------------------------------------------------------------------------
ID=JBOD Target ID|EID=Enclosure Device ID|Slt=Slot No.|DID=Device ID|Onln=Online|
Offln=Offline|Intf=Interface|Med=Media Type|SeSz=Sector Size
Physical Drives = 22
PD LIST :
=======
-----------------------------------------------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type
-----------------------------------------------------------------------------------------------------
12:0 18 JBOD - 745.211 GB SATA SSD N N 512B SSDSC2BB800G7N 01KR467 01KR470LEN U -
12:1 19 JBOD - 745.211 GB SATA SSD N N 512B SSDSC2BB800G7N 01KR467 01KR470LEN U -
12:2 20 JBOD - 1.090 TB SAS HDD N N 512B ST1200MM0088 U -
12:3 14 JBOD - 1.090 TB SAS HDD N N 512B AL14SEB120N U -
12:18 23 JBOD - 1.090 TB SAS HDD N N 512B ST1200MM0018 U -
12:19 22 JBOD - 1.090 TB SAS HDD N N 512B ST1200MM0018 U -
12:20 21 JBOD - 1.090 TB SAS HDD N N 512B ST1200MM0018 U -
12:21 38 JBOD - 1.090 TB SAS HDD N N 512B ST1200MM0018 U -
12:22 37 JBOD - 1.090 TB SAS HDD N N 512B ST1200MM0018 U -
12:23 39 JBOD - 1.090 TB SAS HDD N N 512B ST1200MM0018 U -
15:4 10 JBOD - 1.090 TB SAS HDD N N 512B AL14SEB120N U -
15:5 13 JBOD - 1.090 TB SAS HDD N N 512B AL14SEB120N U -
15:6 8 JBOD - 1.090 TB SAS HDD N N 512B AL14SEB120N U -
15:7 9 JBOD - 1.090 TB SAS HDD N N 512B AL14SEB120N U -
15:8 40 JBOD - 558.911 GB SAS HDD N Y 512B AL13SEB600 U -
15:9 30 JBOD - 745.211 GB SATA SSD N N 512B INTEL SSDSC2BB800G4R U -
15:10 24 JBOD - 1.090 TB SAS HDD N N 512B HUC101812CSS200 U -
15:11 25 JBOD - 1.090 TB SAS HDD N N 512B HUC101812CSS200 U -
15:12 26 JBOD - 1.090 TB SAS HDD N N 512B HUC101812CSS200 U -
15:13 27 JBOD - 1.090 TB SAS HDD N N 512B HUC101812CSS200 U -
15:14 28 JBOD - 1.090 TB SAS HDD N N 512B HUC101812CSS200 U -
15:15 29 JBOD - 1.090 TB SAS HDD N N 512B HUC101812CSS200 U -
-----------------------------------------------------------------------------------------------------
EID=Enclosure Device ID|Slt=Slot No.|DID=Device ID|DG=DriveGroup
DHS=Dedicated Hot Spare|UGood=Unconfigured Good|GHS=Global Hotspare
UBad=Unconfigured Bad|Sntze=Sanitize|Onln=Online|Offln=Offline|Intf=Interface
Med=Media Type|SED=Self Encryptive Drive|PI=Protection Info
SeSz=Sector Size|Sp=Spun|U=Up|D=Down|T=Transition|F=Foreign
UGUnsp=UGood Unsupported|UGShld=UGood shielded|HSPShld=Hotspare shielded
CFShld=Configured shielded|Cpybck=CopyBack|CBShld=Copyback Shielded
UBUnsp=UBad Unsupported|Rbld=Rebuild
Enclosures = 3
Enclosure LIST :
==============
--------------------------------------------------------------------------------
EID State Slots PD PS Fans TSs Alms SIM Port# ProdID VendorSpecific
--------------------------------------------------------------------------------
12 OK 32 10 0 0 0 0 0 Port 0 - 3 x4 SAS EXP BP 00000007
15 OK 32 12 0 0 0 0 0 Port 4 - 7 x4 SAS EXP BP 00000007
62 OK 8 0 0 0 0 0 1 - SGPIO
--------------------------------------------------------------------------------
EID=Enclosure Device ID |PD=Physical drive count |PS=Power Supply count|
TSs=Temperature sensor count |Alms=Alarm count |SIM=SIM Count ||ProdID=Product ID
通过storcli查看磁盘信息(可查看磁盘序列号)
基本语法:
storcli64 /call show all
/c 控制器号 输出结果中的Controller 值
/v RAID号
/e 背板号 输出结果EID值
/f 外部配置
/s 槽位号 输出结果的Slt值
[root@ESXi-003:/opt/lsi/storcli64] ./storcli64 /c0/e15/s4 show all
CLI Version = 007.1504.0000.0000 June 22, 2020
Operating system = VMkernel 7.0.2
Controller = 0
Status = Success
Description = Show Drive Information Succeeded.
Drive /c0/e15/s4 :
================
----------------------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type
----------------------------------------------------------------------------
15:4 10 JBOD - 1.090 TB SAS HDD N N 512B AL14SEB120N U -
----------------------------------------------------------------------------
EID=Enclosure Device ID|Slt=Slot No.|DID=Device ID|DG=DriveGroup
DHS=Dedicated Hot Spare|UGood=Unconfigured Good|GHS=Global Hotspare
UBad=Unconfigured Bad|Sntze=Sanitize|Onln=Online|Offln=Offline|Intf=Interface
Med=Media Type|SED=Self Encryptive Drive|PI=Protection Info
SeSz=Sector Size|Sp=Spun|U=Up|D=Down|T=Transition|F=Foreign
UGUnsp=UGood Unsupported|UGShld=UGood shielded|HSPShld=Hotspare shielded
CFShld=Configured shielded|Cpybck=CopyBack|CBShld=Copyback Shielded
UBUnsp=UBad Unsupported|Rbld=Rebuild
Drive /c0/e15/s4 - Detailed Information :
=======================================
Drive /c0/e15/s4 State :
======================
Shield Counter = 0
Media Error Count = 0
Other Error Count = 5
Drive Temperature = N/A
Predictive Failure Count = 0
S.M.A.R.T alert flagged by drive = No
Drive /c0/e15/s4 Device attributes :
==================================
SN = Z7W0A0FFF11F
Manufacturer Id = LENOVO-X
Model Number = AL14SEB120N
NAND Vendor = NA
FRU/CRU = 00WG701
WWN = 5000039858390774
Firmware Revision = TB47
Firmware Release Number = N/A
Raw size = 1.090 TB [0x8bba0cb0 Sectors]
Coerced size = 1.089 TB [0x8b94f800 Sectors]
Non Coerced size = 1.090 TB [0x8baa0cb0 Sectors]
Device Speed = 12.0Gb/s
Link Speed = 12.0Gb/s
Write Cache = N/A
Logical Sector Size = 512B
Physical Sector Size = 512B
Connector Name = Port 4 - 7
Drive /c0/e15/s4 Policies/Settings :
==================================
Enclosure position = 1
Connected Port Number = 0(path0)
Sequence Number = 2
Commissioned Spare = No
Emergency Spare = No
Last Predictive Failure Event Sequence Number = 0
Successful diagnostics completion on = N/A
FDE Type = None
SED Capable = No
SED Enabled = No
Secured = No
Cryptographic Erase Capable = No
Sanitize Support = Not supported
Locked = No
Needs EKM Attention = No
PI Eligible = No
Certified = No
Wide Port Capable = No
Multipath = No
Port Information :
================
-----------------------------------------
Port Status Linkspeed SAS address
-----------------------------------------
0 Active 12.0Gb/s 0x5000039858390776
1 Active 12.0Gb/s 0x0
-----------------------------------------
Inquiry Data =
00 00 06 12 9f 01 10 02 4c 45 4e 4f 56 4f 2d 58
41 4c 31 34 53 45 42 31 32 30 4e 20 20 20 20 20
54 42 34 37 54 33 45 35 4c 33 56 4e 54 42 34 37
54 42 34 37 54 42 34 37 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 30 30 30 31 31 37 33 35 37 00 30 30 30 31
32 32 30 30 57 47 37 30 31 20 20 20 20 20 4e 33
[root@ESXi-003:/opt/lsi/storcli64]
常用命令
卸载软件
删除VIB命令:esxcli software vib remove --vibname <software name>
示例:
[root@ESXi-003:~] esxcli software vib list | grep MegaCli
vmware-esx-MegaCli-8.04.07 8.04.07-01 LSI PartnerSupported 2023-03-13
esxcli software vib remove --vibname vmware-esx-MegaCli-8.04.07
Removal Result
Message: Operation finished successfully.
Reboot Required: false
VIBs Installed:
VIBs Removed: LSI_bootbank_vmware-esx-MegaCli-8.04.07_8.04.07-01
VIBs Skipped:
[root@ESXi-003:~]