linux性能监控之lsof

lsof：list open files，显示所有打开的文件以及进程信息，我们通常用来检查特定的文件被哪些进程打开

[root@k8s-master ~]# lsof --help
lsof: illegal option character: -
lsof: -e not followed by a file system path: "lp"
lsof 4.87latest revision: ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/latest FAQ: ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/FAQlatest man page: ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/lsof_manusage: [-?abhKlnNoOPRtUvVX] [+|-c c] [+|-d s] [+D D] [+|-f[gG]] [+|-e s][-F [f]] [-g [s]] [-i [i]] [+|-L [l]] [+m [m]] [+|-M] [-o [o]] [-p s]
[+|-r [t]] [-s [p:s]] [-S [t]] [-T [t]] [-u s] [+|-w] [-x [fl]] [--] [names]
Defaults in parentheses; comma-separated set (s) items; dash-separated ranges.-?|-h list help          -a AND selections (OR)     -b avoid kernel blocks-c c  cmd c ^c /c/[bix]  +c w  COMMAND width (9)    +d s  dir s files-d s  select by FD set   +D D  dir D tree *SLOW?*   +|-e s  exempt s *RISKY*-i select IPv[46] files  -K list tasKs (threads)    -l list UID numbers-n no host names         -N select NFS files        -o list file offset-O no overhead *RISKY*   -P no port names           -R list paRent PID-s list file size        -t terse listing           -T disable TCP/TPI info-U select Unix socket    -v list version info       -V verbose search+|-w  Warnings (+)       -X skip TCP&UDP* files     -Z Z  context [Z]-- end option scan     +f|-f  +filesystem or -file names     +|-f[gG] flaGs -F [f] select fields; -F? for help  +|-L [l] list (+) suppress (-) link counts < l (0 = all; default = 0)+m [m] use|create mount supplement+|-M   portMap registration (-)       -o o   o 0t offset digits (8)-p s   exclude(^)|select PIDs         -S [t] t second stat timeout (15)-T qs TCP/TPI Q,St (s) info-g [s] exclude(^)|select and print process group IDs-i i   select by IPv[46] address: [46][proto][@host|addr][:svc_list|port_list]+|-r [t[m<fmt>]] repeat every t seconds (15);  + until no files, - forever.An optional suffix to t is m<fmt>; m must separate t from <fmt> and<fmt> is an strftime(3) format for the marker line.-s p:s  exclude(^)|select protocol (p = TCP|UDP) states by name(s).-u s   exclude(^)|select login|UID set s-x [fl] cross over +d|+D File systems or symbolic Linksnames  select named files or files on named file systems
Anyone can list all files; /dev warnings disabled; kernel ID check disabled.

选项说明：

-a	列出打开文件存在的进程
-c<进程名>	列出指定进程所打开的文件
-g	列出GID号进程详情
-d<文件号>	列出占用该文件号的进程
+d<目录>	列出目录下被打开的文件
+D<目录>	递归列出目录下被打开的文件
-n<目录>	列出使用NFS的文件
-i<条件>	列出符合条件的进程。（4、6、协议、:端口、 @ip ）
-p<进程号>	列出指定进程号所打开的文件
-u	列出UID号进程详情

COMMAND	命令名称
PID	进程ID
TID	线程ID，如果为空代表列出的是进程
USER	用户ID号或登录名
FD	文件描述符
TYPE	与文件关联结点的类型
DEVICE	设备号
SIZE/OFF	文件大小/偏移量，以字节为单位
NODE	文件结点
NAME	文件挂载点和文件所在的系统

我们常用的命令组合：

1.列出所有打开的文件：lsof | more

[root@k8s-master ~]# lsof | more
COMMAND      PID   TID           USER   FD      TYPE             DEVICE  SIZE/OFF       NODE NAME
systemd        1                 root  cwd       DIR                8,3       244         64 /
systemd        1                 root  rtd       DIR                8,3       244         64 /
systemd        1                 root  txt       REG                8,3   1632960   17033053 /usr/lib/systemd/systemd
systemd        1                 root  mem       REG                8,3     20064     167261 /usr/lib64/libuuid.so.1.3.0
systemd        1                 root  mem       REG                8,3    265576    1183915 /usr/lib64/libblkid.so.1.1.0
systemd        1                 root  mem       REG                8,3     90160     167255 /usr/lib64/libz.so.1.2.7
systemd        1                 root  mem       REG                8,3    157440     167263 /usr/lib64/liblzma.so.5.2.2
systemd        1                 root  mem       REG                8,3     23968     167346 /usr/lib64/libcap-ng.so.0.0.0
systemd        1                 root  mem       REG                8,3     19896     180865 /usr/lib64/libattr.so.1.1.0
systemd        1                 root  mem       REG                8,3     19248     164854 /usr/lib64/libdl-2.17.so
systemd        1                 root  mem       REG                8,3    402384     167265 /usr/lib64/libpcre.so.1.2.0
systemd        1                 root  mem       REG                8,3   2156592     164847 /usr/lib64/libc-2.17.so
systemd        1                 root  mem       REG                8,3    142144     165873 /usr/lib64/libpthread-2.17.so

2.列出特定文件系统打开的文件：lsof /poc

[root@k8s-master ~]# lsof /proc
COMMAND      PID  USER   FD   TYPE DEVICE SIZE/OFF       NODE NAME
systemd        1  root    9r   REG    0,3        0        300 /proc/1/mountinfo
systemd        1  root   11r   REG    0,3        0 4026532019 /proc/swaps
systemd-j    384  root    9r   REG    0,3        0        446 /proc/sys/kernel/hostname
udisksd      594  root    8r   REG    0,3        0      28850 /proc/594/mountinfo
udisksd      594  root    9r   REG    0,3        0 4026532019 /proc/swaps
rtkit-dae    646 rtkit  cwd    DIR    0,3        0          1 /proc
rtkit-dae    646 rtkit  rtd    DIR    0,3        0          1 /proc
libvirtd     972  root   19w   REG    0,3        0 4026531960 /proc/mtrr
dockerd     1287  root   11r   REG    0,3        0 4026531956 net
X           1490  root    9w   REG    0,3        0 4026531960 /proc/mtrr
X           1490  root   10w   REG    0,3        0 4026531960 /proc/mtrr
gnome-she   1778   gdm   28r   REG    0,3        0      40875 /proc/1778/mountinfo
packageki   1937  root   10r   REG    0,3        0      40842 /proc/1937/mountinfo
lsof      119641  root    3r   DIR    0,3        0          1 /proc
lsof      119641  root    6r   DIR    0,3        0    2160211 /proc/119641/fd

3.列出root用户打开的文件： lsof -u root | more

[root@k8s-master ~]# lsof -u root | more
COMMAND      PID USER   FD      TYPE             DEVICE  SIZE/OFF       NODE NAME
systemd        1 root  cwd       DIR                8,3       244         64 /
systemd        1 root  rtd       DIR                8,3       244         64 /
systemd        1 root  txt       REG                8,3   1632960   17033053 /usr/lib/systemd/systemd

4.列出所有打开的 IPv4 网络文件：lsof -i 4

[root@k8s-master ~]# lsof -i 4
COMMAND      PID   USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
rpcbind      597    rpc    6u  IPv4   23732      0t0  UDP *:sunrpc 
rpcbind      597    rpc    7u  IPv4   23784      0t0  UDP *:webster 
rpcbind      597    rpc    8u  IPv4   23785      0t0  TCP *:sunrpc (LISTEN)
avahi-dae    612  avahi   12u  IPv4   25053      0t0  UDP *:mdns 
avahi-dae    612  avahi   13u  IPv4   25054      0t0  UDP *:60388 
sshd         946   root    3u  IPv4   30305      0t0  TCP *:ssh (LISTEN)
cupsd        961   root   11u  IPv4   33818      0t0  TCP localhost:ipp (LISTEN)
container    968   root   15u  IPv4   34287      0t0  TCP localhost:45763 (LISTEN)
dovecot     1047   root   25u  IPv4   32668      0t0  TCP *:pop3 (LISTEN)

5.列出在特定端口上运行的所有 TCP 和 UDP 进程：lsof -i TCP/UDP:port

[root@k8s-master ~]# lsof -i TCP:6443
COMMAND    PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
kubelet   2050 root   21u  IPv4   49332      0t0  TCP mail.linux.com:35288->mail.linux.com:sun-sr-https (ESTABLISHED)
kube-sche 2304 root    7u  IPv4   48456      0t0  TCP mail.linux.com:35298->mail.linux.com:sun-sr-https (ESTABLISHED)
kube-sche 2304 root    8u  IPv4   48619      0t0  TCP mail.linux.com:35490->mail.linux.com:sun-sr-https (ESTABLISHED)
kube-apis 2362 root    3u  IPv6   48445      0t0  TCP *:sun-sr-https (LISTEN)
kube-apis 2362 root   67u  IPv6   49518      0t0  TCP mail.linux.com:sun-sr-https->mail.linux.com:35288 (ESTABLISHED)
kube-apis 2362 root   68u  IPv6   49546      0t0  TCP mail.linux.com:sun-sr-https->mail.linux.com:35490 (ESTABLISHED)
kube-apis 2362 root   69u  IPv6   52350      0t0  TCP mail.linux.com:sun-sr-https->mail.linux.com:35498 (ESTABLISHED)

6.查看指定设备的所有打开文件：lsof <device-name>

[root@k8s-master ~]# lsof /dev/sda3 |more
COMMAND      PID           USER   FD   TYPE DEVICE  SIZE/OFF     NODE NAME
systemd        1           root  cwd    DIR    8,3       244       64 /
systemd        1           root  rtd    DIR    8,3       244       64 /
systemd        1           root  txt    REG    8,3   1632960 17033053 /usr/lib/systemd/systemd
systemd        1           root  mem    REG    8,3     20064   167261 /usr/lib64/libuuid.so.1.3.0
systemd        1           root  mem    REG    8,3    265576  1183915 /usr/lib64/libblkid.so.1.1.0
systemd        1           root  mem    REG    8,3     90160   167255 /usr/lib64/libz.so.1.2.7
systemd        1           root  mem    REG    8,3    157440   167263 /usr/lib64/liblzma.so.5.2.2
systemd        1           root  mem    REG    8,3     23968   167346 /usr/lib64/libcap-ng.so.0.0.0

7.列出与kube-api应用程序关联的打开文件:lsof -c kube-api

[root@k8s-master ~]# lsof -c kube-api
COMMAND    PID USER   FD      TYPE  DEVICE  SIZE/OFF     NODE NAME
kube-apis 2362 root  cwd       DIR    0,95        17 35602229 /
kube-apis 2362 root  rtd       DIR    0,95        17 35602229 /
kube-apis 2362 root  txt       REG    0,95 116572160 35602231 /usr/local/bin/kube-apiserver
kube-apis 2362 root  mem       REG     8,3           35602231 /usr/local/bin/kube-apiserver (stat: No such file or directory)
kube-apis 2362 root    0u      CHR     1,3       0t0    49190 /dev/null
kube-apis 2362 root    1w     FIFO     0,9       0t0    48104 pipe
kube-apis 2362 root    2w     FIFO     0,9       0t0    48105 pipe
kube-apis 2362 root    3u     IPv6   48445       0t0      TCP *:sun-sr-https (LISTEN)
kube-apis 2362 root    4u  a_inode    0,10         0     7512 [eventpoll]
kube-apis 2362 root    5r     FIFO     0,9       0t0    48433 pipe
kube-apis 2362 root    6w     FIFO     0,9       0t0    48433 pipe
kube-apis 2362 root    7u     IPv4   48451       0t0      TCP localhost:35190->localhost:2379 (ESTABLISHED)
kube-apis 2362 root    8u     IPv4   48457       0t0      TCP localhost:35196->localhost:2379 (ESTABLISHED)
kube-apis 2362 root    9u     IPv4   49417       0t0      TCP localhost:35198->localhost:2379 (ESTABLISHED)
kube-apis 2362 root   10u     IPv4   48485       0t0      TCP localhost:35212->localhost:2379 (ESTABLISHED)
kube-apis 2362 root   11u     IPv4   49444       0t0      TCP localhost:35218->localhost:2379 (ESTABLISHED)

8.列出属于某个进程 ID 的所有进程: lsof -p $pid

[root@k8s-master ~]# lsof -p 2362
COMMAND    PID USER   FD      TYPE  DEVICE  SIZE/OFF     NODE NAME
kube-apis 2362 root  cwd       DIR    0,95        17 35602229 /
kube-apis 2362 root  rtd       DIR    0,95        17 35602229 /
kube-apis 2362 root  txt       REG    0,95 116572160 35602231 /usr/local/bin/kube-apiserver
kube-apis 2362 root  mem       REG     8,3           35602231 /usr/local/bin/kube-apiserver (stat: No such file or directory)
kube-apis 2362 root    0u      CHR     1,3       0t0    49190 /dev/null
kube-apis 2362 root    1w     FIFO     0,9       0t0    48104 pipe
kube-apis 2362 root    2w     FIFO     0,9       0t0    48105 pipe
kube-apis 2362 root    3u     IPv6   48445       0t0      TCP *:sun-sr-https (LISTEN)
kube-apis 2362 root    4u  a_inode    0,10         0     7512 [eventpoll]
kube-apis 2362 root    5r     FIFO     0,9       0t0    48433 pipe
kube-apis 2362 root    6w     FIFO     0,9       0t0    48433 pipe