linux下程序启动即推出的原因定位分析过程

现象：

安装 todesk 的 deb包并配置 daemon 后，todesk无法运行.
查看程序 status 提示：
Process: 168846 ExecStart=/opt/todesk/bin/ToDesk_Service (code=killed, signal=ILL)

结论：

引发当前 todesk 启动即退出的原因是当前 CPU指令集不受软件功能函数调用支持。

同样有此困扰的软件还有 MongoBD MySQL ，原因都是较新的软件版本函数特性与较旧的 CPU指令集不详匹配。

分析过程：

root@Debian-H81:~#
root@Debian-H81:~# todesk
root@Debian-H81:~#
root@Debian-H81:~# systemctl status todeskd.service
● todeskd.service - ToDesk Daemon Service
Loaded: loaded (/etc/systemd/system/todeskd.service; enabled; preset: enabled)
Active: activating (auto-restart) (Result: signal) since Sun 2024-02-25 09:31:48 CST; 2s ago
Process: 168763 ExecStart=/opt/todesk/bin/ToDesk_Service (code=killed, signal=ILL)
Main PID: 168763 (code=killed, signal=ILL)
CPU: 28ms
root@Debian-H81:~#
root@Debian-H81:~# systemctl start todeskd.service
root@Debian-H81:~#
root@Debian-H81:~# systemctl status todeskd.service
● todeskd.service - ToDesk Daemon Service
Loaded: loaded (/etc/systemd/system/todeskd.service; enabled; preset: enabled)
Active: activating (auto-restart) (Result: signal) since Sun 2024-02-25 09:32:11 CST; 459ms ago
Process: 168846 ExecStart=/opt/todesk/bin/ToDesk_Service (code=killed, signal=ILL)
Main PID: 168846 (code=killed, signal=ILL)
CPU: 23ms
root@Debian-H81:~#
root@Debian-H81:~#

分析过程：
先查看 todesk的日志，发现日至文件大小为0（即文件内容为空）

root@Debian-H81:~#
root@Debian-H81:~# ls -Fl /var/log/todesk/
total 0
-rw-r--r-- 1 root root 0 Feb 24 20:10 sdkservice_2024_02_24.log
-rw-r--r-- 1 root root 0 Feb 25 00:00 sdkservice_2024_02_25.log
-rw-r--r-- 1 root root 0 Feb 24 20:10 service_2024_02_24.log
-rw-r--r-- 1 root root 0 Feb 25 00:00 service_2024_02_25.log
-rw-r--r-- 1 root root 0 Feb 24 20:10 zrtcservice_2024_02_24.log
-rw-r--r-- 1 root root 0 Feb 25 00:00 zrtcservice_2024_02_25.log
root@Debian-H81:~#

再查看OS的系统日志，sys-log 打印如下

root@Debian-H81:~#
root@Debian-H81:~# dmesg -T | grep -i "todesk" | tail -n 20
[Sun Feb 25 09:37:27 2024] traps: ToDesk_Service[169950] trap invalid opcode ip:4d06da sp:7fff8f513f20 error:0 in ToDesk_Service[400000+140f000]
[Sun Feb 25 09:37:31 2024] traps: ToDesk_Service[169961] trap invalid opcode ip:4d06da sp:7ffe83d255a0 error:0 in ToDesk_Service[400000+140f000]
[Sun Feb 25 09:37:34 2024] traps: ToDesk_Service[169974] trap invalid opcode ip:4d06da sp:7ffe8d386380 error:0 in ToDesk_Service[400000+140f000]
[Sun Feb 25 09:37:37 2024] traps: ToDesk_Service[169985] trap invalid opcode ip:4d06da sp:7ffdea6e0780 error:0 in ToDesk_Service[400000+140f000]
[Sun Feb 25 09:37:40 2024] traps: ToDesk_Service[169996] trap invalid opcode ip:4d06da sp:7ffd9eadb140 error:0 in ToDesk_Service[400000+140f000]
[Sun Feb 25 09:37:44 2024] traps: ToDesk_Service[170007] trap invalid opcode ip:4d06da sp:7ffcf65ea380 error:0 in ToDesk_Service[400000+140f000]
[Sun Feb 25 09:37:47 2024] traps: ToDesk_Service[170018] trap invalid opcode ip:4d06da sp:7fffabdb8700 error:0 in ToDesk_Service[400000+140f000]
[Sun Feb 25 09:37:50 2024] traps: ToDesk_Service[170029] trap invalid opcode ip:4d06da sp:7ffd4ba89180 error:0 in ToDesk_Service[400000+140f000]
[Sun Feb 25 09:37:53 2024] traps: ToDesk_Service[170042] trap invalid opcode ip:4d06da sp:7fff5ef6ea00 error:0 in ToDesk_Service[400000+140f000]
[Sun Feb 25 09:37:57 2024] traps: ToDesk_Service[170053] trap invalid opcode ip:4d06da sp:7ffeb02b4000 error:0 in ToDesk_Service[400000+140f000]
[Sun Feb 25 09:38:00 2024] traps: ToDesk_Service[170064] trap invalid opcode ip:4d06da sp:7fffe46610a0 error:0 in ToDesk_Service[400000+140f000]
[Sun Feb 25 09:38:03 2024] traps: ToDesk_Service[170075] trap invalid opcode ip:4d06da sp:7fff2ec8da80 error:0 in ToDesk_Service[400000+140f000]
[Sun Feb 25 09:38:06 2024] traps: ToDesk_Service[170086] trap invalid opcode ip:4d06da sp:7fff337ba8a0 error:0 in ToDesk_Service[400000+140f000]
[Sun Feb 25 09:38:10 2024] traps: ToDesk_Service[170097] trap invalid opcode ip:4d06da sp:7ffce5d91560 error:0 in ToDesk_Service[400000+140f000]
[Sun Feb 25 09:38:13 2024] traps: ToDesk_Service[170110] trap invalid opcode ip:4d06da sp:7fff21955b80 error:0 in ToDesk_Service[400000+140f000]
[Sun Feb 25 09:38:16 2024] traps: ToDesk_Service[170121] trap invalid opcode ip:4d06da sp:7ffd8449b880 error:0 in ToDesk_Service[400000+140f000]
[Sun Feb 25 09:38:19 2024] traps: ToDesk_Service[170132] trap invalid opcode ip:4d06da sp:7ffe8c3f7540 error:0 in ToDesk_Service[400000+140f000]
[Sun Feb 25 09:38:23 2024] traps: ToDesk_Service[170145] trap invalid opcode ip:4d06da sp:7ffcec227100 error:0 in ToDesk_Service[400000+140f000]
[Sun Feb 25 09:38:26 2024] traps: ToDesk_Service[170156] trap invalid opcode ip:4d06da sp:7fff210b6d40 error:0 in ToDesk_Service[400000+140f000]
[Sun Feb 25 09:38:29 2024] traps: ToDesk_Service[170167] trap invalid opcode ip:4d06da sp:7fff36d02b20 error:0 in ToDesk_Service[400000+140f000]
root@Debian-H81:~#

相关内容输出条目都是 trap 打印了不可用的操作代码，怀疑这是 linux-kernel 打印出来的内容。

再次查看 system 级别的日志：

root@Debian-H81:~#
root@Debian-H81:~# journalctl --system | grep -i "todesk" | tail -n 20
Feb 25 09:41:03 Debian-H81 systemd[1]: todeskd.service: Failed with result 'signal'.
Feb 25 09:41:03 Debian-H81 kernel: traps: ToDesk_Service[170702] trap invalid opcode ip:4d06da sp:7ffe59e10620 error:0 in ToDesk_Service[400000+140f000]
Feb 25 09:41:06 Debian-H81 systemd[1]: todeskd.service: Scheduled restart job, restart counter is at 13365.
Feb 25 09:41:06 Debian-H81 systemd[1]: Stopped todeskd.service - ToDesk Daemon Service.
Feb 25 09:41:06 Debian-H81 systemd[1]: Started todeskd.service - ToDesk Daemon Service.
Feb 25 09:41:08 Debian-H81 systemd[1]: todeskd.service: Main process exited, code=killed, status=4/ILL
Feb 25 09:41:08 Debian-H81 systemd[1]: todeskd.service: Failed with result 'signal'.
Feb 25 09:41:08 Debian-H81 kernel: traps: ToDesk_Service[170713] trap invalid opcode ip:4d06da sp:7ffd651d0760 error:0 in ToDesk_Service[400000+140f000]
Feb 25 09:41:12 Debian-H81 systemd[1]: todeskd.service: Scheduled restart job, restart counter is at 13366.
Feb 25 09:41:12 Debian-H81 systemd[1]: Stopped todeskd.service - ToDesk Daemon Service.
Feb 25 09:41:12 Debian-H81 systemd[1]: Started todeskd.service - ToDesk Daemon Service.
Feb 25 09:41:13 Debian-H81 systemd[1]: todeskd.service: Main process exited, code=killed, status=4/ILL
Feb 25 09:41:13 Debian-H81 systemd[1]: todeskd.service: Failed with result 'signal'.
Feb 25 09:41:13 Debian-H81 kernel: traps: ToDesk_Service[170726] trap invalid opcode ip:4d06da sp:7ffe931fece0 error:0 in ToDesk_Service[400000+140f000]
Feb 25 09:41:16 Debian-H81 systemd[1]: todeskd.service: Scheduled restart job, restart counter is at 13367.
Feb 25 09:41:16 Debian-H81 systemd[1]: Stopped todeskd.service - ToDesk Daemon Service.
Feb 25 09:41:16 Debian-H81 systemd[1]: Started todeskd.service - ToDesk Daemon Service.
Feb 25 09:41:16 Debian-H81 systemd[1]: todeskd.service: Main process exited, code=killed, status=4/ILL
Feb 25 09:41:16 Debian-H81 systemd[1]: todeskd.service: Failed with result 'signal'.
Feb 25 09:41:16 Debian-H81 kernel: traps: ToDesk_Service[170737] trap invalid opcode ip:4d06da sp:7fff94b542a0 error:0 in ToDesk_Service[400000+140f000]
root@Debian-H81:~#

从日志输出上看，这的确是kernel报告的情况，需要到 linux kernel 的对应模块中区查找对应的日志内容含义。

当前要做的就是确定自己的 OS上使用了那个版本的 linux - kernel ：

root@Debian-H81:~#
root@Debian-H81:~# hostnamectl status
Static hostname: Debian-H81
Icon name: computer-desktop
Chassis: desktop ️
Machine ID: 025c1b94eb45483b85ca811168c89854
Boot ID: 978473a80fc746638881c8fdfb9f63b9
Operating System: Debian GNU/Linux 12 (bookworm)
Kernel: Linux 6.1.0-18-amd64
Architecture: x86-64
Hardware Vendor: BIOSTAR Group
Hardware Model: H81MLV3
Firmware Version: 4.6.5
root@Debian-H81:~#

我当前的 Debian 12 上使用的 kernel 版本是： Linux 6.1.0-18-amd64

来到 kernel 站点下载对应的 kernel 程序包，并确认程序报中的 trap 模块所在

通过在 Debian 站点上搜索 kernel 信息，可知它对应了 Linus Tovals 的 linux-6.1.76

Package linux-headers-6.1.0-18-amd64

bookworm (stable) (kernel): Header files for Linux 6.1.0-18-amd64
6.1.76-1: amd64

Package linux-image-6.1.0-18-amd64
bookworm (stable) (kernel): Linux 6.1 for 64-bit PCs (signed)
6.1.76-1: amd64

对应版本的kernel下载连接：

https://www.kernel.org/
https://mirrors.edge.kernel.org/pub/linux/kernel/v6.x/linux-6.1.76.tar.gz

查看源文件中的 traps.c

debian@Debian-H81:~/Downloads$
debian@Debian-H81:~/Downloads$ cat -n /home/debian/Downloads/linux-6.1.76/arch/x86/kernel/traps.c | grep -i "invalid opcode"
297 do_error_trap(regs, 0, "invalid opcode", X86_TRAP_UD, SIGILL,
debian@Debian-H81:~/Downloads$
debian@Debian-H81:~/Downloads$ cat -n /home/debian/Downloads/linux-6.1.76/arch/x86/kernel/traps.c | grep -C 30 -i "invalid opcode"
267 UNWIND_HINT_FUNC
268 ANNOTATE_NOENDBR
269 " nop\\n\\t"
270
271 : "=a" (ret) : : "memory");
272
273 return !ret;
274 }
275
276 static int __init ibt_setup(char *str)
277 {
278 if (!strcmp(str, "off"))
279 setup_clear_cpu_cap(X86_FEATURE_IBT);
280
281 if (!strcmp(str, "warn"))
282 ibt_fatal = false;
283
284 return 1;
285 }
286
287 __setup("ibt=", ibt_setup);
288
289 #endif / CONFIG_X86_KERNEL_IBT /
290
291 #ifdef CONFIG_X86_F00F_BUG
292 void handle_invalid_op(struct pt_regs *regs)
293 #else
294 static inline void handle_invalid_op(struct pt_regs *regs)
295 #endif
296 {
297 do_error_trap(regs, 0, "invalid opcode", X86_TRAP_UD, SIGILL,
298 ILL_ILLOPN, error_get_trap_addr(regs));
299 }
300
301 static noinstr bool handle_bug(struct pt_regs *regs)
302 {
303 bool handled = false;
304
305 /*
306 * Normally @regs are unpoisoned by irqentry_enter(), but handle_bug()
307 * is a rare case that uses @regs without passing them to
308 * irqentry_enter().
309 */
310 kmsan_unpoison_entry_regs(regs);
311 if (!is_valid_bugaddr(regs->ip))
312 return handled;
313
314 /*
315 * All lies, just get the WARN/BUG out.
316 */
317 instrumentation_begin();
318 /*
319 * Since we're emulating a CALL with exceptions, restore the interrupt
320 * state to what it was at the exception site.
321 */
322 if (regs->flags & X86_EFLAGS_IF)
323 raw_local_irq_enable();
324 if (report_bug(regs->ip, regs) == BUG_TRAP_TYPE_WARN ||
325 handle_cfi_failure(regs) == BUG_TRAP_TYPE_WARN) {
326 regs->ip += LEN_UD2;
327 handled = true;
debian@Debian-H81:~/Downloads$

从第287行到300行的内容看，导致 todesk 启动即退出（Feb 25 10:26:00 Debian-H81 kernel: traps: ToDesk_Service[180347] trap invalid opcode ip:4d06da sp:7ffea5ceb1c0 error:0 in ToDesk_Service[400000+140f000 ）的原因应该是 KERNEL_IBT 无法处理传入到CPU寄存器的运算数引发的CPU诱骗关口报错。

那么 KERNEL_IBT 是用来干什么的呢？
IBT, 全称为 “indirect branch tracking“，是 Control-flow integrity 思想的一种具体实现，用于防御面向跳转编程的控制流，确保每个 indirect branch 的目标确实都是适合作为跳转目标的。英特尔处理器的 IBT 功能得到了硬件的支持，如果 IBT 被启用，那么 CPU 将确保每个间接跳转都落在一条特殊指令（endbr32 或 endbr64）上，该指令执行时跟 no-op 效果一致。如果发生意外，那么处理器将引发一次 control-protection（#CP）exception。