crash —— 查看进程的内核栈的内容

发布时间 2023-11-04 20:56:58作者: 摩斯电码

有时我们有这样的需求,想查看某个进程的内核栈里的全部内容,介绍下面的几种做法:

方法1:得到进程的栈底地址以及栈大小,然后使用rd指令读取

  • 使用mach指令获取内核栈的大小
crash> mach | grep SIZE
           MEMORY SIZE: 64 GB
             PAGE SIZE: 4096
     KERNEL STACK SIZE: 16384
        IRQ STACK SIZE: 16384
DOUBLEFAULT STACK SIZE: 8192
        NMI STACK SIZE: 8192
      DEBUG STACK SIZE: 8192
        MCE STACK SIZE: 8192
         VC STACK SIZE: 8192

上面KERNEL STACK SIZE表示的就是内核栈的大小,这里是16KB

  • 获取进程的内核栈栈底

就以出发宕机的CPU上当时运行的进程为例。

crash> task -R stack
PID: 80876    TASK: ffff9519080da880  CPU: 42   COMMAND: "bash"
  stack = 0xffffa4e249fb4000,
  • 读取

rd命令默认是按8字节为单位,所以16KB的话,需要读取2KB,也就是0x800,此外,加入-s选项,这样可以将内核栈里的函数符号翻译成符号名加偏移的格式。

点击查看代码 ```bash crash> rd -s 0xffffa4e249fb4000 0x800 ffffa4e249fb4000: 0000000057ac6e9d 0000000000000000 ffffa4e249fb4010: 0000000000000000 0000000000000000 ffffa4e249fb4020: 0000000000000000 0000000000000000 ffffa4e249fb4030: 0000000000000000 0000000000000000 ... ffffa4e249fb7810: b16b5403c9b4b600 .LC6+977 ffffa4e249fb7820: ffffa4e249fb794f ffffa4e249fb78a8 ffffa4e249fb7830: ffffa4e249fb79bf ffffa4e2c9fb79b7 ffffa4e249fb7840: 0000000000000006 number+834 ffffa4e249fb7850: 00000000000e104d 0010000000000001 ffffa4e249fb7860: 0000000000ffff0a 0000393231363737 ffffa4e249fb7870: 0000000000000000 0000000000000000 ffffa4e249fb7880: b16b5403c9b4b600 .LC6+977 ffffa4e249fb7890: ffffa4e249fb79bf ffffa4e249fb7918 ffffa4e249fb78a0: .LC6+977 .LC6+977 ffffa4e249fb78b0: ffffa4e249fb79c5 vsnprintf+710 ffffa4e249fb78c0: ffffa4e249fb79b8 0000000000000001 ffffa4e249fb78d0: ffff0a1000000600 b16b5403c9b4b600 ffffa4e249fb78e0: 0000000000000001 ffffa4e249fb7968 ffffa4e249fb78f0: 0000000000000000 ffffa4e249fb7a40 ffffa4e249fb7900: 0000000000000400 0000000000000000 ffffa4e249fb7910: sprintf+94 0000000000000020 ffffa4e249fb7920: ffffa4e249fb7978 ffffa4e249fb7938 ffffa4e249fb7930: b16b5403c9b4b600 0000000000000000 ffffa4e249fb7940: ffffa4e249fb79b0 desc_read_finalized_seq+55 ffffa4e249fb7950: 0000000000000000 b16b5403c9b4b600 ffffa4e249fb7960: 00000000000005ae ffffa4e249fb79a8 ffffa4e249fb7970: _prb_read_valid+636 ffff95280f6a0000 ffffa4e249fb7980: 0000000000000000 ffffa4e249fb7a28 ffffa4e249fb7990: 0000000000000000 ffffffff00b21d20 ffffa4e249fb79a0: _raw_spin_trylock+19 0000000000000000 ffffa4e249fb79b0: c0000000ffffbfff 0000000000000001 ffffa4e249fb79c0: 0000000000000001 b16b5403c9b4b600 ffffa4e249fb79d0: vt_console_driver 0000000000000000 ffffa4e249fb79e0: ffffa4e249fb7b1f dropped_text.12 ffffa4e249fb79f0: vt_console_driver 0000000000000001 ffffa4e249fb7a00: prb_read_valid+23 00000000000005ae ffffa4e249fb7a10: console_emit_next_record.constprop.0+131 0000000049fb7aac ffffa4e249fb7a20: 0000000000000082 ffffa4e249fb7a40 ffffa4e249fb7a30: text.14 0000000000000400 ffffa4e249fb7a40: 0000000000000000 desc_read_finalized_seq+55 ffffa4e249fb7a50: append_elf_note+103 ffff95200fd7b400 ffffa4e249fb7a60: ffffa4e249fb7ad0 ffffa4e249fb7d90 ffffa4e249fb7a70: 0000000000000000 0000000000000000 ffffa4e249fb7a80: sysrq_crash_op crash_save_cpu+596 ffffa4e249fb7a90: 0000000000000003 ffffa4e249fb7aa8 ffffa4e249fb7aa0: 0000000000000000 0000000000000000 ffffa4e249fb7ab0: 0000000000000000 0000000000000001 ffffa4e249fb7ac0: 0000000000000000 ffffa4e249fb7d08 ffffa4e249fb7ad0: 0000000000000000 0000000000000000 ffffa4e249fb7ae0: 0000000000000000 0000000000000000 ffffa4e249fb7af0: 0000000000013bec 0000000000000000 ffffa4e249fb7b00: 0000000000000000 0000000000000000 ffffa4e249fb7b10: 0000000000000000 0000000000000000 ffffa4e249fb7b20: 0000000000000000 0000000000000000 ffffa4e249fb7b30: 0000000000000000 0000000000000000 ffffa4e249fb7b40: 0000000000000000 sysrq_crash_op ffffa4e249fb7b50: 0000000000000000 0000000000000000 ffffa4e249fb7b60: ffffa4e249fb7d90 ffffa4e249fb7c60 ffffa4e249fb7b70: ffff95282ff43c28 0000000000000003 ffffa4e249fb7b80: ffffa4e249fb7aa8 0000000000000000 ffffa4e249fb7b90: 0000000000000000 0000000000000000 ffffa4e249fb7ba0: 0000000000000001 0000000000000000 ffffa4e249fb7bb0: ffffa4e249fb7d08 0000000000000000 ffffa4e249fb7bc0: 0000000000000000 0000000000000046 ffffa4e249fb7bd0: 0000000000000000 sysrq_crash_op ffffa4e249fb7be0: 0000000000000000 0000000000000001 ffffa4e249fb7bf0: ffff952036371c00 ffff9518ad003000 ffffa4e249fb7c00: machine_kexec+460 0000000000000000 ffffa4e249fb7c10: 0000000000000000 00000000ad003000 ffffa4e249fb7c20: ffff9518ad003000 00000000ad002000 ffffa4e249fb7c30: 0000000000000000 b16b5403c9b4b600 ffffa4e249fb7c40: ffffa4e249fb7c60 ffffa4e249fb7d90 ffffa4e249fb7c50: 0000000000000000 __crash_kexec+119 ffffa4e249fb7c60: 0000000000000000 sysrq_crash_op ffffa4e249fb7c70: 0000000000000000 0000000000000000 ffffa4e249fb7c80: ffffa4e249fb7d90 ffffa4e249fb7c60 ffffa4e249fb7c90: ffff95282ff43c28 0000000000000003 ffffa4e249fb7ca0: ffffa4e249fb7aa8 0000000000000000 ffffa4e249fb7cb0: 0000000000000000 0000000000000000 ffffa4e249fb7cc0: 0000000000000001 0000000000000000 ffffa4e249fb7cd0: ffffa4e249fb7d08 0000000000000000 ffffa4e249fb7ce0: __crash_kexec+165 0000000000000010 ffffa4e249fb7cf0: 0000000000000046 ffffa4e249fb7c60 ffffa4e249fb7d00: 0000000000000018 b16b5403c9b4b600 ffffa4e249fb7d10: .LC3+721 panic+769 ffffa4e249fb7d20: 0000000000000008 ffffa4e249fb7da0 ffffa4e249fb7d30: ffffa4e249fb7d40 b16b5403c9b4b600 ffffa4e249fb7d40: _printk+96 .LC0+9759 ffffa4e249fb7d50: 0000000000000000 0000000000000000 ffffa4e249fb7d60: 0000000000000000 ffffa4e249fb7c50 ffffa4e249fb7d70: 0000000000000000 0000000000000063 ffffa4e249fb7d80: 0000000000000000 sysrq_crash_op ffffa4e249fb7d90: 0000000000000001 sysrq_handle_crash+22 ffffa4e249fb7da0: __handle_sysrq+166 0000000000000002 ffffa4e249fb7db0: ffff95191401f500 ffffa4e249fb7e98 ffffa4e249fb7dc0: 0000564e2104fae0 ffff95190d21d000 ffffa4e249fb7dd0: write_sysrq_trigger+36 ffff952035325680 ffffa4e249fb7de0: proc_reg_write+86 preempt_count_add+71 ffffa4e249fb7df0: 0000000000000002 vfs_write+199 ffffa4e249fb7e00: 0000000000000cc0 0000000564e2104f ffffa4e249fb7e10: 0000564e2104f000 0000564e2104fae0 ffffa4e249fb7e20: 0000000000000a55 ffff951910f2e840 ffffa4e249fb7e30: ffff95190a4029c0 8000000182121865 ffffa4e249fb7e40: 0000000000000000 ffffcf21c6084840 ffffa4e249fb7e50: ffff95192eac9278 b16b5403c9b4b600 ffffa4e249fb7e60: ffff95191401f500 ffff95191401f500 ffffa4e249fb7e70: 0000000000000002 0000564e2104fae0 ffffa4e249fb7e80: 0000000000000000 0000000000000000 ffffa4e249fb7e90: ksys_write+107 0000000000000000 ffffa4e249fb7ea0: b16b5403c9b4b600 ffffa4e249fb7f58 ffffa4e249fb7eb0: ffffa4e249fb7f48 0000000000000000 ffffa4e249fb7ec0: 0000000000000000 do_syscall_64+91 ffffa4e249fb7ed0: 0000000000000007 do_user_addr_fault+495 ffffa4e249fb7ee0: 0000000000000000 0000000000000002 ffffa4e249fb7ef0: 0000000000000000 ffffa4e249fb7f58 ffffa4e249fb7f00: 0000564e2104fae0 0000000000000000 ffffa4e249fb7f10: 0000000000000007 0000000000000000 ffffa4e249fb7f20: 0000000000000000 exc_page_fault+112 ffffa4e249fb7f30: 0000000000000000 0000000000000000 ffffa4e249fb7f40: 0000000000000000 0000000000000000 ffffa4e249fb7f50: entry_SYSCALL_64_after_hwframe+114 0000000000000002 ffffa4e249fb7f60: 00007fa6121b4780 0000564e2104fae0 ffffa4e249fb7f70: 0000000000000002 00007ffe5cf897a0 ffffa4e249fb7f80: 0000000000000002 0000000000000202 ffffa4e249fb7f90: 0000000000000000 0000000100000000 ffffa4e249fb7fa0: 0000000000001428 ffffffffffffffda ffffa4e249fb7fb0: 00007fa6120e0164 0000000000000002 ffffa4e249fb7fc0: 0000564e2104fae0 0000000000000001 ffffa4e249fb7fd0: 0000000000000001 00007fa6120e0164 ffffa4e249fb7fe0: 0000000000000033 0000000000000202 ffffa4e249fb7ff0: 00007ffe5cf89778 000000000000002b ```

方法2:直接使用bt -r指令

点击查看代码
crash> bt -r
PID: 80876    TASK: ffff9519080da880  CPU: 42   COMMAND: "bash"
ffffa4e249fb4000:  0000000057ac6e9d 0000000000000000
ffffa4e249fb4010:  0000000000000000 0000000000000000
ffffa4e249fb4020:  0000000000000000 0000000000000000
ffffa4e249fb4030:  0000000000000000 0000000000000000
ffffa4e249fb4040:  0000000000000000 0000000000000000
ffffa4e249fb4050:  0000000000000000 0000000000000000
ffffa4e249fb4060:  0000000000000000 0000000000000000
ffffa4e249fb4070:  0000000000000000 0000000000000000
ffffa4e249fb4080:  0000000000000000 0000000000000000
ffffa4e249fb4090:  0000000000000000 0000000000000000
ffffa4e249fb40a0:  0000000000000000 0000000000000000
ffffa4e249fb40b0:  0000000000000000 0000000000000000
...
ffffa4e249fb7c40:  ffffa4e249fb7c60 ffffa4e249fb7d90
ffffa4e249fb7c50:  0000000000000000 __crash_kexec+119
ffffa4e249fb7c60:  0000000000000000 sysrq_crash_op
ffffa4e249fb7c70:  0000000000000000 0000000000000000
ffffa4e249fb7c80:  ffffa4e249fb7d90 ffffa4e249fb7c60
ffffa4e249fb7c90:  ffff95282ff43c28 0000000000000003
ffffa4e249fb7ca0:  ffffa4e249fb7aa8 0000000000000000
ffffa4e249fb7cb0:  0000000000000000 0000000000000000
ffffa4e249fb7cc0:  0000000000000001 0000000000000000
ffffa4e249fb7cd0:  ffffa4e249fb7d08 0000000000000000
ffffa4e249fb7ce0:  __crash_kexec+165 0000000000000010
ffffa4e249fb7cf0:  0000000000000046 ffffa4e249fb7c60
ffffa4e249fb7d00:  0000000000000018 b16b5403c9b4b600
ffffa4e249fb7d10:  .LC3+721         panic+769
ffffa4e249fb7d20:  0000000000000008 ffffa4e249fb7da0
ffffa4e249fb7d30:  ffffa4e249fb7d40 b16b5403c9b4b600
ffffa4e249fb7d40:  _printk+96       .LC0+9759
ffffa4e249fb7d50:  0000000000000000 0000000000000000
ffffa4e249fb7d60:  0000000000000000 ffffa4e249fb7c50
ffffa4e249fb7d70:  0000000000000000 0000000000000063
ffffa4e249fb7d80:  0000000000000000 sysrq_crash_op
ffffa4e249fb7d90:  0000000000000001 sysrq_handle_crash+22
ffffa4e249fb7da0:  __handle_sysrq+166 0000000000000002
ffffa4e249fb7db0:  ffff95191401f500 ffffa4e249fb7e98
ffffa4e249fb7dc0:  0000564e2104fae0 ffff95190d21d000
ffffa4e249fb7dd0:  write_sysrq_trigger+36 ffff952035325680
ffffa4e249fb7de0:  proc_reg_write+86 preempt_count_add+71
ffffa4e249fb7df0:  0000000000000002 vfs_write+199
ffffa4e249fb7e00:  0000000000000cc0 0000000564e2104f
ffffa4e249fb7e10:  0000564e2104f000 0000564e2104fae0
ffffa4e249fb7e20:  0000000000000a55 ffff951910f2e840
ffffa4e249fb7e30:  ffff95190a4029c0 8000000182121865
ffffa4e249fb7e40:  0000000000000000 ffffcf21c6084840
ffffa4e249fb7e50:  ffff95192eac9278 b16b5403c9b4b600
ffffa4e249fb7e60:  ffff95191401f500 ffff95191401f500
ffffa4e249fb7e70:  0000000000000002 0000564e2104fae0
ffffa4e249fb7e80:  0000000000000000 0000000000000000
ffffa4e249fb7e90:  ksys_write+107   0000000000000000
ffffa4e249fb7ea0:  b16b5403c9b4b600 ffffa4e249fb7f58
ffffa4e249fb7eb0:  ffffa4e249fb7f48 0000000000000000
ffffa4e249fb7ec0:  0000000000000000 do_syscall_64+91
ffffa4e249fb7ed0:  0000000000000007 do_user_addr_fault+495
ffffa4e249fb7ee0:  0000000000000000 0000000000000002
ffffa4e249fb7ef0:  0000000000000000 ffffa4e249fb7f58
ffffa4e249fb7f00:  0000564e2104fae0 0000000000000000
ffffa4e249fb7f10:  0000000000000007 0000000000000000
ffffa4e249fb7f20:  0000000000000000 exc_page_fault+112
ffffa4e249fb7f30:  0000000000000000 0000000000000000
ffffa4e249fb7f40:  0000000000000000 0000000000000000
ffffa4e249fb7f50:  entry_SYSCALL_64_after_hwframe+114 0000000000000002
ffffa4e249fb7f60:  00007fa6121b4780 0000564e2104fae0
ffffa4e249fb7f70:  0000000000000002 00007ffe5cf897a0
ffffa4e249fb7f80:  0000000000000002 0000000000000202
ffffa4e249fb7f90:  0000000000000000 0000000100000000
ffffa4e249fb7fa0:  0000000000001428 ffffffffffffffda
ffffa4e249fb7fb0:  00007fa6120e0164 0000000000000002
ffffa4e249fb7fc0:  0000564e2104fae0 0000000000000001
ffffa4e249fb7fd0:  0000000000000001 00007fa6120e0164
ffffa4e249fb7fe0:  0000000000000033 0000000000000202
ffffa4e249fb7ff0:  00007ffe5cf89778 000000000000002b

方法3:还是使用bt -T/t指令

方法2使用bt指令把内核栈的内容全部输出了出来,输出内容中有很多没有用的内容,此外,bt还提供了-T/t参数,这样会把内核栈里可以解析的部分打印出来

crash> bt -T
PID: 80876    TASK: ffff9519080da880  CPU: 42   COMMAND: "bash"
  [ffffa4e249fb7578] vsnprintf at ffffffffb9f4b8a4
  [ffffa4e249fb75d0] sprintf at ffffffffb9f4bb6e
  [ffffa4e249fb7630] __sprint_symbol at ffffffffb91fbf34
  [ffffa4e249fb7688] symbol_string at ffffffffb9f48185
  [ffffa4e249fb7758] number at ffffffffb9f464e2
  [ffffa4e249fb77d8] number at ffffffffb9f464e2
  [ffffa4e249fb7848] number at ffffffffb9f464e2
  [ffffa4e249fb78b8] vsnprintf at ffffffffb9f4b776
  [ffffa4e249fb7910] sprintf at ffffffffb9f4bb6e
  [ffffa4e249fb7948] desc_read_finalized_seq at ffffffffb919d0b7
  [ffffa4e249fb7970] _prb_read_valid at ffffffffb919d3ac
  [ffffa4e249fb79a0] _raw_spin_trylock at ffffffffb9f71743
  [ffffa4e249fb7a00] prb_read_valid at ffffffffb919e0d7
  [ffffa4e249fb7a10] console_emit_next_record.constprop.0 at ffffffffb9199f83
  [ffffa4e249fb7a48] desc_read_finalized_seq at ffffffffb919d0b7
  [ffffa4e249fb7a50] append_elf_note at ffffffffb91fdb37
  [ffffa4e249fb7a88] crash_save_cpu at ffffffffb91ff6b4
  [ffffa4e249fb7c00] machine_kexec at ffffffffb90839cc
  [ffffa4e249fb7c58] __crash_kexec at ffffffffb91fe047
  [ffffa4e249fb7ce0] __crash_kexec at ffffffffb91fe075
  [ffffa4e249fb7d18] panic at ffffffffb910c471
  [ffffa4e249fb7d40] _printk at ffffffffb91987e0
  [ffffa4e249fb7d98] sysrq_handle_crash at ffffffffb99d2596
  [ffffa4e249fb7da0] __handle_sysrq at ffffffffb99d2b16
  [ffffa4e249fb7dd0] write_sysrq_trigger at ffffffffb99d31b4
  [ffffa4e249fb7de0] proc_reg_write at ffffffffb94ea8e6
  [ffffa4e249fb7de8] preempt_count_add at ffffffffb914c8e7
  [ffffa4e249fb7df8] vfs_write at ffffffffb94404d7
  [ffffa4e249fb7e90] ksys_write at ffffffffb94409eb
  [ffffa4e249fb7ec8] do_syscall_64 at ffffffffb9f5618b
  [ffffa4e249fb7ed8] do_user_addr_fault at ffffffffb909706f
  [ffffa4e249fb7f28] exc_page_fault at ffffffffb9f5b4c0
  [ffffa4e249fb7f50] entry_SYSCALL_64_after_hwframe at ffffffffba0000aa
    RIP: 00007fa6120e0164  RSP: 00007ffe5cf89778  RFLAGS: 00000202
    RAX: ffffffffffffffda  RBX: 0000000000000002  RCX: 00007fa6120e0164
    RDX: 0000000000000002  RSI: 0000564e2104fae0  RDI: 0000000000000001
    RBP: 00007ffe5cf897a0   R8: 0000000000001428   R9: 0000000100000000
    R10: 0000000000000000  R11: 0000000000000202  R12: 0000000000000002
    R13: 0000564e2104fae0  R14: 00007fa6121b4780  R15: 0000000000000002
    ORIG_RAX: 0000000000000001  CS: 0033  SS: 002b