Last night while I was reading "Systems Performance: Enterprise and the Cloud" and trying different BCC tools, I ran into a stack buffer overflow in offcputime:
1 λ sudo offcputime
2 *** stack smashing detected ***: terminated
3 [1] 535011 abort sudo offcputime
"stack smashing" aka "stack buffer overflow" occurs when you write beyond the bounds of a stack buffer. To detect this, compilers often insert a stack canary: a random value inserted between local variables and saved data in the stack (e.g. return address). Before the function returns, it checks whether the value has changed, to abort with the message above.
I immediately launched gdb to make sense of this overflow
1 λ sudo -E gdb offcputime
2
3 gef> run
4 Starting program: /usr/bin/offcputime
5 *** stack smashing detected ***: terminated
6
7 -------------------------------------------
8 reason: SIGABRT
9 -------------------------------------------
10 ...
11 [ #2] 0x7ffff7c2567d <abort+0x26>
12 ...
13 [ #5] 0x7ffff7d28cb4 <NO_SYMBOL>
14 [ #6] 0x55555555aab8 <NO_SYMBOL>
15 [ #7] 0x555555557c52 <NO_SYMBOL>
16 ...
So something in frame #6 near 0x55555555aab8 caused an overflow. And as you can see, the offcputime binary had no debugging information (NO_SYMBOL). Usually, I would use a decompiler such as Binary Ninja, but why go through the trouble when the program is open source?
After cloning iovisor/bcc and compiling offcputime with debug info the issue became obvious:
1 λ sudo -E gdb offcputime
2
3 gef> run
4 Starting program: /usr/bin/offcputime
5 *** stack smashing detected ***: terminated
6 ------------------------------------------------------------------
7 [*Thread Id:1, tid:536367] Name: "offcputime", stopped at 0x7ffff7c9981c <NO_SYMBOL>, reason: SIGABRT
8 ------------------------------------------------------------------
9 [*#0] 0x7ffff7c9981c <NO_SYMBOL>
10 [ #1] 0x7ffff7c3e150 <raise+0x20>
11 [ #2] 0x7ffff7c2567d <abort+0x26>
12 [ #3] 0x7ffff7c266fc <NO_SYMBOL>
13 [ #4] 0x7ffff7d278d0 <__fprintf_chk> (frame name: __fortify_fail)
14 [ #5] 0x7ffff7d28cb4 <NO_SYMBOL>
15 [ #6] 0x55555555fad7 <NO_SYMBOL>
16 [ #7] 0x55555555dcd0 <main+0x340>
17 [ #8] 0x7ffff7c27741 <NO_SYMBOL>
18 [ #9] 0x7ffff7c27879 <__libc_start_main+0x89>
19 [...]
20 -----------------------------------------------------------------------------------------------
21 gef> frame 7
22 #7 0x000055555555dcd0 in main (argc=<optimized out>, argv=<optimized out>)
23 at offcputime.c:411
24 411 ksyms = ksyms__load();
25 gef> frame 6
26 #6 0x000055555555fad7 in ksyms__load () at trace_helpers.c:138
27 138 }
28 gef>
29
The overflow occurs in the ksyms__load function
1 struct ksyms *ksyms__load(void)
2 {
3 char sym_type, sym_name[256];
4 struct ksyms *ksyms;
5 unsigned long sym_addr;
6 int i, ret;
7 FILE *f;
8
9 f = fopen("/proc/kallsyms", "r");
10 if (!f)
11 return NULL;
12 /* .. omitted for brevity .. */
13 while (true) {
14 ret = fscanf(f, "%lx %c %s%*[^\n]\n",
15 &sym_addr, &sym_type, sym_name);
16 }
If you didn't know, /proc/kallsyms shows all the symbols exposed by the kernel. Here is an example:
1 λ head /proc/kallsyms 2 0000000000000000 T srso_alias_untrain_ret 3 0000000000000000 T _stext 4 0000000000000000 T _text 5 0000000000000000 T __pi__text 6 0000000000000000 T __entry_text_start 7 0000000000000000 T x86_verw_sel 8 0000000000000000 T entry_SYSCALL_64 9 0000000000000000 T entry_SYSCALL_64_safe_stack 10 0000000000000000 t syscall_return_via_sysret The first column is the address of the symbol in kernel space. The reason they're all zero here is because I ran the command as a regular user, and only root can see the addresses, otherwise attackers can use /proc/kallsyms to defeat KASLR
The problem is that ksyms__load reads /proc/kallsyms and assumes the maximum symbol length is 255. Let's see what's the longest symbol in my kernel:
λ cat /proc/kallsyms | awk '{if(length($3) > max) {max=length($3); word=$3}} END {word}'
__pfx__RINvMs_NtNtCscc4isTt46bi_11rust_binder11range_alloc4treeINtB5_10DescriptorNtNtB9_10allocation14AllocationInfoE16try_change_stateNCNvMB5_INtB5_18TreeRangeAllocatorB1a_E17reservation_aborts_0TNtB7_11ReservationINtNtCsgUPw2o9Q1aM_6kernel6rbtree21RBTreeNodeReservationTjjEuEEEB9_
That's 282 bytes! So fscanf into sym_name would obviously overflow.
It's apparently a mangled Rust symbol, and I'm using Linux 7.1 which has Rust in it. I imagine the BCC CI uses an older kernel without Rust, or without that particular symbol.
I started wondering if there is a max length. I checked the kernel source and found this defined in kallsyms.h:
1 // include/linux/kallsyms.h
2 #define KSYM_NAME_LEN 512
I then checked the last time this changed, which was 5 years ago:
kallsyms: increase maximum kernel symbol length to 512
Rust symbols can become quite long due to namespacing...
...
At the moment, Rust symbols may reach up to 300 in length. Setting 512 as the maximum seems like a reasonable choice to keep some headroom.
This explains the discrepancy, ksyms__load was written before this change.
I was ready to report this and submit a patch, but I found it's already reported and diagnosed and there is a pending PR to increase the buffer size to 2048.