This is the mail archive of the systemtap@sourceware.org mailing list for the systemtap project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

[Bug runtime/7082] New: uprobes probes causing kernel problems


On both old and new kernels, uprobes probes can cause system hangs.  Here are
the testcases:

[1]
# stap -ve 'probe process("bash").function("*").return {}' -w -c bash
# exit

[2]
# stap -ve 'probe process("bash").function("*").return {}' -w -c bash
# bash
# exit
# exit

[3]
# stap -ve 'probe process("bash").function("main").return {}' -w -c bash
# exit

[4]
# stap -ve 'probe process("bash").function("main").return {}' -w -c bash
# bash
# exit
# exit


SYS1: On a beta RHEL5.3 i386 system running kernel 2.6.18-125.el5, systemtap
head, I get the following results:

[1] works
[2] system hangs:

uretprobes disabled for pid 16866: cannot set uretprobe trampoline at bfb9d000
------------[ cut here ]------------
kernel BUG at /usr/local/share/systemtap/runtime/uprobes/uprobes.c:2082!
invalid opcode: 0000 [#1]
SMP 
last sysfs file: /module/xfrm_nalgo/sections/__versions
Modules linked in: stap_2e548661a0b51d40f2e960c9c4d7a741_308985(U) uprobes(U)
autofs4 hidp rfcomm l2cap bluetooth sunrpc ipv6 xfrm_nalgo crypto_api
cpufreq_ondemand acpi_cpufreq dm_multipath scsi_dh video hwmon backlight sbs
i2c_ec button battery asus_acpi ac lp floppy snd_hda_intel snd_seq_dummy
snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss
snd_pcm snd_timer snd_page_alloc sr_mod snd_hwdep cdrom i2c_i801 snd e1000e
parport_pc e100 serio_raw parport soundcore i2c_core mii pcspkr sg dm_snapshot
dm_zero dm_mirror dm_log dm_mod ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd
ohci_hcd ehci_hcd
CPU:    3
EIP:    0060:[<f9012ac9>]    Tainted: G      VLI
EFLAGS: 00010202   (2.6.18-125.el5 #1) 
EIP is at uprobe_report_clone+0x1ad/0x2fd [uprobes]
eax: f5133a00   ebx: fffffff4   ecx: 00000202   edx: f5133400
esi: f5133400   edi: ef8d2240   ebp: efd35ac0   esp: f01cef30
ds: 007b   es: 007b   ss: 0068
Process bash (pid: 16865, ti=f01ce000 task=f4d3b000 task.ti=f01ce000)
Stack: f5133a00 f5133400 f4d3b000 fffffff4 f9013620 ef8d7c80 efb2e2e0 00000000 
       c0452eda f7fdb550 f7fdb550 01200011 f4d3b000 efb1f200 00000001 f7fdb550 
       01200011 00000000 c042356c 000041e2 00000073 bfb79f48 493d7a48 0832363f 
Call Trace:
 [<c0452eda>] utrace_report_clone+0x4e/0x148
 [<c042356c>] do_fork+0xd7/0x166
 [<c040318b>] sys_clone+0x28/0x2d
 [<c0404f17>] syscall_call+0x7/0xb
 =======================
Code: ff 85 c0 89 c5 75 08 0f 0b 6a 08 7b 37 01 f9 8b 55 18 8b 47 18 89 54 24 04
8b 9a c8 00 00 00 89 5c 24 0c 3b 98 c8 00 00 00 74 08 <0f> 0b 22 08 7b 37 01 f9
8b 7f 34 c7 44 24 08 00 00 00 00 e9 87 
EIP: [<f9012ac9>] uprobe_report_clone+0x1ad/0x2fd [uprobes] SS:ESP 0068:f01cef30
 <0>Kernel panic - not syncing: Fatal exception

[3] works

[4] works occasionally, but normally hangs system:

uretprobes disabled for pid 3508: cannot set uretprobe trampoline at bfe50000
------------[ cut here ]------------
kernel BUG at /usr/local/share/systemtap/runtime/uprobes/uprobes.c:2082!
invalid opcode: 0000 [#1]
SMP 
last sysfs file: /module/xfrm_nalgo/sections/__versions
Modules linked in: stap_005b533ff342bc467c6a9db8368dd2ae_324(U) uprobes(U)
autofs4 hidp rfcomm l2cap bluetooth sunrpc ipv6 xfrm_nalgo crypto_api
cpufreq_ondemand acpi_cpufreq dm_multipath scsi_dh video hwmon backlight sbs
i2c_ec button battery asus_acpi ac lp floppy snd_hda_intel snd_seq_dummy sr_mod
snd_seq_oss cdrom snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss
snd_mixer_oss snd_pcm snd_timer snd_page_alloc snd_hwdep e1000e snd i2c_i801
e100 sg parport_pc mii i2c_core parport serio_raw soundcore pcspkr dm_snapshot
dm_zero dm_mirror dm_log dm_mod ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd
ohci_hcd ehci_hcd
CPU:    3
EIP:    0060:[<f900eac9>]    Tainted: G      VLI
EFLAGS: 00010202   (2.6.18-125.el5 #1) 
EIP is at uprobe_report_clone+0x1ad/0x2fd [uprobes]
eax: f369d200   ebx: fffffff4   ecx: 00000202   edx: f369da00
esi: f369da00   edi: f35d6140   ebp: f32b3740   esp: f16baf30
ds: 007b   es: 007b   ss: 0068
Process bash (pid: 3507, ti=f16ba000 task=f329c000 task.ti=f16ba000)
Stack: f369d200 f369da00 f329c000 fffffff4 f900f620 f16c5ca0 f16c3500 00000000 
       c0452eda f329caa0 f329caa0 01200011 f329c000 f16c52c0 00000001 f329caa0 
       01200011 00000000 c042356c 00000db4 00000073 f319e400 493d7c90 2a879fc4 
Call Trace:
 [<c0452eda>] utrace_report_clone+0x4e/0x148
 [<c042356c>] do_fork+0xd7/0x166
 [<c040318b>] sys_clone+0x28/0x2d
 [<c0404f17>] syscall_call+0x7/0xb
 =======================
Code: ff 85 c0 89 c5 75 08 0f 0b 6a 08 7b f7 00 f9 8b 55 18 8b 47 18 89 54 24 04
8b 9a c8 00 00 00 89 5c 24 0c 3b 98 c8 00 00 00 74 08 <0f> 0b 22 08 7b f7 00 f9
8b 7f 34 c7 44 24 08 00 00 00 00 e9 87 
EIP: [<f900eac9>] uprobe_report_clone+0x1ad/0x2fd [uprobes] SS:ESP 0068:f16baf30
 <0>Kernel panic - not syncing: Fatal exception



SYS2: On a f9 (kvm) system running kernel 2.6.18-125.el5, systemtap head, I get
the following results:

[1] works

[2] bash reports "Trace/breakpoint trap" and the 2nd bash process never gets created

[3] works

[4] OOPS, but no hang:
kernel: Failed to tag uprobes SSOL vma: pid/tgid=6713/6713, vaddr=00007fff22001000
kernel: uretprobes disabled for pid 6714: cannot set uretprobe trampoline at
00007fff22001000
kernel: ------------[ cut here ]------------
kernel: kernel BUG at
/usr/local/share/systemtap/runtime/uprobes/../uprobes2/uprobes.c:2226!
kernel: invalid opcode: 0000 [1] SMP 
kernel: CPU 0 
kernel: Modules linked in: stap_6437 uprobes netconsole configfs bridge stp bnep
rfcomm l2cap bluetooth autofs4 nfs lockd nfs_acl fuse sunrpc ipt_REJECT
nf_conntrack_ipv4 iptable_filter ip_tables ip6t_REJECT xt_tcpudp
nf_conntrack_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables x_tables ipv6
dm_multipath scsi_dh floppy pcspkr 8139too 8139cp mii i2c_piix4 i2c_core sr_mod
cdrom sg dm_snapshot dm_zero dm_mirror dm_log dm_mod pata_acpi ata_piix
ata_generic libata sd_mod scsi_mod crc_t10dif ext3 jbd mbcache uhci_hcd ohci_hcd
ehci_hcd [last unloaded: stap_6138]
kernel: Pid: 6713, comm: bash Tainted: G S        2.6.27.5-41.fc9.x86_64 #1
kernel: RIP: 0010:[<ffffffffa02dcdc0>]  [<ffffffffa02dcdc0>]
uprobe_report_clone+0x241/0x420 [uprobes]
kernel: RSP: 0018:ffff8800089efdd8  EFLAGS: 00010286
kernel: RAX: ffff8800088b0400 RBX: ffff8800088b03d8 RCX: fffffffffffffff4
kernel: RDX: ffff8800088b0200 RSI: 0000000000000282 RDI: 0000000000000282
kernel: RBP: ffff8800089efe38 R08: ffffffff81408b30 R09: e3ff5be89d05db80
kernel: R10: 00000050620f9371 R11: 0000000100000000 R12: ffff880000f00d80
kernel: R13: ffff8800088b0200 R14: ffff88000b16dbc0 R15: ffff88000b16cfc0
kernel: FS:  00007f6e0aaee6f0(0000) GS:ffffffff814af100(0000) knlGS:0000000000000000
kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
kernel: CR2: 0000003178ea6400 CR3: 00000000089ce000 CR4: 00000000000006e0
kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
kernel: DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 0000000000000400
kernel: Process bash (pid: 6713, threadinfo ffff8800089ee000, task ffff88000b05c4a0)
kernel: Stack:  ffff88000b05db80 ffff88000b05c4a0 ffff8800088b0400 fffffffffffffff4
kernel: ffff88000b05c4a0 ffff8800089efe68 ffff8800089efe38 ffff880001e8b750
kernel: ffff88000b10c9d8 ffff880001e8b780 ffff88000b05c4a0 ffff8800089efe68
kernel: Call Trace:
kernel: [<ffffffff81074055>] utrace_report_clone+0xa3/0x119
kernel: [<ffffffff8103f403>] do_fork+0x169/0x257
kernel: [<ffffffff810c5b2f>] ? path_put+0x1d/0x21
kernel: [<ffffffff8101024a>] ? system_call_fastpath+0x16/0x1b
kernel: [<ffffffff8100e5e1>] sys_clone+0x23/0x25
kernel: [<ffffffff81010707>] ptregscall_common+0x67/0xb0
kernel:
kernel:
kernel: Code: e3 ff ff 48 85 c0 49 89 c7 75 04 0f 0b eb fe 48 8b 50 38 48 8b 8a
88 01 00 00 48 89 4d b8 49 8b 46 38 48 3b 88 88 01 00 00 74 04 <0f> 0b eb fe 49
8d 47 68 48 81 c2 80 01 00 00 4d 8b 76 68 31 db 
kernel: RIP  [<ffffffffa02dcdc0>] uprobe_report_clone+0x241/0x420 [uprobes]
kernel: RSP <ffff8800089efdd8>
kernel: ---[ end trace 8e1387874d8ed9ab ]---

-- 
           Summary: uprobes probes causing kernel problems
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: runtime
        AssignedTo: systemtap at sources dot redhat dot com
        ReportedBy: dsmith at redhat dot com


http://sourceware.org/bugzilla/show_bug.cgi?id=7082

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]