This is the mail archive of the
systemtap@sourceware.org
mailing list for the systemtap project.
[Bug runtime/7082] New: uprobes probes causing kernel problems
- From: "dsmith at redhat dot com" <sourceware-bugzilla at sourceware dot org>
- To: systemtap at sources dot redhat dot com
- Date: 8 Dec 2008 20:08:21 -0000
- Subject: [Bug runtime/7082] New: uprobes probes causing kernel problems
- Reply-to: sourceware-bugzilla at sourceware dot org
On both old and new kernels, uprobes probes can cause system hangs. Here are
the testcases:
[1]
# stap -ve 'probe process("bash").function("*").return {}' -w -c bash
# exit
[2]
# stap -ve 'probe process("bash").function("*").return {}' -w -c bash
# bash
# exit
# exit
[3]
# stap -ve 'probe process("bash").function("main").return {}' -w -c bash
# exit
[4]
# stap -ve 'probe process("bash").function("main").return {}' -w -c bash
# bash
# exit
# exit
SYS1: On a beta RHEL5.3 i386 system running kernel 2.6.18-125.el5, systemtap
head, I get the following results:
[1] works
[2] system hangs:
uretprobes disabled for pid 16866: cannot set uretprobe trampoline at bfb9d000
------------[ cut here ]------------
kernel BUG at /usr/local/share/systemtap/runtime/uprobes/uprobes.c:2082!
invalid opcode: 0000 [#1]
SMP
last sysfs file: /module/xfrm_nalgo/sections/__versions
Modules linked in: stap_2e548661a0b51d40f2e960c9c4d7a741_308985(U) uprobes(U)
autofs4 hidp rfcomm l2cap bluetooth sunrpc ipv6 xfrm_nalgo crypto_api
cpufreq_ondemand acpi_cpufreq dm_multipath scsi_dh video hwmon backlight sbs
i2c_ec button battery asus_acpi ac lp floppy snd_hda_intel snd_seq_dummy
snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss
snd_pcm snd_timer snd_page_alloc sr_mod snd_hwdep cdrom i2c_i801 snd e1000e
parport_pc e100 serio_raw parport soundcore i2c_core mii pcspkr sg dm_snapshot
dm_zero dm_mirror dm_log dm_mod ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd
ohci_hcd ehci_hcd
CPU: 3
EIP: 0060:[<f9012ac9>] Tainted: G VLI
EFLAGS: 00010202 (2.6.18-125.el5 #1)
EIP is at uprobe_report_clone+0x1ad/0x2fd [uprobes]
eax: f5133a00 ebx: fffffff4 ecx: 00000202 edx: f5133400
esi: f5133400 edi: ef8d2240 ebp: efd35ac0 esp: f01cef30
ds: 007b es: 007b ss: 0068
Process bash (pid: 16865, ti=f01ce000 task=f4d3b000 task.ti=f01ce000)
Stack: f5133a00 f5133400 f4d3b000 fffffff4 f9013620 ef8d7c80 efb2e2e0 00000000
c0452eda f7fdb550 f7fdb550 01200011 f4d3b000 efb1f200 00000001 f7fdb550
01200011 00000000 c042356c 000041e2 00000073 bfb79f48 493d7a48 0832363f
Call Trace:
[<c0452eda>] utrace_report_clone+0x4e/0x148
[<c042356c>] do_fork+0xd7/0x166
[<c040318b>] sys_clone+0x28/0x2d
[<c0404f17>] syscall_call+0x7/0xb
=======================
Code: ff 85 c0 89 c5 75 08 0f 0b 6a 08 7b 37 01 f9 8b 55 18 8b 47 18 89 54 24 04
8b 9a c8 00 00 00 89 5c 24 0c 3b 98 c8 00 00 00 74 08 <0f> 0b 22 08 7b 37 01 f9
8b 7f 34 c7 44 24 08 00 00 00 00 e9 87
EIP: [<f9012ac9>] uprobe_report_clone+0x1ad/0x2fd [uprobes] SS:ESP 0068:f01cef30
<0>Kernel panic - not syncing: Fatal exception
[3] works
[4] works occasionally, but normally hangs system:
uretprobes disabled for pid 3508: cannot set uretprobe trampoline at bfe50000
------------[ cut here ]------------
kernel BUG at /usr/local/share/systemtap/runtime/uprobes/uprobes.c:2082!
invalid opcode: 0000 [#1]
SMP
last sysfs file: /module/xfrm_nalgo/sections/__versions
Modules linked in: stap_005b533ff342bc467c6a9db8368dd2ae_324(U) uprobes(U)
autofs4 hidp rfcomm l2cap bluetooth sunrpc ipv6 xfrm_nalgo crypto_api
cpufreq_ondemand acpi_cpufreq dm_multipath scsi_dh video hwmon backlight sbs
i2c_ec button battery asus_acpi ac lp floppy snd_hda_intel snd_seq_dummy sr_mod
snd_seq_oss cdrom snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss
snd_mixer_oss snd_pcm snd_timer snd_page_alloc snd_hwdep e1000e snd i2c_i801
e100 sg parport_pc mii i2c_core parport serio_raw soundcore pcspkr dm_snapshot
dm_zero dm_mirror dm_log dm_mod ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd
ohci_hcd ehci_hcd
CPU: 3
EIP: 0060:[<f900eac9>] Tainted: G VLI
EFLAGS: 00010202 (2.6.18-125.el5 #1)
EIP is at uprobe_report_clone+0x1ad/0x2fd [uprobes]
eax: f369d200 ebx: fffffff4 ecx: 00000202 edx: f369da00
esi: f369da00 edi: f35d6140 ebp: f32b3740 esp: f16baf30
ds: 007b es: 007b ss: 0068
Process bash (pid: 3507, ti=f16ba000 task=f329c000 task.ti=f16ba000)
Stack: f369d200 f369da00 f329c000 fffffff4 f900f620 f16c5ca0 f16c3500 00000000
c0452eda f329caa0 f329caa0 01200011 f329c000 f16c52c0 00000001 f329caa0
01200011 00000000 c042356c 00000db4 00000073 f319e400 493d7c90 2a879fc4
Call Trace:
[<c0452eda>] utrace_report_clone+0x4e/0x148
[<c042356c>] do_fork+0xd7/0x166
[<c040318b>] sys_clone+0x28/0x2d
[<c0404f17>] syscall_call+0x7/0xb
=======================
Code: ff 85 c0 89 c5 75 08 0f 0b 6a 08 7b f7 00 f9 8b 55 18 8b 47 18 89 54 24 04
8b 9a c8 00 00 00 89 5c 24 0c 3b 98 c8 00 00 00 74 08 <0f> 0b 22 08 7b f7 00 f9
8b 7f 34 c7 44 24 08 00 00 00 00 e9 87
EIP: [<f900eac9>] uprobe_report_clone+0x1ad/0x2fd [uprobes] SS:ESP 0068:f16baf30
<0>Kernel panic - not syncing: Fatal exception
SYS2: On a f9 (kvm) system running kernel 2.6.18-125.el5, systemtap head, I get
the following results:
[1] works
[2] bash reports "Trace/breakpoint trap" and the 2nd bash process never gets created
[3] works
[4] OOPS, but no hang:
kernel: Failed to tag uprobes SSOL vma: pid/tgid=6713/6713, vaddr=00007fff22001000
kernel: uretprobes disabled for pid 6714: cannot set uretprobe trampoline at
00007fff22001000
kernel: ------------[ cut here ]------------
kernel: kernel BUG at
/usr/local/share/systemtap/runtime/uprobes/../uprobes2/uprobes.c:2226!
kernel: invalid opcode: 0000 [1] SMP
kernel: CPU 0
kernel: Modules linked in: stap_6437 uprobes netconsole configfs bridge stp bnep
rfcomm l2cap bluetooth autofs4 nfs lockd nfs_acl fuse sunrpc ipt_REJECT
nf_conntrack_ipv4 iptable_filter ip_tables ip6t_REJECT xt_tcpudp
nf_conntrack_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables x_tables ipv6
dm_multipath scsi_dh floppy pcspkr 8139too 8139cp mii i2c_piix4 i2c_core sr_mod
cdrom sg dm_snapshot dm_zero dm_mirror dm_log dm_mod pata_acpi ata_piix
ata_generic libata sd_mod scsi_mod crc_t10dif ext3 jbd mbcache uhci_hcd ohci_hcd
ehci_hcd [last unloaded: stap_6138]
kernel: Pid: 6713, comm: bash Tainted: G S 2.6.27.5-41.fc9.x86_64 #1
kernel: RIP: 0010:[<ffffffffa02dcdc0>] [<ffffffffa02dcdc0>]
uprobe_report_clone+0x241/0x420 [uprobes]
kernel: RSP: 0018:ffff8800089efdd8 EFLAGS: 00010286
kernel: RAX: ffff8800088b0400 RBX: ffff8800088b03d8 RCX: fffffffffffffff4
kernel: RDX: ffff8800088b0200 RSI: 0000000000000282 RDI: 0000000000000282
kernel: RBP: ffff8800089efe38 R08: ffffffff81408b30 R09: e3ff5be89d05db80
kernel: R10: 00000050620f9371 R11: 0000000100000000 R12: ffff880000f00d80
kernel: R13: ffff8800088b0200 R14: ffff88000b16dbc0 R15: ffff88000b16cfc0
kernel: FS: 00007f6e0aaee6f0(0000) GS:ffffffff814af100(0000) knlGS:0000000000000000
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
kernel: CR2: 0000003178ea6400 CR3: 00000000089ce000 CR4: 00000000000006e0
kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
kernel: DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 0000000000000400
kernel: Process bash (pid: 6713, threadinfo ffff8800089ee000, task ffff88000b05c4a0)
kernel: Stack: ffff88000b05db80 ffff88000b05c4a0 ffff8800088b0400 fffffffffffffff4
kernel: ffff88000b05c4a0 ffff8800089efe68 ffff8800089efe38 ffff880001e8b750
kernel: ffff88000b10c9d8 ffff880001e8b780 ffff88000b05c4a0 ffff8800089efe68
kernel: Call Trace:
kernel: [<ffffffff81074055>] utrace_report_clone+0xa3/0x119
kernel: [<ffffffff8103f403>] do_fork+0x169/0x257
kernel: [<ffffffff810c5b2f>] ? path_put+0x1d/0x21
kernel: [<ffffffff8101024a>] ? system_call_fastpath+0x16/0x1b
kernel: [<ffffffff8100e5e1>] sys_clone+0x23/0x25
kernel: [<ffffffff81010707>] ptregscall_common+0x67/0xb0
kernel:
kernel:
kernel: Code: e3 ff ff 48 85 c0 49 89 c7 75 04 0f 0b eb fe 48 8b 50 38 48 8b 8a
88 01 00 00 48 89 4d b8 49 8b 46 38 48 3b 88 88 01 00 00 74 04 <0f> 0b eb fe 49
8d 47 68 48 81 c2 80 01 00 00 4d 8b 76 68 31 db
kernel: RIP [<ffffffffa02dcdc0>] uprobe_report_clone+0x241/0x420 [uprobes]
kernel: RSP <ffff8800089efdd8>
kernel: ---[ end trace 8e1387874d8ed9ab ]---
--
Summary: uprobes probes causing kernel problems
Product: systemtap
Version: unspecified
Status: NEW
Severity: normal
Priority: P2
Component: runtime
AssignedTo: systemtap at sources dot redhat dot com
ReportedBy: dsmith at redhat dot com
http://sourceware.org/bugzilla/show_bug.cgi?id=7082
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.