[Bug kprobes/2071] New: Probes on ISR with probes on task thread's prehandler crash the system

["anil dot s dot keshavamurthy at intel dot com" <sourceware-bugzilla at sourceware dot org>]

I was in the process of testing reentrant probes, so I was calling a routing 
from inside a task thread's prehandler which had a probe, just to test 
reentrancy and this reentrancy test worked just fine with out any problem.

However when I inserted another test module which inserted probes on ISR 
routine (__do_ISR), I see system crash.

Here is what I think is happening.
Our current kprobes design supports reentrancy only from one thread. If while 
in the process of reentrancy and before completing the single-stepping, if  
another probes on ISR fires, then we loose or overwrite the previous kprobes 
state and eventually crash the system.

Will disabling interrupts while servicing the reentrant probes solve the 
problem? Need to try.


The attached test case has
1)probes on my_test_reentrant_export_function().
2)probes on schedule() and the pre_handler for schedule() calls 
my_test_reentrant_export_function()
3)probes on __do_ISR and the pre_handler for __do_ISR() calls
my_test_reentrant_export_function().

Here is the system crash stack back trace while executing the above test on 
IA64. I think this problem should exist on PPC64 too and not sure on Ia32 as 
Ia32 disables interrupt while servicing the breakfault handler. ppc64??

[<a0000001000122a0>] show_stack+0x80/0xa0
                                sp=e000000001feed10 bsp=e000000001fe9360
 [<a000000100012bb0>] show_regs+0x890/0x8c0
                                sp=e000000001feeee0 bsp=e000000001fe9318
 [<a00000010003a560>] die+0x1a0/0x2a0
                                sp=e000000001feef00 bsp=e000000001fe92c8
 [<a00000010003a6a0>] die_if_kernel+0x40/0x60
                                sp=e000000001feef20 bsp=e000000001fe9298
 [<a000000100736a10>] ia64_bad_break+0x550/0x6c0
                                sp=e000000001feef20 bsp=e000000001fe9270
 [<a00000010000c520>] ia64_leave_kernel+0x0/0x280
                                sp=e000000001feeff0 bsp=e000000001fe9270
 [<a000000100739780>] kprobe_exceptions_notify+0x8a0/0x900
                                sp=e000000001fef1c0 bsp=e000000001fe91c0
 [<a00000010073a560>] notifier_call_chain+0x80/0xe0
                                sp=e000000001fef1d0 bsp=e000000001fe9188
 [<a000000100736b50>] ia64_bad_break+0x690/0x6c0
                                sp=e000000001fef1d0 bsp=e000000001fe9160
 [<a00000010000c520>] ia64_leave_kernel+0x0/0x280
                                sp=e000000001fef2a0 bsp=e000000001fe9160
 [<a0000001000ec220>] __do_IRQ+0x0/0x440
                                sp=e000000001fef470 bsp=e000000001fe9150
 [<a0000001000112e0>] indle_irq+0xa0/0x140
                                sp=e000000001fef470 bsp=e000000001fe9118
 [<a00000010000c520>] ia64_leave_kernel+0x0/0x280
                                sp=e00000000fe9118
 [<a00000010073aca0>] kprobes_inc_nmissed_count+0x0/0x120
                                sp=e000000001fef640 bsp=e000000001fe9100
 [<a0000001007392e0>] kprobe_exceptions_notify+0x                         
sp=e000000001fef640 bsp=e000000001fe9070
 [<a00000010073a560>] notifier_call_chain+0x80/0xe0
                                sp=e000000001fef650 bsp=e000000001fe900>] 
ia64_bad_break+0x690/0x6c0
                                sp=e000000001fef650 bsp=e000000001fe9010
 [<a00000010000c520>] ia64_leave_kernel+0x0/0x280
                                s=e000000001fe9010
 [<a00000020008c000>] my_test_reentrant_export_function+0x0/0x40 [mon_dummy]
                                sp=e000000001fef8f0 bsp=e000000001fe9010
 [<a0000002000e4140on_sched]
                                sp=e000000001fef8f0 bsp=e000000001fe8ff0
 [<a00000010073a840>] aggr_pre_handler+0x180/0x1c0
                                sp=e000000001fef8f0 b8
 [<a000000100739570>] kprobe_exceptions_notify+0x690/0x900
                                sp=e000000001fef8f0 bsp=e000000001fe8f18
 [<a00000010073a560>] notifier_call_chain+0x80/0xe0
          sp=e000000001fef900 bsp=e000000001fe8ee0
 [<a000000100736b50>] ia64_bad_break+0x690/0x6c0
                                sp=e000000001fef900 bsp=e000000001fe8eb8
 [<a0000001000nel+0x0/0x280
                                sp=e000000001fef9d0 bsp=e000000001fe8eb8
 [<a0000001007312e0>] schedule+0x0/0x15c0
                                sp=e000000001fefba0 bsp=e0<a00000010005d420>] 
kretprobe_trampoline+0x0/0x20
                                sp=e000000001fefba0 bsp=e000000001fe8e68
 <0>Kernel panic - not syncing: Aiee, killing interrupt handler!

-- 
           Summary: Probes on ISR with probes on task thread's prehandler
                    crash the system
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P1
         Component: kprobes
        AssignedTo: systemtap at sources dot redhat dot com
        ReportedBy: anil dot s dot keshavamurthy at intel dot com


http://sourceware.org/bugzilla/show_bug.cgi?id=2071

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.