Re: Unknown symbol error using stap in RHEL 5.4

[Maynard Johnson <maynardj at us dot ibm dot com>]

Jim Keniston wrote:
> On Wed, 2010-05-12 at 17:48 -0500, Maynard Johnson wrote:
>> Hi,
>> I maintain a package called stapitrace which has been in Fedora for some time.  Then late last year, we also put it in EPEL.  At that time, I did a fresh install of RHEL 5.4 on a ppc64 system and installed my package from the EPEL repository.  I tested it, and everything worked fine.  But it was brought to my attention recently that the package appeared broken on a user's RHEL 5.4 system.  When I started to debug the problem, I noted it was failing with "unknown symbol" errors, and dmesg showed the following:
>>
>> stap_6e2a90fc5508789a23a65a03a12b9273_46646: Unknown symbol unmap_uretprobe
>> stap_6e2a90fc5508789a23a65a03a12b9273_46646: Unknown symbol unmap_uprobe
>> stap_6e2a90fc5508789a23a65a03a12b9273_46646: disagrees about version of symbol register_uretprobe
>> stap_6e2a90fc5508789a23a65a03a12b9273_46646: Unknown symbol register_uretprobe
>> stap_6e2a90fc5508789a23a65a03a12b9273_46646: disagrees about version of symbol unregister_uretprobe
>> stap_6e2a90fc5508789a23a65a03a12b9273_46646: Unknown symbol unregister_uretprobe
>> stap_6e2a90fc5508789a23a65a03a12b9273_46646: Unknown symbol unmap_uretprobe
>> stap_6e2a90fc5508789a23a65a03a12b9273_46646: Unknown symbol unmap_uprobe
>> stap_6e2a90fc5508789a23a65a03a12b9273_46646: disagrees about version of symbol register_uretprobe
>> stap_6e2a90fc5508789a23a65a03a12b9273_46646: Unknown symbol register_uretprobe
>> stap_6e2a90fc5508789a23a65a03a12b9273_46646: disagrees about version of symbol unregister_uretprobe
>> stap_6e2a90fc5508789a23a65a03a12b9273_46646: Unknown symbol unregister_uretprobe
>>
>> ====================
>>
>> This looked suspiciously like the problem reported in https://bugzilla.redhat.com/show_bug.cgi?id=499677.  But I tried 'rmmod uprobes', "make -C /usr/share/systemtap/runtime/uprobes clean", and "rm -rf ~/.systemtap", but was not able to clear the problem.  I noted that the systemtap level was 1.1-3, which is different from a fresh RHEL 5.4 install.  So I tried uninstalling systemtap and installing the 0.9.7-5 rpm from the install DVD, but that didn't clear the error either.  As a sanity check, I did a fresh install of RHEL 5.4 on another system, and was able to use stapitrace with no problem there.
>>
>> Any thoughts on what might be causing this problem?
> 
> It appears that the source from which the kernel was built includes some
> non-SystemTap version of uprobes, and that CONFIG_UPROBES is y or m in
> the .config file.  If that's the case, stap sees that there are uprobes
> functions in the kernel's Module.symvers file and uses the "native"
> version of uprobes (running modprobe uprobes, if appropriate) instead of
> its own version.  Unfortunately, functions like unmap_u[ret]probe, which
> exist in SystemTap's version of uprobes, don't exist in many "native"
> versions.
> 
> If the above is your problem, then I think the best recourse is to
> rebuild the kernel without uprobes, so that SystemTap's version will get
> used.
Jim, thanks for your reply, but this is not really an option for users.

-Maynard
> 
>>
>> Thanks!
>> -Maynard
>>
> 
> Jim
> 
>