Re: Utrace usage queries

[Roland McGrath <roland at redhat dot com>]

[Btw, I'm on the systemtap mailing list, so separate CC'ing is not required.]

> 1. Is there a specific mechanism you had in mind to insert a breakpoint
> at a userspace virtual address?

Not in particular.  That is, with current facilities, this is not a
utrace-level operation but a complex higher-level job involving numerous
operations.  So to begin with, utrace only has so much to say about how you
go about it.  As designated pontificator on what anyone hacking on
systemtap ought to be doing differently, I'll tell you what to do all the
live long day.  But as the authority on utrace, I'll just tell you what
utrace can help you with and that what to do with it is up to you.

> This is our current understanding:
> - A request to register a breakpoint at a virtual address will first
> register a engine with a set of report callbacks and sets the QUIESCE
> flag, and waits for the callback.
> - Once the callback happens, the breakpoint is set at the virtual
> address requested (via, say, access_process_vm()).

This is indeed the way to use utrace to stop one user thread.  utrace calls
work on individual threads.  So if the intent is to stop any thread that
might run the code you're modifying, you'll have to do this for each thread
sharing the memory.  When the last thread you attached to runs its
report_quiesce callback, then you know they are all stopped.

Also note that setting the QUIESCE flag has the perturbing effect on system
calls that PTRACE_ATTACH does, which is essentially the same as catching a
signal during a system call.  A thread that is blocked in a system call (or
concurrently entering one on another CPU via preemption) when you set its
QUIESCE flag might see a system call return early or give an EINTR error
that it would not have seen without you being there.  This is an
unfortunate side effect of the low-level parts of the implementation that
have not changed from the original ptrace.  I expect that in the future
utrace will provide an alternate form of QUIESCE that does not perturb
system calls in this way (it would prevent a thread from running any more
user instructions, but not interfere with its in-kernel state at all).
Until then, you just have to be aware that you are poking at things (the
same caveat you would have now for doing e.g. "strace -p" on some process).

> Is the approach right? Do you have better suggestions? This is important
> in the sense that a request to register a breakpoint at a userspace
> address will ideally have to block till the breakpoint is in place, ie.,
> the registration "event" is complete only _after_ the QUIESCE callback
> completes inserting the breakpoint.

I assume what matters here is machine-dependent.  e.g., on the x86 the
breakpoint instruction is one byte and you can just do a single-byte write
and win without synchronization, I think.  But anyway, all the machine
mechanics here should be well understood from the kprobes implementation
experience.

I do assume that in at least many cases on many machines, you do indeed
need to make sure every thread has stopped.  The report_quiesce callback is
indeed intended for such needs.

> 2. What is the preferred mechanism to return errors from report_*
> callbacks? Eg., say, a genregs_set failed when in a report_signal callback.
> AFAICS, the report callbacks return the new utrace engine state bitmask.

The utrace framework itself does not include anyone for you to report
errors to.  That is your business.  The callbacks are a mechanism to give
your code notification and synchronization with user thread state, that's
all.  So the only specific answers that are meaningful depend on the
context of your code that uses utrace.  

Where does an error report, or any other information, need to get to?


Thanks,
Roland