RE: Controlling probe overhead

["Stone, Joshua I" <joshua dot i dot stone at intel dot com>]

fche@redhat.com wrote:
> David Smith <dsmith@redhat.com> writes:
>> [...] I've have one stress test (that Frank wrote) that will make a
>> RHEL5 system non-responsive.  The system doesn't crash - just
>> decides to no longer take any input.  The overload code kills the
>> script in less than 3 minutes.
> 
> 3 minutes is almost certainly too long for a default overload
> detection interval.  I would expect something on the order of a few
> seconds.

I haven't seen your test, so I can only guess what's going on.  Part of
the problem is that the error state doesn't actually disarm any kprobes.
The overload detection should detect the problem quite rapidly, but the
recovery may take a while to come about while probes are still firing.

I don't know what we can do to improve this without some kprobes help to
disable a probe dynamically.  And given all the synchronization
requirements in dynamic instrumentation, it may not even be possible.

>> +  o->newline() << "#ifndef STP_OVERLOAD_INTERVAL";
>> +  o->newline() << "#define STP_OVERLOAD_INTERVAL 1000000000LL"; + 
>> o->newline() << "#endif"; +  o->newline() << "#ifndef
>> STP_OVERLOAD_THRESHOLD"; +  o->newline() << "#define
>> STP_OVERLOAD_THRESHOLD 500000000LL"; +  o->newline() << "#endif";
> 
> These quantities should probably depend on the processor, so that
> overload intervals are measured in units of time rather than cycles.

I prefer the simplicity of cycles, since you get around all of the
issues that we're having with gettimeofday cycles->time conversions.  I
suppose for this you could make a one-time estimation of X cycles == Y
seconds, and come up with an appropriate interval, but I'm not convinced
that it matters.

I chose a billion-cycle interval arbitrarily, but it turns out pretty
nice because it will come out to something less than a second most
platforms.  I don't see any reason to try to make the interval equal the
same time on all processors, as long as the user still perceives the
system being recovered without "too much" delay.


Josh