Re: Why does ldd not show cyg*.dll in its output?

[David Macek <david dot macek dot 0 at gmail dot com>]

On 29. 6. 2016 23:45, David Macek wrote:
> I can try watching them side by side in debuggers tomorrow, maybe I'll find something.

Yep, found something. TL;DR the issue is that Windows spins a thread in the process before our DLLs are loaded. Detailed analysis below.

On my Win8.x, the ldd "debugger" receives these events:

 1. CREATE_PROCESS_DEBUG_EVENT
 2. LOAD_DLL_DEBUG_EVENT for ntdll.dll
 3. LOAD_DLL_DEBUG_EVENT for KERNEL32.DLL
 4. LOAD_DLL_DEBUG_EVENT for KERNELBASE.dll
 5. LOAD_DLL_DEBUG_EVENT for msys-intl-8.dll
 6. LOAD_DLL_DEBUG_EVENT for msys-2.0.dll
 7. LOAD_DLL_DEBUG_EVENT for msys-iconv-2.dll
 8. EXCEPTION_DEBUG_EVENT ev.u.Exception = {ExceptionRecord = {ExceptionCode = 2147483651, ExceptionFlags = 0, ExceptionRecord = 0x0, ExceptionAddress = 0x7ffda38d1b90 <ntdll!LdrInitShimEngineDynamic+816>, NumberParameters = 1, ExceptionInformation = {0 <repeats 15 times>}}, dwFirstChance = 1}
 9. OUTPUT_DEBUG_STRING_EVENT ev.u.DebugString = {lpDebugStringData = 0x23f2a0 <error: Cannot access memory at address 0x23f2a0>, fUnicode = 0, nDebugStringLength = 24}
10. CREATE_THREAD_DEBUG_EVENT ev.u.CreateThread = {hThread = 0x88, lpThreadLocalBase = 0x7ff5ffffb000, lpStartAddress = 0x180044cf0 <cygthread::stub(void*)>} 
11. EXIT_THREAD_DEBUG_EVENT
12. EXIT_PROCESS_DEBUG_EVENT

I would say events #8 and #9 are not important (ldd ignores them as well). Event #10 causes ldd to call `TerminateProcess` because we don't want any of the analyzed program's code to run. This in turn causes events #11 and #12, after which ldd exits.

On my Win10, events are like this:

 1. CREATE_PROCESS_DEBUG_EVENT
 2. LOAD_DLL_DEBUG_EVENT for ntdll.dll
 3. LOAD_DLL_DEBUG_EVENT for KERNEL32.DLL
 4. LOAD_DLL_DEBUG_EVENT for KERNELBASE.dll
(5) LOAD_DLL_DEBUG_EVENT for msys-intl-8.dll
 6. CREATE_THREAD_DEBUG_EVENT ev.u.CreateThread = {hThread = 0x168, lpThreadLocalBase = 0x22c000, lpStartAddress = 0x7fff19e80720 <ntdll!RtlFindActivationContextSectionString+3456>}
 7. EXIT_THREAD_DEBUG_EVENT
 8. EXIT_PROCESS_DEBUG_EVENT

Event #6 is the culprit here. NT apparently starts a thread in our process which confuses ldd into believing that the process has finished loading DLL from its import table and kills it, as explained above. The thread appears non-deterministically, too, sometimes I see event #5 (and the associated DLL in ldd's output), sometimes not. The thread's entry point is always the same and ProcExp reports the same location (I checked in case of some symbol mash-up, as it often happens when debugging across the Cygwin-Windows boundary). When I broke ldd at `case CREATE_THREAD_DEBUG_EVENT`, the new thread's stack trace (from ProcExp) looked like this:

ntoskrnl.exe!KeSynchronizeExecution+0x3f26
ntoskrnl.exe!KeWaitForMultipleObjects+0x10b5
ntoskrnl.exe!KeWaitForMultipleObjects+0xb6f
ntoskrnl.exe!KeWaitForSingleObject+0x370
ntoskrnl.exe!CmUnRegisterCallback+0x1b02c
ntoskrnl.exe!CmUnRegisterCallback+0x1c225
ntoskrnl.exe!NtVdmControl+0x45822
ntoskrnl.exe!FsRtlFreeExtraCreateParameter+0x44b
ntoskrnl.exe!KeSynchronizeExecution+0x4716
ntoskrnl.exe!KeSynchronizeExecution+0x4690
!RtlUserThreadStart

I came up with this naive fix:

--- ldd.cc.orig 2016-04-28 21:54:57.556500700 +0200
+++ ldd.cc      2016-06-30 22:18:05.332452300 +0200
@@ -357,6 +357,9 @@
            }
          break;
        case CREATE_THREAD_DEBUG_EVENT:
+          if (ev.u.CreateThread.lpStartAddress == (void*)0x7fff19e80720) {
+            break;
+          }
          TerminateProcess (hProcess, 0);
          break;
        case EXIT_PROCESS_DEBUG_EVENT:

After which the output switched back to normal. One observed sequence of events:

 1. CREATE_PROCESS_DEBUG_EVENT
 2. LOAD_DLL_DEBUG_EVENT for ntdll.dll
 3. LOAD_DLL_DEBUG_EVENT for KERNEL32.DLL
 4. LOAD_DLL_DEBUG_EVENT for KERNELBASE.dll
 5. CREATE_THREAD_DEBUG_EVENT ev.u.CreateThread = {hThread = 0x14c, lpThreadLocalBase = 0x26a000, lpStartAddress = 0x7fff19e80720 <ntdll!RtlFindActivationContextSectionString+3456>}
 6. LOAD_DLL_DEBUG_EVENT for msys-intl-8.dll
 7. CREATE_THREAD_DEBUG_EVENT ev.u.CreateThread = {hThread = 0x138, lpThreadLocalBase = 0x26c000, lpStartAddress = 0x7fff19e80720 <ntdll!RtlFindActivationContextSectionString+3456>}
 8. LOAD_DLL_DEBUG_EVENT for msys-2.0.dll
 9. LOAD_DLL_DEBUG_EVENT for msys-iconv-2.dll
10. CREATE_THREAD_DEBUG_EVENT ev.u.CreateThread = {hThread = 0x18c, lpThreadLocalBase = 0x274000, lpStartAddress = 0x7fff19e80720 <ntdll!RtlFindActivationContextSectionString+3456>}
11. EXCEPTION_DEBUG_EVENT ev.u.Exception = {ExceptionRecord = {ExceptionCode = 2147483651, ExceptionFlags = 0, ExceptionRecord = 0x0, ExceptionAddress = 0x7fff19f2fd00 <ntdll!LdrInitShimEngineDynamic+864>, NumberParameters = 1, ExceptionInformation = {0 <repeats 15 times>}}, dwFirstChance = 1}
12. OUTPUT_DEBUG_STRING_EVENT ev.u.DebugString = {lpDebugStringData = 0x5ff1a0 <error: Cannot access memory at address 0x5ff1a0>, fUnicode = 0, nDebugStringLength = 24}
13. CREATE_THREAD_DEBUG_EVENT ev.u.CreateThread = {hThread = 0x118, lpThreadLocalBase = 0x276000, lpStartAddress = 0x180044cf0 <cygthread::stub(void*)>}
14. EXIT_THREAD_DEBUG_EVENT
15. EXIT_THREAD_DEBUG_EVENT
16. EXIT_THREAD_DEBUG_EVENT
17. EXIT_THREAD_DEBUG_EVENT
18. EXIT_PROCESS_DEBUG_EVENT

Here are some more stacktraces from various points in life of the cuckoo threads, in case they are significant:

ntoskrnl.exe!KeSynchronizeExecution+0x3f26
ntoskrnl.exe!KeWaitForMultipleObjects+0x10b5
ntoskrnl.exe!KeWaitForMultipleObjects+0xb6f
ntoskrnl.exe!KeWaitForSingleObject+0x370
ntoskrnl.exe!KeTestAlertThread+0x591
ntoskrnl.exe!KeReleaseSemaphore+0x4d1
ntoskrnl.exe!KiCheckForKernelApcDelivery+0x23
ntoskrnl.exe!NtDeviceIoControlFile+0x163f
ntoskrnl.exe!NtClose+0x1c37
ntoskrnl.exe!NtClose+0xcb
ntoskrnl.exe!setjmpex+0x3a63
!NtClose+0x14
!RtlGetActiveActivationContext+0x51c
!RtlGetFullPathName_UstrEx+0x4ab
!RtlReleaseRelativeName+0x269
!RtlReleaseRelativeName+0x1f3
!RtlFindActivationContextSectionString+0x3039
!RtlFindActivationContextSectionString+0x11f4
!BaseThreadInitThunk+0x14
!RtlUserThreadStart+0x21

ntoskrnl.exe!KeSynchronizeExecution+0x3f26
ntoskrnl.exe!KeSetEvent+0x6a6
ntoskrnl.exe!MmUnlockPagableImageSection+0xdd
ntoskrnl.exe!KeSynchronizeExecution+0x4b52
!RtlLookupFunctionEntry+0x113f
!RtlReleaseRelativeName+0x248
!RtlReleaseRelativeName+0x1f3
!RtlFindActivationContextSectionString+0x3039
!RtlFindActivationContextSectionString+0x11f4
!BaseThreadInitThunk+0x14
!RtlUserThreadStart+0x21

After some experimentation, I came up with a patch that tries to determine whether to ignore a thread based on the thread's entry point lying inside or outside of the memory occupied by ntdll.dll:

--- ldd.cc.orig 2016-04-28 21:54:57.556500700 +0200
+++ ldd.cc      2016-06-30 23:24:20.394384800 +0200
@@ -327,6 +327,10 @@
     {
       bool exitnow = false;
       DWORD cont = DBG_CONTINUE;
+      MODULEINFO mi;
+      HMODULE ntdll = LoadLibrary("ntdll.dll");
+      HMODULE ntdllend = NULL;
+      void *entryPoint;
       if (!WaitForDebugEvent (&ev, INFINITE))
        break;
       switch (ev.dwDebugEventCode)
@@ -357,6 +361,31 @@
            }
          break;
        case CREATE_THREAD_DEBUG_EVENT:
+          if (ntdll != NULL)
+            {
+              if (ntdllend == NULL)
+                {
+                  /* Using our ntdll.dll HMODULE with a foreign process
+                     should be fine because ntdll.dll is mapped to the same
+                     address in every concurrently executing process and
+                     HMODULEs are just pointers to the image in the process
+                     memory. Using GetModuleHandle(NULL) didn't work for the
+                     author of this code (-> ERROR_INVALID_HANDLE). */
+                  if (GetModuleInformation(hProcess, ntdll, &mi, sizeof(mi)))
+                    ntdllend = ntdll + (ptrdiff_t)mi.SizeOfImage;
+                  else
+                    ntdll = NULL;
+                }
+              if (ntdllend != NULL)
+                {
+                  entryPoint = (void*)ev.u.CreateThread.lpStartAddress;
+                  /* If the thread's entry point is in ntdll.dll, let's
+                    assume that this isn't the program's main thread and
+                    ignore it. */
+                  if (ntdll <= entryPoint && entryPoint < ntdllend)
+                    break;
+                }
+            }
          TerminateProcess (hProcess, 0);
          break;
        case EXIT_PROCESS_DEBUG_EVENT:
---end---

I can send the patch through proper channels if it passes a sanity check from someone.

HTH. Cheers.

-- 
David Macek

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature