[Bug libc/2775] New: malloc(0xFFFFnnnn) corruption from 32-bit wraparound

["mec at google dot com" <sourceware-bugzilla at sourceware dot org>]

We were running a unit test on malloc and ran into an edge case that causes
corruption.  I will attach the test program + an strace.

The critical part of the test is:

  /* malloc a lot of blocks near the max possible size. */
  for (i = 0; i < 10000; i++) {
    if (malloc(kMaxSize - i) != NULL) {
      fprintf(stderr, "Oops\n");
      return 1;
    }
  }

When I strace this, I see these mmap calls:

...
mmap2(0xade06220, 110048, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0)
= -1 EINVAL (Invalid argument)
mmap2(NULL, 2097152, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0)
= 0xb7d94000
munmap(0xb7d94000, 442368)              = 0
munmap(0xb7f00000, 606208)              = 0
mprotect(0xb7e00000, 135168, PROT_READ|PROT_WRITE) = 0
munmap(0xb7e00000, 1048576)             = 0
brk(0x80a0000)                          = 0x80a0000
mmap2(0xade06220, 110048, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0)
= -1 EINVAL (Invalid argument)
mmap2(NULL, 2097152, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0)
= 0xb7d94000
munmap(0xb7d94000, 442368)              = 0
munmap(0xb7f00000, 606208)              = 0
mprotect(0xb7e00000, 135168, PROT_READ|PROT_WRITE) = 0
munmap(0xb7e00000, 1048576)             = 0
...

It's an mmap on a fixed address which is not an integral page.

Later on:
mmap2(0xade06008, 110584, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0)
= -1 EINVAL (Invalid argument)
mmap2(NULL, 2097152, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0)
= 0xb7d94000
munmap(0xb7d94000, 442368)              = 0
munmap(0xb7f00000, 606208)              = 0
mprotect(0xb7e00000, 131072, PROT_READ|PROT_WRITE) = 0
munmap(0xb7e00000, 1048576)             = 0
mmap2(0xade06008, 110584, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0)
= -1 EINVAL (Invalid argument)
mmap2(NULL, 2097152, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0)
= 0xb7d94000
munmap(0xb7d94000, 442368)              = 0
munmap(0xb7f00000, 606208)              = 0
mprotect(0xb7e00000, 131072, PROT_READ|PROT_WRITE) = 0
munmap(0xb7e00000, 1048576)             = 0
mmap2(0xade06000, 110592, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0)
= 0xade06000
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++

When the test program runs long enough, eventually it mmap's a fixed address
which is on a page boundary.  It's a PROT_NONE so it disables access to
something.  Shortly after that, _int_malloc gets a SIGSEGV and dies.  Here is a
stack trace:

(gdb) bt
#0  0x47270ce3 in _int_malloc () from /lib/tls/i686/cmov/libc.so.6
#1  0x472723b1 in malloc () from /lib/tls/i686/cmov/libc.so.6
#2  0x08048601 in main ()

I think the problem is in this code:

  /* sYSMALLOc */
  if (grow_heap(old_heap, MINSIZE + nb - old_size) == 0) {

And then in grow_heap:

    if((char *)MMAP((char *)h + new_size, -diff, PROT_NONE,
                    MAP_PRIVATE|MAP_FIXED) == (char *) MAP_FAILED)

That line in grow_heap is the only place mmap call with a fixed address and
MAP_PRIVATE|MAP_FIXED.

Our environment is an in-house distro, native i686-pc-linux-gnu, with glibc 2.3.5.

This isn't an important bug because normal programs don't try to allocate
0xFFFFnnnn bytes of memory.  Since we turned it up, I thought it would be useful
to report it.

Let me know if you need any more info, and thanks for your attention.

-- 
           Summary: malloc(0xFFFFnnnn) corruption from 32-bit wraparound
           Product: glibc
           Version: 2.3.5
            Status: NEW
          Severity: minor
          Priority: P2
         Component: libc
        AssignedTo: drepper at redhat dot com
        ReportedBy: mec at google dot com
                CC: glibc-bugs at sources dot redhat dot com
 GCC build triplet: i686-pc-linux-gnu
  GCC host triplet: i686-pc-linux-gnu
GCC target triplet: i686-pc-linux-gnu


http://sourceware.org/bugzilla/show_bug.cgi?id=2775

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.