glibc malloc issue when process stack is unlimited

[Bhaskar Dutta <bhaskie at gmail dot com>]

[I'd posted this earlier today but seems something went wrong.
Apologies if anyone receives a double post]

Hi,

I am facing a strange issue with libc malloc. I was using libc 2.6
earlier but even upgrading to glibc-2.9-2.8 (latest libc from SuSE
11.1), I face the same problem.

Scenario: process stack is unlimited (ulimit -s unlimited)

The process map shows that the following locations for heap, vdso and heap.

08875000-09569000 rw-p 08875000 00:00 0          [heap]
40017000-40018000 r-xp 40017000 00:00 0          [vdso]
bfeaf000-bff17000 rwxp bfeaf000 00:00 0          [stack]

My process allocates large memory chunks (through malloc) at process
initialization (around 1.9 GB) and then makes small memory requests.

Using strace, I can see that around 1.9 GB memory (from address
43572000 to bb9f1000) being allocated through mmap.

This is verified through mallinfo data:
Total size of memory allocated with sbrk by malloc in bytes           : 2605056
Number of chunks not in use                                           : 17
Total number of chunks allocated with mmap                            : 47
Total size of memory allocated with mmap in bytes
: 2038579200
Total size of memory occupied by chunks handed out by malloc          : 2442936
Total size of memory occupied by free (not in use) chunks             : 162120
Size of the top-most releasable chunk bordering the end of the heap   : 48976

Once I start running calls through the process (each call makes
hundreds of small malloc calls), mmap2 starts failing with ENOMEM and
the process gets memory through brk. This is explanable since Stack is
unlimited in this case and in mmap kernel code, mmap_is_legacy returns
1, so arch_pick_mmap_layout sets mmap_base to TASK_UNMAPPED_BASE which
comes out to be 1 GB (0x40000000) in intel 32 bit machine with 4 GB
memory. So the process gets 2 GB of mmap'ed memory.

The heap barely grows to 29MB (from memory obtained from brk) and then
suddenly brk is not called anymore from libc. Strace output shows that
mmap2 has failed, but no brk calls are seen anymore. The process map
shows nothing in the 900 MB of memory that can otherwise be obtained
through brk system call. This results in malloc failure and my process
cores with a segfault at size 2.1 GB.

When I start the process with exactly same parametes and set
MALLOC_CHECK_ to 2, the process memory map is exactly same, but this
time brk is called till the heap reaches 3ff00000 (890 MB) and the
process finally cores at 3 GB.

Also, when the process is started with a stack of 100 MB, the memory
layout is totally different.
The VDSO and libraries are loaded near the stack (bf000000) and this
time the process scales up to 3 GB easily.

Is this a bug in glibc malloc?
Why does the heap grow to 900 MB in case of MALLOC_CHECK_ and fails at
30 MB in the normal case?

Thanks,
Bhaskar