[PATCH] Fix malloc with really large sizes [BZ #2775]

[Jakub Jelinek <jakub at redhat dot com>]

Hi!

As the BZ#2775 testcase shows (not sure if it is appropriate for make check,
as it uses many threads to force initial thread to allocate on a different
arena than main_arena), we sometimes call grow_heap to grow a heap, but
the size is so large that the diff is negative (and in that case
grow_heap acts as shrink_heap and unlike growing e.g. assumes the diff
is already a multiple of page size).

The grow_heap change is just trying to be really safe, if e.g. old heap size
is 1MB and diff is e.g. 0x7ff01000 on 32-bit 4KB pagesize arch, then we
could easily create heap bigger than HEAP_MAX_SIZE.

2006-09-07  Jakub Jelinek  <jakub@redhat.com>

	[BZ #2775]
	* malloc/malloc.c (sYSMALLOc): Only call grow_heap if
	(long) (MINSIZE + nb - old_size) is positive.

	* malloc/arena.c (grow_heap): When growing bail even if new_size
	is negative.

--- libc/malloc/arena.c.jj	2006-09-07 16:46:50.000000000 +0200
+++ libc/malloc/arena.c	2006-09-07 17:35:38.000000000 +0200
@@ -712,7 +712,7 @@ grow_heap(h, diff) heap_info *h; long di
   if(diff >= 0) {
     diff = (diff + page_mask) & ~page_mask;
     new_size = (long)h->size + diff;
-    if(new_size > HEAP_MAX_SIZE)
+    if((unsigned long) new_size > (unsigned long) HEAP_MAX_SIZE)
       return -1;
     if(mprotect((char *)h + h->size, diff, PROT_READ|PROT_WRITE) != 0)
       return -2;
--- libc/malloc/malloc.c.jj	2006-09-04 16:42:01.000000000 +0200
+++ libc/malloc/malloc.c	2006-09-07 17:39:59.000000000 +0200
@@ -2970,7 +2970,8 @@ static Void_t* sYSMALLOc(nb, av) INTERNA
     /* First try to extend the current heap. */
     old_heap = heap_for_ptr(old_top);
     old_heap_size = old_heap->size;
-    if (grow_heap(old_heap, MINSIZE + nb - old_size) == 0) {
+    if ((long) (MINSIZE + nb - old_size) > 0
+	&& grow_heap(old_heap, MINSIZE + nb - old_size) == 0) {
       av->system_mem += old_heap->size - old_heap_size;
       arena_mem += old_heap->size - old_heap_size;
 #if 0

	Jakub