Re: [PATCH] Use malloca instead alloca

[KOSAKI Motohiro <kosaki dot motohiro at gmail dot com>]

> Could you provide a citation for this? I'm pretty sure the kernel
> ensures this does not happen by never allocating maps in the
> RLIMIT_STACK range. On the other hand, I have seen buggy
> vendor-patched kernels which would allocate maps in the stack range
> even when there is no memory pressure.

OK, Maybe I need more long explanation.

1) x86 has special stack care.

generic version.

linux/mm/util.c
-----------------------------------------
#if defined(CONFIG_MMU) && !defined(HAVE_ARCH_PICK_MMAP_LAYOUT)
void arch_pick_mmap_layout(struct mm_struct *mm)
{
  mm->mmap_base = TASK_UNMAPPED_BASE;
  mm->get_unmapped_area = arch_get_unmapped_area;
  mm->unmap_area = arch_unmap_area;
}
#endif


x86 version
linux/arch/x86/mm/mmap.c
-----------------------------------------------------------------
static unsigned long mmap_base(void)
{
   unsigned long gap = rlimit(RLIMIT_STACK);

  if (gap < MIN_GAP)
    gap = MIN_GAP;
  else if (gap > MAX_GAP)
    gap = MAX_GAP;

  return PAGE_ALIGN(TASK_SIZE - gap - mmap_rnd());
}

/*
 * This function, called very early during the creation of a new
 * process VM image, sets up which VM layout function to use:
 */
void arch_pick_mmap_layout(struct mm_struct *mm)
{
  if (mmap_is_legacy()) {
    mm->mmap_base = mmap_legacy_base();
    mm->get_unmapped_area = arch_get_unmapped_area;
    mm->unmap_area = arch_unmap_area;
  } else {
    mm->mmap_base = mmap_base();
    mm->get_unmapped_area = arch_get_unmapped_area_topdown;
   mm->unmap_area = arch_unmap_area_topdown;
  }
}


x86 maintainer is wise. mm->mmap_base (aka area search starting address)
is lower than stack and search order is top to bottom.  however there
are several
bottom to top searching case.

1) no free vm area (x86 only)

x86 has arch specific arch_get_unmapped_area_topdown() and it fall back
bottom-to-top search when top-to-bottom search is failure.

see http://lxr.linux.no/linux+v3.7/arch/x86/kernel/sys_x86_64.c#L266

2) arch generic arch_pick_mmap_layout uses bottom-to-top search.

see http://lxr.linux.no/linux+v3.7/mm/util.c#L292

3) x86 uses bottom-to-top search when mmap_is_legacy() return true.

the code is here.

static int mmap_is_legacy(void)
{
  if (current->personality & ADDR_COMPAT_LAYOUT)
    return 1;

  if (rlimit(RLIMIT_STACK) == RLIM_INFINITY)
    return 1;

  return sysctl_legacy_va_layout;
}

i.e.
 - uses setarch -L or uses personality syscall directly.
 - RLIMIT_STACK == RLIM_INFINITY.
 - /proc/sys/vm/legacy_va_layout = 1.


Is this sufficient pointer?