This is the mail archive of the libc-hacker@sources.redhat.com mailing list for the glibc project.

Note that libc-hacker is a closed list. You may look at the archives of this list, but subscription and posting are not open.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: [PATCH] xdr_array and calloc security fix


> > But (a > a * b || b > a * b) should work, shouldn't it?
> 
> No. For a=1 b=2 this will give the correct answer (no overflow), but
> for a=0x6000000 b=64 it will give incorrect one (no overflow, while
> 0x180000000LL certainly doesn't fit into 32-bits (but 0x80000000 is
> still bigger than any of the operands).

Ok, if we're going to have two comparisions anyway, I'd suggest we
assume at least 32bits and use

a >= 46340 || b >= 46340

(46340 <= sqrt(2^31), if I did my math correctly)
Of course this will detect some cases as overflow which actually
aren't, but that is harmless.

Regards,
Wolfram.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]