Bug 303

Summary: AIX ulimits not set correctly due to getyuserattr/setrlimit differences
Product: Sudo Reporter: Dale King <daleking>
Component: SudoAssignee: Todd C. Miller <Todd.Miller>
Status: RESOLVED FIXED    
Severity: normal    
Priority: normal    
Version: 1.7.0   
Hardware: IBM   
OS: AIX   
Attachments: patch for correct aix ulimit setting

Description Dale King 2008-10-02 21:37:10 MDT
Created attachment 231 [details]
patch for correct aix ulimit setting

Hi,

I've found a problem with 1.7.0rc2 (and CVS) under AIX.  The new ulimit code doesn't set the target ulimits correctly.  This is because the getuserattr call returns values in blocks but the setrlimit call takes an argument in bytes.

The effect of this is that the ulimits are set too small by a factor of 512.

Another problem is that the default hard stack size in newer AIX releases is actually RLIM_SAVED_MAX rather than -1.

The attached patch fixes this behaviour for me, tested on AIX 5.2/5.3/4.3.3.  The fix was based on the approach taken by old openssh code.

Dale

## su Behaviour:
#
$ su - testlim
testlim's Password:
$ ulimit -a
time(seconds)        unlimited
file(blocks)         2097151
data(kbytes)         262144
stack(kbytes)        262144
memory(kbytes)       32768
coredump(blocks)     524288
nofiles(descriptors) 2000

## sudo 1.7.0rc2 Behaviour:
#
$ sudo -i -u testlim
$ ulimit -a
time(seconds)        unlimited
file(blocks)         4096	#wrong - should be * 512
data(kbytes)         131072	#wrong - default taken - should be 262144
stack(kbytes)        512	#wrong - should be * 512
memory(kbytes)       64		#wrong - should be * 512
coredump(blocks)     1024	#wrong - should be * 512
nofiles(descriptors) 2000

## sudo 1.7.0rc2 Behaviour with multiplier fix:
#
$ sudo -i -u testlim
$ ulimit -a
time(seconds)        unlimited
file(blocks)         2097151
data(kbytes)         262144
stack(kbytes)        262144
memory(kbytes)       32768
coredump(blocks)     524288
nofiles(descriptors) 2000
Comment 1 Todd C. Miller 2008-10-03 09:20:54 MDT
I've committed your fix.  I don't currently have access to an AIX machine so the AIX-specific code in sudo was written "blind".  Thanks for checking and fixing this!
Comment 2 Dale King 2008-10-06 18:50:12 MDT
Hi Todd,

Thanks for committing the fix.

Looking at CVS I think that the default fallback position for setting stack size when RLIM_SAVED_MAX is not defined should be to RLIM_INFINITY, not to 0x400000 (which in any case needs to be multiplied by 512 and have 1 subtracted since the value is set in bytes not blocks).

AIX 4.3.3/5.1/5.2/5.3:
  $ grep RLIM_INF /usr/include/sys/resource.h
  #define RLIM_INFINITY   0x7fffffffffffffffL
  #define RLIM_INFINITY   0x7FFFFFFF
  #define RLIM_SAVED_MAX  (RLIM_INFINITY-1)
  #define RLIM_SAVED_CUR  (RLIM_INFINITY-2)

AIX 4.1/4.2:
grep RLIM_INF /usr/include/sys/resource.h
  #define RLIM_INFINITY   0x7fffffff

Regards,
Dale
Comment 3 Todd C. Miller 2008-11-05 19:44:26 MST
I think the best thing to do here is to just use RLIM_INFINITY if RLIM_SAVED_MAX is not defined.  Both RLIM_SAVED_MAX and RLIM_SAVED_MAX are special values interpreted by the kernel so using RLIM_INFINITY-1 on a kernel that doesn't know about RLIM_SAVED_MAX is not going to have the desired effects.