pkg/32445: x11/Xaw3d causes graphics/xfig to hang under kde

[thesing%cs.uni-sb.de@localhost]

>Number:         32445
>Category:       pkg
>Synopsis:       x11/Xaw3d has an overflow error in geometry computation code, 
>causing xfig to hang under KDE
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    pkg-manager
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue Jan 03 14:05:00 +0000 2006
>Originator:     Stephan Thesing
>Release:        NetBSD 3.99.11
>Organization:
=  Tel.: +49-681-302-5571      = Universitaet des Saarlandes =
=  Fax.: +49-681-302-3065      = Postfach 15 11 50           =
=  Compiler Research Group     = 66041 Saarbruecken          =
=  FR 6.2 - Informatik         = GERMANY                     =
>Environment:
        
        
System: NetBSD gargoyle.cs.uni-sb.de 3.99.11 NetBSD 3.99.11 (Gargoyle) #5: Mon 
Nov 7 08:53:22 CET 2005 
thesing%gargoyle.cs.uni-sb.de@localhost:/local/thesing/netbsd/current/obj/sys/arch/i386/compile.i386/Gargoyle
 i386
Architecture: i386
Machine: i386
>Description:
 The x11/Xaw3d library has an integer overflow error in the computation of the
  geometry for a Box Layout (file Box.c).  There, the box tries to extend its 
width until its
   height fits within the constraint height (influenced by the window manager).
 Unfortunately, widths are 16bit  unsigned and in the error case (occuring 
under KDE, windowmaker, twm
   work fine here:-), the constraint width is 65535, i.e. maximal.
 The code loops until either the computed height is smaller than the constraint 
height or
  the width exceeds the constraint width.  In each loop iteration, the width of 
the box layout is
  doubled.  This loop does not terminate, if one chooses unfortunate initial 
width, as the width
   wraps around if it overflows 16 bits and if the maximal constraint width is 
SHORT_INT_MAX.
>How-To-Repeat:
 Try to use xfig under KDE.
>Fix:
The following patch for X11/Xaw3d tries to capture the overflow before it wraps 
around the
 `width' variable, setting the width to the maximal one.
It makes xfig work under KDE.

--- Box.c.orig  1996-10-15 14:41:18.000000000 +0000
+++ Box.c       2006-01-02 15:13:32.000000000 +0000
@@ -352,8 +352,12 @@
        }
        else {
            width = preferred_width;
+           if (0==width) width=1;
            do { /* find some width big enough to stay within this height */
-               width *= 2;
+               if (width>=32768) /* overflow */
+                 width=constraint->width;
+               else
+                 width*=2;
                if (width > constraint->width) width = constraint->width;
                DoLayout(w, width, 0, &preferred_width, &preferred_height, 
FALSE);
            } while (preferred_height > constraint->height &&

>Unformatted: