MesaLib/gallium, r600, vl_mpeg12 bug: crashes

[tlaronde%kergis.com@localhost]

[Resending to tech-x11, probably a better match than tech-userlevel
even if all lists seem dead.]

At least xine(1) and vlc(1) crashes, in various circumstances (xine(1)
directly if invoked without argument; vlc(1) when trying to read a
dvd) but the failure in both cases comes from MesaLib, gallium,
namely, in MesaLib sources:

src/gallium/auxiliary/vl/vl_mpeg12_decoder.c

There are functions (not procedures) but treated like procedures i.e.
the return value is in fact ignored, taken as granted with assertions
being used to catch the failing cases by aborting.

And it aborts.

The problem is that a buffer is initted (guessing by the names),
from samples. 3 planes are expected (a guess once more: red,
green, blue in whatever order) but in my case there is only sample (one
plane).

	=> My screen has pixel depth = 16 bits stored in 2 bytes.
	XXX I should try if going 24 bits exhibit the problem or not...

But a loop goes for 3 iterations (3 planes) and encounters NULL for the
second one leading to the trigger of the assertions.

Modifying the code to stop the iteration when a NULL is encountered
and then bypassing the assertion in the caller, the DVD is rendered
with a rough image, partially scrambled, mainly green in my case.

	=> This ressembles what I had with the 640 permissions on
/dev/dri/cardN...
	XXX I should definitively test going 24 bits...

1) So does anyone know anything about the processing, this part of the
code and the way the "samples" (if this is it) are supposed to be
acquired. From where?

2) Is a 16 bits screen depth known to cause problems and is the code
normally ready for whatever color depth? (both in kernel and in
userland)

-- 
        Thierry Laronde <tlaronde +AT+ kergis +dot+ com>
                     http://www.kergis.com/
                    http://kertex.kergis.com/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89  250D 52B1 AE95 6006 F40C