Re: Support for multibyte/wide characters in vis(3)

[christos%astron.com@localhost (Christos Zoulas)]

In article <20130210131249.2d5596d3@shibato>, J.R. Oldroyd 
<jr%opal.com@localhost> wrote:
>Hi all,
>
>Back in mid-2012 I did some work on the FreeBSD version of vis(3) to
>add multibyte/wide character support.  This allows printable characters
>in non-ASCII locales to be displayed as such, rather than displaying
>them as \M-x sequences when LC_CTYPE is set to other than ASCII locales.
>
>By the time I got round to submitting this patch, I found that Brooks
>Davis had recently removed the FreeBSD vis(3) and imported the NetBSD
>version instead in order to add the additional interfaces to the FreeBSD
>library.  I have therefore had to merge my multibyte/wide char support
>into the NetBSD version.
>
>In the process of doing so, I found a bug in the NetBSD version in
>which istrnvis() and istrnvisx() both handle expansion of the "extra"
>argument and then call istrsnvis() and istrsnvisx() which also handle
>expansion of the "extra" argument.  This results in double processing
>of the extra argument, which is an error.  I have fixed this error.
>
>Furthermore, it became clear that the five internal ixxx() functions
>which all did something similar to one another were unnecessary.  It
>is possible to simplify this code to just one internal function,
>istrsnvisx() which is used by all the stub functions.  My update also
>makes this code simplification.
>
>In summary, this vis(3) update offers these improvements:
>
>       - bug fix of double processing of the "extra" arg in
>         some cases
>
>       - code simplification by removing four ixxx() functions
>
>       - multibyte/wide character support
>
>I have tested this on FreeBSD and verified that vis(1), ps(1) and
>mtree(1) still work as expected.  I've also done some additional
>testing with a small program that exercises the new interfaces.
>
>The FreeBSD vis(1) does require a small patch of its own in order to
>properly read in multibyte characters.  I have not checked whether or
>not the NetBSD vis(1) also needs this patch, but I'm including a
>link to the FreeBSD vis(1) diff in case something similar is also
>needed on NetBSD.
>
>The FreeBSD ps(1) and mtree(1) did not require any changes to work
>with the multibyte vis(3).
>
>Do let me know of any errors or problems with this version and I'll
>be happy to take a look at things.
>
>Here are links to the updates:
>       http://opal.com/jr/freebsd/vis/vis3/netbsd-vis.c
>       http://opal.com/jr/freebsd/vis/vis1/vis.c.diff
>
>The changes to vis(3) are rather extensive, so some explanation may be
>helpful.  The changes come down into the following:
>
>       - using wchar_t and wide char string library functions
>         for the internal processing
>
>       - changing "src", "dst", "extra" and associated vars to
>         use wchar_t and renaming function arguments to be "mbsrc"
>         etc to distinguish the multibyte strings
>
>       - adding "L" to all wide char strings and character constants
>
>       - using calloc() to allocate space for the internal wide
>         char arrays and free() to clean them up
>
>       - removal of check of isascii()
>
>       - simplification of five ixxx() functions into one which
>         also allowed removal of the HAVE() macro
>
>       - fix double-processing of "extra" arg in two cases
>
>       - minor style(9) adjustments
>
>Thanks for considering this.

Thanks for doing this; unfortunately it does not work. The problem is that
you are using mbstowcs() which works for NUL terminated strings. This does
not work for str*visx() which encode a fixed length array of characters that
may include NUL's. You can try the unit tests in:

    /usr/src/tests/lib/libc/gen/t_vis.c

and see them all fail.

Best,

christos