Re: Support for multibyte/wide characters in vis(3)

["J.R. Oldroyd" <fbsd%opal.com@localhost>]

On Mon, 11 Feb 2013 03:33:20 +0000 (UTC) christos%astron.com@localhost 
(Christos Zoulas) wrote:
>
> In article <20130210131249.2d5596d3@shibato>, J.R. Oldroyd 
> <jr%opal.com@localhost> wrote:
> >Hi all,
> >
> >Back in mid-2012 I did some work on the FreeBSD version of vis(3) to
> >add multibyte/wide character support.  This allows printable characters
> >in non-ASCII locales to be displayed as such, rather than displaying
> >them as \M-x sequences when LC_CTYPE is set to other than ASCII locales.
> >
> 
> Thanks for doing this; unfortunately it does not work. The problem is that
> you are using mbstowcs() which works for NUL terminated strings. This does
> not work for str*visx() which encode a fixed length array of characters that
> may include NUL's. You can try the unit tests in:
> 

Just following up on this to let folk know that the problem with block
input containing NULs was fixed, and then further work was done to ensure
that a vis-encoded string would always unvis back to the original input
even under circumstances when the locale of the data being encoded does
not match the current locale setting.  This is done by falling back to
byte-by-byte encoding if a multibyte character decode error is detected.

The result of all this is that the content of a vis-encoded string may
vary based on LC_CTYPE locale setting, and programs that output a
vis-encoded string will now display all graphics characters of the locale
without unnecessarily encoding them.  Unvis will always decode a vis-encoded
string back to the original input.

In practical terms, this means programs like ps(1), that use vis, will
now display 8-bit characters as their locale's characters rather than
encoding them to \M-x\M-y.

The former behavior can be achieved by setting LC_CTYPE=C for the vis
operation which will ensure all 8-bit characters are vis-encoded.

Any problems/comments, please let me know, preferably while this is
still fresh in my mind!

        -jr