Subject: new iconv module for GNU libiconv compatibility.
To: None <tech-userlevel@NetBSD.ORG>
From: Takehiko NOZAKI <sigsegv@s25.xrea.com>
List: tech-userlevel
Date: 09/26/2004 02:46:57
--Multipart_Sun_Sep_26_02:46:57_2004-1
Content-Type: text/plain; charset=US-ASCII

Hello, all.

I wrote new iconv module for Unicode Escape Sequence
(Unicode Character Entity References?)
for GNU libiconv(not glibc iconv) compatibility.

this module can treat following multibyte(?) sequence.
    [ CES(CCS) ]	[ escape + codepoint(hex) ]	[ -f/-t argument ]
    US-ASCII(UCS2)		"\u0000"		C99, JAVA
    US-ASCII(UCS4/ISO10646)	"\U00000000"		C99

please test this module.

NOTE:
you can use this module for mbrtowc/wcrtomb conversion
(you must add mklocale src), but take care that:

following byte sequence may arise inconsistency to ISO-C99/SUSv3.
	
    const char *str = "\u000Z\u3042"; /* Z is not hex */
    mbstate_t st;
    size_t n;
    do {
        n = mbrtowc(&wc, str, 1, &st);
        str++;
    } while (n == (size_t)-2);
	
1st:
	n = -2(insufficient, feed more multibyte)
	mbstate_t { '\' }
	s = "u000Z\u3042"
...
6th:
	wc = L'\'
	n = 1
	mbstate_t { 'u', '0', '0', '0', 'Z' }
	str = "\u3042"
7th:
	wc = L'u'
	n = 0
	mbstate_t { '0', '0', '0', 'Z' }
	str = "\u3042"
...

in 7th loop, 1byte converted to wc, but use only mbstate_t's cache,
str doesn't consumed, so that n = 0.
but ISO-C99/SUSv3 said that when mbrtowc return 0, wc may be L'\0'...

more detailed information for this inconsistency, see
	http://open-std.org/JTC1/SC22/WG14/www/docs/dr_288.htm


very truly yours.
--
Takehiko NOZAKI <tnozaki@NetBSD.ORG>


--Multipart_Sun_Sep_26_02:46:57_2004-1
Content-Type: application/octet-stream; type=gzip
Content-Disposition: attachment; filename="citrus_ues-20040925.patch.gz"
Content-Transfer-Encoding: base64

H4sICKyPVUEAA2NpdHJ1c191ZXMtMjAwNDA5MjUucGF0Y2gA7Txrd9pIsp/Fr+gke2IwYCTwCxxn
B4PsKGODF4k8zkyOVhbCaAISFwnHuZv8963qbonWA4yTzMx+CCcxUld1vbuquhFo3si5b5GRG4QL
96YWOGFQm8JNULuxAqc2cwun3/8qDDo6GbtTp0VqE3/m1Oy7oNZzwjO9WwsWdm0998pdYeEAzLlz
vVuygLfA9T2i7B3ISmHkjsekuiTVBb0nIGy1Wt2gi6Q0iO7MSV2W90m90WoctJQmqcrwkhjJcrm8
aX79YDVfOWjVFfjH5hd++YVUlcPmUeWYlOm7IpNffimQvdoyWNSCibVwaq5y7NWcYHRT0/T+4f4h
f6u+H+7hqIRsqsHnYBm60yqdInnTYB2NK03vSFLOpE0Tamfu7QHjto5decPsTrO5efIm1h1DfWd8
+/SLM+VYbsjfLvzr9pv2t7P/ta8dVwffOX/47fP1iTsOzdeavjFWtM3rOZhM95zp+O9c1FyEtSv7
uJlY2MdNEgm9aXFzHEmpk/bylq/Qw5Z8BP/iFQ6kNy3wiEZ6le+36ofCKm8cVQ5JuYErfbXAp0CP
OgwuYFXX5Xp9L/D39qMFijBwE0DzZlzpv1reHy7MeCx+LodydsZQ1fOpr8XdVvahca4cNupby77C
fwSH43zyW0T731vBmARrY71+LCeCHe4JF/qhYE9XM6W1v9862I9jHUk/FOzZiqY0W41ERTvGWIe/
R2ti/eLs1/qv1Jd7je28Ka4OnPPIcM+bsiGGt5VKjMotpeJhuT0P1/a9O9PzPWctm/UBbfuz+d/X
jnHuawP5qJFsx+B+QzvGqUnKvhB8jdZBM26nJEZyTQBH89OJut5qNMTgPajv0/DF980BbIEHkSrz
xQORuxVyFLNZ5DXRiogPk43idCsZaIRuhSmEZhafB2WEb8780XLq1EDo2pX10cFI+sFhuZEVi7Pe
ZiSMvNrIuat5y+lUUshrC8K0eSQTWW7RfzxOMMI2EcrE2FGyGZArMgRYZR+jq/yM/IMto38UyoWy
PuhcD9RT2w0Xy8AEt7uePV2OHPLiJhjtAdO92ceXonHhv11j6DUmSsBvzaUT7Nl/gpEfZCka+0Hk
xxr9IYIZ4zfjbi5h/CO61SrXdiP7k90aeqC2Wy2UyS7p+PPPC/d2EpKiXaLEOpQLuV74fzh2WKFY
7emUUKwAMlvgLO6c0R4CKHDg8DwE/TZkPMsbkWXgENcjgb9c2A4duXE9a/GZjP3FLKiQT244If6C
vvvLkJIBLd2xa1tIpEKgaSdzZzFzw9AZkfnCv3NHcBFOrBD+OEBoOvU/YaqF5TlycVJAyeDEmRO2
6I2yl5IuIP44Esv2IeBmyyAEnUILxEW61o1/hyBuFUoFXp4furZTARQ3IJhpkc6KM1UxKRZwtaeW
O3MW1FKknhUFWApmiUQBXUdLEO9PkoYwRSNSI99ezhwvtCLf1cAtPiAsyMwKnYVrTYOV+anfkLKo
yCoSjFeaTvT+ufG2PVAJXF8P+m+0rtolZ+8BqJL20HjVH5B2r0s6/Z4x0M6GRn+gk3//u60D/s4O
gljE9d4T9R0kCV0nMEO7ur7UgA4QHrR7hqbqFaL1OpfDrta7qBAgQ3p9g1xqV5oBaEa/gvwopexU
0j8nV+qg8wpu22fapWa8pyKda0YP+Z2jiOS6PTC0zvCyPSDXw8F1X2fkULMubDov29qV2t0jIAaw
JuobtWcQ/VX78lLUFP4lFD1TQcb22SWjRRmBol1toHYM1Gh11QG7gXiXFaJfqx0NL9R3KujSHryv
cLq6+q8hIAGQkuu2r9oXoF7xAcuAVzrDgXqFEoMp9OGZbmjG0FDJRb/f1SktYKCrgzdaR9VPyGVf
p0Yb6moFuBhtKgCQAYsBGK7PhrpGbaf1DHUwGF4bWr9XopRe9d+CcUDeNkzvUkP3e1RtsFN/8B4J
o02oHyrk7SsVxgdoVmq5NppDBwt2DEpOQAW+YFRD0Jf01ItL7ULtdVSE9pHSW01XS+A2TUcEjbF+
237P1BxSE6DHQDp2KcRwhfqVaOek3X2jofgcGcJB13jo9M8pKX3YecVdEK8ISLUEal9c3YLPkMVH
zjjYm7zE8TGBG9dzRsVL7czUOx29RJ4/J0+i0anrhWBF04TSpHWLT6Ma+rR0AtMdWPBjUkObsMl0
4UNyIDiPJ/ok8/Dz3ImYR+Mzy54AtxrMMcczWNEpuBVAxg9Tg85i4aURMSV4t5nBUYYijnlpijCI
hT8zCLZIDU5dKAtpLT7ZE2tBx4TRp7xketbMCeaW7exNnuaAuVXyQKz25sNsnJcPArkdz86HLTmv
Qhnng3oEDLe0Q/KfQllCxwFPB/3LokC66ndVs9NsSkW5lBrFYzupqMDwV2JCb6Z6UNTAB5o39k82
cLAnU8cDBAmNBne/KfUPJxJE0u/Le3ixv6RKluG4qhySYLlY+LdQDsjcchc0sBg/HeqGs45RWiDJ
cZGlgIEcofSR8TJcYtleTkO3OvVtawq1y7LdqRt+ptwkKeYmgU9uuPSp0cWa4dD/ZOeMrxkOKH6Q
gXyyATC7yRkPKCBvRjThK9gFxhCFGa5jgL1iL0U+NTuqZhp9U9WKpu24ZkmSis/5ZfWl45ayqLrR
NlSGUiHmeOnZMCuewrjuBeazZwyW4HY+7HV67Su1OANGksnDE6QD9JmAB+m0j9nV1HrnfUnKOFaU
ynh/rYp4saJ59KjwkmC0PKSrMxOqlXnVflc0mU2Ueq5wOqNndtVrtYfFU5JkAZEBe6ra1U3oLC61
jmaApJpRNOcBkJXRNmgw1yam6XqQQx1y57sjbJbbgwsdyxcGo2go13NDk1q5mLYKlATThG4ZkiKE
u+NWMDxWqibBAbgWVwQsCMcl1nzuWIuAhD65ccjSg3Z6xNaB2dXaF20dSrNRDMiTU9IbXl6WaBBJ
M2cWOGGxiDKT3RK02XKFBO7/O/64uBuUEOvrt+gIefPjo3RkAgiAeTBfuHcMCG0q9KtbmSFwnNkD
NmCURUM8ZCJ7/pnPqsCWhwqzMtgPsNbSe7S9yFpzRHBRTpIx7GONFqy1V9acK6MJcRXZL8dc0XKj
hzG4aKFq3Z/Da3hZSkAb9Qh6dM5eFCNrcKhXYOFR6LtFWrq4tgHsRewJ7lppKbEt2HLuyDstwq6U
+KoeXzXiq/2dVjznIB49jK+O4qvj+KpJ50iwV1wuPOAL5RH4URMyhHaMehZfdQRG3XhUja/Ok0QV
2LQTSrktUrZi/Jv4yhYoj+JRJ74ar6NsMcrUWzG4qmQ8CLm7fQ3Zeef333fSfgWYtLPcSTuUDg93
Yh+ysKU9hhPYv30gp+Q/hNGtkJgSvWSzK+TrSd7s3ZF7C5Ofykq9sX9weHTctG6wj356sj5ilqEP
TItsPgTtEsAhsAnJfQVRYO8d8lhiOHPgQIv47rxchmsUWf6QGoBJpEbke4VBRj5rZDgGiPlb8Z68
fEmKiFg9Jfsl6OgB//wD6wM+TVxobij0JZH5Cou8MAf/BNQRZL1eE9eM+7EiVQrMCHZSDrk2cYRC
5/byFHiPjmUZtxU48IIO3IzHaxMc4zL1H83FTnMZZ7kw4kK65O2ZiQll63RJe32QJpkLP9nJdMm8
mkqoNGfBVO+B1DsPoH+PcPicBIIHV9CvFspRDMU8pd1APmGNtiTNfeAJfWkFD5kg7hAQR6JkN+rx
ANpXmriAzZpCNn88tW4DYTNwbtJ1A63NvSKO6sPBoH+B7ZR8X6fTxbwOdUFM+VApwFpkZn3GMoHD
jysSYJq1UG6XVA2BfWqxGMiwRKBYkFMOY2tnTT8F7mZeoCyk3YjwKSkyf5Q29X4nYvKThYyHklCy
1Zd0FwTLUMENlCTd+lA2Xe/OmrojJjW4B9nF/ipRv05cGJz6MQQdRyEAA3fDOHUavFN05HFKjvES
9zrFk5NIb9GmMA8WjSgYU1tEYtIyZZgeyC01i9GmYK9aJS+IEg1BXGKGo0JJuQat1jksaTgJTQd/
Yz6/iRzL5Q/UrXK5TJG/pqSTRZEEEn7wAYOElYNYRhooeOqRwgS974/GwxLD4r6aBs7/cYnpgBXY
ritI/JU4UzyJXgmjbBAGwHFNiuVhrvxySvjC49QFwhmN4mpGvnzhCYRA8qq+xH09com37jGXh6Qv
MjGeR1LQUyKu0uFW9s3lsyWHo80cVkbbxIRRgMCCYGHdXI6bebghH8R8QeR8mrioqlV2mcxz3MpP
BCuDG0hauxWfjN5CIK5gcXqN4ZmV+QJq/UkOLFmwJ24pxmI5BP6+wLnkC5aICIbsGdXV+mWjT5K1
eeqXokWRNRO8IFtVoybgBNNWNSrWMQpPc4AJgkCj+AXQStAuYpcjC3iU+tJzbXY+RcdYZkg6OSdr
nQrGofmTc1un9ilOSOudMeQKIZuqJysT5AoeSZ5Y4rGb0+qJq54z5RzxLatIwn1izkpI8TXOlxD8
PH1CHxG63pKdqlFX0j4+o99q4bBOVBJzMl8btN64tBARMHeiUMBIuVxK5XUXU7lwC0GgMOo3C8f6
CFdc/lac5MVSWiUKrjxQhhFOVdSkjFF9jEkLBZpqe8pXv4C7SldsEDqKatRRiMjVBm3bsxWOnxAK
lZAq8WnVz7BuHgZgEu8zS6xNE5pe3rhz1VoCRNV6b9q066HVqZUvRlU5Scy51NV/Pdgq8wPIx7XK
rAl+oAeO+mn6XvlhLfGWfe6GRvXPbUgT0fgkiqCsJ/M6QWqqE/zw5d27dy3+WUscu0js3jqg2yHM
D3gLmyGWOsTWbLVMnPqNe5tcJDltFt3O8ngUy3e0YJAplDvOND6EETkD38PNfNEcZajSfAcdj1co
3ecrshXCS2lKkhdRE5RkvbEHimRT6mLJX0lHXUBrl1CUWClBCGy5FZkVLVrpWJfgR2DcgjfQA+VE
8dtCZYzUuGHYAn/qr/ATBSNWUF6j4Hb2r5BGvRQ32iKHtXkmJ9HQyfxgL6gQgUVmCyKQTdQPAOWk
82ySpBpulwfrZ9rF5nOJ9eeu7BM3+pmLHWydHolpB+4onchwbIXh4mNIaRR3dF8R8uZjj2CRw9qM
BbRT2WqX4vMiuItwsGEkWIl+jJU2/HfY0A7wSGZ7G+Yex6xOY2IbU7Ou7AlqPNZsn+ycNB7ZMp2+
41jfTVZz4Lutvdhhvzm3FoFj3lkL17qZbnmqv+7I/g4rLK+esGzglhmBH+WQq7bReVWE4LLssMQa
1N8LZTxn5M0s3kb9JczG3rrID+Kf3cOKwp1bhIY7T+Dr4ZGwPZsX5xXy7D4+t+fofI9OJ0jA9WTF
RuI8IO+Kc05i+BwTVS7o60ra+NSztosPrRjwB0KPuks8qZvTlAKeAo54ySdFasYHCdHJ/+6cD8RH
7YmjcUliprSbzYqw/SbRx+k8pcfdJyfzOibzR4LMH9adlaVDK1iaEOvoeT/PxGcN+ddkmD30UZLD
Y4w/hGBC7+25YU740U+Gkwsp58PDTHO5Pec1fPPywYZPq9aF/oYOsJD5WBOPBYXPNR2+n06vUUSj
/DijdSseLFb9IS/6DNB8eTNl1nUWY8t26NMN9DER/jQQmLmjGYOhzj8p76qdS70IhkWB0qBzs38d
A9c9f2KGzmw+hWadP1PyFyjEykRGI93oqr1Ovkox7CGdeAkSldr+MeDJX/8Y8OQxjwFPfvRjwJPM
Y8DHLaWefQy4cfTzKeCfTwH/fAr451PAP58C/mufAubP4I49fDgyqobYYr0yVz1/ehyf+D1TL7Qe
K6bpzuBCNaCI0sf31tTZPAxT7XUjeuLDw0nmKDEvuGu/5R4sfvT3bTayigrsRqTHFNZNhDLf+txP
fuuTF9SjnC814aOUkvQU7AiNWPTRdHT/BkIPV7z0FBjBPXRC+Eg34nf0p/RJRXreiEPyfeNcekZ2
/rmz0RX0O/t/kS9iXhudEWN9tzciSjnuUBo/1B3I6cf4A6/2rKlrBXt4/ye4JvcrqdvKsvYLqvuJ
r6fukzw90KHbMpLqMnm9nKLPGuD1Vv2wdbD6oYH99U7PpZXqbw9byrHg/2alQcrNygH9ButH3z2u
LnEPyn5QokDcwK+6i6qiKNHgoAD7Mec+RCz6oxuFst1s0rtms1DGXT7eYFQ87OyRu/gfcbUoyeMc
ndBhs5tF1O9zcpJSxsWy+DXSQ3TxIXcx8yF6KP3DIxw2TMKGHEZdHYNWv7ZSKNOvUESA+GdQCuUO
i4rET7sU/gsFG8GeE0gAAA==


--Multipart_Sun_Sep_26_02:46:57_2004-1--