Re: Unicode to ASCII

[RVP <rvp%SDF.ORG@localhost>]

On Sun, 21 Feb 2021, Johnny Billquist wrote:

> pdftotext is really ugly if it converts text and creates a stream of bytes, 
> but if it's a Unicode character, it just creates all the bytes
> required to encode the character. How can you in that case even differentiate 
> between U+6161 and "AA" for example?

I presume, in such a case, that pdftotext will choose the non-surprising
behaviour of printing "AA" as "AA" rather than \x{61}\x{61} ;-)


> ''Converting Unicode to UTF-8 in a "lossless" manner'' makes no sense.
>
> UTF-8 already is Unicode characters.

Well, they're separate things, actually (code points vs. an encoding
format)--better discussed in its own thread. You can use other
encoding formats for the same Unicode code points: UCS-2, UTF-16, UTF-32,
UTF-7, UCS-4, ...

> It's always lossless. You can convert 
> back and forth all day long.

Again. not quite, which is why I put quotes around my lossless. (Also,
my peculiar sense of humour getting in the way of good explanations.)

For example, here are 3 different UTF-8 encodings of the same Unicode
code-point for the character ASCII 'A':

A = 0x41
A = 0xC1 0x81
A = 0xE0 0x81 0x81

Proper implementations of UTF-8 are supposed to treat all 3 (or more!)
as the same, but, roll-your-own implementations generally don't--which
leads to black-hats cracking your website... (Also an interesting
topic better discussed elsewhere.)

-RVP