Re: libcodecs(3), take 4

[yamt%mwd.biglobe.ne.jp@localhost (YAMAMOTO Takashi)]

hi,

> On Tue, Oct 05, 2010 at 11:28:14PM +0000, YAMAMOTO Takashi wrote:
>> > So you advocate putting all the transformations into libc?  That is a
>> > backwards step, I believe.
>> 
>> no, i don't advocate it.
>> majority of your codecs are already in libc, libz, etc anyway, aren't they?
> 
> Not really - let's count them:
> 
> With the 19 hash functions counting as 1 transform, there are 54
> transforms.  The ones (to some extent) in libc or libz or libbz2 are:
> 
> strvis
> strvisc
> strunvis
> bzip2
> bunzip2
> gzip
> gunzip
> md5
> rmd160
> sha1
> sha256
> sha512
> zero
> 
> let's also ignore the listing transformation. The dubious ones are:
> 
> bswap16/32/64 - ntoh/hton is done as cpp macros. we have htonle etc
> random - we have random(3), but it only produces a 32 bit entity, dd
>        from /dev/random will do it, too
> 
> So counting the hash functions as 1, that gives us a total of:
> 
> 14/54, or 26%
> 
> so, no, I don't think most transformations are in our standard libraries.
> 
> To give an example of what I want them for - to analyse the contents of
> a base64-encoded ssh public key, it's easy to do:
> 
> osx-vm1: {1} cat .ssh/id_rsa.pub 
> ssh-rsa 
> AAAAB3NzaC1yc2EAAAABIwAAAQEAvpYW8UheVK2zxWkrQ3mCTQ+YnvZqvEiwUSP7YdCpUog7LKwwCT+8YivSdHpR9kHvn2E7vIphxeC4saoc/EiuaukZGw2JSSi/eRFe6lPpFz+bmRPSk15BD6FI2bCiIYSyEU5Zl1bzssOaEqgA9IV+ymsJzAyb1p5cngq9cUbR7r7yQZTY+079qtX4NyHP2Tn2urqIC8nxZ3//1xZmHMRS/MmwSkePWXom8EfFerrbzMiLwQmj1lRbEZnWo+ckDdI7sBrlflv0+9wCdqLHOxMcfYBjtgK8FrIIBO5FbuOCunTBjVHaSKMbCcj9eJZB6taUUph8RpDaxi6qd2OceK3rlQ==
>  eurobsdcon%osx-vm1.crowthorne.alistaircrooks.co.uk@localhost
> osx-vm1: {2} awk '{ print $2 }' .ssh/id_rsa.pub | codecs atob | codecs hexdump
> 00000 |  00 00 00 07 73 73 68 2d 72 73 61 00 00 00 01 23  | ....ssh-rsa....#
> 00016 |  00 00 01 01 00 be 96 16 f1 48 5e 54 ad b3 c5 69  | .........H^T...i
> 00032 |  2b 43 79 82 4d 0f 98 9e f6 6a bc 48 b0 51 23 fb  | +Cy.M....j.H.Q#.
> 00048 |  61 d0 a9 52 88 3b 2c ac 30 09 3f bc 62 2b d2 74  | a..R.;,.0.?.b+.t
> 00064 |  7a 51 f6 41 ef 9f 61 3b bc 8a 61 c5 e0 b8 b1 aa  | zQ.A..a;..a.....
> 00080 |  1c fc 48 ae 6a e9 19 1b 0d 89 49 28 bf 79 11 5e  | ..H.j.....I(.y.^
> 00096 |  ea 53 e9 17 3f 9b 99 13 d2 93 5e 41 0f a1 48 d9  | .S..?.....^A..H.
> 00112 |  b0 a2 21 84 b2 11 4e 59 97 56 f3 b2 c3 9a 12 a8  | ..!...NY.V......
> 00128 |  00 f4 85 7e ca 6b 09 cc 0c 9b d6 9e 5c 9e 0a bd  | ...~.k......\...
> 00144 |  71 46 d1 ee be f2 41 94 d8 fb 4e fd aa d5 f8 37  | qF....A...N....7
> 00160 |  21 cf d9 39 f6 ba ba 88 0b c9 f1 67 7f ff d7 16  | !..9.......g....
> 00176 |  66 1c c4 52 fc c9 b0 4a 47 8f 59 7a 26 f0 47 c5  | f..R...JG.Yz&.G.
> 00192 |  7a ba db cc c8 8b c1 09 a3 d6 54 5b 11 99 d6 a3  | z.........T[....
> 00208 |  e7 24 0d d2 3b b0 1a e5 7e 5b f4 fb dc 02 76 a2  | .$..;...~[....v.
> 00224 |  c7 3b 13 1c 7d 80 63 b6 02 bc 16 b2 08 04 ee 45  | .;..}.c........E
> 00240 |  6e e3 82 ba 74 c1 8d 51 da 48 a3 1b 09 c8 fd 78  | n...t..Q.H.....x
> 00256 |  96 41 ea d6 94 52 98 7c 46 90 da c6 2e aa 77 63  | .A...R.|F.....wc
> 00272 |  9c 78 ad eb 95                                   | .x...           
> osx-vm1: {2} 
> 
> (Hi, Karlsruhe people at the back! - this is what the slide said)
> 
> I see no easy way we have of doing that otherwise right now.
> 
> base64 is an integral part of web communications.  base85 apparently
> underpins PDF documents.  I believe that we need to start looking
> ahead.

ok, the implementation of those encodings might be as useful as other
encodings we currently have.  i still don't understand how libcodecs is the
good api for them, tho.

> 
>> > With a separate libcodecs, the transformations can be called from other
>> > libraries and programs. Bloating libc is not one of my goals for this.
>> > 
>> >> > At the same time, there is a case for getting rid of the following
>> >> > programs from base:
>> >> 
>> >> what will be a replacement of them?
>> > 
>> > A small calling wrapper around libcodecs(3).
>> 
>> can you provide it so that i can understand that the api work well?
>> (sorry if you already did and i missed it)
> 
> Sure, I'll provide it - but I'd need to write it first.  The primary
> motivation was not to replace things, but to give us functionality
> which we need. 
> 
> As to the API itself, I want a simple input and output byte stream -
> anything else would detract, I believe from the simplicity.  - and
> that's why I believe libcodecs(3) provides the right functinoality
> with the right API.

my gut feeling is that
- it's too simple for encodings for which it's difficult to estimate
  the size of output.
- it's too simple to handle large input/output.
- tables, subsets, and regex stuff seem too complex and unnecessary.

YAMAMOTO Takashi

> 
> Regards,
> Alistair