Re: libcodecs(3), take 4

[Alistair Crooks <agc%pkgsrc.org@localhost>]

On Tue, Oct 05, 2010 at 11:28:14PM +0000, YAMAMOTO Takashi wrote:
> > So you advocate putting all the transformations into libc?  That is a
> > backwards step, I believe.
> 
> no, i don't advocate it.
> majority of your codecs are already in libc, libz, etc anyway, aren't they?

Not really - let's count them:

With the 19 hash functions counting as 1 transform, there are 54
transforms.  The ones (to some extent) in libc or libz or libbz2 are:

strvis
strvisc
strunvis
bzip2
bunzip2
gzip
gunzip
md5
rmd160
sha1
sha256
sha512
zero

let's also ignore the listing transformation. The dubious ones are:

bswap16/32/64 - ntoh/hton is done as cpp macros. we have htonle etc
random - we have random(3), but it only produces a 32 bit entity, dd
         from /dev/random will do it, too

So counting the hash functions as 1, that gives us a total of:

14/54, or 26%

so, no, I don't think most transformations are in our standard libraries.

To give an example of what I want them for - to analyse the contents of
a base64-encoded ssh public key, it's easy to do:

osx-vm1: {1} cat .ssh/id_rsa.pub 
ssh-rsa 
AAAAB3NzaC1yc2EAAAABIwAAAQEAvpYW8UheVK2zxWkrQ3mCTQ+YnvZqvEiwUSP7YdCpUog7LKwwCT+8YivSdHpR9kHvn2E7vIphxeC4saoc/EiuaukZGw2JSSi/eRFe6lPpFz+bmRPSk15BD6FI2bCiIYSyEU5Zl1bzssOaEqgA9IV+ymsJzAyb1p5cngq9cUbR7r7yQZTY+079qtX4NyHP2Tn2urqIC8nxZ3//1xZmHMRS/MmwSkePWXom8EfFerrbzMiLwQmj1lRbEZnWo+ckDdI7sBrlflv0+9wCdqLHOxMcfYBjtgK8FrIIBO5FbuOCunTBjVHaSKMbCcj9eJZB6taUUph8RpDaxi6qd2OceK3rlQ==
 eurobsdcon%osx-vm1.crowthorne.alistaircrooks.co.uk@localhost
osx-vm1: {2} awk '{ print $2 }' .ssh/id_rsa.pub | codecs atob | codecs hexdump
00000 |  00 00 00 07 73 73 68 2d 72 73 61 00 00 00 01 23  | ....ssh-rsa....#
00016 |  00 00 01 01 00 be 96 16 f1 48 5e 54 ad b3 c5 69  | .........H^T...i
00032 |  2b 43 79 82 4d 0f 98 9e f6 6a bc 48 b0 51 23 fb  | +Cy.M....j.H.Q#.
00048 |  61 d0 a9 52 88 3b 2c ac 30 09 3f bc 62 2b d2 74  | a..R.;,.0.?.b+.t
00064 |  7a 51 f6 41 ef 9f 61 3b bc 8a 61 c5 e0 b8 b1 aa  | zQ.A..a;..a.....
00080 |  1c fc 48 ae 6a e9 19 1b 0d 89 49 28 bf 79 11 5e  | ..H.j.....I(.y.^
00096 |  ea 53 e9 17 3f 9b 99 13 d2 93 5e 41 0f a1 48 d9  | .S..?.....^A..H.
00112 |  b0 a2 21 84 b2 11 4e 59 97 56 f3 b2 c3 9a 12 a8  | ..!...NY.V......
00128 |  00 f4 85 7e ca 6b 09 cc 0c 9b d6 9e 5c 9e 0a bd  | ...~.k......\...
00144 |  71 46 d1 ee be f2 41 94 d8 fb 4e fd aa d5 f8 37  | qF....A...N....7
00160 |  21 cf d9 39 f6 ba ba 88 0b c9 f1 67 7f ff d7 16  | !..9.......g....
00176 |  66 1c c4 52 fc c9 b0 4a 47 8f 59 7a 26 f0 47 c5  | f..R...JG.Yz&.G.
00192 |  7a ba db cc c8 8b c1 09 a3 d6 54 5b 11 99 d6 a3  | z.........T[....
00208 |  e7 24 0d d2 3b b0 1a e5 7e 5b f4 fb dc 02 76 a2  | .$..;...~[....v.
00224 |  c7 3b 13 1c 7d 80 63 b6 02 bc 16 b2 08 04 ee 45  | .;..}.c........E
00240 |  6e e3 82 ba 74 c1 8d 51 da 48 a3 1b 09 c8 fd 78  | n...t..Q.H.....x
00256 |  96 41 ea d6 94 52 98 7c 46 90 da c6 2e aa 77 63  | .A...R.|F.....wc
00272 |  9c 78 ad eb 95                                   | .x...           
osx-vm1: {2} 

(Hi, Karlsruhe people at the back! - this is what the slide said)

I see no easy way we have of doing that otherwise right now.

base64 is an integral part of web communications.  base85 apparently
underpins PDF documents.  I believe that we need to start looking
ahead.

> > With a separate libcodecs, the transformations can be called from other
> > libraries and programs. Bloating libc is not one of my goals for this.
> > 
> >> > At the same time, there is a case for getting rid of the following
> >> > programs from base:
> >> 
> >> what will be a replacement of them?
> > 
> > A small calling wrapper around libcodecs(3).
> 
> can you provide it so that i can understand that the api work well?
> (sorry if you already did and i missed it)

Sure, I'll provide it - but I'd need to write it first.  The primary
motivation was not to replace things, but to give us functionality
which we need. 

As to the API itself, I want a simple input and output byte stream -
anything else would detract, I believe from the simplicity.  - and
that's why I believe libcodecs(3) provides the right functinoality
with the right API.

Regards,
Alistair