Re: The lamentation of proplib(3)

[Mindaugas Rasiukevicius <rmind%netbsd.org@localhost>]

Jean-Yves Migeon <jeanyves.migeon%free.fr@localhost> wrote:
> >> Secondly, they are tons of interchange formats out there. libnv is one
> >> more, with its original author stating that it is not really meant as a
> >> replacement for XML/JSON.
> >
> > The library provides an interface to pack and transport the data.  As
> > far as the caller is concerned, it does not matter what serialisation
> > format it uses.
> 
> That's the purpose of the lib.
> 
> However I disagree for the caller: it does matter, indirectly. A 
> horribly inefficient serialization means that the lib will not get 
> widespread use.

Well, proplib has a horribly inefficient format and as you see it is being
used.  In fact, there are many formats and protocols in the industry which
are horribly inefficient, but are very popular and widespread.  Anyway, I
do agree that the format should be reasonably efficient just for a different
reason: there are use cases where it does matter and I have at least one
potential use case in the NetBSD source tree.

> Besides if the serialization format has limitations (no nesting allowed, 
> key unicity, ...), it cannot replace proplib(3) 1:1.

Right, but most mainstream formats can.

> > Why?  Most of the use cases in our tree do not really need granularity
> > on errors - you either retrieve (or construct) the whole thing or you
> > fail.
> 
> Well, you have to know /why/ it failed when you construct it. EINVAL is 
> not really informative: duplicate key, depth limit, out of memory, out 
> of bound (for string or int encoding)...

The point is that failure is rarely a case.  It can be useful as a debug
information while developing, though.

> > That is why accumulated error is so useful, it would simplify many cases
> > in our tree.  If we add support for schemas, then the schema validation
> > code is the routine which could be more informative.
> 
> I cannot see how nvlist_error() can carry this information. How is the 
> API supposed to inform the caller that the schema validation code is 
> wrong and not nvl?

The schema validator would be a part of the library.  As I said/agreed,
it could return more than just an error code.

> This is probably badly expressed on my part. Two things:
> 
> 1 - I was thinking about sysctl.
> 
> *stat(8) binaries use sysctl(3) to query about structures (io_sysctl, 
> clockinfo, ...) that get shared between userland and kernel. There is no 
> reflection here, the caller has to use the correct structure if it wants 
> to get the proper decoding. Else it ends badly.
> 
> An interchange format has to detect decoding mismatches, especially when 
> they pose security/integrity issues (information leak, out of bound
> values).
> 
> 2 - regular polling of statistics
> 
> Following the sysctl example, in the case of top (but any other stat 
> would do: sysstat, iostat, netstat, ...) the values are regularly 
> updated by copying them from kernel back in userland.
> 
> I have met from time to time system-specific APIs to map such values in 
> userland read-only, to avoid pinging back the kernel for their update 
> (Xen iorings, L4 flexpage, can't remember for the others), but there was 
> no library to manipulate them through a higher level interface (for 
> example when you want to pass driver hardware counters). So I had to 
> roll my own. Not difficult when you access atomic-friendly values 
> (integers and such), less so about strings or objects.

I suppose you are talking about various kmem grovellers.  The structures
would not be passed as is.  The kernel code would always have to take the
data and form it as key-value pairs in the dictionaries.  There are cases
beyond the capability or applicability of such library, e.g. it does not
make sense to convert something like crash(8) to such library - as a low
level debugger it will always access the structures from an image directly.

-- 
Mindaugas