Re: SoC: Improve syslogd

["Rainer Gerhards" <rgerhards%gmail.com@localhost>]

On Thu, Mar 27, 2008 at 4:12 PM, Martin Schütte 
<lists%mschuette.name@localhost> wrote:
> Rainer Gerhards schrieb:
>
> > The biggie, at least IMHO is Unicode support. While I followed every
>  > IETF move with rsyslog (in fact, I used it partly for prototyping), I
>  > was hesitant to include Unicode support. That has the potential to
>  > break many existing applications.
>
>  Indeed. Having Unicode in logfiles also requires all used shell tools
>  and parsers to handle it correctly.

Having unicode in the syslog spec does not necessarily mean that it
also needs to be persisted to log files. One approach may be to accept
it as is and then convert it to whatever local charset you intend to
have. So the end result inside the logfiles will be local script, just
as is today.

I forgot to mention one thing that needs to be looked at:
syslog-protocol permits NULs to be present inside the MSG. It has some
provisioning that permits a syslogd to remove/modify them, but that,
of course, affects syslog-sign. NULs are quite problematic in all
syslog code I have seen ;)

>
>  > Also, the new log line format is something that can break existing log
>  > parsers, so one needs to be very careful about how it is implemented.
>
>  The usual transition problem: noone uses the format --> no parsers, and
>  no parsers --> noone uses the format.
>
>  Apart from that I would think users with centralized logging or
>  automatic analysis would be the first to appreciate the enhanced format;
>  while desktop users who check their logs manually simply do not care if
>  the lines look a bit different.
>  Since the new format (-protocol) is a real superset, It is also easy to
>  provide a filter (a sed or awk command to re-format) for compatibility.

I like the idea with the filters. That's probably a solution to the issue.

>  Another problem is the API and the handling of structured data
>  (name=value-pairs).
>  I expect all applications to build their own structured data-strings and
>  pass them as messages to syslog(3), just to be portable. -- Thus a
>  syslog library which modifies the structured data (to insert additional
>  information) will have to parse every message. :-/
>  In the long term it would be nice to introduce a new API call which
>  receives the structured data as a seperate argument.

The right course of action, IMHO, is to provide the underlying
infrastructure and then approach the glibc folks with a new syslog
call. It's not something that is hard to do, but it requires a lot of
community consensus - and the (near-) universal availability of
syslogds that understand the format.

Rainer