Re: A log monitoring tool

[David Holland <dholland-tech%netbsd.org@localhost>]

On Tue, Jan 03, 2012 at 10:13:48PM +0530, Abhinav Upadhyay wrote:
 > > That's about right; it's not really research (it's more about applying
 > > current research in machine learning) but it's not trivial. Also, the
 > > first thing to do is go find out what other people have done along the
 > > same lines. I think there has been some since I originally wrote that.
 > 
 > So, I looked around a bit. My Google searches lead me to two
 > similar projects.
 > [string matching stuff]

Right, those aren't machine learning. While we probably ought to have
functionalitiy like that in NetBSD, it's a SMOP.

What I actually meant was to look for research in the machine learning
community, or possibly stuff appearing at forums like LISA; I think
there has been some, but it won't have made it into production tools
like those yet.

If you have (a) professor(s) on hand, ask them; otherwise, I'm not
really sure where to start but I can probably find someone to ask
myself.

 > > I'm not qualified to advise it though :-/
 > 
 > I am not sure I can do it as well, but I just learned some machine
 > learning and I thought this would be a cool project to try on.

Yup, it should be.

 > If I
 > can come to an understanding of what is really expected of such a tool
 > then perhaps the job of implementing it will become much easier. I
 > think you can advise or help with that :)

That's probably true :-)

The ideal, I guess, is a daemon that sits somewhere secured and
monitors logs (maybe from a whole server room full of machines) and
reports stuff of interest on a big screen in the ops center, possibly
including alarm bells for critical events (hackers, bad h/w failures,
etc.)

For the moment I'd spend time on algorithms and not on the engineering
issues (e.g. protecting the daemon, collecting logs securely, etc.) as
many of those problems are difficult in practice but not very
interesting. syslogd itself and the whole way logging works in Unix is
a big pile of fail, for the most part, and that isn't going to be
solved easily or quickly.

-- 
David A. Holland
dholland%netbsd.org@localhost