Re: A log monitoring tool

[Abhinav Upadhyay <er.abhinav.upadhyay%gmail.com@localhost>]

Hi,

On Mon, Jan 2, 2012 at 2:42 AM, Julian Djamil Fagir <gnrp%komkon2.de@localhost> 
wrote:
> I cannot really imagine how such a tool should work, but I'm absolutely
> clueless about AI. ;-)
> I simply cannot think of any possibility of summarizing logs except for e.g.
> searching for terms like 'error', 'fail' or 'warning', in many cases they are
> just too different and unorganized.

I believe if there is sufficient amount of training data available (in
the form of logs which have been manually marked as critical and
others non-critical or so) then using supervised machine learning and
probably natural language processing, a model can be developed for
classifying important log entries from non-important ones.

>> Now, I am not a sys admin, nor I have any experience of doing any
>> significant sys admin related work. So in other words I don't have any
>> idea what logs are important from a sys admin's point of view, what
>> does a sys admin consider important in a log and what is really not
>> important. How do they distinguish between important and non-important
>> stuff in the logs.
> The problematic events I can think of immediately are:
>  * reoccuring events (often with varying parameters)
>  * mass events
>  * logins or login attemps, especially the usual scanning ones if they reoccur
>  * events with high log levels

Yes, I think those are the kind of events which interest a system
administrator. A crude way to identify events like login attempts (to
break in the system), is to watch for log entries which are rare but
when the do occur, their frequencies are quite high.

>> Any details about these topics or pointers to the literature which
>> tells about these kind of things will be highly appreciated, as I
>> think if I want to play with this idea then I need to setup such an
>> environment on my system as well.
> I don't know of any, but simply having a look at the great services (Apache,
> openldap, postfix, pam, raidframe) might give you an insight what you have to
> expect.

 ok. I guess I will play with some of these to see the data.

Thanks,
Abhinav