Results 1 to 1 of 1
  1. #1
    Join Date
    Aug 2006
    Posts
    1

    Unanswered: Parse log files accross servers > single database

    I am wondering if anyone can point me in the right direction. My problem is this (and I think it is large):

    I need to take a log file from several web servers, and combine the output into one database which is searchable by customers.

    The log file is plain text and looks like this:

    Thu Aug 24 10:06:30 2006 lfd: Watching /var/log/secure...
    Thu Aug 24 10:06:30 2006 lfd: Watching /var/log/messages...
    Thu Aug 24 10:06:30 2006 lfd: Watching /usr/local/apache/logs/error_log...
    Thu Aug 24 10:06:30 2006 lfd: Watching /var/log/maillog...
    Thu Aug 24 10:06:31 2006 lfd: DSHIELD - retrieving and blocking IP address ranges
    Thu Aug 24 10:06:31 2006 lfd: SPAMHAUS - retrieving and blocking IP address ranges
    Thu Aug 24 10:14:55 2006 lfd: mod_security triggered by 211.2xx.1xx.1xx - 1 failure(s) in the last 10 secs
    Thu Aug 24 10:23:26 2006 lfd: mod_security triggered by 200.6x.1xx.1xx - 1 failure(s) in the last 25 secs
    Thu Aug 24 10:24:16 2006 lfd: mod_security triggered by 220.1xx.3x.6x - 1 failure(s) in the last 30 secs
    Thu Aug 24 10:44:16 2006 lfd: mod_security triggered by 88.2xx.2xx.4x - 1 failure(s) in the last 15 secs

    I need to filter out the useless lines of information like Thu Aug 24 10:06:30 2006 lfd: Watching /var/log/secure...

    And only include information valuable to the customer like this:

    Thu Aug 24 10:24:16 2006 lfd: mod_security triggered by 220.1xx.3x.6x - 1 failure(s) in the last 30 secs

    lfd: pop3d - 61 logins in 3025 secs from 6x.1xx.8x.1x for email@d$main.com exceeds 60/hour - *Blocked in csf* (flush in 575 secs)

    and on the last line the email address also needs to be removed, or munged.

    ---

    The log file is only delimited by spaces, but as you can see the data also contains spaces, so breaking it down may not be possible. Listing it as one string of data per line is prabably all that is needed though.

    The end result should be a page customers can check to see if their or their clients IP was blocked by the firewall, with output of individual details any greater than just IP are not disclosed.


    I knowing very little about mysql database programming, but does this look like a large job? Or better yet, a tool or program that already does this.

    So far all I've managed to do was get the information symlinked to a browser viewable .txt page and thats about where my manager skill ends and the hiring begins. The original plan was to create a page with Iframes to each servers .txt output page, but the log occasional lists email addesses and/or usernames, and it may grow to a few hundred kb each over the course of a month. So I am assuming putting the info into a searchable database is the best way to go.
    Last edited by pgzn; 08-24-06 at 13:59.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •