Sunday, September 16, 2012

OSSIM hands-on 3: Creating a custom OSSIM plugin

This one is the third post regarding the series of practical exercises for OSSIM users. Now we are going to create a custom plugin to process Exchange Web Server logs through the SIEM engine.

Exchange Web SMTP server logs

The log file used for this practical hands-on exercise can be downloaded here: exchangews.log

Once downloaded, open the file to see the logs we are going to parse. Here are some sample lines:
 2011-10-09 05:00:19 1.1.1.1 36A42160 SMTPSVC1 MEE-PDC 192.168.1.2 0 QUIT - 36A42160 240 6219 68 4 0 SMTP - - - -  
 1.1.1.10 - 1.1.1.9 [11/Oct/2011:13:16:40 -0600] "HELO -?+1.1.1.9 SMTP" 250 46  

Creating the plugin configuration file - exchangews.cfg

1.- Global plugin configuration settings
  • Copy ssh.cfg into another one named exchangews.cfg
  • Change the plugin_id field (use 9001 as it is part of the user range that goes up to 10000)
  • Change location to point to the log file /var/log/exchangews.log
  • Delete startup and shutdown fields since those are not going to be used (there is no application associated to this plugin)
  • Create new translation table:
    • HELO=1
    • MAIL=2
    • RCPT=3
    • DATA=4
    • QUIT=5
    • xxxx=6
    • DEFAULT_=9999

2.- Creating new rules, filling up the fields below

We will create two regular expressions to parse the data, since we have two different formats in the log file.
 [exchangews - Generic rule]
 #2011-10-09 05:00:15 1.1.1.1 36A42160 SMTPSVC1 MEE-PDC 192.168.1.2 0 HELO - +36A42160 250 0 48 13 0 SMTP - - - -  
 #2011-10-09 05:00:16 1.1.1.1 36A42160 SMTPSVC1 MEE-PDC 192.168.1.2 0 MAIL - +FROM:+<test@sample1.com> 250 0 57 45 0 SMTP - - - -  
 event_type=event  
 regexp="(?P<date>\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2})\s(?P<src_ip>\IPV4)\s(?P<userdata2>\S+)\s(?P<hostname>\S+)\s(?P<userdata3>\S+)\s(?P<dst_IP>\IPV4)\s\d\s(?P<type>\w+)"  
 date={normalize_date($date)}  
 plugin_sid={translate($type)}  
 dst_ip={resolv($dst_ip)}  
 src_ip={resolv($src_ip)}  
 hostname={$hostname}  
 userdata2={$userdata2}  
 userdata3={$userdata3}  
 [exchangews = Generic rule 2 NCSA Format]  
 #1.1.1.10 - 1.1.1.9 [11/Oct/2011:13:16:40 -0600] "HELO -?+1.1.1.9 SMTP" 250 46  
 #1.1.1.10 - 1.1.1.9 [11/Oct/2011:13:16:41 -0600] "MAIL -?+FROM:+<Keith@testdomain.com> SMTP" 250 46  
 event_type=event  
 regexp="(?P<src_ip>\IPV4)\s-\s(?P<dst_ip>\S+)\s\[(?P<date>\d\d\/\w{3}\/\d{4}:\d\d:\d\d:\d\d)\s-\d{4}\]\s\"(?P<type>\w+)"  
 date={normalize_date($date)}  
 plugin_sid={translate($type)}  
 dst_ip={resolv($dst_ip)}  
 src_ip={resolv($src_ip)}  

3.- test exchangews.cfg file against the log file (/var/log/exchangews.log)
/usr/share/ossim/scripts/regexp.py /var/log/exchangews.log /etc/ossim/agent/plugins/exchangews.cfg q

Creating database file - exchangews.sql

1.- Create the file using the following examples

INSERT INTO plugin (id, type, name, description) VALUES (9001, 1, 'exchangews', 'Exchange E-mail Web server');
INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority, reliability) VALUES (9001, 1, NULL, NULL, 'exchangews: HELO' ,3, 2);
INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority, reliability) VALUES (9001, 9999, NULL, NULL, 'exchangews: Generic exchange event' ,3, 2);

2.- Insert file values into the database at the server box (10.0.0.30 in case you are following the other hands-on exercises)

cat exchangews.sql | ossim-db

3.- Apply changes in the SIEM

/etc/init.d/ossim-server restart

Enable the plugin on the sensor

  • Edit /etc/ossim/agent/config.cfg.orig and add a new line with exchangews plugin.
  • Enable the plugin using ossim-setup command.
  • Check that ossim-agent is reading the log file with lsof +d command.
  • Insert new logs into the file (using cat >>).
  • Check in the web interface that those events have been processed.


Results and troubleshooting

For troubleshooting check the log files at:
  • /var/log/ossim/agent.log
  • /var/log/ossim/server.log

This exercise resulting files can be downloaded here:

References

http://alienvault.com
http://communities.alienvault.com

4 comments:

  1. Hello

    where we have to write the reg expression ? in /var/log/exchangews.log or /usr/share/ossim/scripts/regexp.py

    ReplyDelete
  2. In 4.4 regexp.py[1] was removed, is n't it?

    [1] http://stuff.ferran.sh/blog/regexp.py

    ReplyDelete
  3. so i got the plugin to work but it works only if i do an echo with the log text to the file i configured in the plugin config "/var/log/mylog.log".

    the problem is that i dont know where does the server store the log of my aplication.

    i configured the ossec agent to colect and send the log like this

    localfile
    log_format syslog /log_format
    location /var/log/mylog.log /location
    /localfile

    any help is appreciated

    ReplyDelete