Project

General

Profile

Prelude Correlator

Prelude-Correlator is a Python rules based correlation engine. It has the ability to connect and fetch alerts from a remote Prelude-Manager server, and correlate incoming alerts based on the provided ruleset. Upon successful correlation, IDMEF correlation alerts are raised.

Initially, the Prelude-Correlator rule language was inspired by SEC, evolving to use a real programming language. At this point, we decided to switch to a Python based rules engine, which provides the great flexibility required for writing correlation rules.

Python-based rule format example

from preludecorrelator.context import Context
from preludecorrelator.pluginmanager import Plugin

class EventScanPlugin(Plugin):
    def run(self, idmef):
        source = idmef.get("alert.source(*).node.address(*).address")
        target = idmef.get("alert.target(*).node.address(*).address")

        if not source or not target:
            return

        for saddr in source:
            for daddr in target:
                ctx = Context(("SCAN EVENTSCAN", saddr, daddr), { "expire": 60, "threshold": 30, "alert_on_expire": True }, update = True, idmef=idmef)
                if ctx.getUpdateCount() == 0:
                    ctx.set("alert.correlation_alert.name", "A single host has played many events against a single target. This may be a vulnerability scan")
                    ctx.set("alert.classification.text", "Eventscan")
                    ctx.set("alert.assessment.impact.severity", "high")

Writing Python rules

In order to write Prelude-Correlator rules, you will need to learn basics of the Python language. Here are some good starting points for learning Python:

Each Python plugin consists of a primary Python class which will run upon reception of an IDMEF event.

from preludecorrelator.pluginmanager import Plugin

print("*** Any global initialization code goes here")

class MyPlugin(Plugin):
    def run(self, idmef):
        print("*** This function is going to be called when Prelude-Correlator receives an IDMEF event")

The primary run() function takes a single argument: the IDMEF message received by Prelude-Correlator.

When running Prelude-Correlator using this simple ruleset, here is what you will get:

$ prelude-correlator 
*** Any global initialization code goes here
09 Feb 09:53:32 (process:14076) INFO: 10 plugin have been loaded.
09 Feb 09:53:32 (process:14076) INFO: Connecting to 127.0.0.1:4690 prelude Manager server.
09 Feb 09:53:32 (process:14076) INFO: TLS authentication succeed with Prelude Manager.

[And when Prelude-Correlator receive an IDMEF message]:

*** This function is going to be called when Prelude-Correlator receive an IDMEF event

Playing with received IDMEF messages

The IDMEF message object ('idmef' in the example above), passed to the primary plugin run() method, provides a number of methods:

The get() method

The get() method allow to retrieve a given field from the IDMEF message using an IDMEF Path. Example:

 classification = idmef.get("alert.classification.text")
 print("*** %s" % classification)

This will retrieve the alert.classification.text IDMEF Path, and print it afterwise:

*** Portscan

Now let see how results are retrieved for more complex object (example: listed object):

sources = idmef.get("alert.source(*).node.address(*).address")
for addr in sources:
    print("*** %s" % addr)

This will retrieve the alert.source(*).node.address(*).address IDMEF Path, iterate the returned value and print them one by one:

*** 192.0.2.50
*** 192.0.2.51
*** 192.0.2.52

The match() method

The match method is similar to the Get method described above, except it allows matching the value against a regular expression:

is_failed_auth = idmef.match("alert.classification.text", "(.+)")

The set() and alert() methods

The set method allows you to set IDMEF Path to a certain value within an IDMEF object. The alert method allow to send an IDMEF object to the Prelude-Manager concentrator.
This is mostly useful when generating CorrelationAlert. Here we will make a simple demonstration of how to use these methods:

  from preludecorrelator.idmef import IDMEF

  # create a new IDMEF object
  idmef = IDMEF() 
  idmef.set("alert.classification.text", "My Classification")
  idmef.alert()

If you run the above code in a Prelude-Correlator plugin, here is what you will get when Prelude-Correlator receives an IDMEF message:

$ prelude-correlator --print-output
25 Jun 14:00:38 (process:13208) INFO: 1 plugins have been loaded.
25 Jun 14:00:38 (process:13208) INFO: Connecting to 127.0.0.1:4690 prelude Manager server.
25 Jun 14:00:38 (process:13208) INFO: TLS authentication succeed with Prelude Manager.

[When Prelude-Correlator receive an IDMEF message]:

version: <empty>
alert:
        messageid: bd32c2a8-1559-11df-8f1a
        analyzer(0): 
                analyzerid: 952281412762694
                name: prelude-correlator
                manufacturer: CS-SI
                model: Prelude-Correlator
                version: 1.2.6
                class: Correlator
                ostype: Linux
                osversion: 2.6.32-573.3.1.el6.x86_64
                process:
                        name: prelude-correlator
                        pid: 14286
                        path: /usr/bin/prelude-correlator
        create_time: 09/11/2015 18:26:39.66467 +01:00
        classification:
                text: My Classification

Python Context

A Context class is available, that greatly eases the writing of Python correlation rules. A Context is a named variable, which you can use to keep track of a given plugin state. They consist of:
  • The context name, used to identify the context
  • An embedded IDMEF message, that might be used to generate a CorrelationAlert
  • An optional expiry timer (the context is destroyed on timer expiration)
  • An optional threshold.

Creating a context:

  ctx = Context("MY_CONTEXT_NAME")

Updating a context (create the context if it does not exist, or reset it's expiration timer if it does):

  ctx = Context("MY_CONTEXT_NAME", update=True)

Additionally, several context creation options might be specified:
  • expire : number of seconds after which the context should be destroyed.
  • alert_on_expire : If a context expires and this option is set, the embedded IDMEF message will be sent.
  • threshold : Internal threshold counter, that might be checked using the getUpdateCount method.

Example, creating (or updating) a context that will expire after 60 seconds, and with the threshold counter set to 30:

 ctx = Context("MY_CONTEXT_NAME", { expire = 60, threshold = 30 }, update=True) 

Setting values within the context embedded IDMEF object:

 ctx.set("alert.classification.text", "My Correlation Alert")

Retrieving a context:

 ctx = context.search("MY_CONTEXT_NAME")

Example: CorrelationAlert on EventStorm (playing excessive events by a single host)

Here is how to write a Python correlation plugin for EventStorm detection: we want to generate a CorrelationAlert if a single source is generating an excessive amount of events.

We initially start by retrieving a list of all the source addresses within the received alert:

source = IDMEF.get("alert.source(*).node.address(*).address")

We then iterate the sources list, and create (or retrieve - if it already exists) the Context associated with each source address.
The Context is set to expire after 120 seconds, and the internal threshold counter is set to 150.

If receiving an IDMEF message with the "x.x.x.x" source address, the created context will be named SCAN_EVENTSTORM_x.x.x.x.

for saddr in source:
    ctx = Context(("SCAN_EVENTSTORM", saddr), { "expire": 120, "threshold": 150, "alert_on_expire": True }, update = True, idmef = idmef)

Notice that we provides the received IDMEF message to the Context creation/update method. This will append the received alert source and target to the Context embedded IDMEF message (which will become the generated CorrelationAlert).

This is equivalent to calling:

   created_idmef_message = IDMEF()
   created_idmef_message.addAlertReference(received_idmef_message)

After creating/updating the context, we check whether the context has been created (and not updated), and we initialize portion of the alert to be generated if this is the case :

            if ctx.getUpdateCount() == 0:
                ctx.set("alert.correlation_alert.name", "A single host is producing an unusual amount of events")
                ctx.set("alert.classification.text", "Eventstorm")
                ctx.set("alert.assessment.impact.severity", "high")

That's it. Here is what will happen with the above plugin (with a threshold of 3, so that the example is readable):

  • Prelude-Correlator receives the first alert: [source address: 1.1.1.1 -> target address: 2.2.2.2]
    • The SCAN_EVENTSTORM_1.1.1.1 context is created, setup to expire in 120 seconds, with an internal threshold of 3.
    • The !getUpdateCount() method returns 0, so we copy some of the current alert data into the Context embedded correlation alert.
  • Prelude-Correlator receives the second alert: [source address: 1.1.1.1 -> target address: 3.3.3.3]
    • The SCAN_EVENTSTORM_1.1.1.1 context is updated, the 120 seconds timer is reset.
    • The !getUpdateCount() method returns 1.
  • Prelude-Correlator receives the third alert: [source address: 1.1.1.1 -> target address: 4.4.4.4]
    • The SCAN_EVENTSTORM_1.1.1.1 context is updated, the 120 seconds timer is reset.
    • The !getUpdateCount() method returns 2.

At this point, we know a CorrelationAlert is going to be generated as soon as the internal context Timer will expire.

Here is the alert generated:

version: <empty>
alert:
        messageid: 12e8fe68-42bd-11dd-9544
        analyzer(0): 
                analyzerid: 2052969743121519
                name: prelude-correlator
                model: prelude-correlator
                version: 1.2.6
                class: Correlator
                ostype: Linux
                osversion: 2.6.32-573.3.1.el6.x86_64

        create_time: 09/11/2015 18:26:39.66467 +01:00
        classification:
                text: Eventstorm
        source(0): 
                spoofed: unknown (0)
                node:
                        category: unknown (0)
                        address(0): 
                                category: ipv4-addr (7)
                                address: 1.1.1.1
        target(0): 
                decoy: unknown (0)
                node:
                        category: unknown (0)
                        address(0): 
                                category: ipv4-addr (7)
                                address: 2.2.2.2
        target(1): 
                decoy: unknown (0)
                node:
                        category: unknown (0)
                        address(0): 
                                category: ipv4-addr (7)
                                address: 3.3.3.3
        target(2): 
                decoy: unknown (0)
                node:
                        category: unknown (0)
                        address(0): 
                                category: ipv4-addr (7)
                                address: 4.4.4.4
        assessment:
                impact:
                        severity: high (4)
                        type: other (0)
        correlation_alert:
                name: A single host is producing an unusual amount of events
                alertident(0): 
                        alertident: 90704f76-42bd-11dd-8410
                        analyzerid: bc-fs-sensor13
                alertident(1): 
                        alertident: 94a7b25a-42bd-11dd-ba64
                        analyzerid: bc-fs-sensor13
                alertident(2): 
                        alertident: 9bc0e962-42bd-11dd-b088
                        analyzerid: bc-fs-sensor13