Project

General

Profile

[prelude-correlator] preludecorrelator context resets timer (expire) for each match

Added by Marcus Smith almost 5 years ago

Hello,

I noticed that the context class of preludecorrelator package resets the timer every time that a match is detected. Let me explain it with some examples using the predefined rule EventStormPlugin.py (https://github.com/Prelude-SIEM/prelude-correlator/blob/master/rules/EventStormPlugin.py)


class EventStormPlugin(Plugin):
    def run(self, idmef):
        source = idmef.get("alert.source(*).node.address(*).address")
        if not source:
            return

        for saddr in source:
            ctx = Context(("SCAN EVENTSTORM", saddr), {"expire": 120, "threshold": 150, "alert_on_expire": True},
                          update=True, idmef=idmef, ruleid=self.name)
            if ctx.getUpdateCount() == 0:
                ctx.set("alert.correlation_alert.name", "A single host is producing an unusual amount of events")
                ctx.set("alert.classification.text", "Eventstorm")
                ctx.set("alert.assessment.impact.severity", "high")

For each idmef, It retrieves all the source addresses and then loops over them creating/updating the contexts ("SCAN EVENTSTORM", saddr). The timer (expire) is set to 120 seconds. The threshold is set to 150.

I modified the code of the class adding the following call:

print("*** %s" % context.stats())

So that I'm able to see the satus of the context. The output generated is:

27 Mar 12:11:33 preludecorrelator.context (pid:22123) INFO: [SCAN EVENTSTORM_192.168.1.1]: tmin=12:09:33 tmax=12:13:33 update=0 threshold=1/150 expire=0/120

27 Mar 12:11:37 preludecorrelator.context (pid:22123) INFO: [SCAN EVENTSTORM_192.168.1.1]: tmin=12:09:33 tmax=12:13:37 update=1 threshold=2/150 expire=0/120

27 Mar 12:11:37 preludecorrelator.context (pid:22123) INFO: [SCAN EVENTSTORM_192.168.1.1]: tmin=12:09:33 tmax=12:13:37 update=2 threshold=3/150 expire=0/120

27 Mar 12:11:39 preludecorrelator.context (pid:22123) INFO: [SCAN EVENTSTORM_192.168.1.1]: tmin=12:09:33 tmax=12:13:39 update=3 threshold=4/150 expire=0/120

27 Mar 12:11:41 preludecorrelator.context (pid:22123) INFO: [SCAN EVENTSTORM_192.168.1.1]: tmin=12:09:33 tmax=12:13:41 update=4 threshold=5/150 expire=0/120

For each update, the threshold and the update counter is increased by one but the timer is being reset.

Every time we call the "ctx = Context(("SCAN EVENTSTORM", saddr),...." a call to new is performed and according to the new method, if update is set to true, it calls the ctx.update, as you can see in the next chunk:

    def __new__(cls, name, options={}, overwrite=True, update=False, idmef=None, ruleid=None):
        if update or (overwrite is False):
            ctx = search(name, idmef, update=True)
            if ctx:
                if update:
                    ctx.update(options, idmef)

                    # If a context was updated, check intersection
                    ctx._mergeIntersect()
                    return ctx

                if overwrite is False:
                    return ctx
        else:
            ctx = search(name, idmef, update=False)
            if ctx:
                ctx.destroy()

        return super(Context, cls).__new__(cls)

The update method has timer_rst set to True as default, so that's the reason of the reset.

    def update(self, options={}, idmef=None, timer_rst=True):
        self._update_count += 1

        if idmef:
            self.addAlertReference(idmef)

        if timer_rst and self.running():
            self.reset()

        self._options.update(options)
        self.setOptions(self._options)
        logger.debug("[update]%s", self.getStat(), level=3)

Taking that into account I did a workaround to the code. As you can see in the next chunk:

class EventStormPlugin(Plugin):
    def run(self, idmef):
        print("*** %s" % context.stats())
        source = idmef.get("alert.source(*).node.address(*).address")
        if not source:
            return

        for saddr in source:
            ctx = context.Context(("SCAN EVENTSTORM", saddr), {"expire": 120, "threshold": 150, "alert_on_expire": True}, overwrite=False, update=False, idmef=idmef, ruleid=self.name)
            if ctx.getUpdateCount() == 0:
                ctx.set("alert.correlation_alert.name", "A single host is producing an unusual amount of events")
                ctx.set("alert.classification.text", "Eventstorm")
                ctx.set("alert.assessment.impact.severity", "high")
                ctx.update(timer_rst = False)
            else:
                ctx.update(idmef = idmef, timer_rst = False)

I set update to false and also overwrite to false, so then, when the _new_method is called, it will enter to the

if update or (overwrite is False):

But instead of calling the ctx.update, it will just return the current ctx


                if overwrite is False:
                    return ctx

And finally, I'm performing the update manually, specifying timer_rst = False. But again, the obtained result is not the expected one...

27 Mar 13:06:30 preludecorrelator.context (pid:22859) INFO: [SCAN EVENTSTORM_192.168.1.1]: tmin=13:04:26 tmax=13:08:26 update=1 threshold=2/150 expire=4/120

27 Mar 13:06:36 preludecorrelator.context (pid:22859) INFO: [SCAN EVENTSTORM_192.168.1.1]: tmin=13:04:26 tmax=13:08:30 update=2 threshold=3/150 expire=5/120

27 Mar 13:06:40 preludecorrelator.context (pid:22859) INFO: [SCAN EVENTSTORM_192.168.1.1]: tmin=13:04:26 tmax=13:08:36 update=3 threshold=4/150 expire=4/120

27 Mar 13:06:42 preludecorrelator.context (pid:22859) INFO: [SCAN EVENTSTORM_192.168.1.1]: tmin=13:04:26 tmax=13:08:40 update=4 threshold=5/150 expire=2/120

27 Mar 13:06:44 preludecorrelator.context (pid:22859) INFO: [SCAN EVENTSTORM_192.168.1.1]: tmin=13:04:26 tmax=13:08:42 update=5 threshold=6/150 expire=2/120

27 Mar 13:06:58 preludecorrelator.context (pid:22859) INFO: [SCAN EVENTSTORM_192.168.1.1]: tmin=13:04:26 tmax=13:08:44 update=6 threshold=7/150 expire=14/120

27 Mar 13:07:00 preludecorrelator.context (pid:22859) INFO: [SCAN EVENTSTORM_192.168.1.1]: tmin=13:04:26 tmax=13:08:58 update=7 threshold=8/150 expire=1/120

The expire is also being reset, but I cannot find in the source code where this reset is taking place. Is there any error in the way I'm calling ctx.update?
For further help I attach you the rule file

Thank you


Replies (3)

RE: [prelude-correlator] preludecorrelator context resets timer (expire) for each match - Added by Marcus Smith almost 5 years ago

I think that the reset is taking place at the search function, due to the update=True

def __new__(cls, name, options={}, overwrite=True, update=False, idmef=None, ruleid=None):
        if update or (overwrite is False):
            ctx = search(name, idmef, update=True)
            if ctx:
                if update:
                    ctx.update(options, idmef)

                    # If a context was updated, check intersection
                    ctx._mergeIntersect()
                    return ctx

                if overwrite is False:
                    return ctx

The search function calls checkTimeWindow

def search(name, idmef=None, update=False):
    name = getName(name)
    for ctx in _CONTEXT_TABLE.get(name, ()):
        ctime = ctx.checkTimeWindow(idmef, update)
        if ctime:
            return ctx

    return None

Which in case that update=True it updates the value of time_min and time_max

    def checkTimeWindow(self, idmef, update=True):
        i = self._intersect(idmef)
        if not i:
            return False

        if update:
            self._time_min = i[0]
            if self._time_max != -1:
                self._time_max = i[1]

reseting its value...

So, apparently, indisctintly of the options (overwrite and update) you choose when creating a context, each time the context is updated, the timer is reset.

¿Why does this functionality bothers me?

Because it seems that in order to raise an alert, two conditions must be fullfiled:

1. the expire (time) of the context ran out
2. the threshold reaches the limit value

So, in a use case when I'm receiving continuous logs, despite the threshold is reached, the timer would be reset continuosly and the alert will not be triggered until we stop receiving logs (I tested it). And that supposes that I won't notice that I'm receiving an EventStorm for example until it ended.

So there are two options to solve this issue:

1. Avoid the timer reset each time a new context is called/updated

Changing ctx = search(name, idmef, update=True) to ctx = search(name, idmef, update=False) at the new function

2. Trigger an alert when a context reaches the threshold value, regardless the expire value. (I didn't analyze where this process takes place)

RE: [prelude-correlator] preludecorrelator context resets timer (expire) for each match - Added by Marcus Smith almost 5 years ago

Workaround:

class EventStormPlugin(Plugin):
    def run(self, idmef):
        source = idmef.get("alert.source(*).node.address(*).address")
        if not source:
            return

        for saddr in source:
            ctx = context.Context(("SCAN EVENTSTORM", saddr), {"expire": 120, "threshold": 150, "alert_on_expire": True}, update=True, idmef=idmef, ruleid=self.name)
            update_counter = ctx.getUpdateCount()
            if update_counter == 0:
                ctx.set("alert.correlation_alert.name", "A single host is producing an unusual amount of events")
                ctx.set("alert.classification.text", "Eventstorm")
                ctx.set("alert.assessment.impact.severity", "high")
            context_options = ctx.getOptions();
            if update_counter == context_options["threshold"]:
                ctx.alert()
                ctx.destoy()
    (1-3/3)