Bug #385: Infinite loop in bufpool_add_message - Prelude Manager - UNITY 360

Bug #385

Infinite loop in bufpool_add_message

Added by Friedrich Delgado over 13 years ago. Updated over 8 years ago.

Status:

Closed

Priority:

High

Assignee:

Yoann VANDOORSELAERE

Target version:

PRELUDE SIEM - Prelude OSS 1.0.1

Start date:

08/02/2010

Due date:

% Done:

100%

Resolution:

Description

After some while (a few hours or a few days) our prelude-manager 1.0.0 with several sensors will go into an infinite loop in bufpool_add_message.

In an attached gdb it looks like this loop will never terminate:

        while ( get_total_mem() + len >= on_disk_threshold ) {
                evicted = evict_from_memory();
                if ( evicted == bp )
                        break;
        }

get_total_mem() always returns 0 and evict_from_memory() appears to always return NULL. If I set a breakpoint in the line after the loop, it is never reached.

Specifically, with a breakpoint in line 227 of bufpool.c and display on 'evicted', 'len' and 'mem_msglen' (the value returned by get_total_mem) when the breakpoint is hit it will always look like this:

Breakpoint 5, bufpool_add_message (bp=0x809c0d8, msg=0xb64e6008) at bufpool.c:227
227                     if ( evicted == bp )
3: mem_msglen = 0
2: len = 1124069
1: evicted = (bufpool_t *) 0x0

0001-Prevent-infinite-loop.patch View (1005 Bytes) Yoann VANDOORSELAERE, 08/02/2010 06:38 PM

Associated revisions

Revision f13f3cf3 (diff)
Added by Yoann Vandoorselaere over 13 years ago

Prevent infinite loop

An infinite loop was possible on insertion of an event in the processing
queue in case the event len was higher than the 'on disk threshold' value.
Closes #385.

Thanks Friedrich Delgado <delgado@pre-sense.de> for reporting the problem,
and the useful debug information!

History

#1 Updated by Friedrich Delgado over 13 years ago

Some additional info:

It appears that it's quite directly dependant on the amount of events coming in, if the manager will go into the loop. After it hasn't been accepting data for a few days and I restart it, it starts receiving lots of messages (at least 288 messages per day per sensor, from about 5 sensors) and then stops pretty quickly. After restarting it once or twice again, it will keep running for a few days.

I've tried to put

sched-buffer-size = 101M

into the prelude-manager.conf as a temporary workaround. This might lead to high memory consumption (101MB of main memory) but if I understand the logic correctly, the loop should not be entered at all.

I'm aware that you might need additional information to be able to reproduce this, but I have no idea which information that might be, so please just feel free to ask!