Bug #385
Infinite loop in bufpool_add_message
100%
Description
After some while (a few hours or a few days) our prelude-manager 1.0.0 with several sensors will go into an infinite loop in bufpool_add_message.
In an attached gdb it looks like this loop will never terminate:
while ( get_total_mem() + len >= on_disk_threshold ) { evicted = evict_from_memory(); if ( evicted == bp ) break; }
get_total_mem() always returns 0 and evict_from_memory() appears to always return NULL. If I set a breakpoint in the line after the loop, it is never reached.
Specifically, with a breakpoint in line 227 of bufpool.c and display on 'evicted', 'len' and 'mem_msglen' (the value returned by get_total_mem) when the breakpoint is hit it will always look like this:
Breakpoint 5, bufpool_add_message (bp=0x809c0d8, msg=0xb64e6008) at bufpool.c:227 227 if ( evicted == bp ) 3: mem_msglen = 0 2: len = 1124069 1: evicted = (bufpool_t *) 0x0
Associated revisions
Prevent infinite loop
An infinite loop was possible on insertion of an event in the processing
queue in case the event len was higher than the 'on disk threshold' value.
Closes #385.
Thanks Friedrich Delgado <delgado@pre-sense.de> for reporting the problem,
and the useful debug information!
History
#1 Updated by Friedrich Delgado almost 14 years ago
Some additional info:
It appears that it's quite directly dependant on the amount of events coming in, if the manager will go into the loop. After it hasn't been accepting data for a few days and I restart it, it starts receiving lots of messages (at least 288 messages per day per sensor, from about 5 sensors) and then stops pretty quickly. After restarting it once or twice again, it will keep running for a few days.
I've tried to put
sched-buffer-size = 101M
into the prelude-manager.conf as a temporary workaround. This might lead to high memory consumption (101MB of main memory) but if I understand the logic correctly, the loop should not be entered at all.
I'm aware that you might need additional information to be able to reproduce this, but I have no idea which information that might be, so please just feel free to ask!
#2 Updated by Yoann VANDOORSELAERE almost 14 years ago
- File 0001-Prevent-infinite-loop.patch added
- Status changed from New to Assigned
- Assignee set to Yoann VANDOORSELAERE
- Priority changed from Normal to High
- Target version set to 1.0.1
Thank you for the detailed report! Could you please try the attached patch, and let us know whether this solve your problem?
Regards,
#3 Updated by Yoann VANDOORSELAERE almost 14 years ago
- File deleted (
0001-Prevent-infinite-loop.patch)
#4 Updated by Yoann VANDOORSELAERE almost 14 years ago
- File 0001-Prevent-infinite-loop.patch View added
#5 Updated by Friedrich Delgado almost 14 years ago
Thanks for the quick response.
I've applied the patch, recompiled and reinstalled and disabled the workaround.
I'm quite sure that there will be no infinite loop any more, but I'll get back to you in a few days, if I see any further problems.
#6 Updated by Friedrich Delgado over 13 years ago
Patched prelude manager 1.0.0 has been running smoothly for 3 days now, no problems in sight. Thanks!
#7 Updated by Anonymous over 13 years ago
- Status changed from Assigned to Resolved
- % Done changed from 0 to 100
Applied in changeset f13f3cf321abf762e589877ac1601de4e89da4b7.
#8 Updated by Thomas ANDREJAK over 8 years ago
- Target version changed from 1.0.1 to Prelude OSS 1.0.1
#9 Updated by Thomas ANDREJAK over 8 years ago
- Status changed from Resolved to Closed