PRELUDE SIEM: Develhttp://www.prelude-siem.org/http://www.prelude-siem.org/welcome/themes/prelude/favicon/Prelude-icon.png2019-07-12T09:22:10ZUNITY 360
Redmine Devel: RE: [prelude-correlator] alert is not triggered when receiving continuous logs in a contexthttp://www.prelude-siem.org/boards/2/topics/222?r=227#message-2272019-07-12T09:22:10ZAntoine LUONGantoine.luong@csnovidys.com
<p>This behavior will change in 5.1.0.</p> Devel: RE: [prelude-correlator] alert is not triggered when receiving continuous logs in a contexthttp://www.prelude-siem.org/boards/2/topics/222?r=226#message-2262019-04-16T15:44:04ZAntoine LUONGantoine.luong@csnovidys.com
<p>Maybe we should have an additional threshold causing the alert to be sent regardless of the timer when reached.</p> Devel: xmlmod plugin without format options writes all the alerts in the same linehttp://www.prelude-siem.org/boards/2/topics/2252019-03-29T12:01:51ZSteven Shawn
<p>Hello,</p>
<p>While trying to forward XML Prelude alerts (using the xmlmod plugin output) to a centralized machine through syslog or filebeat I realized that If you haven't enabled the format option, each new alert is written at the same line, so, for example, filebeat doesn't detect that a new alert has been written and consecuently is not sent. The format option I'm referring to is:</p>
<pre>
# Tells Xmlmod to produce a pretty, human-readable xml output:
# format
</pre>
<p>As I'm not interested in having a human-redeable xml output I disabled it.</p>
<p>A quick workaround to solve this:</p>
<pre><code class="python syntaxhl"><span class="CodeRay">static <span class="predefined">int</span> file_write(void *context, const char *buf, <span class="predefined">int</span> <span class="predefined">len</span>)
{
size_t ret;
ret = fwrite(buf, <span class="integer">1</span>, (size_t) <span class="predefined">len</span>, context);
<span class="keyword">if</span> ( ret != (size_t ) <span class="predefined">len</span> && ferror((FILE *) context) ) {
prelude_log(PRELUDE_LOG_ERR, <span class="string"><span class="delimiter">"</span><span class="content">could not write IDMEF-XML data: '%s'.</span><span class="char">\n</span><span class="delimiter">"</span></span>, strerror(errno));
<span class="keyword">return</span> -<span class="integer">1</span>;
}
fwrite(<span class="string"><span class="delimiter">"</span><span class="char">\n</span><span class="delimiter">"</span></span>, sizeof(char), <span class="integer">1</span>, context);
<span class="keyword">return</span> (<span class="predefined">int</span>) ret;
}
</span></code></pre>
<p>The addition is</p>
<pre><code class="python syntaxhl"><span class="CodeRay">fwrite(<span class="string"><span class="delimiter">"</span><span class="char">\n</span><span class="delimiter">"</span></span>, sizeof(char), <span class="integer">1</span>, context);
</span></code></pre>
<p>This is not the perfect solution, so I recommend to analyze the distinct options. The best one, in my opinion, is to append "/n" at the end of buf.</p>
<p>Thank you</p>
<p>Steven</p> Devel: [prelude-correlator] alert is not triggered when receiving continuous logs in a contexthttp://www.prelude-siem.org/boards/2/topics/2222019-03-28T10:15:06ZMarcus Smith
<p>Hello,</p>
<p>The detected issue is described at</p>
<p><a class="external" href="https://www.prelude-siem.org/boards/1/topics/218">https://www.prelude-siem.org/boards/1/topics/218</a>.</p>
<p>To sum up, it seems that in order to raise an alert, two conditions must be fullfiled:</p>
<p>1. the expire (time) of the context ran out<br />2. the threshold reaches the limit value</p>
<p>So, in a use case when I'm receiving continuous logs, despite the threshold is reached, the timer would be reset continuosly and the alert will not be triggered until we stop receiving logs (I tested it). And that supposes that I won't notice that I'm receiving an EventStorm for example until it ended.</p>
<p>So there are two options to solve this issue:</p>
<p>1. Avoid the timer reset each time a new context is called/updated</p>
<p>Changing ctx = search(name, idmef, update=True) to ctx = search(name, idmef, update=False) at the new function</p>
<p>2. Trigger an alert when a context reaches the threshold value, regardless the expire value. (I didn't analyze where this process takes place)</p> Devel: RE: Just a comment on possible roadmap to the futurehttp://www.prelude-siem.org/boards/2/topics/107?r=115#message-1152016-12-21T20:51:58ZTony Sutonysu@su-networking.com
<p>Thx for the clarification!</p> Devel: RE: Just a comment on possible roadmap to the futurehttp://www.prelude-siem.org/boards/2/topics/107?r=110#message-1102016-12-20T08:17:44ZThomas ANDREJAKthomas.andrejak@csgroup.eu
<p>Hello</p>
<p>Thanks for explaining your point around NoSQL and Elasticsearch.</p>
<p>But, this is Prelude OSS and not Prelude SIEM. Prelude OSS do only the alert part (real time part).<br />With Prelude SIEM (Commercial), we include the raw data (Syslog and others) through elasticsearch since 2 years. We also include many other things : behavior analytics, dashboard, reporting, incidents, administration, authentication, etc.</p>
<p>The roadmap of Prelude OSS does not include the integration of the raw data part, sorry. But if you want, you can contribute to the projet to add the support.</p>
<p>Every year, we do an audit of the performance part with experts in database (relational and NoSQL) and for now, our needs keep that relational database is the best choice for IDMEF database.</p>
<p>If you have more than 10000 alerts peer day, then you use Prelude in the wrong way, it is not a log management system. <br />Normally, you have to check every alerts, why it comes, and so on.</p>
<p>Regards</p> Devel: Just a comment on possible roadmap to the futurehttp://www.prelude-siem.org/boards/2/topics/1072016-12-19T19:30:17ZTony Sutonysu@su-networking.com
<p>Hello,</p>
<p>As I'm working my way through deploying Prelude for the first time, there are a number of things in the documented architecture where I'd like to at least just raise suggestions... Which would address some things I anticipate down the road and maybe are not immediately imperative.</p>
<p>First,<br />The use of a relational database.<br />Although proven for its reliability and performance, I wonder if it's really the best choice for an app like Prelude which requires aggregating enormous amounts of data, in fact the more that can be accumulated from more sources and over a longer period of time, the better the analysis should typically become.</p>
<p>A relational database has two fundamental limitations that are hard obstructions... The inability to modify the original schema and physical data storage limitations. Clustering can incrementally increase storage but with great effort.</p>
<p>NoSQL databases, particularly Hadoop style storage have no such limitations. Static schemas are replaced with the ability to just add new data types as you wish on demand and relational aspects are abstracted into a meta layer that can be re-configured easily. Also, hadoop type storage and NoSQL like Cassandra can expand storage simply by just bringing up a new node as a member of the cluster, and today the various administrative tasks like joining, node communication and data mapping are done automatically.</p>
<p>Secondly,<br />I can see that Prelude is in its nascient beginnings of implementing an Agent based distributed architecture with its many advantages (decentralized computing load distribution, local administration and configuration) compared to centralization (better centralized control).</p>
<p>To address the above issues and objectives,<br />You may or may not know about the Elasticsearch project (<a class="external" href="http://elastic.co">http://elastic.co</a>) which I've also been using. Elasticsearch is a competitor to, and a solid alternative to the pure Hadoop/Solr/Pig/Hive Big Data analytical solutions typically used for the biggest Web search engines, IBM Watson which was a Jeopardy! game contestant against humans in 2011(?), and much more. As a re-imagination of the traditional Hadoop stack, a number of features were implemented in Elasticsearch</p>
<p>- As much as possible, instead of requiring a different language for each component in the Hadoop stack, everything in the Elasticsearch stack is based on the same web languages of HTML, javascript and JSON, optionally secured with SSL/TLS. By supporting web protocols and interfaces, curl is typically used for consoles.<br />- All analytics, data structures and data movement is based on JSON<br />- As described above, limitless storage by adding inexpensive nodes to the cluster.<br />- Use Logstash as the main data aggregator and conversion agent, which uses standard grok to create filters that parse data. Links to existing plugins and filters and more is here,<br /><a class="external" href="https://www.elastic.co/guide/en/logstash/current/index.html">https://www.elastic.co/guide/en/logstash/current/index.html</a></p>
<p>I would think that you only need to create IDMEF and IODEF filters(actually definitions) to immediately import or export data from everything else Logstash can already translate into and out of Prelude. And, if you want to inject something into the data like metadata tags, Logstash can do that for you, too.</p>
<p>In fact, should you wish to take a closer look at the Elasticsearch stack to see what you might like to assimilate, you'll notice that its three major components (Kibana which is the web query interface, Elasticsearch itself which is generally storage and Logstash which is the data aggregator and convertor) are completely independent components on their own which can be deployed completely independently or replaced... You just need to need to know how to "talk JSON."</p>
<p>In any case, am very interested in getting Prelude as it now exists off the ground...<br /><img src="/plugin_assets/redmine_wiki_extensions/images/smile.png" alt=":)"></p> Devel: RE: Development of an "attack map" pluginhttp://www.prelude-siem.org/boards/2/topics/70?r=75#message-752016-08-01T16:41:44ZDavid Casierdavid@casier-deroland.fr
<p>Thanks, with your helpp, I've been able to produce a first basic fonctionnal version of my plugin.</p>
<p>You can find it <a href="https://github.com/davidcasier/preludeplugin-alertmap" class="external">here</a></p>
<p>There is still however a bug; Any Ajax request prevent the map from loading and unbind javascript functions linked to buttons. Reloading the map fix the issue, but in order to use more function of the map API, I still need to fix that bug.</p>
<p>I suppose this is caused by the fact that the javascript is not loaded a second time after an ajax request, but i didnt found how to force the javascript reloading.</p>
<p>Regards</p> Devel: Plugin Prewikkahttp://www.prelude-siem.org/boards/2/topics/742016-08-01T14:07:50ZAnis OUAREDanis.ouared@c-s.fr
<p>Hello,<br />I am currently developing an API for Prelude I want to know what are the functions of prewikka to use in order to make a Prewikka plugin ?</p>
<p>Best regards.</p> Devel: RE: Development of an "attack map" pluginhttp://www.prelude-siem.org/boards/2/topics/70?r=73#message-732016-07-21T17:46:36ZAntoine LUONGantoine.luong@csnovidys.com
<p>You should use the libpreludedb API for this. For example in Python:</p>
<pre><code class="python syntaxhl"><span class="CodeRay"><span class="keyword">import</span> <span class="include">preludedb</span>
db = preludedb.DB(preludedb.SQL(<span class="string"><span class="delimiter">"</span><span class="content">type=mysql host=localhost name=prelude user=prelude pass=prelude</span><span class="delimiter">"</span></span>))
<span class="comment"># Get the first 10 classifications of high-severity alerts</span>
results = db.getValues([<span class="string"><span class="delimiter">"</span><span class="content">count(alert.classification.text)/order_desc</span><span class="delimiter">"</span></span>, <span class="string"><span class="delimiter">"</span><span class="content">alert.classification.text/group_by</span><span class="delimiter">"</span></span>],
criteria=<span class="string"><span class="delimiter">"</span><span class="content">alert.assessment.impact.severity == 'high'</span><span class="delimiter">"</span></span>, limit=<span class="integer">10</span>, offset=<span class="integer">0</span>)
</span></code></pre>
<p>More info here: <a class="wiki-page" href="http://www.prelude-siem.org/projects/prelude/wiki/LibpreludedbAPI">LibpreludedbAPI</a></p>
<p>Regards</p> Devel: RE: Development of an "attack map" pluginhttp://www.prelude-siem.org/boards/2/topics/70?r=72#message-722016-07-21T11:28:56ZDavid Casierdavid@casier-deroland.fr
<p>Hello,</p>
<p>Thanks for your help. I've found a better script for the map visualization, using Javascript rather than Python. <br />Also, is it possible to query the alert database from the plugin through a simple SQL command or is there already implemented method for that? I'm trying to query the IP in order to add markers on the map.</p>
<p>Regards</p> Devel: RE: Development of an "attack map" pluginhttp://www.prelude-siem.org/boards/2/topics/70?r=71#message-712016-07-19T18:25:33ZAntoine LUONGantoine.luong@csnovidys.com
<p>Hello,</p>
<p>The pyplot.show() function is not adapted to web development, you cannot use it here. You need to export to png format, and then use <img> html tags.</p>
<p>The ConfigParserSection.get() function is the same as the classic dict get() since ConfigParserSection inherits from OrderedDict.</p>
<p>Regards</p> Devel: Development of an "attack map" pluginhttp://www.prelude-siem.org/boards/2/topics/702016-07-19T11:09:35ZDavid Casierdavid@casier-deroland.fr
<p>Hello,</p>
<p>I'm a student in computer science and I’m currently creating a prewikka plugin which would display the localisation of attacks on a map, through the geolocalisation of Ip addresses.<br />I currently use a modified version of <a href="https://github.com/pierrrrrrre/PyGeoIpMap" class="external">this</a> python script to geolocalize IPs and create the map as an png file.</p>
<p>The problem of this method is that the generation of the map (as a png image) can take a bit of time, which can be troublesome if it need to be updated regularly.</p>
<p>Do you happen to know if it is possible to include a pyplot image in a cheetah template? I didn’t found any documentation about this. I tried to create a simple pyplot graph and use pyplot.show() in the plugin, but it then keep loading endlessly without displaying an error.</p>
<p>Also how does the _config.get() function work in prewikka? The example in the plugin tutorial isn't clear about what the parameters do.</p>
<p>Regards</p> Devel: RE: Creating a more advanced pluginhttp://www.prelude-siem.org/boards/2/topics/57?r=66#message-662016-05-20T12:38:41ZAntoine LUONGantoine.luong@csnovidys.com
<p>Can we see the output of the "python setup.py install" command?</p>
<p>Regards</p> Devel: RE: Creating a more advanced pluginhttp://www.prelude-siem.org/boards/2/topics/57?r=65#message-652016-05-18T14:13:21Zhacen banihana8attia@gmail.com
<p>thank you for the response , <br />well , when I run it from python console I get this :</p>
<pre>
>>> from prewikka import view
>>>
>>> view.View
<class 'prewikka.view.View'>
>>>
</pre>
<p>Regards</p>