When the string representation of a JSON message grew bigger than
64k, the JSON record would just be truncated. This lead to errors
in the parser(s) of the JSON stream.
This patch changes the buffer logic to grow the buffer on demand.
By adding the key in the root of *flow and fileinfo events it
will be possible to get all events for one application layer by
using a 'event_type:proto OR app_proto:proto' filter. This will
permit to the analyst to get a good view of events related to
one protocol.
This patch also fixes a regression in file logging where app_proto
was available before 94dbd303e4 create
the regression.
The cast of data to AlertJsonThread was not correct as the real
type of the void pointer is a OutputJsonCtx. This was working by
luck because they both have a file_ctx as first element.
To remain constistent with the other logs, set the event type to
the same name as the structure containing the defails. In this
case fileinfo.
Part of bug #1127.
Due to what appears to be an issue in logstash, the 'file' part of
the file event types was masked by a field that logstash-forwarder
added itself.
Since logstash-forwarder is an important part of the logstash stack,
this patch works around the issue by renaming our 'file' structure
to 'fileinfo', thus resolving the naming conflict.
Bug #1127
The TM_FLAG_LOGAPI_TM flag indicates that a module is run by the log
api, not by the 'regular' thread module call functions.
Set flag in all all Log API users' registration code.
Purpose of this flag is in profiling. In profiling output it will be
used to list log api thread modules separately.
In various places SCStrndup was used to 'dup' a bstr string, however
libhtp provides bstr_util_strdup_to_c for this. As this is a cleaner
interface, it's preferred.
This patch separates http keys from file to have a different value
list:
{
"time":"01\/31\/2014-12:04:52.837245","event_type":"file","src_ip":"5.3.1.1","src_port":80,"dest_ip":"1.8.1.9","dest_port":9539,"proto":"TCP",
"http":{"url":"/foo/","hostname":"bar.com","http_refer":"http:\/\/bar.org","http_user_agent":"Mozilla\/5.0"},
"file":{"filename":"bar","magic":"unknown","state":"CLOSED","stored":false,"size":21}
}
One interest of this modification is that it is possible to use the
same key as the one used in http events. Thus correlating both type
of events is trivial. On code side, this will permit to factorize
the code by simply asking the underlying protocol to output its
info in a json object.
Second interest is that adding file extraction for a new protocol
will result in only changing the protocol specific json list.
This patch adds an event_type key to the generated events. Current
value is one of "dns", "alert, "file", "tls", "http", "drop". It is
then easy to differentiate in log analysis tools the events based on
source inside Suricata.
- restore json output-file.[ch] as output-json-file.[ch] after rebase conflict
- fix Makefile.am after merge conflict
- some dev-log-api-v4.0 rebase json fallout cleanup
maxtors
Victor Julien
Tom DeCanio
Eric Leblond
Ken Steele