Add a keyword configuration dump-all-headers, with allowed values
{both, request, response}, dumping all HTTP headers in the eve-log http
object. Each header is a single object in the list request_headers
(response_headers) with the following notation:
{
"name": <header name>,
"value": <header value>
}
To avoid forged malicious headers, the header name size is capped at 256
bytes, the header value size at 2048.
By default, dump-all-headers is disabled.
Substitute json_string with SCJsonString custom function.
SCJsonString will ensure string passed is json-encodable (utf-8).
If it's not, the string will be converted in such a way that any
non-printable character will be encoded in its hex form.
The resulting json object will be returned.
rust modification will encode any non-printable character during its
conversion in to_cstring.
XFF configuration is already set in app-layer-htp-xff, and in
output-json-alert. Extending XFF configuration to files and HTTP allow
to get the same behavior as for alerts.
Extend the configuration of filestore json to let filestore metafile
dump be aware of xff. This is available only if write-fileinfo is set
to yes and file-store version is 2.
The new OutputInitResult is a struct return type that allows
logger init functions to return a NULL context without
raising error.
Instead of returning NULL to signal error, the "ok" field will
be set to false. If ok, but the ctx is NULL, then silently
move on to the next logger.
Use case: multiple versions of a specific logger, and one
implementation decides the configuration is not for that
implemenation. It can return NULL, ok.
Set flags by default:
-Wmissing-prototypes
-Wmissing-declarations
-Wstrict-prototypes
-Wwrite-strings
-Wcast-align
-Wbad-function-cast
-Wformat-security
-Wno-format-nonliteral
-Wmissing-format-attribute
-funsigned-char
Fix minor compiler warnings for these new flags on gcc and clang.
A warning log is already emitted if eve-log is enabled in the
configuration but json support is not built so the logger
registration functions can be silent.
As the logging modules are no longer threading modules, rename
them so they don't look like they are being registered as
threading modules.
Also, move the registration to the output.c which will handle
registration of the loggers.
When the string representation of a JSON message grew bigger than
64k, the JSON record would just be truncated. This lead to errors
in the parser(s) of the JSON stream.
This patch changes the buffer logic to grow the buffer on demand.
The cast of data to AlertJsonThread was not correct as the real
type of the void pointer is a OutputJsonCtx. This was working by
luck because they both have a file_ctx as first element.
*** CID 1211009: Bad bit shift operation (BAD_SHIFT)
/src/output-json-http.c: 265 in JsonHttpLogJSON()
259 /* log custom fields if configured */
260 if (http_ctx->fields != 0)
261 {
262 HttpField f;
263 for (f = HTTP_FIELD_ACCEPT; f < HTTP_FIELD_SIZE; f++)
264 {
>>> CID 1211009: Bad bit shift operation (BAD_SHIFT)
>>> In expression "1 << f", left shifting by more than 31 bits has undefined behavior. The shift amount, "f", is as much as 46.
265 if ((http_ctx->fields & (1<<f)) != 0)
266 {
267 /* prevent logging a field twice if extended logging is
268 enabled */
269 if (((http_ctx->flags & LOG_HTTP_EXTENDED) == 0) ||
270 ((http_ctx->flags & LOG_HTTP_EXTENDED) !=
________________________________________________________________________________________________________
*** CID 1211010: Bad bit shift operation (BAD_SHIFT)
/src/output-json-http.c: 492 in OutputHttpLogInitSub()
486 {
487 if ((strcmp(http_fields[f].config_field,
488 field->val) == 0) ||
489 (strcasecmp(http_fields[f].htp_field,
490 field->val) == 0))
491 {
>>> CID 1211010: Bad bit shift operation (BAD_SHIFT)
>>> In expression "1 << f", left shifting by more than 31 bits has undefined behavior. The shift amount, "f", is as much as 46.
492 http_ctx->fields |= (1<<f);
493 break;
494 }
495 }
496 }
497 }
httplog_ctx->fields would not be initialized before setting flags in
it:
Scanbuild:
output-json-http.c:491:46: warning: The left expression of the compound assignment is an uninitialized value. The computed value will also be garbage
http_ctx->fields |= (1<<f);
~~~~~~~~~~~~~~~~ ^
1 warning generated.
Drmemory:
~~27874~~ Error #1: UNINITIALIZED READ: reading register eax
~~27874~~ # 0 JsonHttpLogJSON [/home/buildbot/qa/buildbot/donkey/drmemory/Suricata/src/output-json-http.c:260]
~~27874~~ # 1 JsonHttpLogger [/home/buildbot/qa/buildbot/donkey/drmemory/Suricata/src/output-json-http.c:375]
Just memset the whole structure right after initialition.
The HTTP module of Eve didn't register itself with the app-layer
for HTTP. This meant that if no other HTTP logger was active, the
HTTP logging in Eve wouldn't work.
This patch makes the HTTP Eve module register itself correctly.
Bug #1133.
The TM_FLAG_LOGAPI_TM flag indicates that a module is run by the log
api, not by the 'regular' thread module call functions.
Set flag in all all Log API users' registration code.
Purpose of this flag is in profiling. In profiling output it will be
used to list log api thread modules separately.
In various places SCStrndup was used to 'dup' a bstr string, however
libhtp provides bstr_util_strdup_to_c for this. As this is a cleaner
interface, it's preferred.
This patch adds an event_type key to the generated events. Current
value is one of "dns", "alert, "file", "tls", "http", "drop". It is
then easy to differentiate in log analysis tools the events based on
source inside Suricata.
This patch is synchronizing key name with Common Information Model.
It updates key name following what is proposed in:
http://docs.splunk.com/Documentation/PCI/2.0/DataSource/CommonInformationModelFieldReference
The interest of these modifications is that using the same key name
as other software will provide an easy to correlate and improve
data. For example, geoip setting in logstash can be applied on
all src_ip fields allowing geoip tagging of data.
Philippe Antoine
Victor Julien
Maurizio Abba
Jason Ish
Giuseppe Longo
maxtors
Tom DeCanio
Eric Leblond
Giuseppe Longo
Ken Steele