This commit modifies the JSON loggers with changes necessary to support
multi-threaded EVE output.
Each "thread-init" function sets up the per-thread log file context for
subsequent calls to the JSON output to buffer function.
Add a keyword configuration dump-all-headers, with allowed values
{both, request, response}, dumping all HTTP headers in the eve-log http
object. Each header is a single object in the list request_headers
(response_headers) with the following notation:
{
"name": <header name>,
"value": <header value>
}
To avoid forged malicious headers, the header name size is capped at 256
bytes, the header value size at 2048.
By default, dump-all-headers is disabled.
Substitute json_string with SCJsonString custom function.
SCJsonString will ensure string passed is json-encodable (utf-8).
If it's not, the string will be converted in such a way that any
non-printable character will be encoded in its hex form.
The resulting json object will be returned.
rust modification will encode any non-printable character during its
conversion in to_cstring.
XFF configuration is already set in app-layer-htp-xff, and in
output-json-alert. Extending XFF configuration to files and HTTP allow
to get the same behavior as for alerts.
Extend the configuration of filestore json to let filestore metafile
dump be aware of xff. This is available only if write-fileinfo is set
to yes and file-store version is 2.
The new OutputInitResult is a struct return type that allows
logger init functions to return a NULL context without
raising error.
Instead of returning NULL to signal error, the "ok" field will
be set to false. If ok, but the ctx is NULL, then silently
move on to the next logger.
Use case: multiple versions of a specific logger, and one
implementation decides the configuration is not for that
implemenation. It can return NULL, ok.
Set flags by default:
-Wmissing-prototypes
-Wmissing-declarations
-Wstrict-prototypes
-Wwrite-strings
-Wcast-align
-Wbad-function-cast
-Wformat-security
-Wno-format-nonliteral
-Wmissing-format-attribute
-funsigned-char
Fix minor compiler warnings for these new flags on gcc and clang.
A warning log is already emitted if eve-log is enabled in the
configuration but json support is not built so the logger
registration functions can be silent.
As the logging modules are no longer threading modules, rename
them so they don't look like they are being registered as
threading modules.
Also, move the registration to the output.c which will handle
registration of the loggers.
When the string representation of a JSON message grew bigger than
64k, the JSON record would just be truncated. This lead to errors
in the parser(s) of the JSON stream.
This patch changes the buffer logic to grow the buffer on demand.
The cast of data to AlertJsonThread was not correct as the real
type of the void pointer is a OutputJsonCtx. This was working by
luck because they both have a file_ctx as first element.
*** CID 1211009: Bad bit shift operation (BAD_SHIFT)
/src/output-json-http.c: 265 in JsonHttpLogJSON()
259 /* log custom fields if configured */
260 if (http_ctx->fields != 0)
261 {
262 HttpField f;
263 for (f = HTTP_FIELD_ACCEPT; f < HTTP_FIELD_SIZE; f++)
264 {
>>> CID 1211009: Bad bit shift operation (BAD_SHIFT)
>>> In expression "1 << f", left shifting by more than 31 bits has undefined behavior. The shift amount, "f", is as much as 46.
265 if ((http_ctx->fields & (1<<f)) != 0)
266 {
267 /* prevent logging a field twice if extended logging is
268 enabled */
269 if (((http_ctx->flags & LOG_HTTP_EXTENDED) == 0) ||
270 ((http_ctx->flags & LOG_HTTP_EXTENDED) !=
________________________________________________________________________________________________________
*** CID 1211010: Bad bit shift operation (BAD_SHIFT)
/src/output-json-http.c: 492 in OutputHttpLogInitSub()
486 {
487 if ((strcmp(http_fields[f].config_field,
488 field->val) == 0) ||
489 (strcasecmp(http_fields[f].htp_field,
490 field->val) == 0))
491 {
>>> CID 1211010: Bad bit shift operation (BAD_SHIFT)
>>> In expression "1 << f", left shifting by more than 31 bits has undefined behavior. The shift amount, "f", is as much as 46.
492 http_ctx->fields |= (1<<f);
493 break;
494 }
495 }
496 }
497 }
httplog_ctx->fields would not be initialized before setting flags in
it:
Scanbuild:
output-json-http.c:491:46: warning: The left expression of the compound assignment is an uninitialized value. The computed value will also be garbage
http_ctx->fields |= (1<<f);
~~~~~~~~~~~~~~~~ ^
1 warning generated.
Drmemory:
~~27874~~ Error #1: UNINITIALIZED READ: reading register eax
~~27874~~ # 0 JsonHttpLogJSON [/home/buildbot/qa/buildbot/donkey/drmemory/Suricata/src/output-json-http.c:260]
~~27874~~ # 1 JsonHttpLogger [/home/buildbot/qa/buildbot/donkey/drmemory/Suricata/src/output-json-http.c:375]
Just memset the whole structure right after initialition.
Jeff Lucovsky
Jason Ish
Philippe Antoine
Victor Julien
Maurizio Abba
Jason Ish
Giuseppe Longo
maxtors
Tom DeCanio
Eric Leblond
Giuseppe Longo
Ken Steele