This patch implements reconnection handling for the redis output.
A reconnect limitation has been implemented with a limitation of
one connection per second.
This patch implements redis pipelining. This consist in contacting
the redis server every N events to minimize the number of TCP
exchange. This is optional and setup via the configuration file.
Introduce a function to realize the parsing and config file and
opening of connection to the database. Only used by output-json
for now it will be usable by other logging modules.
Introduce a function LogFileWrite that will handle the writing with
respect of the type defined in the configuration. This is used in
this patch to remove the write complexity from output-json.
When using redis output, we are loosing the host key (added by
logstash or logstash-forwarder) and we can't find anymore what
Suricata did cause the alert.
This patch is adding this key during message generation using the
'sensor-name' variable or the hostname is 'sensor-name' is not
defined.
This adds a counter indicating how many times
the flow max memcap has been reached
Since there is no always a reference to FlowManagerThreadData,
the counter is put in DecodeThreadVars.
Currently when there is no counter increase in one call of FlowGetNew
because we don't have tv or dtv at the time of the call.
The following is a snippet of the generated EVE entry:
"flow":{"memcap":0,"spare":10000,"emerg_mode_entered":0,"emerg_mode_over":0,"tcp_reuse":0,"memuse":7085248}
The code was not correct and coverity did detect a potential
overflow problem that should not happen because of the structure
of md5 string and of format.
Unlike other app-layer protocols which are enabled by default,
disable this one by default as it likely shouldn't be enabled
in production use of Suricata.
Two structure fields were uninitialized and used has a bit field.
Weird behavior were seen in list of logged fields due to that with
some build options.
Multiple events can be applied on a transaction so we may need to
log the same header twice.
The HDR_IS_LOGGED flag was making it impossible. And this system
is usless as email application layer is transaction based.
Add a boolean option named "payload" to the unified2-alert output type.
Such options makes suricata omit the payload in the resulting unified2
file. The default value is true in order to preserve the current behaviour.
- Some tests always passed due to the result value being
initialized to 1.
- Fix tests that now fail. Looks like just the test were wrong
and that the code does the right thing.
Remove the old tracker reset macro which is no longer being used.
Clear last_seen and remove flags on initialization.
Remove extra call to DefragTrackerInit as it was being called 2x
for each new tracker.
Now that DefragTrackerNew is just a wrapper for DefragTrackerAlloc,
remove it and just call DefragTrackerAlloc directly.
These trackers are likely for completed fragments, but have
not been cleaned up. If a packet on the same flow with an
already seen IP ID is seen, it could be reused prior to
being properly reinitialized.
Previous code was wrongly supposing the lines are ending with CRLF.
But this is not the case so we must propagate the length of end of
line delimiter to be able to compute the md5 of the mail body.
To be able to correlate between events, it is better to have the
tx_id information in the root object. This function adds a new
function to automate the addition of the field.
To be able to identify mails with identical subjects without
using the subject itself as a key, it is possible to use the md5
hash of the subjet string. This allows to limit the privacy impact.
Some mail clients are using tabulation and/or space for comma
separated list. This patch removes them so the event will contain
only significative characters.
This patch adds a way to specify which MIME fields to log via
the custom keyword in the EVE configuration. it also adds an
extended logging where some fields are added. The logging support
mono value fields as well as multivalue fields via the use of
JSON array.
This patch introduces a new function that can be used to handle
multivalued MIME fields. A callback function can be called for
each corresponding field value.
If the status is not PARSE_DONE then in that case we may have
imcomplete information. Increasing the stream reassemly depth
in that case would be a good idea.
The body_md5 has been added and contain the value of the md5sum
of the body.
This patch is using the state PARSE_DONE on the MIME state to
detect when a message has been completely parsed.
This patch is computing the md5 sum of the body of the MIME message.
This will allow to detect messages with same content and sent to
different people.
This patch changes the way smtp message are written. It is using
the "email" key to store the email related fields. This will
allow to do the same search through SMTP and IMAP if we implement
this last one.
The mpm_uricontent_maxlen logic was meant to track the shortest
possible pattern in the MPM of a SGH. So a minlen more than a maxlen.
This patch replaces the complicated tracking logic by a simpler
scheme. When the SGH's are finalize, the minlen is calculated.
It also fixes a small corner case where the calculated "maxlen" could
be wrong. This would require a smaller pattern in a rule to be forced
as fast pattern.
More exceptional cases for protocol detection. In very unbalanced flows,
where just a few bytes are sent toserver and many toclient, proto detect
might not complete in time on the toserver direction. This can lead to
queuing up many segments in the toclient direction.
Another case is that in come cases the stream is flagged as proto detect
done, but the flows proto detect flags are not set. This is now handled
by the ProtoDetectDone() check.
Due to a broken sequence number check, detect could fail to process
smsgs in case of a sequence wrap. This could lead to excessive use
of smsg's but also of segments, since these aren't cleared until the
smsg containing them is.
Use the reassembly fast paths only after protocol detection has completed.
In some corner cases the sending of smaller segments lead to protocol
detection failing.
If the protocol required TOSERVER data first, but the SSN started with
a GAP, then the TOCLIENT side would get stuck in an expensive path:
1. it would run detection on TOCLIENT
2. it would try to force reassembly for TOSERVER
3. it would reset the detected protocol as TOSERVER failed
4. it would not evict any segment
This had 2 consequences:
1. on long running sessions this could lead to using lots of memory
on segments, denying other sessions resources
2. wasted cycles on protocol detection and segment list management
This patch introduces a fix. It checks in the (2) stage above, whether
the opposing stream (that we depend on) it is a NOREASSEMBLY state. If
so, it gives up on this side of the session as well.
In case of protocol detection not yet being complete, the segment
list was walked unconditionally to unset the app layer processed
flag. Optimize this to bail on the first segment that doesn't have
the flag set.
Support TLS in Lua detection scripts.
function init (args)
local needs = {}
needs["tls"] = tostring(true)
return needs
end
function match(args)
version, subject, issuer, fingerprint = TlsGetCertInfo();
if version == nil then
return 0
end
str = string.format("Version %s\nIssuer %s\nSubject %s\nFingerprint %s",
version, issuer, subject, fingerprint)
SCLogInfo(str);
return 1
end
Use simple bool values to track the transaction state in both directions.
A tx is only created in two cases:
1. full request parsed
2. response parsed (request missing)
This is true even for multi-packet TCP requests.
This leads to the following tx completion logic for the request side:
the presence of a tx implies the request is complete
On the response side, we consider the tx complete when we have seen
the response. If the DNS parser thinks the response was lost, we also
flag the response side as complete.
Remove most of the CFLAGS updates from configure. Flags are now (mostly)
set in AM_CLFLAGS.
Update all -DBLAH additions to CFLAGS to use AC_DEFINE([BLAH], ...)
Improve Lua vs LuaJIT checking.
Improve the configure output a bit.
Lots of smaller cleanups.
It's not uncommon to see an header like:
X-Forwarded-For: 1.2.3.4:56789
This patch recognizes this case and ignores the port. It also supports
this for IPv6 if the address has the following notation:
X-Forwarded-For: [12::34]:1234
This patch also adds unittests.
Move the tenant load and reload commands to be executed by the detect
loader thread(s).
Limitation: no yaml parsing in parallel. The Conf API is currently not
thread safe, so don't load the tenant config (yaml) in parallel.
To speed up startup with many tenants, tenant loading will be parallelized.
As no tempary threads should be used for these memory allocation heavy
tasks, this patch adds new type of 'command' thread that can be used to
load and reload tenants.
This patch hardcodes the number of loaders to 4. Future work will make it
dynamic.
The loader thread essentially sleeps constantly. When a tasks is sent to
it, it will wake up and execute it.
Store the tenant id in the flow and use the stored id when setting
up pesudo packets.
For tunnel and defrag packets, get tenant from parent. This will only
pass tenant_id's set at capture time.
For defrag packets, the tenant selector based on vlan id will still
work as the vlan id(s) are stored in the defrag tracker before being
passed on.
Make threshold loading prefix aware, so it can be part of tenant
configuration.
If the setting is missing from the tenant, the global setting is tried
and if that too is missing, the global default is used.
Note: currently per host thresholds are tracked globally and NOT per
tenant.
Make reference loading prefix aware, so it can be part of tenant
configuration.
If the setting is missing from the tenant, the global setting is tried
and if that too is missing, the global default is used.
Make classification loading prefix aware, so it can be part of tenant
configuration.
If the setting is missing from the tenant, the global setting is tried
and if that too is missing, the global default is used.
Allow for a tenant to be reloaded. The command is the same as the
register-tenant command, so with a yaml and tenant-id as argument.
However this replaces an existing tenant.
Eric Leblond
Giuseppe Longo
cardigliano
Jason Ish
Torgeir Natvig
Mats Klepsland
Alessandro Guido
Victor Julien
Alexander Gozman
Antti Tönkyrä
gureedo