The mpm_uricontent_maxlen logic was meant to track the shortest
possible pattern in the MPM of a SGH. So a minlen more than a maxlen.
This patch replaces the complicated tracking logic by a simpler
scheme. When the SGH's are finalize, the minlen is calculated.
It also fixes a small corner case where the calculated "maxlen" could
be wrong. This would require a smaller pattern in a rule to be forced
as fast pattern.
More exceptional cases for protocol detection. In very unbalanced flows,
where just a few bytes are sent toserver and many toclient, proto detect
might not complete in time on the toserver direction. This can lead to
queuing up many segments in the toclient direction.
Another case is that in come cases the stream is flagged as proto detect
done, but the flows proto detect flags are not set. This is now handled
by the ProtoDetectDone() check.
Due to a broken sequence number check, detect could fail to process
smsgs in case of a sequence wrap. This could lead to excessive use
of smsg's but also of segments, since these aren't cleared until the
smsg containing them is.
Use the reassembly fast paths only after protocol detection has completed.
In some corner cases the sending of smaller segments lead to protocol
detection failing.
If the protocol required TOSERVER data first, but the SSN started with
a GAP, then the TOCLIENT side would get stuck in an expensive path:
1. it would run detection on TOCLIENT
2. it would try to force reassembly for TOSERVER
3. it would reset the detected protocol as TOSERVER failed
4. it would not evict any segment
This had 2 consequences:
1. on long running sessions this could lead to using lots of memory
on segments, denying other sessions resources
2. wasted cycles on protocol detection and segment list management
This patch introduces a fix. It checks in the (2) stage above, whether
the opposing stream (that we depend on) it is a NOREASSEMBLY state. If
so, it gives up on this side of the session as well.
In case of protocol detection not yet being complete, the segment
list was walked unconditionally to unset the app layer processed
flag. Optimize this to bail on the first segment that doesn't have
the flag set.
It was not possible to simply specify PKG_CONFIG_PATH to build
with an non bundled libhtp. With this patch we don't need anymore
the htp lib and include configure options.
setup-app-layer.sh sets up an application layer detector and
parser template.
setup-app-layer-logger.sh sets up a JSON application layer
transaction logger for an application parser that has
already been provisioned.
setup-app-layer-detect.sh sets up a keyword for performing
content inspections on buffers created by the application
layer.
Simplify creation of a new detect module by creating a copy of the
detect template. It sets the name and registers it in the build
system. Finally it's registration function is called.
Support TLS in Lua detection scripts.
function init (args)
local needs = {}
needs["tls"] = tostring(true)
return needs
end
function match(args)
version, subject, issuer, fingerprint = TlsGetCertInfo();
if version == nil then
return 0
end
str = string.format("Version %s\nIssuer %s\nSubject %s\nFingerprint %s",
version, issuer, subject, fingerprint)
SCLogInfo(str);
return 1
end
Use simple bool values to track the transaction state in both directions.
A tx is only created in two cases:
1. full request parsed
2. response parsed (request missing)
This is true even for multi-packet TCP requests.
This leads to the following tx completion logic for the request side:
the presence of a tx implies the request is complete
On the response side, we consider the tx complete when we have seen
the response. If the DNS parser thinks the response was lost, we also
flag the response side as complete.
Rollover option is causing issue with TCP streaming code because
packets from the same flow to be treated out of order. As long as
the situation is not fixed in the streaming engine, it is a bad idea
to enable it by default.
Remove most of the CFLAGS updates from configure. Flags are now (mostly)
set in AM_CLFLAGS.
Update all -DBLAH additions to CFLAGS to use AC_DEFINE([BLAH], ...)
Improve Lua vs LuaJIT checking.
Improve the configure output a bit.
Lots of smaller cleanups.
It's not uncommon to see an header like:
X-Forwarded-For: 1.2.3.4:56789
This patch recognizes this case and ignores the port. It also supports
this for IPv6 if the address has the following notation:
X-Forwarded-For: [12::34]:1234
This patch also adds unittests.
Victor Julien
Jason Ish
Eric Leblond
Antti Tönkyrä
gureedo