If the status is not PARSE_DONE then in that case we may have
imcomplete information. Increasing the stream reassemly depth
in that case would be a good idea.
The body_md5 has been added and contain the value of the md5sum
of the body.
This patch is using the state PARSE_DONE on the MIME state to
detect when a message has been completely parsed.
This patch is computing the md5 sum of the body of the MIME message.
This will allow to detect messages with same content and sent to
different people.
This patch changes the way smtp message are written. It is using
the "email" key to store the email related fields. This will
allow to do the same search through SMTP and IMAP if we implement
this last one.
The mpm_uricontent_maxlen logic was meant to track the shortest
possible pattern in the MPM of a SGH. So a minlen more than a maxlen.
This patch replaces the complicated tracking logic by a simpler
scheme. When the SGH's are finalize, the minlen is calculated.
It also fixes a small corner case where the calculated "maxlen" could
be wrong. This would require a smaller pattern in a rule to be forced
as fast pattern.
More exceptional cases for protocol detection. In very unbalanced flows,
where just a few bytes are sent toserver and many toclient, proto detect
might not complete in time on the toserver direction. This can lead to
queuing up many segments in the toclient direction.
Another case is that in come cases the stream is flagged as proto detect
done, but the flows proto detect flags are not set. This is now handled
by the ProtoDetectDone() check.
Due to a broken sequence number check, detect could fail to process
smsgs in case of a sequence wrap. This could lead to excessive use
of smsg's but also of segments, since these aren't cleared until the
smsg containing them is.
Use the reassembly fast paths only after protocol detection has completed.
In some corner cases the sending of smaller segments lead to protocol
detection failing.
If the protocol required TOSERVER data first, but the SSN started with
a GAP, then the TOCLIENT side would get stuck in an expensive path:
1. it would run detection on TOCLIENT
2. it would try to force reassembly for TOSERVER
3. it would reset the detected protocol as TOSERVER failed
4. it would not evict any segment
This had 2 consequences:
1. on long running sessions this could lead to using lots of memory
on segments, denying other sessions resources
2. wasted cycles on protocol detection and segment list management
This patch introduces a fix. It checks in the (2) stage above, whether
the opposing stream (that we depend on) it is a NOREASSEMBLY state. If
so, it gives up on this side of the session as well.
In case of protocol detection not yet being complete, the segment
list was walked unconditionally to unset the app layer processed
flag. Optimize this to bail on the first segment that doesn't have
the flag set.
It was not possible to simply specify PKG_CONFIG_PATH to build
with an non bundled libhtp. With this patch we don't need anymore
the htp lib and include configure options.
setup-app-layer.sh sets up an application layer detector and
parser template.
setup-app-layer-logger.sh sets up a JSON application layer
transaction logger for an application parser that has
already been provisioned.
setup-app-layer-detect.sh sets up a keyword for performing
content inspections on buffers created by the application
layer.