Change SCFlowAppLayerProto to return 5 values:
<alproto> <alproto_ts> <alproto_tc> <alproto_orig> <alproto_expect>:
alproto: detected protocol
alproto_ts: detected protocol in toserver direction
alproto_tc: detected protocol in toclient direction
alproto_orig: pre-change/upgrade protocol
alproto_expected: expected protocol in change/upgrade
Orig and expect are used when changing and upgrading protocols. In a
SMTP STARTTLS case, orig would normally be set to "smtp" and expect
to "tls".
When switching protocol from http to tls the following corner case
was observed:
pkt 6, TC "200 connection established"
pkt 7, TS acks pkt 6 + adds "client hello"
pkt 8 TC, acks pkt 7
pkt 8 is where normally the detect on the 200 connection established
would run however before detection runs the app-layer is called
and it resets the state
So the issue is missed detection on the last data in the original
protocol before the switch.
Another case was:
TS -> STARTTLS
TC -> Ack "STARTTLS data"
220
TS -> Ack "220 data"
Client Hello
In IDS mode, this made a rule that wanted to look at content:"STARTTLS"
in combination with the protocol SMTP 'alert smtp ... content:"STARTTLS";'
impossible. By the time the content would match, the protocol was already
switched.
This patch fixes this case by creating a 'Detect/Log Flush' packet in
both directions. This will force final inspection and logging of the
pre-upgrade protocol (SMTP in this example) before doing the final
switch.
Add API calls to upgrade to TLS or to request a protocol change
without a specific protocol expectation.
If the HTTP CONNECT session includes a port on the url, use that to
look up the probing parser during protocol detection. Solves a
missed detection of a SSLv2 session that upgrades to TLSv1. SSLv2
relies on the probing parser which is limited to certain ports.
In case of STARTTLS in SMTP and FTP, the port is hardcoded to 443.
A new event APPLAYER_UNEXPECTED_PROTOCOL is set if there was a
mismatch.
Support changing the application level protocol for a flow. This is
needed by STARTTLS and HTTP CONNECT to switch from the original
alproto to tls.
This commit allows a flag to be set 'FLOW_CHANGE_PROTO', which
triggers a new protocol detection on the next packet for a flow.
Set flags by default:
-Wmissing-prototypes
-Wmissing-declarations
-Wstrict-prototypes
-Wwrite-strings
-Wcast-align
-Wbad-function-cast
-Wformat-security
-Wno-format-nonliteral
-Wmissing-format-attribute
-funsigned-char
Fix minor compiler warnings for these new flags on gcc and clang.
In preparation of turning input to keyword parsers to const add
options to the common rule parser to enforce and strip double
quotes and parse negation support.
At registration, the keyword can register 3 extra flags:
SIGMATCH_QUOTES_MANDATORY: value to keyword must be quoted
SIGMATCH_QUOTES_OPTIONAL: value to keyword may be quoted
SIGMATCH_HANDLE_NEGATION: leading ! is parsed
In all cases leading spaces are removed. If the 'quote' flags are
set, the quotes are removed from the input as well.
The expression 'isdataat:!1,relative' is used to make sure a match
is at the end of a buffer quite often. This patch optimizes this case
for 'content' followed by the expression. It enforces it by setting
and 'ends with' flag on the content and then taking that flag into
account while doing the pattern match.
Content inspection optimization: when just distance is used without
within we don't need to search recursively.
E.g. content:"a"; content:"b"; distance:1; will scan the buffer for
'a' and when it finds 'a' it will scan the remainder for 'b'. Until
now, the failure to find 'b' would lead to looking for the next 'a'
and then for 'b' after that. However, we already inspected the
entire buffer for 'b', so we know this will fail.
Now that MPM runs when the TX progress is right, stateful detection
operates differently.
Changes:
1. raw stream inspection is now also an inspect engine
Since this engine doesn't take the transactions into account, it
could potentially run multiple times on the same data. To avoid
this, basic result caching is in place.
2. the engines are sorted by progress, but the 'MPM' engine is first
even if the progress is higher
If MPM flags a rule to be inspected, the inspect engine for that
buffer runs first. If this step fails, the rule is no longer
evaluated. No state is stored.
Previously the MPM/Prefilter engines would suggest the same rule
candidates multiple times.
For example, while processing the request body, the http headers
would be inspected by MPM multiple times.
The mask check was one way to quickly decide which rules could be
skipped.
Now that the MPM engines normally return a rule just once, this
mask check no longer makes sense. If the rule meets the ip/port/
direction based conditions, it needs to be evaluated if the MPM
said so. Even if not all conditions are yet true.
WIP disable mask as it no longer makes sense
WIP redo mask match
In various scenarios buffers would be checked my MPM more than
once. This was because the buffers would be inspected for a
certain progress value or higher.
For example, for each packet in a file upload, the engine would
not just rerun the 'http client body' MPM on the new data, it
would also rerun the method, uri, headers, cookie, etc MPMs.
This was obviously inefficent, so this patch changes the logic.
The patch only runs the MPM engines when the progress is exactly
the intended progress. If the progress is beyond the desired
value, it is run once. A tracker is added to the app layer API,
where the completed MPMs are tracked.
Implemented for HTTP, TLS and SSH.
Victor Julien
Jason Ish
Mats Klepsland