Add API calls to upgrade to TLS or to request a protocol change
without a specific protocol expectation.
If the HTTP CONNECT session includes a port on the url, use that to
look up the probing parser during protocol detection. Solves a
missed detection of a SSLv2 session that upgrades to TLSv1. SSLv2
relies on the probing parser which is limited to certain ports.
In case of STARTTLS in SMTP and FTP, the port is hardcoded to 443.
A new event APPLAYER_UNEXPECTED_PROTOCOL is set if there was a
mismatch.
The Flow::data_al_so_far was used for tracking data already
parsed when protocol for the current direction wasn't known yet. As
this behaviour has changed the tracking can be removed.
Now that the flow lookup is done in the worker threads the flow
queue handlers running after the capture thread(s) no longer have
access to the flow. This limits the options of how flow balancing
can be done.
This patch removes all code that is now useless. The only 2 methods
that still make sense are 'hash' and 'ippair'.
Store the tenant id in the flow and use the stored id when setting
up pesudo packets.
For tunnel and defrag packets, get tenant from parent. This will only
pass tenant_id's set at capture time.
For defrag packets, the tenant selector based on vlan id will still
work as the vlan id(s) are stored in the defrag tracker before being
passed on.
Use separate data structures for storing TX and FLOW (AMATCH) detect
state.
- move state storing into util funcs
- remove de_state_m
- simplify reset state logic on reload
A flow has 3 states: NEW, ESTABLISHED and CLOSED.
For all protocols except TCP, a flow is in state NEW as long as just one
side of the conversation has been seen. When both sides have been
observed the state is moved to ESTABLISHED.
TCP has a different logic, controlled by the stream engine. Here the TCP
state is leading.
Until now, when parts of the engine needed to know the flow state, it
would invoke a per protocol callback 'GetProtoState'. For TCP this would
return the state based on the TcpSession.
This patch changes this logic. It introduces an atomic variable in the
flow 'flow_state'. It defaults to NEW and is set to ESTABLISHED for non-
TCP protocols when we've seen both sides of the conversation.
For TCP, the state is updated from the TCP engine directly.
The goal is to allow for access to the state without holding the Flow's
main mutex lock. This will later allow the Flow Manager(s) to evaluate
the Flow w/o interupting it.
The flow end flags field is filled by the flow manager or the flow
hash (in case of forced timeout of a flow) to record the timeout
conditions in the flow:
- emergency mode
- state
- reason (timed out or forced)
Add logging to the flow logger.
app-layer.[ch], app-layer-detect-proto.[ch] and app-layer-parser.[ch].
Things addressed in this commit:
- Brings out a proper separation between protocol detection phase and the
parser phase.
- The dns app layer now is registered such that we don't use "dnstcp" and
"dnsudp" in the rules. A user who previously wrote a rule like this -
"alert dnstcp....." or
"alert dnsudp....."
would now have to use,
alert dns (ipproto:tcp;) or
alert udp (app-layer-protocol:dns;) or
alert ip (ipproto:udp; app-layer-protocol:dns;)
The same rules extend to other another such protocol, dcerpc.
- The app layer parser api now takes in the ipproto while registering
callbacks.
- The app inspection/detection engine also takes an ipproto.
- All app layer parser functions now take direction as STREAM_TOSERVER or
STREAM_TOCLIENT, as opposed to 0 or 1, which was taken by some of the
functions.
- FlowInitialize() and FlowRecycle() now resets proto to 0. This is
needed by unittests, which would try to clean the flow, and that would
call the api, AppLayerParserCleanupParserState(), which would try to
clean the app state, but the app layer now needs an ipproto to figure
out which api to internally call to clean the state, and if the ipproto
is 0, it would return without trying to clean the state.
- A lot of unittests are now updated where if they are using a flow and
they need to use the app layer, we would set a flow ipproto.
- The "app-layer" section in the yaml conf has also been updated as well.
By moving FlowReference() out of FlowGetFlowFromHash() and into the one
function that calls it, all the flow functions take const Packet * instead
of Packet *.
This patch is updating the flow tag system to use the flow
storage API. The tag_list member of Flow structure is suppressed
and its cleaning operation are suppressed too as this is handled
transparently by the flow storage API.
Improved accuracy, improved performance. Performance improvement
noticeable with http heavy traffic and ruleset.
A lot of other cosmetic changes carried out as well. Wrappers introduced
for a lot of app layer functions.
Failing dce unittests disabled. Will be reintroduced in the updated dce
engine.
Cross transaction matching taken care of. FPs emanating from these
matches have now disappeared. Double inspection of transactions taken
care of as well.
clang was issuing some warnings related to unused return in function.
This patch adds some needed error treatment and ignore the rest of the
warnings by adding a cast to void.
Major redesign of the flow engine. Remove the flow queues that turned
out to be major choke points when using many threads. Flow manager now
walks the hash table directly. Simplify the way we get a new flow in
case of emergency.
Removal of per flow 'aldata' array. It contained a ptr for each ALPROTO. Instead now we have 2 ptrs in the flow: alparser and alstate.
Various cleanups and dead code removal from the app layer API.
Should safe 100+ bytes memory per flow on 64 bit.
Updated lots of unittests to reflect these changes.
Victor Julien
Anoop Saldanha
Ken Steele
Eric Leblond
Anoop Saldanha
Pablo Rincon