Fixed counter for ippair.memcap by using the correct function,
IPPairGetMemcap.
Until now IPPairGetMemuse was used both for memuse and memcap.
Ticket: 7827
The mDNS support is based heavily on the DNS support, reusing the
existing DNS parser where possible. This meant adding variations on
DNS, as mDNS is a little different. Mainly being that *all* mDNS
traffic is to_server, yet there is still the concept of request and
responses.
Keywords added are:
- mdns.queries.rrname
- mdns.answers.rrname
- mdns.additionals.rrname
- mdns.authorities.rrname
- mdns.response.rrname
They are mostly in-line with the DNS keywords, except
mdns.answers.rdata which is a better than that mdns.response.rrname,
as its actually looking at the rdata, and not rrnames.
mDNS has its own logger that differs from the DNS logger:
- No grouped logging
- In answers/additionals/authorities, the rdata is logged in a field
that is named after the rdata type. For example, "txt" data is no
longer logged in the "rdata" field, but instead a "txt" field. We
currently already did this in DNS for fields that were not a single
buffer, like SOA, SRV, etc. So this makes things more consistent. And
gives query like semantics that the "grouped" object was trying to
provide.
- Types are logged in lower case ("txt" instead of "TXT")
- Flags are logged as an array: "flags": ["aa", "z"]
Ticket: #3952
Ticket: 7552
When we use midstream, and the first packet we see of a flow is
a response from server, and we want to match on some signature
to client :
- we had first set sgh_toserver/FLOW_SGH_TOSERVER as we first
thought this was a packet to server
- we then swap/reverse the flow, so sgh_toclient becomes sgh_toserver
but it contains signatures to server and cannot match our
to_client signature
The detect engine with DetectRunSetup will set again the
signatures group heads properly
If alproto for the current direction was not detected but the opposite
side was successfully detected, if the Pattern Matching and Pattern
Probing on the flow was also successfully done and the current
direction's alproto is still unknown, a decoder event is set to indicate
that the protocol detection only happened in one direction.
This event is set after having sent the current data to the applayer
parser. Now, the respective applayer parser may or may not successfully
parse the data. However, the alproto on flow is already set from the
other direction so there will be a flow event generated by Suricata. In
order to keep this consistent with the stats, also make sure to
increment the flow counter when the decode event is set so that the flow
counter is incremented irrespective of the parsing status reported by
the applayer parser.
This patch makes stats for several specific applayer flow count equal to
the number of flow events logged for those specific applayer protocols.
Bug 7238
Ticket: 5053
The names are now dynamically registered at runtime.
The AppProto alproto enum identifiers are still static for now.
This is the final step before app-layer plugins.
TCP urgent handling is a complex topic due to conflicting RFCs and
implementations.
Until now the URG flag and urgent pointer values were simply ignored,
leading to an effective "inline" processing of urgent data. Many
implementations however, do not default to this behavior.
Many actual implementations use the urgent mechanism to send 1 byte of
data out of band to the application.
Complicating the matter is that the way the urgent logic is handled is
generally configurable both of the OS and the app level. So from the
network it is impossible to know with confidence what the settings are.
This patch adds the following policies:
`stream.reassembly.urgent.policy`:
- drop: drop URG packets before they affect the stream engine
- inline: ignore the urgent pointer and process all data inline
- oob (out of band): treat the last byte as out of band
- gap: skip the last byte, but do no adjust sequence offsets, leading to
gaps in the data
For the `oob` option, tracking of a sequence number offset is required,
as the OOB data does "consume" sequence number space. This is limited to
64k. For this reason, there is a second policy:
`stream.reassembly.urgent.oob-limit-policy`:
- drop: drop URG packets before they affect the stream engine
- inline: ignore the urgent pointer and process all data inline
- gap: skip the last byte, but do no adjust sequence offsets, leading to
gaps in the data
Bug: #7411.
In 23323a961f ("app-layer: reduce app cleanup and output-tx calls"), flag
was set per packet updating the app-layer state. However this was missing a
common pattern: in IDS mode most updates are done in the opposite direction
of the traffic due to updates getting triggered by ACK's. This meant that
file store processing might not happen for a long time, or at all. Also,
app layer cleanup might not be called, which includes file pruning.
This patch sets per flow set of flags to indicate app layer is (potentially)
updated. It sets this per direction, based on how the parsers were invoked.
If an ACK triggers an app update, the flow is tagged for the opposite
direction and the next packet in that direction triggers output and cleanup.
Fixes: 23323a961f ("app-layer: reduce app cleanup and output-tx calls")
Bug: #6120.
Tested on Fedora 37 with clang 15.
app-layer.c:1055:27: error: a function declaration without a prototype is deprecated in all versions of C [-Werror,-Wstrict-prototypes]
void AppLayerSetupCounters()
^
void
app-layer.c:1176:29: error: a function declaration without a prototype is deprecated in all versions of C [-Werror,-Wstrict-prototypes]
void AppLayerDeSetupCounters()
^
void
2 errors generated.
In the case of a gap, or in the case of a flow where one side
is recognized, but the other is not before the end, we still
increase the counters to have consistency between
jq 'select(.event_type=="flow" and .app_proto=="ftp") | .app_proto' log/eve.json | wc -l
jq 'select(.event_type=="stats") | .stats."app_layer".flow.ftp' log/eve.json
Ticket: #5769
Track packets that updated the app-layer, and for those run
the transaction housekeeping and output-tx logging loops.
Do the same of end of flow packets.
This skips needless iterations over the transaction stores.
Work towards making `suricata-common.h` only introduce system headers
and other things that are independent of complex internal Suricata
data structures.
Update files to compile after this.
Remove special DPDK handling for strlcpy and strlcat, as this caused
many compilation failures w/o including DPDK headers for all files.
Remove packet macros from decode.h and move them into their own file,
turn them into functions and rename them to match our function naming
policy.
In the case of midstream SYN/ACK pickup, we reverse the flow based on
the SYN/ACK. If we then later get traffic that appears to be in the
reverse direction based on the app-layer, we would reverse it again.
This isn't correct. When we have the SYN/ACK we know the flow's real
direction.
Adds a framework for setting exception policies. These would be called
when the engine reaches some kind of exception condition, like hitting
a memcap or some traffic processing error.
The policy gives control over what should happen next: drop the packet,
drop the packet and flow, bypass, etc.
Implements the policy for:
stream: If stream session or reassembly memcaps are hit call the
memcap policy on the packet and flow.
flow: Apply policy when memcap is reached and no flow could be
freed up.
defrag: Apply policy when no tracker could be picked up.
app-layer: Apply ppolicy if a parser reaches an error state.
All options default to 'ignore', which means the default behavior
is unchanged.
Adds commandline options: add simulation options for exceptions. These
are only exposed if compiled with `--enable-debug`.
Ticket: #5214.
Ticket: #5215.
Ticket: #5216.
Ticket: #5218.
Ticket: #5194.
The idea of stream frames is that the applayer parsers can tag PDUs and
other arbitrary frames in the stream while parsing. These frames can then
be inspected from the rule language. This will allow rules that are more
precise and less costly.
The frames are stored per direction in the `AppLayerParserState` and will only
be initialized when actual frames are in use. The per direction storage has a
fixed size static portion and dynamic support for a larger number. This is done
for effeciency.
When the Stream Buffer slides, frames are updated as they use offsets relative
to the stream. A negative offset is used for frames that started before the
current window.
Frames have events to inspect/log parser errors that don't fit the TX model.
Frame id starts at 1. So implementations can keep track of frame ids where 0
is not set.
Frames affect TCP window sliding. The frames keep a "left edge" which
signifies how much data to keep for frames that are still in progress.
Ticket: #4562
As the data which triggered the opposing side
was the same protocol and not another one,
that means the protocol change failed.
Prevents a memory leak in later call of AppLayerParserParse
which would allocate a new state and leak the old one
Adam Kiripolsky
Philippe Antoine
Jason Ish
Juliana Fajardini
Victor Julien
Shivani Bhardwaj
Jeff Lucovsky
Philippe Antoine
Victor Julien