Data on RST packets is not invalid, but also shouldn't be used
in reassembly.
RFC 1122:
4.2.2.12 RST Segment: RFC-793 Section 3.4
A TCP SHOULD allow a received RST segment to include data.
DISCUSSION
It has been suggested that a RST segment could contain
ASCII text that encoded and explained the cause of the
RST. No standard has yet been established for such
data.
RST data will be presented to the detection engine per packet,
but will not be part of stream reassembly.
Bug: #6244.
Add wrappers for the default allocators to set SC_ENOMEM.
The stream reassembly wrappers can set both SC_ENOMEM (alloc failed)
and SC_ELIMIT (memcap reached).
In 23323a961f ("app-layer: reduce app cleanup and output-tx calls"), flag
was set per packet updating the app-layer state. However this was missing a
common pattern: in IDS mode most updates are done in the opposite direction
of the traffic due to updates getting triggered by ACK's. This meant that
file store processing might not happen for a long time, or at all. Also,
app layer cleanup might not be called, which includes file pruning.
This patch sets per flow set of flags to indicate app layer is (potentially)
updated. It sets this per direction, based on how the parsers were invoked.
If an ACK triggers an app update, the flow is tagged for the opposite
direction and the next packet in that direction triggers output and cleanup.
Fixes: 23323a961f ("app-layer: reduce app cleanup and output-tx calls")
Bug: #6120.
In TCP, large gaps in the data could lead to an extremely poor utilization
of the streaming buffer memory. This was caused by the implementation using
a single continues memory allocation from the "stream offset" to the
current data. If a 100 byte segment was inserted for ISN + 20MiB, we would
allocate 20MiB, even if only 100 bytes were actually used.
This patch addresses the issue by implementing a list of memory regions.
The StreamingBuffer structure holds a static "main" region, which can be
extended in the form of a simple list of regions.
[ main region ] [ gap ] [ aux region ]
[ sbb ] [ sbb ]
On insert, find the correct region and see if the new data fits. If it
doesn't, see if we can expand the current region, or than we need to add
a new region. If expanding the current region means we overlap or get
too close to the next region, we merge them.
On sliding, we free any regions that slide out of window and consolidate
auxilary regions into main where needed.
Bug: #4580.
Tested on Fedora 37 with clang 15.
app-layer.c:1055:27: error: a function declaration without a prototype is deprecated in all versions of C [-Werror,-Wstrict-prototypes]
void AppLayerSetupCounters()
^
void
app-layer.c:1176:29: error: a function declaration without a prototype is deprecated in all versions of C [-Werror,-Wstrict-prototypes]
void AppLayerDeSetupCounters()
^
void
2 errors generated.
Add thread local cache to avoid locking overhead for ssns and segments.
A thread will return segments/ssns to a local cache first, and if that
is full, to a return queue where the actual return to the pool returns
a batch, to amortize locking overhead.
Adds segment and session pool/cache counters to see where how effective
the cache is.
Track packets that updated the app-layer, and for those run
the transaction housekeeping and output-tx logging loops.
Do the same of end of flow packets.
This skips needless iterations over the transaction stores.
This avoids long lasting inactive flows because in the most likely
case the RST did in fact end the connection. However Suricata may
still consider it to be "established".
Adds a framework for setting exception policies. These would be called
when the engine reaches some kind of exception condition, like hitting
a memcap or some traffic processing error.
The policy gives control over what should happen next: drop the packet,
drop the packet and flow, bypass, etc.
Implements the policy for:
stream: If stream session or reassembly memcaps are hit call the
memcap policy on the packet and flow.
flow: Apply policy when memcap is reached and no flow could be
freed up.
defrag: Apply policy when no tracker could be picked up.
app-layer: Apply ppolicy if a parser reaches an error state.
All options default to 'ignore', which means the default behavior
is unchanged.
Adds commandline options: add simulation options for exceptions. These
are only exposed if compiled with `--enable-debug`.
Ticket: #5214.
Ticket: #5215.
Ticket: #5216.
Ticket: #5218.
Ticket: #5194.
This patch optionally adds packet header to the TCP segment
and update the for each segment function by changing the
callback.
This patch is based on the work by Scott Jordan <scottfgjordan@gmail.com>
Move raw detection logic out of main StreamReassembleRawDo() so that
it can be reused for other parts of the engine.
The caller now has to specify a right edge of the data.
Jeff Lucovsky
Victor Julien
Eric Leblond
Philippe Antoine
Scott Jordan
Eric Leblond