Fix spurious retransmissions getting dropped, stalling connections in IPS
mode.
There are several reasons why benign spurious retransmissions can happen,
with the most obvious one that an ACK is lost so the sender retransmits
while the receiver has ACK'd it. If Suricata sees the ACK but afterwards
it gets lost, we can get in this condition. Packet loss can have a wide
range of causes here, including packets reaching a host but getting
dropped in the NIC queue or kernel queues due to resource constraints.
So these packets are no longer an "error" in this patch.
Next to this, the accuracy of the spurious retransmission has been
improved. Use SEQ macros to compare sequence numbers. Only use base_seq
if reassembly is still enabled for a stream.
A special case is added for cases where a segment is before last_ack
but after base_seq, which can happen when protocol detection isn't
finished yet. In this case the segment is tagged as spurious, but still
processed. This way we can check for overlaps.
Bug: #5875.
Linux is slightly more permissive wrt timestamps than many
other OS'. To avoid many events/issues with linux hosts, add an
option to allow for this slightly more permissive behavior.
Ideally the host-os config would be used, but in practice this
setting is rarely set up correctly, if at all.
This option is enabled by default.
On every accepted packet in established state, update next_seq if
packet seq+len is larger than existing next_seq. This allows it to
catch up after large gaps that are filled again a bit later.
Bug: #5877.
An ACK that ACK'd older data while still being in-window could
lead to FIN_WAIT1 to FIN_WAIT2 state transition. Detect this
case and generally harden the check.
Bug: #5877.
Support case where there are multiple SYN retransmits, where
each has a new timestamp.
Before this patch, Suricata would only accept a SYN/ACK that
matches the last timestamp. However, observed behavior is that
the server may choose to only respond to the first. In IPS mode
this could lead to a connection timing out as Suricata drops
the SYN/ACK it considers wrong, and the server continues to
retransmit it.
This patch reuses the SYN/ACK queuing logic to keep a list
of SYN packets and their window, timestamp, wscale and sackok
settings. Then when the SYN/ACK arrives, it is first evaluated
against the normal session state. But if it fails due to a
timestamp mismatch, it will look for queued SYN's and see if
any of them match the timestamp. If one does, the ssn is updated
to use that SYN and the SYN/ACK is accepted.
Bug: #5856.
When a "stream starter" packet finds an existing TCP flow, the flow will be
evaluated for reuse.
The following scenario wasn't handled well:
1. Suricata starts after a tool has just stopped using lots of connections
(e.g. ab stress testing a webserver)
2. even though the client is closed already, the server is still doing
connection cleanup sending many FINs and later RSTs
3. Suricata creates flows for these packets, but no TCP sessions
4. client resumes testing, creating flows that have the same 5 tuple as the
flows created for the FIN/RST packets
5. Suricata refuses to "reuse" the flows as the condition "tcp flow w/o session"
is not considered valid for session reuse
6. new TCP connection is not properly tracked and evaluated in parsing and
detection
There may be other vectors into this, like a flow w/o session because of
memcap issues.
Bug: #5843.
Packets only ever reference the flow while holding its lock. This
means than any code possibly evicting the flow will have to wait
for the existing users to complete their work. Therefore the use_cnt
serves no function anymore and can be removed.
Set the engine to ignore the stream.midstream-policy if stream.midstream
is enabled.
If we had both stream.midstream AND stream.midstream_policy enabled,
this could lead to midstream flows being dropped (or bypassed, or...)
instead of being accepted by the engine, as it was probably meant when
the user enabled midstream flows.
Bug #5765
This allows all traffic Exception Policies to be set from one
configuration point. All exception policy options are available in IPS
mode. Bypass, pass and auto (disabled) are also available in iDS mode
Exception Policies set up individually will overwrite this setup for the
given traffic exception.
Task #5219
Issue: 5718
This commit switches the majority of time handling to a new type --
SCTime_t -- which is a 64 bit container for time:
- 44 bits -- seconds
- 20 bits -- useconds
Add thread local cache to avoid locking overhead for ssns and segments.
A thread will return segments/ssns to a local cache first, and if that
is full, to a return queue where the actual return to the pool returns
a batch, to amortize locking overhead.
Adds segment and session pool/cache counters to see where how effective
the cache is.
This allows to set a midstream-policy that can:
- fail closed (stream.midstream-policy=drop-flow)
- fail open (stream.midstream-policy=pass-flow)
- bypass stream (stream.midstream-policy=bypass)
- do nothing (default behavior)
Usage and behavior:
If stream.midstream-policy is set then if Suricata identifies a midstream flow
it will apply the corresponding action associated with the policy.
No setting means Suricata will not apply such policies, either inspecting the
flow (if stream.midstream=true) or ignoring it stream.midstream=false.
Task #5468
Victor Julien
Philippe Antoine
Juliana Fajardini
Jeff Lucovsky