Remove unused TM_FLAG_STREAM_TM.
Rename TM_FLAG_DETECT_TM to TM_FLAG_FLOWWORKER_TM as it was mostly used
to check if a thread is a flow worker. TM_FLAG_DETECT_TM was always set
for a flow worker, even when there was no detection in use.
The flow worker needs to get the opportunity to run the flow update
before globally making it's current timestamp available. This is to
avoid another thread using the time to evict the flow that is about to
get a legitimate update.
Ticket: #7455.
The initdata argument to OutputFlowThreadInit was always NULL, remove
it. Internally the ThreadInit functions still get initdata, but this
is the data provided when that logging instance was registered.
Ticket: #7227
Add new flags to trigger FLOW_TS_APP_UPDATED/FLOW_TC_APP_UPDATED flags
to be set for the next packet in the relevant direction.
This allows for app relevant work to be done in the next packet in our
direction.
Ticket: 6578
When a protocol such as SSH sets no_inspection, we still have to
flush the current streams and packets that contain clear-text
for detection.
Ticket: #6305
When running FlowWorkerStreamTCPUpdate, one of the dequeued packet
may set the flow action to drop, without updating the not-pseudo
packet action, as is done usually with a previous call to
FlowHandlePacketUpdate
In the FlowFinish logic, one or two pseudo packets are used to finish flow
handling. In the case of 2 (one per direction), the logic first set up the
2 packets, then it would process them one by one. This lead to poor cache
locality.
This patch processes the first packet entirely first, followed by the second
packet.
In 23323a961f ("app-layer: reduce app cleanup and output-tx calls"), flag
was set per packet updating the app-layer state. However this was missing a
common pattern: in IDS mode most updates are done in the opposite direction
of the traffic due to updates getting triggered by ACK's. This meant that
file store processing might not happen for a long time, or at all. Also,
app layer cleanup might not be called, which includes file pruning.
This patch sets per flow set of flags to indicate app layer is (potentially)
updated. It sets this per direction, based on how the parsers were invoked.
If an ACK triggers an app update, the flow is tagged for the opposite
direction and the next packet in that direction triggers output and cleanup.
Fixes: 23323a961f ("app-layer: reduce app cleanup and output-tx calls")
Bug: #6120.
Pseudo packets originating in the flow worker do not need to leave the
flow worker. Putting those in the ThreadVars::decode_pq will make them
be evaluated by the next steps in the pipeline, but those will all
ignore pseudo packets.
Instead, this patch returns them to the packet pool, while still honoring
the IPS verdict logic.
This patch adds a new unix command that allows the user to trigger
a dump of the ruleset profile to the file without having to stop
Suricata.
This will be really useful to debug performance issue related to
signatures in production environment.
Flow house keeping can accumulate work that wasn't taken into account
during shutdown. This could lead to flows still in the flowworker
thread context when being it was freed, leading to missed work and
memory leaks.
This patch adds a new way of checking if a thread module is still
busy.
Bug: #6062.
Until this patch the logic the flow worker flow house keeping used was:
- at most 2 flows are handled per packet
- pseudo packets could flush the entire queue
This patch changes that. Pseudo packets are fairly common, and can lead
to packet stalls / latency spikes if the number of flows in the queue
is large.
It does that by adding a new packet type only used at shutdown, which
flushes out the queues completely. All other packets will now stick
to the 2 flow rate limit.
During housekeeping multiple flows are processed. If a rule reload happens
at that time, we need to use the new detect thread as soon as possible.
Bug: #5969.
Packets only ever reference the flow while holding its lock. This
means than any code possibly evicting the flow will have to wait
for the existing users to complete their work. Therefore the use_cnt
serves no function anymore and can be removed.
Issue: 5718
This commit switches the majority of time handling to a new type --
SCTime_t -- which is a 64 bit container for time:
- 44 bits -- seconds
- 20 bits -- useconds
Worker threads are responsible for final processing of timed out flows.
These are selected by the Flow Manager and inserted into a per thread
queue. The Flow Worker then checks this queue after each packet. Due to
the burstiness of this process, the packet threads would sometimes process
a lot of these flows in the context of a single packet, leading to spike
in latency which might cause packet loss.
This patch changes the behavior to only process at max 2 flows per packet.
This way added processing cost is amortized over many packets.
Track packets that updated the app-layer, and for those run
the transaction housekeeping and output-tx logging loops.
Do the same of end of flow packets.
This skips needless iterations over the transaction stores.
Update APIs to store files in transactions instead of the per flow state.
Goal is to avoid the overhead of matching up files and transactions in
cases where there are many of both.
Update all protocol implementations to support this.
Update file logging logic to account for having files in transactions. Instead
of it acting separately on file containers, it is now tied into the
transaction logging.
Update the filestore keyword to consider a match if filestore output not
enabled.
Victor Julien
Philippe Antoine
Jeff Lucovsky
Jason Ish
Shivani Bhardwaj
Eric Leblond