When a Tracker is set up for a IPID, frags come in for it and it's
reassembled and complete, the `DefragTracker::remove` flag is set. This
is mean to tell the hash cleanup code to recyle the tracker and to let
the lookup code skip the tracker during lookup.
A logic error lead to the following scenario:
1. there are sufficient frag trackers to make sure the hash table is
filled with trackers
2. frags for a Packet with IPID X are processed correctly (X1)
3. frags for a new Packet that also has IPID X come in quickly after the
first (X2).
4. during the lookup, the frag for X2 hashes to a hash row that holds
more than one tracker
5. as the trackers in hash row are evaluated, it finds the tracker for
X1, but since the `remove` bit is not checked, it is returned as the
tracker for X2.
6. reassembly fails, as the tracker is already complete
The logic error is that only for the first tracker in a row the `remove`
bit was checked, leading to reuse to a closed tracker if there were more
trackers in the hash row.
Ticket: #7042.
This commit modifies the logic used to determine the disposition of a
flow/packet.
If the rule doesn't require a stream and only contains properties for
packet matching, then the alert is not marked as applying to the
flow and hence, the flow won't be dropped.
Issue: 5578
Ticket: 6889
To avoid regexp dos with too much backtracking.
This is already done on pcre keyword, and pcrexform transform.
We use the same default limits for rules parsing.
(cherry picked from commit 316cc528f7)
Using pcre1 in master6
The list of fragments may still contain overlaps, so adding up the
fragment lengths is flawed. Instead track the largest size of
contiguous data that can be re-assembled.
Bug: #6675
(cherry picked from commit d226d0a3fc)
Fix the BSD policy case where a subsequent fragment starts before an
original fragment and overlaps the beginning of the original
fragment. In this case the overlapping data from the new fragment is
preferred.
Suricata was preferring the data from the original fragment, but it
should only do that when the original fragment has an offset <= to the
new fragment.
- Adds tests for this case
Bug: #6669
(cherry picked from commit f1709ea551)
Instead of breaking the loop when the current fragment does not have
any more fragments, set a flag and continue to the next fragment as
the next fragment may have data that occurs before this fragment, but
overlaps it.
Then break if the next fragment does not overlap the previous.
Bug: #6668
(cherry picked from commit d0fd078250)
Eve's packet_info.linktype should correctly indicated what the `packet`
field contains. Until now it was using DLT_RAW even if Ethernet or other
L2+ headers were present.
This commit records the datalink of the packet creating the first
fragment, which can include the L2+ header data.
Bug: #6887.
(cherry picked from commit 49c67b2bb1)
Ticket: 6878
Follow up on 15649424a7
When adding many sequence nodes, either from start or scalar event
We add "sequence nodes" whose name is an integer cf sequence_node_name
and then run ConfNodeLookupChild to see if it had been already set
(from the command line cf comment in the code)
And ConfNodeLookupChild iterates the whole linked list...
1. We add node 1
2. To add node 2, we check if node 1 equals this new node
3. To add node 3, we check if nodes 1, or 2 equals this new node's name
And so on...
This commits avoids these checks ig the list is empty at the beginning
(cherry picked from commit 240e068b81)
Ticket: #6305
When running FlowWorkerStreamTCPUpdate, one of the dequeued packet
may set the flow action to drop, without updating the not-pseudo
packet action, as is done usually with a previous call to
FlowHandlePacketUpdate
(cherry picked from commit 4c4f7ff1a2)
Fix memory leak at util-decode-mime:MimeDecInitParser, which
root cause is not-freeing allocated memory for mimeMsg
Bug: #6745
(cherry picked from commit 231c892bef)
A dead lock could occur at start up, where a loader thread would
get stuck on it's condition variable, while the main thread was
polling the loaders task results.
The vector to the dead lock is as follows:
main loader
DetectEngineMultiTenantSetup
-DetectLoaderSetupLoadTenant
--DetectLoaderQueueTask
---lock loader
---add task
---unlock loader
lock loader
check/exec tasks
unlock loader
---wake up threads
lock ctrl mutx
cond wait ctrl
unlock ctrl
-DetectLoadersSync
--lock loader
--check tasks
--unlock loader
Between the main thread unlocking the loader and waking up the
threads, it is possible that the loader has already moved ahead
but not yet entered its conditional wait. The main thread sends
its condition signal, but since the loader isn't yet waiting on
it the signal is ignored. Then when the loader does enter its
conditional wait, the signal is not sent again.
This patch updates the logic to send signals much more often.
It also makes sure that the signal is sent under lock, as the
API requires.
Bug: #6767.
Co-authored-by: Shivani Bhardwaj <shivani@oisf.net>
Ticket: 6477
So as to avoid ending up with too many empty transactions.
This happens when Suricata sees a DATA command in the current
transaction but did not have a confirmation response for it.
Then, if Suricata receives another DATA command, it will
create another new transaction, even if the previous one
is empty. And so, a malicious client can create many empty
transactions by just sending a repeated amount of DATA commands
without having a confirmation code for them.
Suricata cannot use state->current_command == SMTP_COMMAND_DATA
to prevent this attack and needs to resort to a new boolean
is_data because the malicious client may send another dummy command
after each DATA command.
This patch leaves only one call to SMTPTransactionCreate
(cherry picked from commit 61f2e4e1e5)
Unlike the original commit, this fix just sets a limit but does not
expose it as a configurable option.
Ticket: #6477
(cherry picked from commit 8f73a0ac55)
Unlike the original commit, this commit just enforces a maximum limit,
but does not expose the configuration of it.
Ticket: #5921
(cherry picked from commit 4175680a8a)
When reloading rules, respect `--set default-rule-path=...` from the
command line if set.
Previously the rule reload would always take the default-rule-path from
the configuration file, even if overrided on the command line.
Issue: #1911
(cherry picked from commit 3ea6572e22)
So that we can have multi-tenant on MacOS without stack
overflows because of the size of the structure...
Ticket: #6263.
(cherry picked from commit 92fce2fdc0)
Issue: 6239
This commit moves the global variables associated with engine analysis
into the detect engine context. Doing so provides encapsulation of the
analysis components as well as thread-safe operation in a multi-tenant
(context) deployment.
(cherry picked from commit c8615bcd47)
If the configuration field "formats" is empty, DNS response records do
not have any relevant information other than that there was a
response, but not much about the response.
I'm pretty sure the intention here was to log the response details if
no formats were provided, which is what happens when the field is
commented out.
So if no formats are specified, use the default of all.
Bug: #6420
(cherry picked from commit a240a93b69)
Issue: 4623
This commit addresses the issues reported in issue 4623 when the jump
value points at the last byte in the buffer.
(cherry picked from commit f363b99fd7)
email_ctx->fields only gets populated when smtp.custom setting is on.
The fn EveEmailLogJSONCustom is called when either
1. smtp.extended setting is on or,
2. email_ctx->fields is populated which means smtp.custom setting is on
In case neither of these are set in suricata.yaml, no call should
ideally be made to the fn EveEmailLogJSONCustom.
However, it turns out that email_ctx->fields is unset and then set only
after the smtp config was found. This leads to email_ctx->fields
sometimes contain value even when no config was given to the smtp
section and can lead to unexpected output.
Fix this by using SCCalloc while initializing OutputJsonEmailCtx struct
instead of SCMalloc.
Bug 6380
(cherry picked from commit 7406ac0fa5)
When the decoders encounter multiple layers of tunneling, multiple tunnel
packets are created. These are then stored in ThreadVars::decode_pq, where
they are processed after the current thread "slot" is done. However, due
to a logic error, the tunnel packets after the first, where not called
for the correct position in the packet pipeline. This would lead to these
packets not going through the FlowWorker module, so skipping everything
from flow tracking, detection and logging.
This would only happen for single and workers, due to how the pipelines
are constructed.
The "slot" holding the decoder, would contain 2 packets in
ThreadVars::decode_pq. Then it would call the pipeline on the first
packet with the next slot of the pipeline through a indirect call to
TmThreadsSlotVarRun(), so it would be called for the FlowWorker.
However when that first (the most inner) packet was done, the call
to TmThreadsSlotVarRun() would again service the ThreadVars::decode_pq
and process it, again moving the slot pointer forward, so past the
FlowWorker.
This patch addresses the issue by making sure only a "decode" thread
slot will service the ThreadVars::decode_pq, thus never moving the
slot past the FlowWorker.
Bug: #6402.
(cherry picked from commit 15947f2173)
Pseudo packets originating in the flow worker do not need to leave the
flow worker. Putting those in the ThreadVars::decode_pq will make them
be evaluated by the next steps in the pipeline, but those will all
ignore pseudo packets.
Instead, this patch returns them to the packet pool, while still honoring
the IPS verdict logic.
(cherry picked from commit 3247e39f0c)
Issue: 4624
Handle negative post_offset values that jump before the buffer as though
they refer to the buffer start.
(cherry picked from commit 2bf9d0fdf9)
Ticket: #6279
If we have the smtp body beginning without headers, we need to
create the md5 context and right away and supply data to it.
Otherwise, on the next line being processed, md5_ctx will be
NULL but body_begin will have been reset to 0
(cherry picked from commit c6afee64d5)
Adapted to use hash functions from master6
A bad pattern in a rule that hyperscan would fail to compile would
exit Suricata. This could happen during a rule reload as well.
In case of a untrusted ruleset, this could potentially be used to
shut down the sensor.
Commit 7d0851b0c2 already blocks the only know case, but this patch
is more defensive.
Ticket: #6195.
(cherry picked from commit 00e00254ea)
Make sure thread ctx registration happens and id remains correct
in case of reloads.
To do so, move id var into the detect ctx.
(cherry picked from commit 2cac440f7d)
Implement a new design for handling var name id's. The old logic
was aware of detection engine versions and generally didn't work
well for multi-tenancy cases. Other than memory leaks and crashes,
logging of var names worked or failed based on which tenant was
loaded last.
This patch implements a new approach, where there is a global store
of vars and their id's for the lifetime of the program.
Overall Design:
Base Store: "base"
Used during keyword registration. Operates under lock. Base is shared
between all detect engines, detect engine versions and tenants.
Each variable name is ref counted.
During the freeing of a detect engine / tenant, unregistration decreases
the ref cnt.
Base has both a string to id and a id to string hash table. String to
id is used during parsing/registration. id to string during unregistration.
Active Store Pointer (atomic)
The "active" store atomic pointer points to the active lookup store. The call
to `VarNameStoreActivate` will build a new lookup store and hot swap
the pointer.
Ensuring memory safety. During the hot swap, the pointer is replaced, so
any new call to the lookup functions will automatically use the new store.
This leaves the case of any lookup happening concurrently with the pointer
swap. For this case we add the old store to a free list. It gets a timestamp
before which it cannot be freed.
Free List
The free list contains old stores that are waiting to get removed. They
contain a timestamp that is checked before they are freed.
Bug: #6044.
Bug: #6201.
(cherry picked from commit b130234b26)
Victor Julien
Jeff Lucovsky
Philippe Antoine
Jason Ish
Alexey Simakov
Ralph Eastwood
Philippe Antoine
jason taylor
Shivani Bhardwaj