Commit Graph

12555 Commits (76131c8cff1c5b895b49f2fecdf37764551cc7bb)
 

Author SHA1 Message Date
Victor Julien ab8f289bb6 flow/worker: run housekeeping for bypassed packets
Run flow eviction and flow inject queues for bypassed packets as well,
to avoid a scenario where these won't get run at all if too much of the
traffic is bypassed.

Bug: #4779.
3 years ago
Victor Julien 41fee41722 flow/manager: remove obsolete code 3 years ago
Victor Julien ec7e0561e8 flow/bypass: use_cnt desync'd on bypassed flows
Locally bypassed flows had unsafe updates to `Flow::use_cnt` leading to a race
issue. For a packet it would do the flow lookup, attach the flow to the packet,
increment the `use_cnt`. Then it would detect that the flow is in the bypass
state, and unlock it while holding a reference (so alos not decrementing the
`use_cnt`). When the packet was then returned to the packet pool, the flow would
be disconnected from the packet, which would decrement `use_cnt` without holding
the flow lock.

This patch addresses this issue by disconnecting the flow from the packet
immediately when the bypassed state is detected. This moves the `use_cnt`
decrement to within the lock.

Bug: #4766.
3 years ago
Philippe Antoine 416575ea02 pcrexform: use substring and not whole match 3 years ago
Philippe Antoine c9d664b0a0 tftp: StringToAppProto case
So, fuzz_applayerparserparse_tftp will fuzz tftp
3 years ago
Philippe Antoine 5bd065cb3c range: checks that end is after start for HTTP2
As was done only for HTTP1 in previous commit

The verification part stays separated from the parsing part,
as we want to keep on logging invalid ranges values.
3 years ago
Jason Ish 07370ed5c0 queue.h: suppress scan-build warnings
If running under scan-build, use our own implementations of all
the macros which include some code to satisfy scan-build
warnings.
3 years ago
Philippe Antoine 77604d86d6 range: move back files ownership in one case
In the case, we receive a range request with expected
overlap then new bytes, but the response does not get to the
new bytes, we are still skipping, but the HttpRangeContainerBlock
had the ownership of the files, and need to give it back
3 years ago
Philippe Antoine bba70607e8 range: checks that end is after start
Otherwise, we end up allocating too much memory
3 years ago
Philippe Antoine 27b4f165b1 loopback: decodes IPv6 from all OSes
As does wireshark
3 years ago
Philippe Antoine accdad7881 ike: do not keep server transforms in state
Fixes #4534

Now, only the tx with the transforms will match
with ike.chosen_sa_attribute
3 years ago
Philippe Antoine 83887510a8 modbus: tx iterator
When there are a lot of open transactions, as is possible with
modbus, the default tx_iterator will loop for the whole
transacations vector to find each transaction, that means
quadratic complexity.

Reusing the tx_iterator from the template, and keeping as a state
the last index where to start looking avoids this quadratic
complexity.
3 years ago
Philippe Antoine b34c025b52 util: avoid calling snprintf in PrintStringsToBuffer
As we print only one character
3 years ago
Philippe Antoine 53ef65d390 http2: enable by default, even if not in config 3 years ago
Philippe Antoine 424dcda2c0 http2: enable by default 3 years ago
Philippe Antoine fa4c7626bd http2: null check during upgrade 3 years ago
Philippe Antoine ea4a509a54 app-layer: disable by default if not in configuration
DNP3, ENIP, HTTP2 and Modbus are supposed to be disabled
by default. That means the default configuration does it,
but that also means that, if they are not in suricata.yaml,
the protocol should stay disabled.
3 years ago
Jason Ish 75bc9d9dd8 queue.h: wrap the system sys/queue.h
Instead of using local implementations for the queue.h macro,
wrap the system provided queue.h and then adding missing
features as needed.

The idea is that Suricata when integrated with another library
that includes sys/queue.h can look at the same source of truth
for these macros.

But not all operating systems include a queue.h with the same
features, and some don't include it at all, like Windows. So
on Windows this will be a full implementation of all the queue.h
features Suricata needs.
3 years ago
Philippe Antoine 6fadb97d5d alert: fixes leak in ThresholdHandlePacketRule
ThresholdHandlePacketRule may take ownership of an allocated
DetectThresholdEntry, and places it in a position of the
array th_entry. But it never got released
3 years ago
Philippe Antoine d21a252238 fuzz: target must use the rules it parsed
DetectEngineReloadThreads does not work for the fuzz targets
as there is no_of_detect_tvs = 0 as we did not register
real threads and slots.

So, we force the flow worker module to use the newly detect engine
conetxt with all it needs
3 years ago
Jason Ish 8b9721b265 github-ci: pin macos build to 10.15
There is currently a build failure with macos-latest (recently updated)
to 11 in the libhtp test suite code. Not sure if there are other
build issues in libhtp or Suricata at this time.
3 years ago
Jason Ish d18fc4f3f0 github-ci: use sccache for gcc in commits workflow
Previously was only used for Rust.
3 years ago
Victor Julien fa72a5add8 flow: free spare pool more aggressively
The flows exceeding the spare pools config setting would be freed
per at max 100 flows a second. After a high speed test this would
lead to excessive memory use for a long time.

This patch updates the logic to free 10% of the excess flows per
run, freeing multiple blocks of flows as needed.

Bug: #4731.
3 years ago
Victor Julien ff97d7c15d threading: force break loop on flow inject
Track availability of break loop callback to avoid overhead.
3 years ago
Victor Julien b788d3345c flow: process evicted flows on low/no traffic
In a scenario where there was suddenly no more traffic flowing, flows
in a threads `flow_queue` would not be processed. The easiest way to
see this would be in a traffic replay scenario. After the replay is done
no more packets come in and these evicted flows got stuck.

In workers mode, the capture part handles timeout this was updated to
take the `ThreadVars::flow_queue` into account.

The autofp mode the logic that puts a flow into a threads `flow_queue`
would already wake a thread up, but the `flow_queue` was then ignored.
This has been updated to take the `flow_queue` into account.

In both cases a "capture timeout" packet is pushed through the pipeline
to "flush" the queues.

Bug: #4722.
3 years ago
Victor Julien 31977170a8 threading: minor cleanups 3 years ago
Jeff Lucovsky 314ec77f88 unittests/template: Register template unittests 3 years ago
Jeff Lucovsky 6e149cdec3 unittests/enip: Register ENIP unittests 3 years ago
Philippe Antoine 8a50edbd10 pcre: fixes a memory leak on alloc error 3 years ago
Philippe Antoine 8536048443 http2: do not try to upgrade if http2 is disabled in config 3 years ago
Philippe Antoine 42ba421ca9 http2: flatten code style 3 years ago
Philippe Antoine 527415dba0 protodetect: handle all gaps, even when depth is reached 3 years ago
Jason Ish 6e3e8530a1 readthedocs: add configuration file 3 years ago
Philippe Antoine 586522e1e9 pcre: local match data for pcrexform 3 years ago
Philippe Antoine c64a1f6a09 pcre: use thread-storage for matches 3 years ago
Philippe Antoine 3b690e53c8 pcre: using de_ctx in unit tests for free function 3 years ago
Philippe Antoine a049a6c29c pcre: creates a match structure per match run
So that DetectPcrePayloadMatch is thread safe
and does not rewrite a shared parse_regex.match structure
3 years ago
Philippe Antoine 78cf9cfc5f http: range: remove assert that can happen 3 years ago
Philippe Antoine 8e8899c90c http2: range: check return value when opening
HttpRangeContainerOpenFile can return NULL
so, http2_range_open can set file_range to NULL
And we should check this before calling http2_range_close
3 years ago
Philippe Antoine 65a6f61004 http: delete obsolete range log
Commit d776d72711
has been transfering ownership of file container

So, we cannot log it
3 years ago
Philippe Antoine 9b3c355c20 fuzz: adds one target with predefined rules 3 years ago
Philippe Antoine c06c4a663e lgtm: adds build instructions to get lgtm to work
LGTM is a statis analysis tool
3 years ago
Jason Ish df0ed6fda4 af-packet: use configured cluster-id when checking for fanout
When testing for fanout support a cluster-id of 1 was always being
used instead of the configured cluster-id. This limited fanout
support to only one Suricata instance.

Instead of hardcoding an ID of 1, use the configured cluster-id.

Also make cluster_id a uint16_t instead of an int in AFPThreadVars.

Redmine issue:
https://redmine.openinfosecfoundation.org/issues/3419
3 years ago
Philippe Antoine 3a230c2208 ipv6: simpler generic overlap condition
This also changes the behavior, as the condition is checked in
every case cf ipv6-malformed-fragments-8
3 years ago
Juliana Fajardini fc958e9e89 userguide: update wiresharkwiki in public datasets 3 years ago
Juliana Fajardini dbeb8bfa1f doc/devguide: add few more explanations & details 3 years ago
Juliana Fajardini 2cd25e8105 devguide/app-layer: rename /img dir to /diagrams
Semantically speaking it makes more sense, because it stores `msc`
files for dynamic image generation.
Updated files that refered to `img` accordingly, too.
3 years ago
Juliana Fajardini f65b3908ed devguide/transactions: add TSL_STATE enum snippet 3 years ago
Juliana Fajardini d6c5dfacc7 devguide/transactions: update & refine diagrams
- DNS sequence diagram was incorrect (transactions should be
unidirectional). After changing it, it made sense to rename the file.
Adjusted spacing, too. Updated transactions.rst accordingly.
- TLS sequence diagram was refined to illustrate how Suricata actually
implements the protocol.
3 years ago
Juliana Fajardini 84311ab151 devguide/transactions: fix wordings 3 years ago