Commit Graph

15923 Commits (41b9836b11bbd653953f5c5dc5f87875e15fae8d)
 

Author SHA1 Message Date
Victor Julien 41b9836b11 threads: give threads more time to get ready
In certain conditions, it can take a long time for threads to start up.
For example in af-packet, setting up the socket, rings, etc has been
observed to take close to half a second per thread, and since the
threads go one by one in a preset order, this means the start up can
take a lot of time if there are many threads. The old logic would just
allow a hard coded 60s. This was not always enough when the number of
threads was high.

This patch makes the wait time take the number of threads into account.
It adds a second of time budget to the base 60s for each thread.

So as an example, if a system has 112 af-packet threads, it would wait
172 seconds (60 + 112) for the threads to get ready.

Ticket: #7048.
1 year ago
Victor Julien 85fd4b2ec7 threads: optimize start up check
When starting a large amount of threads, the loop was inefficient. It
would loop over the threads and if one wasn't yet ready it would sleep a
bit and then reevaluate all the threads. This reevaluation of threads
already checked was inefficient, and could lead to the time budget
running out.

This patch splits the check, and keeps track of the threads that have
already passed. This avoids the rescanning of already checked threads.
1 year ago
Victor Julien 121955d5c1 tls-store: support client logging
Adds a `client-` prefix to the logged certs and meta files.

Ticket: #7045.
1 year ago
Victor Julien 6fe5b739b4 tlsstore: remove stale FIXME 1 year ago
Victor Julien 032bc04a1e detect/tls.store: fix direction check
STREAM_* flags are invalid for `Flow::flags`.

Fixes: dfcb429524 ("detect/cert: Use client side certs")
1 year ago
Jason Ish daa6f6f7f3 github-ci: re-add --disable-lua to commit check
This is required for some older versions in the pull request to build
as the commits change some compile time options with respect to Lua.
1 year ago
Jason Ish 3eb8c728fd doc: update lua sandbox docs for allowed packages/functions 1 year ago
Jason Ish 10e6028175 lua: track memory limit exceede errors
Update the Lua allocated to set a code on memory allocation limit
exceeded errors so an appropriate error message can be logged and a
state incremented.

Fixes the tracking of the allocated size by using the difference
between original size, and new size and toss in some debug
validations.
1 year ago
Jason Ish 011f0ba994 lua: remove sandbox lib for now
Not sure if I see a use for it, some extra debug logging might be just
as useful for those writing Lua scripts.
1 year ago
Jason Ish 5a1cba72f0 lua: add logging and counter for instruction limit being exceeded 1 year ago
Jason Ish c8fa454cb2 lua: add blocked functions as a special log type plus stat
Distinguish between a generic Lua script error and an error created by a
function being blocked, so each is logged once respective of each other.

Also add a stat that is incremented when a script fails due to a
blocked function.

NOTE: This does not catch calls to functions that are blocked by not
having the library loaded, such as "io.open", as they are blocked by
not even loading the "io" library.
1 year ago
Jason Ish 86f9e43068 lua: use a function allow list instead of a deny list
The Lua library surface area is small enough to manage an allow list,
which is generally better than a deny list, as we'll explicitly need
to opt-in to new functions provided by the Lua runtime.
1 year ago
Jason Ish 936930778c rust/Makefile: cleanup "clean" targets
Remove maintainer-clean-local, this is not needed.

In distclean-local, remove "rust/dist" and "rust/vendor" as they are
created during "make dist".

In "clean-local", remove "rust/target" and "rust/gen" as they are
created during a normal "make".
1 year ago
Jason Ish 7897043144 github-ci/scan-build: exclude rust (lua)
The vendored Lua code triggers some scan-build failures, so exclude
the rust/ directory for now. Might want to look at these separately
though.
1 year ago
Jason Ish 4788d684da github-ci: test make after clean without cbindgen
Modify the CentOS 9 Stream build to not have cbdingen available, as
its already building from the dist.  But add a "make clean" followed
by a "make" to test that it still builds after a clean.
1 year ago
Jason Ish 2e440169d6 lua: remove lua as a compile time feature
Its always built-in. However, can be disabled at runtime.
1 year ago
Jason Ish 1fd2c1a379 rust/lua: remove lua_int8 feature
Now that we're fixed to Lua 5.4, the integer size is always 8.
1 year ago
Jason Ish bc011f2205 lua: use rust crate to vendor (bundle) lua
Remove lua-dev(el) from all CI tests.
1 year ago
Jason Ish afb705d278 lua: reset instruction counter before calling script 1 year ago
Jason Ish 1f05a17fb9 lua: misc cleanups in sandbox implementation
Including:
- rename guards
- SCMalloc to SCCalloc
- remove unused enum
- rename public functions to our naming standard
1 year ago
Jo Johnson ba6a976e06 doc: Initial doc for lua sandbox 1 year ago
Jo Johnson 04adb0c0f6 lua: Add config to allow sandbox bypass 1 year ago
Jo Johnson e946b20e0f lua: Add config override for lua sandbox limits 1 year ago
Jo Johnson 8428b0b9d7 lua: Add lua sandbox for detection rules 1 year ago
Jo Johnson 01c8af766c lua: remove internal references to luajit 1 year ago
Jo Johnson d5c6c3a21c lua: build lua by default
Ticket: #4776

[Edits by Jason Ish]
- Add Lua in CI where needed
- Disable Lua for builds that don't have Lua 5.4
1 year ago
Jo Johnson 712496bb3f lua: Remove luajit support
lua 5.4 support is not available in luajit

Ticket: #4776
1 year ago
Jo Johnson 586c92d9d5 lua: require lua 5.4
github-ci: Disable lua on debian 10 as it doesn't have Lua 5.4.

Ticket: #4776
1 year ago
Victor Julien e0411878fc github-actions: bump scan-build to Ubuntu 24.04 / clang 18 1 year ago
Jason Ish 6e2a1ec5d6 misc: move prototypes to correct header
Move prototypes for functions that exist in util-port-interval-tree.c
from detect-engine-port.h to util-port-interval-tree.h.

Fix header guard names while there.
1 year ago
jason taylor 47d6c3a3ab doc: add source verification docs
Ticket: #6908

Signed-off-by: jason taylor <jtfas90@gmail.com>
1 year ago
Philippe Antoine b91e7fe2ae detect/http-server-body: clean up tests
Ticket: 4083
1 year ago
Philippe Antoine ce16a56a1f detect: unify functions for multi-buffer
Ticket: 6575

Multi buffers keywords now use a single registration function
DetectAppLayerMultiRegister with a GetBuffer argument.

This GetBuffer function pointer is similar to the ones used by
single-buffer keyword, except that it takes an additional
parameter which is the index of the buffer to get.
Under the hood, an anonymous union between these 2 functions
pointers types is used.

In the end, this deduplicates code, especially the calls to
DetectEngineContentInspection
1 year ago
Philippe Antoine 55bc5f2290 detect/template: make template use DetectEngineInspectBufferGeneric 1 year ago
Philippe Antoine ce9bfba76a ci: fix and test with Wunused-macros
Ticket: 6937
1 year ago
Philippe Antoine b3eb1c4f81 clean: remove unused struct definitions
Found with
git grep "typedef struct" src/ | awk '{print $3}' | sort |
uniq | sed 's/_$//' | while read i; do
echo -n $i; git grep $i | wc -l; done | awk '$2 < 3'

Ticket: 4083
1 year ago
Richard McConnell fc2e49f84a app-layer: Set sc_errno upon error return
Bug: https://redmine.openinfosecfoundation.org/issues/6782

Callers to these allocators often use ``sc_errno`` to provide context of
the error. And in the case of the above bug, they return ``sc_errno``,
but as it has not been set ``sc_errno = 0; == SC_OK``.

This patch simply sets this variable to ensure there is context provided
upon error.
1 year ago
Victor Julien 52a008e358 detect/http-host: clean up tests 1 year ago
Victor Julien 6c937a9243 pcap-log: use correct pkthdr size for limit enforcement
The on-disk pcap pkthdr is 16 bytes. This was calculated using
`sizeof(struct pcap_pkthdr)`, which is 24 bytes on 64 bit Linux. On
Macos, it's even worse, as a comment field grows the struct to 280
bytes.

Address this by hardcoding the value of 16.

Bug: #7037.
1 year ago
Victor Julien 9b980b18a8 pcap-log: minor cleanups 1 year ago
Victor Julien ea8c283dc7 pcap-log: minor cleanup
Use same pointer to one location consistently.
1 year ago
Victor Julien 8c4b96129f pcap-log: always pass 'comp' to PcapWrite
The variable is always available.
1 year ago
Victor Julien 5aa00aae0a pcap-log: rename connp to comp
Matches other variable names for the compression settings.
1 year ago
Victor Julien 3eb74c9992 pcap-log: don't check variable that is never set
`rotate` was never modified.
1 year ago
Victor Julien 5455799795 time: only consider packet threads
In offline mode, a timestamp is kept per thread, and the lowest
timestamp of the active threads is used. This was also considering the
non-packet threads, which could lead to the used timestamp being further
behind that needed. This would happen at the start of the program, as
the non-packet threads were set up the same way as the packet threads.

This patch both no longer sets up the timestamp for non-packet threads
as well as not considering non-packet threads during timestamp
retrieval.

Fixes: 6f560144c1 ("time: improve offline time handling")

Bug: #7034.
1 year ago
Shivani Bhardwaj 0aaec69303 flow: use debug validate macro 1 year ago
Shivani Bhardwaj d6b63b38ca flow: add defensive check on memuse 1 year ago
Shivani Bhardwaj 7144b9421d tcp: use bool wherever possible 1 year ago
Shivani Bhardwaj f4b8f706fa flow: use bool wherever possible 1 year ago
Shivani Bhardwaj 2c751dba73 flow: remove unneeded else 1 year ago