Commit Graph

11600 Commits (c95850c6ce334af9174ef6efeeb19347e2804b1b)
 

Author SHA1 Message Date
Philippe Antoine 61c327dd80 signature: checks for integer overflow in limits propagation 5 years ago
Victor Julien 5db1d9b841 eve/yaml: move mqtt down 5 years ago
Victor Julien 1c748f394b fuzz/pcap: enable MQTT parser 5 years ago
Shivani Bhardwaj e9fe5ada7f datasets: reload static sets 5 years ago
Sascha Steinbiss c31360070b rust/mqtt: add MQTT parser 5 years ago
Philippe Antoine a5572890a9 detect: adds engine for u8 keywords 5 years ago
Jeff Lucovsky a5d30a3220 doc/output: Document multithreaded eve option 5 years ago
Jeff Lucovsky 30ae98f658 output/json: Multi-threaded EVE logging support
This commit modifies the JSON loggers with changes necessary to support
multi-threaded EVE output.

Each "thread-init" function sets up the per-thread log file context for
subsequent calls to the JSON output to buffer function.
5 years ago
Jeff Lucovsky aa20770277 log: Support multi-threaded eve output. 5 years ago
Jeff Lucovsky 15b4554ab3 output: Check for fwrite_unlocked
This commit creates a macro for fwrite_unlocked which is probed during
configuration time.
5 years ago
Jeff Lucovsky 3d0f353ee5 output: Correct typos 5 years ago
Jeff Lucovsky 1b791f34a5 output: Remove unused variables/define 5 years ago
Jeff Lucovsky 92e2e2ec8d log: remove unused include files 5 years ago
Jeff Lucovsky 60658cbe01 output/flow: Eliminate unnecessary parameter
This commit removes a parameter to an internal-only function call.
Removing the parameter allows an JSON builder optimization to be used.
5 years ago
Jeff Lucovsky 4aa7c988e8 output/netflow: Eliminate unneeded parameter
This commit changes an internal-only function to remove a parameter
that's invariant in all use cases. This allows an JSON builder
optimization to be used.
5 years ago
Philippe Antoine 1569f3e349 transform: adds url_decode keyword
Fixes https://redmine.openinfosecfoundation.org/issues/2689

Adds a new source file to handle this keyword.
And modifies documentation, Makefile, and registration accordingly.

url_decode decodes url-encoded data, ie replacing '+' with space
and '%HH' with its value.
5 years ago
Victor Julien 4c8af9cb96 stream: fix endless loop in traffic with gaps 5 years ago
Victor Julien 9b13c1b804 flow: avoid double state update on reuse
Avoids an unnecessary atomic operation.
5 years ago
Victor Julien fd2dff8542 flow: minor cleanups 5 years ago
Victor Julien 7bf000731c flow: validate emergency timeout settings
Make sure they are below the regular values.
5 years ago
Victor Julien 611c991f27 flow: improve performance in emergency mode
When the flow engine enters emergency mode, 3 things happen:

1. a different set of (lower) timeout values are applied
2. the flow manager runs more often
3. worker threads go get a flow directly from the hash table

Testing showed that performance went down significantly due to concurrency
issues:

1. worker threads would fight each other over the hash access
2. flow manager would get in the way of workers

This patch changes the behavior in 2 ways:

1. it makes the flow manager slightly less aggressive. It will still
   try to run ~3 times per second, but no longer 10 times.

   This should be reducing the contention. At the same time flows
   won't time out faster if they are checked many times per second.

2. The 'get a used flow' logic optimizes the use of atomics by only
   doing an atomic operation once, and while doing so reserving
   a slice of the hash per worker.

   The worker will also give up much quicker, to avoid the overhead
   of hash walking and taking and releasing locks.

These combined changes show much better 'under stress' behavior, esp
on multi-NUMA systems.
5 years ago
Philippe Antoine 0da4dc0dea enip: use status for probing parser 5 years ago
Philippe Antoine a99ad4c1e4 signature: checks for integer overflow in limits propagation 5 years ago
Philippe Antoine 5c31383d1c detect: fix read overflow in DetectGetLastSMByListId 5 years ago
Victor Julien b99ffd9ece eve: remove unused jansson code 5 years ago
Victor Julien a8e2399ea9 eve/metadata: create preformatted json string at start up
Avoid runtime overhead of assembling metadata json string by
pre-creating it at rule parsing time.
5 years ago
Victor Julien 1a18081a59 detect/profile: convert match dumps to jsonbuilder
Remove unused code and do minor misc cleanups as well.
5 years ago
Victor Julien 1639dfa36e pfring: fix compile warning 5 years ago
Victor Julien 38fe11f3b2 eve: remove unused json_t common functions
These are no longer used as all callers have switched to
the JsonBuilder equivalents.
5 years ago
Victor Julien 7ccfa177de eve/tls: minor cleanups 5 years ago
Victor Julien 04dad483c1 eve/metadata: convert to jsonbuilder 5 years ago
Victor Julien 556bee2adc stream: call parser with 0 data on EOF
This way both sides can call the EOF logic.
5 years ago
Victor Julien 616d7f256b app-layer/tcp: don't use un-ACK'd data
Still use un-ACK'd data in unclean shutdown. This means any state
before TCP_CLOSED, or TCP_CLOSED that was caused by a RST.
5 years ago
Victor Julien 42205006d1 flow/timeout: flag last pseudo packet
Flag the last flow timeout pseudo packet so that we can force
TX logging w/o setting both app-layer flags.

Case this fixes:

1. flow times out when only TS TCP data received, but non of it is ACK'd.
   So there is no app-layer proto yet, or app state or Flow::alparser. So
   EOF flags can't be set.

2. Flow timeout sees no reason to create pseudo packet in TC direction.

3. TS pseudo packet finds HTTP, creates HTTP state, flag EOF TS.

4. TX logging skips HTTP logging because:
   - TC progress not reached
   - EOF TC flag not set.

The solution has been to flag the very last packet for the flow as such
and use it has a master-EOF flag.
5 years ago
Victor Julien c825f83633 stream/tcp: track if ssn has been closed with RST 5 years ago
Victor Julien 1b3582325b app-layer: set EOFs on app-layer disable 5 years ago
Victor Julien 1cbbc82647 flow/worker: set EOF flags on change proto 5 years ago
Victor Julien a9f2540203 flow-timeout: set app-layer EOF flag 5 years ago
Victor Julien ecd7862c36 app-layer: add debug 5 years ago
Victor Julien d8d59ac9b5 stream: minor debug fixup 5 years ago
Victor Julien 5fd9386665 app-layer/pd: improve size check in bail conditions 5 years ago
Victor Julien 4f73943df9 app-layer: split EOF flag per direction 5 years ago
Victor Julien 57b75f89da stream: app update from loop
When the stream engine has data ready for the app-layer it will call
this API from a loop instead of just once. The loop is to ensure that
if we have a very lossy stream where between 'app_progress' and
'last_ack' there are multiple chunks of data and multiple gaps we
process all the chunks.
5 years ago
Victor Julien e822b30cc2 stream: improve gap handling with 'incomplete'
Make sure stream requiring more data because of 'incomplete' records
properly move ahead if there is a GAP in the window of required data.
5 years ago
Victor Julien c7d59a61ea stream: fix IDS mode using un-ACK'd data 5 years ago
Victor Julien 49eba6ac23 stream: code cleanup 5 years ago
Victor Julien f65bf4c7ea flow/tcp: consider pkts established based on 3whs 5 years ago
Victor Julien 7309c97eda detect/flow: test cleanup 5 years ago
Jeff Lucovsky 52cb1b8167 detect/dns-query: Splice UT to rust 5 years ago
Jeff Lucovsky 961b314b86 dns: Remove parser buffering code 5 years ago