Commit Graph

11430 Commits (6978d8dac6fbf4c8e09d411850a2e9aa016da635)
 

Author SHA1 Message Date
Victor Julien 611c991f27 flow: improve performance in emergency mode
When the flow engine enters emergency mode, 3 things happen:

1. a different set of (lower) timeout values are applied
2. the flow manager runs more often
3. worker threads go get a flow directly from the hash table

Testing showed that performance went down significantly due to concurrency
issues:

1. worker threads would fight each other over the hash access
2. flow manager would get in the way of workers

This patch changes the behavior in 2 ways:

1. it makes the flow manager slightly less aggressive. It will still
   try to run ~3 times per second, but no longer 10 times.

   This should be reducing the contention. At the same time flows
   won't time out faster if they are checked many times per second.

2. The 'get a used flow' logic optimizes the use of atomics by only
   doing an atomic operation once, and while doing so reserving
   a slice of the hash per worker.

   The worker will also give up much quicker, to avoid the overhead
   of hash walking and taking and releasing locks.

These combined changes show much better 'under stress' behavior, esp
on multi-NUMA systems.
5 years ago
Philippe Antoine 0da4dc0dea enip: use status for probing parser 5 years ago
Philippe Antoine a99ad4c1e4 signature: checks for integer overflow in limits propagation 5 years ago
Philippe Antoine 5c31383d1c detect: fix read overflow in DetectGetLastSMByListId 5 years ago
Victor Julien b99ffd9ece eve: remove unused jansson code 5 years ago
Victor Julien a8e2399ea9 eve/metadata: create preformatted json string at start up
Avoid runtime overhead of assembling metadata json string by
pre-creating it at rule parsing time.
5 years ago
Victor Julien 1a18081a59 detect/profile: convert match dumps to jsonbuilder
Remove unused code and do minor misc cleanups as well.
5 years ago
Victor Julien 1639dfa36e pfring: fix compile warning 5 years ago
Victor Julien 38fe11f3b2 eve: remove unused json_t common functions
These are no longer used as all callers have switched to
the JsonBuilder equivalents.
5 years ago
Victor Julien 7ccfa177de eve/tls: minor cleanups 5 years ago
Victor Julien 04dad483c1 eve/metadata: convert to jsonbuilder 5 years ago
Victor Julien 556bee2adc stream: call parser with 0 data on EOF
This way both sides can call the EOF logic.
5 years ago
Victor Julien 616d7f256b app-layer/tcp: don't use un-ACK'd data
Still use un-ACK'd data in unclean shutdown. This means any state
before TCP_CLOSED, or TCP_CLOSED that was caused by a RST.
5 years ago
Victor Julien 42205006d1 flow/timeout: flag last pseudo packet
Flag the last flow timeout pseudo packet so that we can force
TX logging w/o setting both app-layer flags.

Case this fixes:

1. flow times out when only TS TCP data received, but non of it is ACK'd.
   So there is no app-layer proto yet, or app state or Flow::alparser. So
   EOF flags can't be set.

2. Flow timeout sees no reason to create pseudo packet in TC direction.

3. TS pseudo packet finds HTTP, creates HTTP state, flag EOF TS.

4. TX logging skips HTTP logging because:
   - TC progress not reached
   - EOF TC flag not set.

The solution has been to flag the very last packet for the flow as such
and use it has a master-EOF flag.
5 years ago
Victor Julien c825f83633 stream/tcp: track if ssn has been closed with RST 5 years ago
Victor Julien 1b3582325b app-layer: set EOFs on app-layer disable 5 years ago
Victor Julien 1cbbc82647 flow/worker: set EOF flags on change proto 5 years ago
Victor Julien a9f2540203 flow-timeout: set app-layer EOF flag 5 years ago
Victor Julien ecd7862c36 app-layer: add debug 5 years ago
Victor Julien d8d59ac9b5 stream: minor debug fixup 5 years ago
Victor Julien 5fd9386665 app-layer/pd: improve size check in bail conditions 5 years ago
Victor Julien 4f73943df9 app-layer: split EOF flag per direction 5 years ago
Victor Julien 57b75f89da stream: app update from loop
When the stream engine has data ready for the app-layer it will call
this API from a loop instead of just once. The loop is to ensure that
if we have a very lossy stream where between 'app_progress' and
'last_ack' there are multiple chunks of data and multiple gaps we
process all the chunks.
5 years ago
Victor Julien e822b30cc2 stream: improve gap handling with 'incomplete'
Make sure stream requiring more data because of 'incomplete' records
properly move ahead if there is a GAP in the window of required data.
5 years ago
Victor Julien c7d59a61ea stream: fix IDS mode using un-ACK'd data 5 years ago
Victor Julien 49eba6ac23 stream: code cleanup 5 years ago
Victor Julien f65bf4c7ea flow/tcp: consider pkts established based on 3whs 5 years ago
Victor Julien 7309c97eda detect/flow: test cleanup 5 years ago
Jeff Lucovsky 52cb1b8167 detect/dns-query: Splice UT to rust 5 years ago
Jeff Lucovsky 961b314b86 dns: Remove parser buffering code 5 years ago
Victor Julien 0025467f90 sources: hide RegisterTests behind ifdef UNITTESTS
Update callers.
5 years ago
Victor Julien 00cc3c7374 eve/ssh: change hassh logging format
Elastic search didn't accept the 'hassh' and 'hassh.string'. It would
see the first 'hassh' as a string and split the second key into a
object 'hassh' with a string member 'string'. So two different types
for 'hassh', so it rejected it.

This patch mimics the ja3(s) logging by creating a 'hassh' object
with 2 members: 'hash', which holds the md5 representation, and
'string' which holds the string representation.
5 years ago
Victor Julien 085eb9fc8e eve/ssh: minor cleanup 5 years ago
Philippe Antoine 0c92b8f7e4 dcerpc: adds invalid signature unit test 5 years ago
Philippe Antoine b8069365f5 dcerpc: check app proto for signature keywords 5 years ago
Victor Julien 6ab323d323 detect: hide RegisterTests behind ifdef UNITTESTS
Update all callers to more aggressively use UNITTESTS guards as well.
5 years ago
Xiaofan Wang 071f55dcd7 ftp: fix direction of expectation for STOR command
Fix direction in active mode.
5 years ago
Victor Julien 0d24066876 sip: minor cleanup 5 years ago
Victor Julien db3b637ada htp: minor UNITTESTS guarding cleanup 5 years ago
Victor Julien 7c364017da ftp: small code cleanup 5 years ago
Victor Julien 0c06d08d94 gitignore: add .vscode and various other files 5 years ago
Jeff Lucovsky 72e2f36f9b ftp: Restrict file name lengths
Restrict file name lengths to PATH_MAX - 1 to avoid over subscribing
memory to FTP file name tracking.
5 years ago
Shivani Bhardwaj c169cfe0a3 bytetest: use ByteExtractString instead of StringParse 5 years ago
Shivani Bhardwaj 6f84515dd9 util: fix trailing char check with ByteExtractString 5 years ago
Philippe Antoine eb5b927787 ssh: fixing incomplete kex parsing
We use the record length from the ssh record header,
and not the size of the parsed data, as is done in other places.
5 years ago
Victor Julien bcd416e6ba nfs: fix 'dangling' files in lossy sessions
In case of lossy connections the NFS state would properly clean up
transactions, including file transactions. However for files the
state was never set to 'truncated', leading to files to stay 'active'.

This would lead these files staying in the NFS's state. In long running
sessions with lots of files this would lead to performance and memory
use issues.

This patch cleans truncates the file that was being transmitted when
a file transaction is being closed.

Based on 65e9a7c31c
5 years ago
Victor Julien fc7cecb4ae nfs: check post-gap timeouts once a second at most
Based on 25f2efe977
5 years ago
Victor Julien 5d65e5a748 nfs: update ts only if it changed
Based on 8aa380600d
5 years ago
Zach Kelly b25de4d99a rdp: remove parser buffering code 5 years ago
Zach Kelly 22a2bee614 rdp/eve: convert to jsonbuilder 5 years ago