Commit Graph

10040 Commits (6fe8bce3b0c653df8e72bdb81bc235672386b8b1)

Author SHA1 Message Date
Philippe Antoine 65e232669c dnp3: better limit for tests when fuzzing 6 years ago
Philippe Antoine 80dc6c6f1e fuzz: improves detect proto target
By mimicking assert message so as to clusterfuzz differentiates
between failures
6 years ago
Victor Julien 4664444067 detect: fix inspection order with stateful rules
When stateful detection rules, for which detection has already started
for a previous packet, are added to the candidates array, the array
is sorted to mantain the correct inspection order. However, due to a
trivial error in the sort helper the array was sorted in descending
instead of ascending order.
6 years ago
Victor Julien ba781265a4 dcerpc/udp: fix transaction handling and logging
Implement missing transaction handling.

Fix logging wrongly casting 'state' to DCERPCState instead of
DCERPCUDPState leading to crashes and malformed output.

Remove unused fields from DCERPCUDPState.
6 years ago
Victor Julien e7b5201016 detect/mpm: remove usused cleanup function 6 years ago
Victor Julien e799357d9f detect: optimize prefilter result handling 6 years ago
Victor Julien ffb0945b4c detect/prefilter: small cleanup 6 years ago
Victor Julien 14aacbd067 decode/null: fix type parsing 6 years ago
Jason Ish 76e011a5ba dnp3: set byte order when logging dnp3 src and dst
DNP3 uses little endian on the wire, for the most part this
is handled as the messages are deserialize. However, the link
header is a cast over raw data, so swap these bytes as they
are being logged.

Redmine issue:
https://redmine.openinfosecfoundation.org/issues/4173
6 years ago
Philippe Antoine 65711f6bc2 app-layer: do not try to parse gaps during protocol change
As this will leak the flow alstate because AppLayerParserParse
relies on FlowChangeProto to know if it should allocate a new
alstate
6 years ago
Philippe Antoine 89ae1a0036 detect: apply transforms to http body 6 years ago
Philippe Antoine 6b50a71d1a app-layer: lower limit for protocol detection on protocol change
So that protocol detection does not run for too long because
TCPProtoDetectCheckBailConditions somehow relies on its TCP stream
to start from zero, which is not the case on protocol change

Adds also debug validation checks, such as
both sides are known on protocol change

And only sets once alproto_orig
6 years ago
Jeff Lucovsky dabd50eeee detect: Register icmpv4 header 6 years ago
Jeff Lucovsky ac8532966b detect: Add icmpv4.hdr sticky buffer
This commit adds a new sticky buffer to access the ICMPv4 header.
6 years ago
Jeff Lucovsky 7cbe7c6463 detect: New enum for icmpv4 header keyword 6 years ago
Jeff Lucovsky 988bb26828 decode: Improved handling of ICMPv4 messages
This commit improves handling of ICMPv4 messages, especially those with
variable sized headers.

This commit also adds a header length variable for use by the new
sticky buffer for the header.
6 years ago
Jeff Lucovsky 828bf6d1d6 detect: Treat offset as a signed value
This commit updates the detector to treat 'offset' as a signed value to
be compatible with Snort.
6 years ago
Phil Young dc5349a30c napatech: Add Deprecation Warning Message for HBA
Added a message that HBA will be deprecated in the future and removed
hba from default conf file.
6 years ago
Jason Ish 8dbc774dfa dns: eve 1 deprecation warning
Redmine issue:
https://redmine.openinfosecfoundation.org/issues/4137
6 years ago
Jason Ish d8242c5d07 dns: fix leak in dns v1 logging
Intermediate JsonBuilder object was not being freed.

Redmine issue:
https://redmine.openinfosecfoundation.org/issues/4086
6 years ago
Phil Young fa77e02cf1 napatech: Removed restriction on use of inline mode
Removed the unnecessary restriction on the use of inline mode only when
bypass is enabled.  Now, Inline can be used independent of bypass
functionality.
6 years ago
Phil Young 789dcbd49d napatech: Fix potential double release of packet
This addresses readmine issue #4018.  There was the potential for a packet
buffer to be released twice in response to an error condition.  This
addresses this by only calling NT_NetRxRelease() when the p->ReleasePacket
is called.
6 years ago
Philippe Antoine 7d594d8710 fuzz: better packet recycling in sigpcap target
need especially to set datalink for every packet
6 years ago
Philippe Antoine c93cbe66c2 fuzz: applayer target uses yaml config
so that every app-layer protocol is enabled
6 years ago
Sascha Steinbiss ea2bc4c962 eve: do not access flow storage in packet context
We must make sure not to access the flow storage (e.g. keeping a
MacSet) before making sure we have a flow to begin with, We can,
for example, run into an alert without a flow with `ip` rules,
in which case the flow might be NULL. See Redmine issue #4109.
6 years ago
Jeff Lucovsky 68418a26db detect/file-data: Improved support for share bufs
This commit improves support for shared buffer usage, i.e., when
multiple rules share the file data (http) buffer and apply different
combinations of transforms and fast_patterns (or none).
6 years ago
Victor Julien 0eb692e500 mingw: fix format string warnings 6 years ago
Victor Julien ba2a9be11a detect/mpm: fix id confusion in mpm_ctx sharing
Mixing of dynamic id's and hardcoded config values could possibly
lead to the settings not getting applied properly.
6 years ago
Victor Julien 82d7f64630 detect/mpm: improve transforms handling
Make sure keywords with transforms get their own mpm ctx, instead of
sharing it with the 'pure' version of the keyword.
6 years ago
Victor Julien ee15bd8076 detect/mpm: rename variable to ease code review 6 years ago
Victor Julien 9f9c29a14a dcerpc: fix stream flag handling
Only hardcoded direction flags were passed to the parser, not the
full range.

Handle receiving an EOF flag w/o data.

Bug: #3856
6 years ago
Angelo Mirabella c72069c3b2 detect/magic: fix crash on rule reloading
This changseset fixes a bug causing a segmentation fault.
When rules are reloaded and a rule using libmagic matches, suricata
crashes due to an improper reinitialization of the thread contexts.

Bug: #3726
6 years ago
Angelo Mirabella ea15282f47 unix-socket: fix alert metadata logging
This changeset fixes a bug that was preventing suricata to dump
alert metadata info when running in unix-socket mode.
When running in unix-socket mode, suricata was skipping the
initialization of the output modules and, as a consequence,
the metadata output module was never invoked.
6 years ago
Victor Julien 62cc0c7acf alert/syslog: fix minor compile warning 6 years ago
Victor Julien 4e925ca260 stream: harden data size handling
Handle edge cases around ACKs and last_ack getting below
'app_progress', which can happen during shutdown of a flow
with multiple GAPs.
6 years ago
Victor Julien 5391746518 stream: bail early if no data after a gap 6 years ago
Victor Julien 27c9280735 stream: harden data size handling 6 years ago
Victor Julien 8a08fe82d4 stream: don't send STREAM_START multiple times 6 years ago
Victor Julien 479078d377 detect/content: add more negation tests 6 years ago
Victor Julien c41e64d637 dnp3: reword warning message 6 years ago
Victor Julien 36a93380a5 defrag: fix pthread warning on OpenBSD 6 years ago
Philippe Antoine 61d0cd1399 signature: checks protocol for file.name keyword
By setting the flags as for the filename keyword (not sticky version)
6 years ago
Philippe Antoine bde0c88984 dnp3: more precise probing for banners 6 years ago
Philippe Antoine 12dfc33e05 dnp3: fix probing test not using final null in string 6 years ago
Philippe Antoine 4706b38866 ssl: do not reuse struct session_id_length
As it can be confused between SSLv2 and TLSv13
In SSLv2, this variable is not used after the function scope, so
we can use a temporary variable.
6 years ago
Philippe Antoine f9c6b60edc detect: fix use of uninit memory in DetectDatarepParse
By initializing to empty string
6 years ago
Philippe Antoine 6b735a2e57 util: fix warning about format string for integer 6 years ago
Philippe Antoine dc7a9d2b46 util: THashInitConfig limits hash size when fuzzing 6 years ago
Philippe Antoine dcd5e4dec9 util: THashInitConfig does not exit but return error 6 years ago
Jason Ish 2b215a45e0 enip: set unidirection transaction flag 6 years ago
Jason Ish 60ebc27c4e app-layer: handle unidirectional transactions 6 years ago
Philippe Antoine 2eacc6a9a8 ssl: adds safety checks
Ensure the client version is valid by checking hello flags

Ensure no integer underflow occurs in SSLv3ParseHandshakeType
6 years ago
Jeff Lucovsky 4624e66cdd detect/transform: Validator for compress-ws
This commit adds a buffer validator for compress whitespace. Buffers
containing two or more consecutive whitespace characters are invalid
with this transform.
6 years ago
Jeff Lucovsky 30b1d7a9c1 detect: Correct typo 6 years ago
Jeff Lucovsky 7ab986250a detect/file-data: Apply transforms on file_data
This commit cause transforms to be applied on the sticky buffer.
6 years ago
Victor Julien e07a4393a9 http: error check htp_list_size
This avoids a potential casting to uint64_t of -1, leading to a very
high upper bound of the tx loop.
6 years ago
Jeff Lucovsky a75e9667ce log/eve: Threaded filename change: eve.N.json
This commit changes the name of the file used with threaded eve logging
to better support log rotation

Instead of using "eve.json.N" and creating potential issues with log
rotation (which also uses a ".N" suffix), the eve logs will be named
"eve.N.json" when threaded.
6 years ago
Jeff Lucovsky 73567272cd path: SCBasename - function to return basename
This commit returns the basename of a file, if it exists
in the same way that `basename(1)` works.
6 years ago
Jeff Lucovsky db9776af64 detect/rules: Fix copy/paste error
This commit corrects a copy/paste error handling the `include-mpm-stats`
configuration setting.
6 years ago
Jeff Lucovsky 41967a53ec detect: Apply clang-format updates 6 years ago
Jeff Lucovsky d911fb87b8 detect/rules: Increase array size to remove SEGV
This commit changes the size of reporting variables to be dynamic based
on the buffer ids in use instead of a fixed value to address a SEGV when
the fixed value was less than the max buffer/type id in use.
6 years ago
Philippe Antoine d57275fa64 protodetect: rerun probing parser if pattern matched
If a pattern matches in the other direction, after
probing parser finished without finding a protocol,
we will rerun the probing parser, which will include
the newly protocol found by its pattern
6 years ago
Philippe Antoine 5f348e9907 protodetect: runs probing parser on protocol found
It a protocol is found in a first direction, we should run the
probing parser, even if it is not in the known ports.

That can happen for HTTP2, where client magic is detected,
then server probe can be run
6 years ago
Philippe Antoine 518e0e66cb applayer: fix a leak in protocol change
TCPProtoDetect can either set f->alproto, change f->alstate
and return error.

When the original alstate gets freed, we shall set the pointer
to NULL, as it can get reused.
6 years ago
Jeff Lucovsky bb71eac9f9 log/anomaly: Move metadata out of anomaly section
This commit moves the metadata from the anomaly object where it was
incorrectly located.
6 years ago
Victor Julien daffcc6e53 app-layer: improve depth handling
Consider txs inspected and done for a direction after depth is reached for
that direction.
6 years ago
Victor Julien a5a46795bd stream: make sure to call app-layer in case of depth reached 6 years ago
Victor Julien 8b2886635f dcerpc/tcp: implement trunc logic
When one side of the connection reaches the STREAM_DEPTH condition the
parser should be aware of this. Otherwise transactions will forever be
waiting for data in that direction.
6 years ago
Victor Julien 4da0d9bdea applayer/rust: expose truncate callback 6 years ago
Victor Julien 2cfa35ccc9 app-layer: don't check tx detect flags if detect is disabled 6 years ago
Victor Julien 34e83b8acf stream: remove GAP flag from stream
This flag was checked in many places, but never set expect in 2 unittests.
6 years ago
Jeff Lucovsky c5ace81a27 log/eve: Rename fileinfo alert object to files
This commit changes the name of the "fileinfo" array in the alert object
to "files" to better support legacy use of "fileinfo" in reporting and
elsewhere.

The "fileinfo" event type is not an array while the alert "fileinfo"
member was.
6 years ago
Shivani Bhardwaj 98285177ba datasets: use default memcap 6 years ago
Philippe Antoine 872de829ea detect: initialize pointer before calling pcre_study
So as not to check an uninitialized value
Found by MSAN
6 years ago
Victor Julien f21a4bc40e datasets: remove experimental warning 6 years ago
Shivani Bhardwaj 1286b0a8f1 datasets: parse defaults section from yaml
Datasets can now have a global defaults setting in suricata.yaml. In
case the settings for memcap and hashsize are not find in the yaml or
rule, this shall be the fallback.

Example:

datasets:
  defaults:
    memcap: 100mb
    hashsize: 2048
  ua-seen:
    type: string
    load: datasets.csv
6 years ago
Shivani Bhardwaj 5ac94fc407 datasets: allow memcap, hashsize be set via yaml or rule
It is now possible to set the memcap and hashsize via suricata.yaml and
rules.

Rule example:

alert http any any -> any any (http.user_agent; dataset:isset,ua-seen,type string,load datasets.csv,memcap 100mb,hashsize 2048; sid:1;)

suricata.yaml example:

datasets:
  ua-seen:
    type: string
    load: datasets.csv
    memcap: 20mb
    hashsize: 2048
6 years ago
Shivani Bhardwaj b2482d6c60 datasets: allow max possible memcap while loading
While using the "load" option of datasets, it should be possible to load
any file from the disk, so set the limit to highest possible.
6 years ago
Jeff Lucovsky ce603d662f log/eve: Ensure eve logs have sequential suffixes
This commit ensures that the eve logs have sequential suffixes without
gaps.
6 years ago
Jeff Lucovsky ad2e18be3e atomics: Add "decl and init with value" function
This commit adds an interface to declare and initialize an atomic with a
specific value. This can help with situations where there's no defined
initialization path to set things up.
6 years ago
Jeff Lucovsky 8395a9201e log: Ensure threaded eve honors SIGHUP
This commit ensures that all logging contexts register for the file
rotation mechanism (SIGHUP and configured).
6 years ago
Philippe Antoine 9b5c923327 http: disables lzma by default for HTTP 6 years ago
Philippe Antoine 7011bddf84 http2: mimic HTTP1 request from upgrade 6 years ago
Philippe Antoine 9d1b030ff0 http2: first connection upgrade from http1 6 years ago
Philippe Antoine 9185a90fc9 dnp3: fix unit tests when fuzzing 6 years ago
Philippe Antoine 82f1758573 applayer: keep running detection on protocol change
ie do not stop on first try if we do not have enough data
6 years ago
Philippe Antoine 21e741795d applayer: on protocol change, use previous state 6 years ago
Philippe Antoine 828ff2dc3c http: removal of connect unit tests
moved to suricata-verify
6 years ago
Philippe Antoine 547d6c2d78 applayer: pass parameter to StateAlloc
This parameter is NULL or the pointer to the previous state
for the previous protocol in the case of a protocol change,
for instance from HTTP1 to HTTP2

This way, the new protocol can use the old protocol context.
For instance, HTTP2 mimicks the HTTP1 request, to have a HTTP2
transaction with both request and response
6 years ago
Jason Ish c4d0a61eca datasets: fix dataset load path construction
Test the full path instead of just the filename provided in the
rule to see if it exists.

Fixes the case where a rule file is loaded from a directory
other than the default-rule-directory.

Redmine issue:
https://redmine.openinfosecfoundation.org/issues/3916
6 years ago
Philippe Antoine 9788b2ec8d signature: frees transform options in SigMatchPrepare 6 years ago
Philippe Antoine 1674239442 detect: checks for overflow when comparing signatures priorities 6 years ago
Victor Julien abca451901 flow: suppress Coverity FP 6 years ago
Victor Julien 204302cbac flow: minor code cleanup 6 years ago
Victor Julien 42ce297e0e flow: turn BUG_ON into debug check 6 years ago
Eric Leblond 6494abc6b1 ebpf: fix invalid description in doc string 6 years ago
Victor Julien 6492fe0841 detect/app-layer-events: improve warnings/errors
Improve handling of outdated events that are no longer supported by the engine.
6 years ago
Victor Julien 8cd82486e2 flow/bypass: don't bypass on flow timeout pseudo packets 6 years ago
Jason Ish cc2c7b731a rule parsing: valid that input rule string is UTF8
Before parsing a rule string, validate that it is UTF-8 first.

Related Redmine issue:
https://redmine.openinfosecfoundation.org/issues/3850
6 years ago
Jason Ish 583593aa1b http/eve: use set_string_from_bytes where appropriate
The JsonBuilder set_string_from_bytes will escape unprintable
characters in the output stream. Using BytesToStringBuffer
can generate invalid UTF-8 which prevents the conversion from
a C string to a Rust string.
6 years ago
Victor Julien d6cf4b3335 detect/dataset: error if set couldn't be fully loaded 6 years ago
Victor Julien 017c038bcb datasets: free old data when reusing a hash container 6 years ago
Jeff Lucovsky 00f77f9643 log: Log errors while writing log info
This commit adds logic to log errors during output. Errors are logged
once and the number of errors is maintained.
6 years ago
Jeff Lucovsky b9458adf8a log: Add log output error code 6 years ago
Jeff Lucovsky 6cdd87f5ad log: Use unlocked variants of stdio functions
This commit uses the unlocked variants of additional stdio functions
6 years ago
Jeff Lucovsky 3c91e14352 log: Add more stdio_unlocked macros
This commit adds additional macros for interfaces in stdio_unlocked
according to their local availability.
6 years ago
Victor Julien fa0b91b18f error: reformat enum 6 years ago
Victor Julien fbdc776525 app-layer: handle parser return code issues more gracefully 6 years ago
Victor Julien 5155982ba5 pcap: fix minor scan-build warnings 6 years ago
Victor Julien aac2b9dbb9 commandline: add static analyzer hint for -r parsing 6 years ago
Victor Julien 71297f575f commandline: minor formatting fixes 6 years ago
Jeff Lucovsky 9efb936697 general: Improve grammar in error messages
This commit corrects a minor grammar issue in address/port error
messages.
6 years ago
Ali Jad Khalil 6be08637c3 decode/teredo: Modified/refactored Teredo logic
This is just a slight refactor to make analagous decoding/encapsulation
schemes - Geneve, Teredo, and VXLAN - be implemented as similarly as
possible.
6 years ago
Ali Jad Khalil 66452dd38a decode/vxlan: Modified/refactored VXLAN logic
This is just a slight refactor to make analagous decoding/encapsulation
schemes - Geneve, Teredo, and VXLAN - be implemented as similarly as
possible.
6 years ago
Ali Jad Khalil 5d955c1836 decode/geneve: Add Geneve decoding functionality
These changes are in response to feature request 3063. Geneve is
very similar to VXLAN, but uses a slightly different encapsulation
scheme.
6 years ago
Victor Julien 77bafe13a3 decode: reformat event table 6 years ago
Jason Ish 665328b29e plugins: require registration function SCPluginRegister
Instead of looking for a symbol, "PluginSpec" look for a function
named SCPluginRegister that returns a SCPlugin.

This makes it much easier to create Rust plugins without having
to deal with dlopen constructors and such, which is rather
straight forward in C, but a bit of advanced boilerplate in Rust
that can be eliminated by simply calling a registration function.
6 years ago
Jason Ish 335e4e728f suricata: expose the SuricataContext with a function
Expose the "SuricataContext" required by Rust as a function. During
normal startup we register this context with the Rust code, but
plugins written in Rust will need to get the same registration
done, but to do this in a plugin, the plugin code must
call and set the context within its address space.
6 years ago
Jason Ish 87a91e6aa8 logging: expose the log level with a function
The log level needs to exposed so Rust plugins can bootstrap
themselves with the correct login to SCLogNotice!(), etc work
as expected.
6 years ago
Jason Ish 17776e098e rust-context: remove unused opaque type Store 6 years ago
Joshua Lumb 6dbc6d6669 runmodes: memory leak on runmode single 6 years ago
Jeff Lucovsky ff29345527 output: Remove unused output functions
This commit removes registration, initialization, and de-initialization
functions no longer needed
6 years ago
Jeff Lucovsky 15caf3eea5 log: Remove standalone output registration
Remove standalone output logger registration since eve is
multi-instance.
6 years ago
Victor Julien 703de67086 flow: fix multi-manager hash range calculation 6 years ago
Victor Julien 260a20bf91 flow: suppress noisy messages 6 years ago
Philippe Antoine c06d8f2463 dcerpc: validate signature with dcerpc keywords
so that they do not use another protocol's keywords
6 years ago
Philippe Antoine d509a78074 http: handles gaps
ie data=NULL and len>0 from libhtp callbacks
6 years ago
Victor Julien 23c0efa2ec flowbits: convert flowbits dumping to json builder 6 years ago
Victor Julien 7facf5785f detect/analyzer: convert to jsonbuilder 6 years ago
Victor Julien d3cf2c21df threshold: fix potential memory leak 6 years ago
Victor Julien 6444f738af reject: minor code cleanup
Use enum for direction to assist compiler.
6 years ago
Victor Julien b6fed6d431 stream: fix GAP size
Fix GAPs sizes sometimes exceeding the ACK'd data size.
6 years ago
Victor Julien 6978d8dac6 stream: fix deinit after incomplete init 6 years ago
Victor Julien a015a31073 output/flow: fix deinit after incomplete init 6 years ago
Victor Julien c39101bbec nfq: suppress debug message using info level 6 years ago
Victor Julien 797a128256 plugins: suppress coverity toctou warning 6 years ago
Victor Julien bd68bd5037 flow: work around harmless coverity warnings 6 years ago
Victor Julien 29148fb563 flow: fix spare sync incomplete counter 6 years ago
Victor Julien 50c7b0d1c3 reject: work around coverity warning 6 years ago
Vadym Malakhatko 65455208a5 plugin: fix typo in long_opts struct 6 years ago
Jeff Lucovsky cc93638d33 napatech: Style -- remove extra space 6 years ago
Jeff Lucovsky 67529bd25a napatech: Use proper parser for type
This commit uses the proper parser call for the value being parsed.
6 years ago
Jeff Lucovsky 45b055aca5 napatech: Improve configuration range handling
This commit corrects issues parsing ranges from the Napatech section of
the configuration file.
6 years ago
Jeff Lucovsky c408b15c51 napatech: Fix compiler issues w/out bypass
This commit fixes compiler errors when Napatech bypass is not configured
6 years ago
Carl Smith 81d7a7aa82 threshold: Change rule parsing to use pcre_copy_substring
Fixes memory leak when parsing threshold rules.
All parsed strings are less than 16 characters except
for the IP address which could be up to 48 characters.
Remove redefinition of MAX_SUBSTRINGS
6 years ago
Jason Ish b8994cdaca plugins: track all loaded plugins in a list
Track the pointer returned from dlopen in a list to prevent a
resource leak by the pointer going out of scope.

Found by Coverity, CID 1465661.

Redmine issue:
https://redmine.openinfosecfoundation.org/issues/3864
6 years ago
Jason Ish dbb5dcb1dd plugins: use closedir to close open directory (not free)
Found by Coverity, CID 1465665: ALLOC_FREE_MISMATCH.

Redmine issue:
https://redmine.openinfosecfoundation.org/issues/3864
6 years ago
Jason Ish f2a1626b51 output-json: fix Coverity USE_AFTER_FREE
Return error if plugin open fails. Fixes Coverity CID 1465664
USE_AFTER_FREE error.

Redmine issue:
https://redmine.openinfosecfoundation.org/issues/3864
6 years ago
Victor Julien ac491c6e8d fuzz/pcap: add missing flow queue 6 years ago
Victor Julien 107ad95c41 fuzz/pcap: enable http2 6 years ago
Victor Julien 928d4820f9 plugins: remove unused func, suppressing compile warning 6 years ago
Victor Julien 9902413745 plugins: add missing guards 6 years ago
Jeff Lucovsky abc8bd11b9 output/ssh: Use correct file context
This commit corrects an issue with the SSH output module that resulted
in a SEGV when SSH output is logged.
6 years ago
Jason Ish e10d107415 plugins: support for capture plugins
Allow a plugin to register itself as a capture source. This isn't that
much different than how current sources register, it just happens
a little later on during startup.

One "slot" is reserved for capture plugins, but multiple plugins
implementing a capture can be loaded.  The --capture-plugin command
line option must be used to tell Suricata which plugin
to use.

This is still very much a work in progress, but can load
PF_RING as a capture plugin.
6 years ago
Jason Ish 8fb35236e6 plugins: initial support for a filetype plugin
A filetype plugin is a plugin that implements an eve filetype. Most
of the current filetypes could likely be implemented as such a plugin.
Such a plugin must implement Open, Close and Write, where Write
is provided the formatted JSON to be logged.

This commit also includes the plumbing for plugin loading. Example
plugin to come.

Plugins are loaded by the "plugin" section in the configuration
file:

  plugins:
    - /path/to/directory/plugins
    - /path/to/plugin_file.so

This can also be done on the command line with:

  --set plugins.0=/path/plugin_file.so
6 years ago
Jason Ish f35c25cef2 util-error: define generic plugin error code 6 years ago
Jason Ish 900f1522b4 plugins: config.h: move into src and rename to autoconf.h
While fixing files that include config.h, just remove the
include if possible.
6 years ago
Jason Ish f26d6eaf98 http2: log as http to abstract http and http2 a little
This commit logs http2 as an http event. The idea is to somewhat
normalize http/http2 so common info can be version agnostic.

This puts the http2 specific fields in an "http2" object inside
the "http" object.

HTTP2 headers/values that are in common with HTTP1 are logged
under the "http" object to be compatible with HTTP1 logging.
6 years ago
Philippe Antoine 1422b18a99 http2: initial support 6 years ago
Philippe Antoine 0507d1e8f8 detect: generic structures for mpm with lists 6 years ago
Victor Julien ee41c0e293 flow/spare: implement pool shrinking
Remove at most one block per run, so it shrinks slowly.
6 years ago
Victor Julien b3599507f4 flow: redesign of flow timeout handling
Goals:
- reduce locking
- take advantage of 'hot' caches
- better locality

Locking reduction

New flow spare pool. The global pool is implmented as a list of blocks,
where each block has a 100 spare flows. Worker threads fetch a block at
a time, storing the block in the local thread storage.

Flow Recycler now returns flows to the pool is blocks as well.

Flow Recycler fetches all flows to be processed in one step instead of
one at a time.

Cache 'hot'ness

Worker threads now check the timeout of flows they evaluate during lookup.
The worker will have to read the flow into cache anyway, so the added
overhead of checking the timeout value is minimal. When a flow is considered
timed out, one of 2 things happens:

- if the flow is 'owned' by the thread it is handled locally. Handling means
  checking if the flow needs 'timeout' work.

- otherwise, the flow is added to a special 'evicted' list in the flow
  bucket where it will be picked up by the flow manager.

Flow Manager timing

By default the flow manager now tries to do passes of the flow hash in
smaller steps, where the goal is to do full pass in 8 x the lowest timeout
value it has to enforce. So if the lowest timeout value is 30s, a full pass
will take 4 minutes. The goal here is to reduce locking overhead and not
get in the way of the workers.

In emergency mode each pass is full, and lower timeouts are used.

Timing of the flow manager is also no longer relying on pthread condition
variables, as these generally cause waking up much quicker than the desired
timout. Instead a simple (u)sleep loop is used.

Both changes reduce the number of hash passes a lot.

Emergency behavior

In emergency mode there a number of changes to the workers. In this scenario
the flow memcap is fully used up and it is unavoidable that some flows won't
be tracked.

1. flow spare pool fetches are reduced to once a second. This avoids locking
   overhead, while the chance of success was very low.

2. getting an active flow directly from the hash skips flows that had very
   recent activity to avoid the scenario where all flows get only into the
   NEW state before getting reused. Rather allow some to have a chance of
   completing.

3. TCP packets that are not SYN packets will not get a used flow, unless
   stream.midstream is enabled. The goal here is again to avoid evicting
   active flows unnecessarily.

Better Localily

Flow Manager injects flows into the worker threads now, instead of one or
two packets. Advantage of this is that the worker threads can get packets
from their local packet pools, avoiding constant overhead of packets returning
to 'foreign' pools.

Counters

A lot of flow counters have been added and some have been renamed.

Overall the worker threads increment 'flow.wrk.*' counters, while the flow
manager increments 'flow.mgr.*'.

Additionally, none of the counters are snapshots anymore, they all increment
over time. The flow.memuse and flow.spare counters are exceptions.

Misc

FlowQueue has been split into a FlowQueuePrivate (unlocked) and FlowQueue.
Flow no longer has 'prev' pointers and used a unified 'next' pointer for
both hash and queue use.
6 years ago
Victor Julien e0aa7c1dbc unittests: check for flow memuse 6 years ago
Victor Julien f50c7b6d11 flow-manager: call other timeouts max once a second
Call Defrag and others only once per second. Flow Manager may wake
up (much) more often when flow engine is under resource pressure.
As this does not affect Defrag and others, it only unnecessarily
adds load.
6 years ago
Victor Julien 6814f08e93 flow-manager: only update FlowBucket::next_ts if it changed 6 years ago
Victor Julien 1d6d7f0858 flow: unref flow at end of flow worker 6 years ago
Victor Julien 2fd7c87f22 flow: do timeout checks before tuple compare 6 years ago
Victor Julien 48605f4867 flow: don't reorder list on lookup
Reduces cache misses.
6 years ago
Victor Julien 2a872ccb86 flow: timeout check on flow lookup 6 years ago
Victor Julien 7583a6c37c flow: simplify hash lookup logic
Remove double compare paths in favor of a single unified path.
6 years ago
Victor Julien 8b016cff4b flow: only move lastts forward
Pcaps with timestamps jumping around could confuse flow timeout
handling otherwise.
6 years ago
Victor Julien afd4a8012e flow/worker: check pkt src using DEBUG_VALIDATE_BUG_ON 6 years ago
Jason Ish fd5d8b78d0 alert/eve: add snmp metadata for rdp alerts
Redmine issue:
https://redmine.openinfosecfoundation.org/issues/3441
6 years ago
Jason Ish ef0ebc9550 alert/eve: add snmp metadata for snmp alerts
Redmine issue:
https://redmine.openinfosecfoundation.org/issues/3441
6 years ago
Shivani Bhardwaj 4c7f55e636 dcerpc: handle gap for TCP streams 6 years ago
James Dutrisac db5b73d9d6 pcap: read directories recursively
Describe Changes
- Added ability to recursively read pcap directories
- src/suricata.c: addition of new command line parameter
    --pcap-file-recursive
- src/source-pcap-file.c: parsing of the command line argument
- src/source-pcap-file-directory-helper.h: two thread vars tracking
    directory depth and should recurse
- src/util-error.c / src/util-error.h:
   Added new warning code "SC_WARN_PATH_READ_ERROR"
- Redmine ticket: https://redmine.openinfosecfoundation.org/issues/2363

Ticket: #2363
6 years ago
James Dutrisac 5a92d0a704 path: introduce path handling util funcs
This commit provides changes to util-path.c and util-path.h
to support the recursive reading of directories. It adds
4 functions.
- SCIsRegularFile to provide OS independent file info.
- SCIsRegularDirectory to provide OS independent directory info.
- SCRealPath is an OS independent wrapper for realpath.
- PathJoin to manage path resolution logic.
6 years ago
Philippe Antoine 3cfc1fcb07 stream/app-layer: break loop on proto change 6 years ago
Shivani Bhardwaj c9a637d854 datasets: fix null pointer deref 6 years ago
Shivani Bhardwaj a8f147d17a datasets: Init even in socket mode
Closes redmine ticket 3476.
6 years ago
Sascha Steinbiss 93eef1da84 detect-mqtt: unify error handling in rule parsing
This is meant to provide a single path to the error case.
This might help make things more clear for static
checkers.
6 years ago
Sascha Steinbiss 5dc21b0e09 detect-engine: initialize struct fields 6 years ago
Philippe Antoine 9a954e944d http: merge duplicated code
HtpRequestBodyHandlePUT and HtpRequestBodyHandlePOST
6 years ago
Philippe Antoine fe6950de08 http: use more precise parameter in HtpRequestBodySetupMultipart
So that we can see that it does not have to handle gaps
6 years ago
Philippe Antoine e13b319beb http: remove unused code
HtpRequestBodySetupPUT function
So that we can see that we do not use data=NULL in there
6 years ago
Philippe Antoine 28050c967f file: handles gaps natively
ie data=NULL and len>0 parameters
6 years ago
Philippe Antoine f0159b2fd2 util: PrintRawDataFp handles null
for gaps which are data=NULL and len>0
6 years ago
Philippe Antoine 6343920dfc applayer: allow rust parsers to have only one probe 6 years ago
Sascha Steinbiss 4e1a41a17d output-json: add MAC address output
This commit adds MAC address output to the EVE-JSON format. We follow the
remarks made in Redmine ticket #962: for packets, log MAC src/dst as a
scalar field in EVE; for flows, log MAC src/dst as lists in EVE. Field names
are different between flow and packet context to avoid type confusion
(src_mac vs. src_macs). Configuration approach and JSON representation is
taken from previous GitHub PR #2700.
6 years ago
Jeff Lucovsky c42574169e output/anomaly: Restrict anomaly logger count
This commit restricts the anomaly logger count. The restriction is
necessary due to state maintenance in the logger that doesn't scale
beyond a single logger.

Until that issue's solved, when multiple anomaly loggers are configured,
an error message will be emitted to highlight the restriction.
6 years ago
Philippe Antoine 61c327dd80 signature: checks for integer overflow in limits propagation 6 years ago
Victor Julien 1c748f394b fuzz/pcap: enable MQTT parser 6 years ago
Shivani Bhardwaj e9fe5ada7f datasets: reload static sets 6 years ago
Sascha Steinbiss c31360070b rust/mqtt: add MQTT parser 6 years ago
Philippe Antoine a5572890a9 detect: adds engine for u8 keywords 6 years ago
Jeff Lucovsky 30ae98f658 output/json: Multi-threaded EVE logging support
This commit modifies the JSON loggers with changes necessary to support
multi-threaded EVE output.

Each "thread-init" function sets up the per-thread log file context for
subsequent calls to the JSON output to buffer function.
6 years ago
Jeff Lucovsky aa20770277 log: Support multi-threaded eve output. 6 years ago
Jeff Lucovsky 15b4554ab3 output: Check for fwrite_unlocked
This commit creates a macro for fwrite_unlocked which is probed during
configuration time.
6 years ago
Jeff Lucovsky 3d0f353ee5 output: Correct typos 6 years ago
Jeff Lucovsky 1b791f34a5 output: Remove unused variables/define 6 years ago
Jeff Lucovsky 92e2e2ec8d log: remove unused include files 6 years ago
Jeff Lucovsky 60658cbe01 output/flow: Eliminate unnecessary parameter
This commit removes a parameter to an internal-only function call.
Removing the parameter allows an JSON builder optimization to be used.
6 years ago
Jeff Lucovsky 4aa7c988e8 output/netflow: Eliminate unneeded parameter
This commit changes an internal-only function to remove a parameter
that's invariant in all use cases. This allows an JSON builder
optimization to be used.
6 years ago
Philippe Antoine 1569f3e349 transform: adds url_decode keyword
Fixes https://redmine.openinfosecfoundation.org/issues/2689

Adds a new source file to handle this keyword.
And modifies documentation, Makefile, and registration accordingly.

url_decode decodes url-encoded data, ie replacing '+' with space
and '%HH' with its value.
6 years ago
Victor Julien 4c8af9cb96 stream: fix endless loop in traffic with gaps 6 years ago
Victor Julien 9b13c1b804 flow: avoid double state update on reuse
Avoids an unnecessary atomic operation.
6 years ago
Victor Julien fd2dff8542 flow: minor cleanups 6 years ago
Victor Julien 7bf000731c flow: validate emergency timeout settings
Make sure they are below the regular values.
6 years ago
Victor Julien 611c991f27 flow: improve performance in emergency mode
When the flow engine enters emergency mode, 3 things happen:

1. a different set of (lower) timeout values are applied
2. the flow manager runs more often
3. worker threads go get a flow directly from the hash table

Testing showed that performance went down significantly due to concurrency
issues:

1. worker threads would fight each other over the hash access
2. flow manager would get in the way of workers

This patch changes the behavior in 2 ways:

1. it makes the flow manager slightly less aggressive. It will still
   try to run ~3 times per second, but no longer 10 times.

   This should be reducing the contention. At the same time flows
   won't time out faster if they are checked many times per second.

2. The 'get a used flow' logic optimizes the use of atomics by only
   doing an atomic operation once, and while doing so reserving
   a slice of the hash per worker.

   The worker will also give up much quicker, to avoid the overhead
   of hash walking and taking and releasing locks.

These combined changes show much better 'under stress' behavior, esp
on multi-NUMA systems.
6 years ago
Philippe Antoine 0da4dc0dea enip: use status for probing parser 6 years ago
Philippe Antoine a99ad4c1e4 signature: checks for integer overflow in limits propagation 6 years ago
Philippe Antoine 5c31383d1c detect: fix read overflow in DetectGetLastSMByListId 6 years ago
Victor Julien b99ffd9ece eve: remove unused jansson code 6 years ago
Victor Julien a8e2399ea9 eve/metadata: create preformatted json string at start up
Avoid runtime overhead of assembling metadata json string by
pre-creating it at rule parsing time.
6 years ago
Victor Julien 1a18081a59 detect/profile: convert match dumps to jsonbuilder
Remove unused code and do minor misc cleanups as well.
6 years ago
Victor Julien 1639dfa36e pfring: fix compile warning 6 years ago
Victor Julien 38fe11f3b2 eve: remove unused json_t common functions
These are no longer used as all callers have switched to
the JsonBuilder equivalents.
6 years ago
Victor Julien 7ccfa177de eve/tls: minor cleanups 6 years ago
Victor Julien 04dad483c1 eve/metadata: convert to jsonbuilder 6 years ago
Victor Julien 556bee2adc stream: call parser with 0 data on EOF
This way both sides can call the EOF logic.
6 years ago
Victor Julien 616d7f256b app-layer/tcp: don't use un-ACK'd data
Still use un-ACK'd data in unclean shutdown. This means any state
before TCP_CLOSED, or TCP_CLOSED that was caused by a RST.
6 years ago
Victor Julien 42205006d1 flow/timeout: flag last pseudo packet
Flag the last flow timeout pseudo packet so that we can force
TX logging w/o setting both app-layer flags.

Case this fixes:

1. flow times out when only TS TCP data received, but non of it is ACK'd.
   So there is no app-layer proto yet, or app state or Flow::alparser. So
   EOF flags can't be set.

2. Flow timeout sees no reason to create pseudo packet in TC direction.

3. TS pseudo packet finds HTTP, creates HTTP state, flag EOF TS.

4. TX logging skips HTTP logging because:
   - TC progress not reached
   - EOF TC flag not set.

The solution has been to flag the very last packet for the flow as such
and use it has a master-EOF flag.
6 years ago
Victor Julien c825f83633 stream/tcp: track if ssn has been closed with RST 6 years ago
Victor Julien 1b3582325b app-layer: set EOFs on app-layer disable 6 years ago
Victor Julien 1cbbc82647 flow/worker: set EOF flags on change proto 6 years ago
Victor Julien a9f2540203 flow-timeout: set app-layer EOF flag 6 years ago
Victor Julien ecd7862c36 app-layer: add debug 6 years ago
Victor Julien d8d59ac9b5 stream: minor debug fixup 6 years ago
Victor Julien 5fd9386665 app-layer/pd: improve size check in bail conditions 6 years ago
Victor Julien 4f73943df9 app-layer: split EOF flag per direction 6 years ago
Victor Julien 57b75f89da stream: app update from loop
When the stream engine has data ready for the app-layer it will call
this API from a loop instead of just once. The loop is to ensure that
if we have a very lossy stream where between 'app_progress' and
'last_ack' there are multiple chunks of data and multiple gaps we
process all the chunks.
6 years ago
Victor Julien e822b30cc2 stream: improve gap handling with 'incomplete'
Make sure stream requiring more data because of 'incomplete' records
properly move ahead if there is a GAP in the window of required data.
6 years ago
Victor Julien c7d59a61ea stream: fix IDS mode using un-ACK'd data 6 years ago
Victor Julien 49eba6ac23 stream: code cleanup 6 years ago
Victor Julien f65bf4c7ea flow/tcp: consider pkts established based on 3whs 6 years ago
Victor Julien 7309c97eda detect/flow: test cleanup 6 years ago
Jeff Lucovsky 52cb1b8167 detect/dns-query: Splice UT to rust 6 years ago
Victor Julien 0025467f90 sources: hide RegisterTests behind ifdef UNITTESTS
Update callers.
6 years ago
Victor Julien 085eb9fc8e eve/ssh: minor cleanup 6 years ago
Philippe Antoine 0c92b8f7e4 dcerpc: adds invalid signature unit test 6 years ago
Philippe Antoine b8069365f5 dcerpc: check app proto for signature keywords 6 years ago
Victor Julien 6ab323d323 detect: hide RegisterTests behind ifdef UNITTESTS
Update all callers to more aggressively use UNITTESTS guards as well.
6 years ago
Xiaofan Wang 071f55dcd7 ftp: fix direction of expectation for STOR command
Fix direction in active mode.
6 years ago
Victor Julien 0d24066876 sip: minor cleanup 6 years ago
Victor Julien db3b637ada htp: minor UNITTESTS guarding cleanup 6 years ago
Victor Julien 7c364017da ftp: small code cleanup 6 years ago
Jeff Lucovsky 72e2f36f9b ftp: Restrict file name lengths
Restrict file name lengths to PATH_MAX - 1 to avoid over subscribing
memory to FTP file name tracking.
6 years ago
Shivani Bhardwaj c169cfe0a3 bytetest: use ByteExtractString instead of StringParse 6 years ago
Shivani Bhardwaj 6f84515dd9 util: fix trailing char check with ByteExtractString 6 years ago
Zach Kelly 22a2bee614 rdp/eve: convert to jsonbuilder 6 years ago
Joshua Lumb f7c4600482 threads/runmode: Changes to thread config behaviour 6 years ago
Shivani Bhardwaj 333a785efd sip: remove extra jsonbuilder close 6 years ago
Shivani Bhardwaj 9f9670ebdc logging: Add DCERPC logger 6 years ago
Shivani Bhardwaj bab497ab2c dcerpc: Add multi transaction support
DCERPC parser so far provided support for single transactions only.
Extend that to support multiple transactions.

In order for multiple transactions to work, there is always a
transaction identifier for any protocol in its header that lets a
response match the request. In DCERPC, for TCP, that param is call_id in
the header which is a 32 bit field. For UDP, however since it uses
different version of RPC (4.x), this is defined by serial number field
defined in the header. This field however is not contiguous and needs to
be assembled by the provided serial_low and serial_hi fields.
6 years ago
Victor Julien 9831839388 detect/mpm: fix hs check 6 years ago
Roland Fischer 9f1efa3c10 pcap: 32bit counters can wrap-around
Fixes issue 2845.

pcap_stats is based on 32bit counters and given a big enough throughput
will overflow them. This was reported by people using Myricom cards which
should only be a happenstance. The problem exists for all pcap-based
interfaces.

Let's use internal 64bit counters that drag along the pcap_stats and
handle pcap_stats wrap-around as we update the 64bit stats "often enough"
before the pcap_stats can wrap around twice.
6 years ago
Shivani Bhardwaj 67e7be633c krb: convert to jsonbuilder
Closes redmine ticket 3754.
6 years ago
Shivani Bhardwaj 72dab0a8b7 snmp: convert to jsonbuilder
Closes redmine ticket 3756.
6 years ago
Jason Ish 53aa967e0b applayer: add flags to parser registration struct
This will allow Rust parsers to register for gap handing from
Rust (some Rust parsers do handle gaps, but they set the flag
from C).
6 years ago
Jason Ish 7476399f43 template: add gap handling 6 years ago
Victor Julien 4726d7027c detect/mpm: 'mpm-algo' parsing cleanups 6 years ago
Victor Julien f2a3d6d834 flow: fix unlikely issue with int handling
Thanks for reporting this magenbluten PR 4575.
6 years ago
Victor Julien fa2b46cdc3 detect/stream_size: minor code cleanups 6 years ago
Victor Julien ac3cf6ff75 detect/config: set config for special cases
Allow app-layer to declare the txs are uni-directional and special
care is needed for applying config.
6 years ago
Victor Julien 2145cf99a3 detect/config: initial version 6 years ago
Victor Julien a2f249cc86 app-layer: handle AppLayerTxData being NULL
Http parser can have 'NULL' user data in case of memcap limit getting
reached.
6 years ago
Victor Julien 5dd4d948d9 app-layer: remove unused detect flags API 6 years ago
Victor Julien f88657206c app-layer: GetTxData callback is mandatory 6 years ago
Victor Julien 8fe9faecb2 app-layer: remove DetectFlags API. Replaced by AppLayerTxData 6 years ago
Victor Julien 9664f73f75 app-layer: remove logged API calls 6 years ago
Victor Julien 455eab370e template: support AppLayerTxData 6 years ago
Victor Julien e0debed0b4 tftp: support AppLayerTxData 6 years ago
Victor Julien a1e06247a6 dcerpc/udp: support AppLayerTxData 6 years ago
Victor Julien 3202d29325 dcerpc: support AppLayerTxData 6 years ago
Victor Julien 8cd55124a3 modbus: support AppLayerTxData 6 years ago
Victor Julien 7d663ed5cf enip: support AppLayerTxData 6 years ago
Victor Julien fb780c7d92 ssl/tls: support AppLayerTxData 6 years ago
Victor Julien bc11a1c23e smtp: support AppLayerTxData 6 years ago
Victor Julien c98f597831 ftp: support AppLayerTxData 6 years ago
Victor Julien 302cf49486 dnp3: support AppLayerTxData 6 years ago
Victor Julien 77a95eddd9 smb: support AppLayerTxData 6 years ago
Victor Julien 7a7805cde6 nfs: support AppLayerTxData 6 years ago
Victor Julien 910922cdc4 htp: support AppLayerTxData 6 years ago
Victor Julien 5665fc8301 app-layer: add ApplyTxConfig API
Optional callback a parser can register for applying configuration
to the 'transaction'. Most parsers have a bidirectional tx. For those
parsers that have different types of transaction handling, this new
callback can be used to properly apply the config.
6 years ago
Victor Julien df27205451 output/tx: implement filtering 6 years ago
Victor Julien e15995e2d2 detect: store detect flags in AppLayerTxData 6 years ago
Victor Julien c797c9f09c app-layer: add logger flags to AppLayerTxData 6 years ago
Victor Julien 411f428a38 app-layer: define AppLayerTxData and AppLayerTxConfig
AppLayerTxData is a structure each tx should include that will contain
the common fields the engine needs for tracking logging, detection and
possibly other things.

AppLayerTxConfig will be used by the detection engine to configure
the transaction.
6 years ago
Victor Julien 274a033d65 htp: alloc user data at tx start
This way the AppLayerTxData is set up from the start. Any type of
processing (logging, detection) will lead to setting up the user
data later on anyway.

Remove other places where it was added.
6 years ago
Victor Julien f7ff7dbaed config: common definitions 6 years ago
Victor Julien 6dcdf394d7 rules: add config action 6 years ago
Jason Ish 8c98fa452f dnp3/eve: update for regenerated dnp3 object logging code
Migration from Jansson to JsonBuilder.
6 years ago
Jason Ish bf8d8c573a dnp3/eve: regenerator object logging code 6 years ago
Jason Ish ccc057fdc9 dnp3/eve: convert to jsonbuilder (non generated code)
First step of converting DNP3 to JsonBuilder by first converting
the non-generated code.
6 years ago
Shivani Bhardwaj 6f7d8e50c8 src: use FatalError whenever possible
Replaces all patterns of SCLogError() followed by exit() with
FatalError(). Cocci script to do this:

@@
constant C;
constant char[] msg;
@@

- SCLogError(C,
+ FatalError(SC_ERR_FATAL,
  msg);
- exit(EXIT_FAILURE);

Closes redmine ticket 3188.
6 years ago
Jeff Lucovsky f6a399f154 general: Correct typos 6 years ago
Jeff Lucovsky fb409664d2 detect: byte_math support 6 years ago
Jeff Lucovsky 1a726731e4 detect: Use byte-math to byte var handling func 6 years ago
Jeff Lucovsky 0e4ba7b13e detect: Add byte_math detector 6 years ago
Jeff Lucovsky ac01adc260 detect: Add utility module for byte var handling 6 years ago