Commit Graph

578 Commits (10150e95ad0c6aa9e5af6f39f709e87b3b3b0ba9)

Author SHA1 Message Date
Victor Julien 975062cf40 detect: track base id for xform buffers
Buffers with transforms are based on the non-transformed "base"
buffer, with a new ID assigned and the transform callbacks added.

This patch stores the id of the original buffer in the new buffer
inspect and prefilter structures. This way the buffers with and
without transforms can share some of the logic are progression
of file and body inspection trackers.

Related tickets: #4361 #4199 #3616
5 years ago
Victor Julien 52692da7cf detect/analyzer: fix pkt engine display 5 years ago
Philippe Antoine c8dbe24fb6 proto: introduce signature protocol, as extension to flow protocol
AppProtoEquals function allows to check if a flow protocol
matches a signature protocol
5 years ago
Victor Julien 13cebb1857 detect: fix heap overflow issue with buffer setup
In some cases, the InspectionBufferGet function would be followed by
a failure to set the buffer up, for example due to a HTTP body limit
not yet being reached. Yet each call to InspectionBufferGet would lead
to the matching list_id to be added to the
DetectEngineThreadCtx::inspect.to_clear_queue. This array is sized to
add each list only once, but in this case the same id could be added
multiple times, potentially overflowing the array.
5 years ago
Victor Julien 84385549fe detect: remove unused arg from generic list inspect 5 years ago
Victor Julien d694dac5a8 detect: remove inspect v1 API 5 years ago
Victor Julien c6c93d1d12 detect/stream: convert to inspect API v2 5 years ago
Victor Julien ee15bd8076 detect/mpm: rename variable to ease code review 5 years ago
Shivani Bhardwaj e9fe5ada7f datasets: reload static sets 5 years ago
Jeff Lucovsky fb409664d2 detect: byte_math support 5 years ago
Jeff Lucovsky 2d055ed1f7 detect: Correct Coverity REVERSE_INULL issue
This commit corrects the "Null pointer dereferences" issue (CID
1465221).
5 years ago
Jeff Lucovsky 197a593078 detect: Add transform validation api
This commit extends the API with a function that validates arguments
against the transforms for the SM list (if any).
5 years ago
Jeff Lucovsky 5d10db88bc detect/transform: Support transform options
This commit adds support for transform-specific options. During Setup,
transforms have the signature string available for options detection.
When a transform detects an option, it should convert the option into an
internal format and supply a pointer to this format as the last argument
to DetectSignatureAddTransform.

Transforms that support options must provide a function in their
Sigmatch table entry. When the transform is freed, a pointer to the
internal format of the option is passed to this function.
5 years ago
Shivani Bhardwaj e7c0f0ad91 src: remove multiple uses of atoi
atoi() and related functions lack a mechanism for reporting errors for
invalid values. Replace them with calls to the appropriate
ByteExtractString* functions.

Partially closes redmine ticket #3053.
5 years ago
Jeff Lucovsky d3a65fe156 detect: Provide `de_ctx` to free functions
This commit makes sure that the `DetectEngineCtx *` is available
to each detector's "free" function.
5 years ago
Jeff Lucovsky d1151f3f8e detect: Provide function to clear per-thread ctx
This commit provides an interface to free previously allocated
per-thread contextual information on the keyword lists.
5 years ago
Victor Julien 7f19da1cc0 detect: more robust against transform issues
In case of transform issues (transform not consumed before pkt_data
for example), the code would hit an ugly BUG_ON.

Address this by a more graceful error message, that will still
invalidate the sig but not crash the engine.
5 years ago
Shivani Bhardwaj 0e4f261224 Use StringParse* for all parsers and configurations 5 years ago
Jeff Lucovsky be4c6b85de general: Convert _Bool to bool
This commit addresses task 3167 and changes usages of '_Bool` to `bool`.
The latter is included from `suricata-common.h`
6 years ago
Jeff Lucovsky 1cddbc085b detect: Fix typos/spelling errors. 6 years ago
Victor Julien d369e54f1d app-layer: all protocols are tx aware now
So remove the runtime check for it.
6 years ago
Victor Julien 550cfdd98d threading: hide 'trans_q' from queue handlers 6 years ago
Victor Julien d7cb0774dd detect: cleanup reload thread handling 6 years ago
Victor Julien 1a8562b3c6 detect: clean up threads handling
Clean up reload and break loop thread handling.
6 years ago
Victor Julien e5010d7704 detect: inject packet cleanup 6 years ago
Jason Ish fdb587d2fc detect-engine: check for tx detect flag support
When registing a detection engine, check that the app-layer
protocol supports tx detect flags. Exit with a fatal
error if it does not as this is a code implementation
error that should be resolved during development.
6 years ago
Victor Julien 0a809bf577 packet: set unique pkt_src 'flush' packets
Set unique type for capture timeout and for detect reload flush
to assist in debugging.
6 years ago
Victor Julien 58e48bcb87 debug: make it easier to trace flush logic 6 years ago
Victor Julien 4dff903b35 detect: introduce pkt mpm engines
Instead of the hardcode L4 matching in MPM that was recently introduced,
add an API similar to the AppLayer MPM and inspect engines.

Share part of the registration code with the AppLayer.

Implement for the tcp.hdr and udp.hdr keywords.
6 years ago
Victor Julien 14896365ef detect: remove Threadvars argument from API calls
Remove it as it's (almost) never used. If it is really needed it can
be accessed through DetectEngineThreadCtx::tv as well.
6 years ago
Victor Julien ba5679de0f profiling: fix undefined profiling code use 6 years ago
Victor Julien 35be8385eb detect: tcp.hdr sticky buffer
Sticky buffer to inspect the TCP header.
6 years ago
Victor Julien 47ef8f5822 detect/mpm: add l4-header support
Prepare MPM part of the detection engine for a new type of per
packet matching, where the L4 header will be inspected.

Preparation for TCP header inspection keyword.
6 years ago
Victor Julien 0965afd661 detect: pkt inspect engines
Instead of hard coded calls to the inspection logic for
payload inspection and 'MATCH'-list inspection use a callback
approach. This will register a callback per 'sm_list' much like
how app-layer inspect engines are registered.

This will allow for adding more types later without adding
runtime overhead.

Implement the callback for the PMATCH and MATCH logic.
6 years ago
Victor Julien d270a7603a detect/inspect: add flags to inspect buffer 6 years ago
Victor Julien 32fb7d773a detect/content-inspect: turn void arg into Packet
Replace the 'void *data' argument by a 'Packet *p' as this was
the only user left of the data pointer.
6 years ago
Victor Julien d78c6ff714 detect/thread: ctx info is allowed to have NULL data 6 years ago
Victor Julien 3f253cb8c0 mpm: rename internal id for ac-tile to ac-ks 7 years ago
Victor Julien 477e46da78 detect/http-client-body: convert to inspect api v2 7 years ago
Victor Julien 76fd666cad detect/http_raw_header: move tests into tests/ 7 years ago
Victor Julien 10e2731f18 detect/http-uri: move tests into tests/ 7 years ago
Victor Julien e30212c5d8 detect: fix crash during startup with malformed yaml
detect-engine:
  custom-values:
    toclient-groups: 200
    toserver-groups: 200

Bug #2745
7 years ago
Victor Julien ad1945aae4 detect: fix content inspection flags
Fix generic inspect function content inspection flags so that
streaming buffers work correctly.
7 years ago
Victor Julien 0b3220a0df detect: improve inspect buffer handling
Fix and Optimize cleanup. For the simple single inspect buffer optimize
the cleanup by keeping track of the actually used buffers. This avoid
looping over unused buffers.

Fix the case of cleaning not being done after a tx if the next tx is
also inspected in the context of the same packet.

Fix cleanup of the multi-inspect buffers. Optimize in 2 ways. First
like with single keep track of which multi-inspect buffers have been
used. Second, keep a max of the buffers within a multi-inspect buffer.
Use this max to limit (nested) looping.
7 years ago
Victor Julien 506f299b09 coverity: suppress warnings 7 years ago
Victor Julien 575fb69a06 coverity: suppress warning for intentional code 7 years ago
Victor Julien 35c5ae3458 detect: limit flush logic to sigs that need it
Limit the early 'flush' logic to sigs that actually need to match
on both stream and http bodies.
7 years ago
Victor Julien c0adff3770 detect: remove STATE_MATCH flag use at runtime
Instead, use it only at init time and use Signature::app_inspect
directly at runtime.
7 years ago
Victor Julien 12fec46d13 multi-tenant: introduce device selector
Add device to tenant mapping support:

  mappings:
  - device: ens5f0
    tenant-id: 1
  - device: ens5f1
    tenant-id: 23

Implemented by assigning the tenant id to the 'livedev', which means
it's only supported for capture methods that use the livedev API.

It's also currently not supported for IPS. In a case like 'eth0 -> eth1'
it's unclear which tenant should be used for the return traffic in a
flow, where the incoming device is 'eth1'.
7 years ago
Victor Julien bfa884c956 detect: avoid potential use-after-free in error path 7 years ago
Victor Julien d062089c60 detect: clean up counter registration 7 years ago
Victor Julien df88c048ba detect: fix delayed detect
Last multi-detect changes broken delayed-detect by refusing to reload
a 'stub' detect engine. This patch distinguishes between a stub for
multi-tenancy and for delayed detect.
7 years ago
Victor Julien 5bcae21653 detect: reload-rules shouldn't reload a stub 7 years ago
Victor Julien 4eaec2dff0 detect/debug: suppress noisy info messages 7 years ago
Victor Julien 8c918a4bea detect/multi-tenant: fix mix of default detect engine and tenants 7 years ago
Victor Julien 6e9d81289d detect: make detect engine types explicit
There are 3 types of detect engine objects:
    1. normal
       The normal detection engine if no multi-tenancy is in use

    2. tenant
       A per tenant detection engine

    3. stub
       A stub (or minimal as it was called before) detect engine
       that is needed to have something in place when there are
       only tenants.

       A stub is also used in case of 'delayed detect', where we
       need a minimal detect engine to start up which is replaced
       by a full (normal type) detect engine after startup.

This patch adds a new field 'type' to the DetectEngineCtx object
to distinguish between the types. This replaces the boolean 'minimal'.
7 years ago
Victor Julien d377eba254 detect/multi-tenant: fix removing of tenant id 0 7 years ago
Victor Julien 4866295bfb detect/tenants: fix crash when adding mapping
When no tenants and mappings are defined in 'live' mode, adding a
mapping resulted in a crash.
7 years ago
Victor Julien 663549d02c detect: remove lock from global keyword logic
The global keyword registration and per thread init handling used
the lock from the DetectEngineMasterCtx. This lead to a dead lock
situation at multi-tenancy tenant reloads.

The lock was unnecessary however, as the only time the registration
list is updated is at engine initialization. At that time Suricata
is still running in a single thread. After this, the data structure
doesn't change anymore.

Bug #2516.
7 years ago
Mats Klepsland 321603de37 detect-engine: add DetectEngineCtx to setup callback function
Add detect engine context as variable to setup callback function
in 'DetectBufferTypeRegisterSetupCallback'.
7 years ago
Giuseppe Longo c620fc3dc4 detect-engine: free events
Events are stored in a detection engine but actually
they are not freed.
7 years ago
Victor Julien d14e51a4aa detect/content: pass START/END flags to inspection 8 years ago
Victor Julien 7823ef721f file_data: update to API v2
As we can have multiple files per TX we use the multi inspect
buffer support.

By using this API file_data supports transforms.

Redo part of the flash decompression as a hard coded built-in sort
of transform.
8 years ago
Victor Julien d64785274e detect/prefilter: move hash into detect engine ctx 8 years ago
Victor Julien efbd901385 detect: move mpm engines into detect engine ctx
This allows safe registration at runtime.
8 years ago
Victor Julien ad16925bc9 detect/inspect engines: copy to detect engine ctx
Register rule-time engines in the detect engine. This is necessary
now that rule parsing can create new buffers.
8 years ago
Victor Julien 0de86211c6 detect: register dynamic buffers into de_ctx
Register buffers that are created during rule parsing. Currently
this means an existing buffer with one or more transformations.
8 years ago
Victor Julien a499a44f7a detect: move buffer type map into detect ctx
Move previously global table into detect engine ctx. Now that we
can register buffers at rule loading time we need to take concurrency
into account.

Move DetectBufferType to detect.h and update DetectBufferCtx API calls
to include a detect engine ctx reference.
8 years ago
Victor Julien f6e5cb1db6 detect: prefilter/inspect API v2, with transforms
Introduce InspectionBuffer a structure for passing data between
prefilters, transforms and inspection engines.

At rule parsing time, we'll register new unique 'DetectBufferType's
for a 'parent' buffer (e.g. pure file_data) with its transformations.
Each unique combination of buffer with transformations gets it's
own buffer id.

Similarly, mpm registration and inspect engine registration will be
copied from the 'parent' (again, e.g. pure file_data) to the new id's.

The transforms are called from within the prefilter engines themselves.

Provide generic MPM matching and setup callbacks. Can be used by
keywords to avoid needless code duplication. Supports transformations.

Use unique name for profiling, to distinguish between pure buffers
and buffers with transformation.

Add new registration calls for mpm/prefilters and inspect engines.

Inspect engine api v2: Pass engine to itself. Add generic engine that
uses GetData callback and other registered settings.

The generic engine should be usable for every 'simple' case where
there is just a single non-streaming buffer. For example HTTP uri.

The v2 API assumes that registered MPM implements transformations.

Add util func to set new transform in rule and add util funcs for rule
parsing.
8 years ago
Giuseppe Longo 884e051671 detect-engine-hsbd: decompress swf files
This checks if a buffer is a swf file and try
to decompress it, if decompression is enabled.
8 years ago
Giuseppe Longo 822faa08f8 detect: set events in inspection phase
During the inspection phase actually is not possible to catch
an error if it occurs.
This patch permits to store events in the detection engine
such that we can match on events and catch them.
8 years ago
Eric Leblond 9864552484 detect-metadata: add a string storage to de_ctx
To avoid to have a lot of string allocations, we use a hash table
stored in de_ctx to point to existing string instead of duplicating
them.
8 years ago
Eric Leblond 6bf00ab289 output-json-alert: conditionaly output metadata
Metadata of the signature can now conditionaly put in the alert
events. This will allow user to get more context about the events
generated by the alert.

detect-metadata: conditional parsing

Only parses metadata if an output module will use the information.
Patch also adds a unittest to check metadata is not parsed if not
asked to.

output-json-alert: optional output keys as array

Update rule metadata configuration to have an option to output
value as array. Also adds an option to log only a series of keys
as array. This is useful in the case of some ruleset where from
instance the `tag` key is used multiple time.

(Jason Ish) rule metadata: always log as lists

After review of rule metadata, we can't make assumptions
on what should be a list or not. So log everything as a list.
8 years ago
Victor Julien 2e4305f504 detect: minor cleanup 8 years ago
Victor Julien cf2feeecf4 detect/prefilter: redo profiling 8 years ago
Victor Julien 31648913e6 detect/profiling: postpone setup
Do this to allow for including of runtime buffer registrations.
8 years ago
Victor Julien 1df00749df detect: fix multiple files per tx inspect
Fix the inspection of multiple files in a single TX, where new files
may be added to the TX after inspection started.

Assign the hard coded id DE_STATE_FLAG_FILE_INSPECT to the file
inspect engine.

Make sure that sigs that do file inspection and don't match on the
current file always store a detailed state. This state will include
the DE_STATE_FLAG_FILE_INSPECT flag.

When the app-layer indicates a new file is available, for each sig
that has the DE_STATE_FLAG_FILE_INSPECT flag set, reset part of the
state so that the sig is evaluated again.
8 years ago
Victor Julien af51e0f5a1 detect: rewrite of the detect engine
Use per tx detect_flags to track prefilter. Detect flags are used for 2
things:
1. marking tx as fully inspected
2. tracking already run prefilter (incl mpm) engines

This supercedes the MpmIDs API for directionless tracking
of the prefilter engines.

When we have no SGH we have to flag the txs that are 'complete'
as inspected as well.

Special handling for the stream engine:

If a rule mixes TX inspection and STREAM inspection, we can encounter
the case where the rule is evaluated against multiple transactions
during a single inspection run. As the stream data is exactly the same
for each of those runs, it's wasteful to rerun inspection of the stream
portion of the rule.

This patch enables caching of the stream 'inspect engine' result in
the local 'RuleMatchCandidateTx' array. This is valid only during the
live of a single inspection run.

Remove stateful inspection from 'mask' (SignatureMask). The mask wasn't
used in most cases for those rules anyway, as there we rely on the
prefilter. Add a alproto check to catch the remaining cases.

When building the active non-mpm/non-prefilter list check not just
the mask, but also the alproto. This especially helps stateful rules
with negated mpm.

Simplify AppLayerParserHasDecoderEvents usage in detection to only
return true if protocol detection events are set. Other detection is done
in inspect engines.

Move rule group lookup and handling into it's own function. Handle
'post lookup' tasks immediately, instead of after the first detect
run. The tasks were independent of the initial detection.

Many cleanups and much refactoring.
8 years ago
Victor Julien 746638b220 cuda: remove
Remove CUDA support as it has been broken for a long time.

Ticket #2382.
8 years ago
Victor Julien ac0ae2dcd1 file_data: smtp file_data to generic file_data
Generalize the SMTP file_data inspection into a 'files'
file_data inspection that can be used for any protocol
that uses the File API.
8 years ago
Andreas Herz d6c3a36113 detect-engine: add missing mutex unlock 8 years ago
Giuseppe Longo 1328ecb8f6 detect: save invalid rules
This keeps the invalid rules in string format into a list,
added in DetectEngineCtx.
8 years ago
Giuseppe Longo 1567f84cd2 detect-engine: remove DONE state
Remove the DONE state to fix a problem with state not being
changed correctly when multiple reload were done. As DONE was
not really useful, we can remove it.
8 years ago
Giuseppe Longo 56000acefb detect-engine: add reload time/rules stats
This patch adds the following stats for
the detect engine:
- time of the last reload
- number of rules loaded
- number of rules failed
8 years ago
Victor Julien b56c0b524b detect: error out on invalid detect.profile option
Bug #891.
8 years ago
Victor Julien ab1200fbd7 compiler: more strict compiler warnings
Set flags by default:

    -Wmissing-prototypes
    -Wmissing-declarations
    -Wstrict-prototypes
    -Wwrite-strings
    -Wcast-align
    -Wbad-function-cast
    -Wformat-security
    -Wno-format-nonliteral
    -Wmissing-format-attribute
    -funsigned-char

Fix minor compiler warnings for these new flags on gcc and clang.
8 years ago
Victor Julien c65a119cc0 debug: suppress notice message 8 years ago
Victor Julien 1bbf555318 detect: improve stateful detection
Now that MPM runs when the TX progress is right, stateful detection
operates differently.

Changes:

1. raw stream inspection is now also an inspect engine

   Since this engine doesn't take the transactions into account, it
   could potentially run multiple times on the same data. To avoid
   this, basic result caching is in place.

2. the engines are sorted by progress, but the 'MPM' engine is first
   even if the progress is higher

   If MPM flags a rule to be inspected, the inspect engine for that
   buffer runs first. If this step fails, the rule is no longer
   evaluated. No state is stored.
8 years ago
Victor Julien d304be5bc3 detect: register progress in inspect engines
Register required progress so we can stop inspecting as soon
as the progress isn't far enough yet.
8 years ago
Victor Julien aba9cd7d02 stream inspection: add debug counters 8 years ago
Victor Julien c859d39f4f coverity: suppress CID 1400648 9 years ago
Victor Julien 511e804915 detect: fix missing unlock in error path 9 years ago
Victor Julien 22f3205664 var-names: expose outside of detect engine
Until now variable names, such as flowbit names, were local to a detect
engine. This made sense as they were only ever used in that context.

For the purpose of logging these names, this needs a different approach.
The loggers live outside of the detect engine. Also, in the case of
reloads and multi-tenancy, there are even multiple detect engines, so
it would be even more tricky to access them from the outside.

This patch brings a new approach. A any time, there is a single active
hash table mapping the variable names and their id's. For multiple
tenants the table is shared between tenants.

The table is set up in a 'staging' area, where locking makes sure that
multiple loading threads don't mess things up. Then when the preparing
of a detection engine is ready, but before the detect threads are made
aware of the new detect engine, the active varname hash is swapped with
the staging instance.

For this to work, all the mappings from the 'current' or active mapping
are added to the staging table.

After the threads have reloaded and the new detection engine is active,
the old table can be freed.

For multi tenancy things are similar. The staging area is used for
setting up until the new detection engines / tenants are applied to
the system.

This patch also changes the variable 'id'/'idx' field to uint32_t. Due
to data structure padding and alignment, this should have no practical
drawback while allowing for a lot more vars.
9 years ago
Victor Julien 43cc06eabe detect: use engine version instead of id
Use engine version based on global detect engine master. This is
incremented between reloads.
9 years ago
Victor Julien fe415ae518 detect: remove DMATCH list 9 years ago
Victor Julien ad238121e3 detect: remove the AMATCH list 9 years ago
Victor Julien f2fc5a255f http_header: convert to use common code 9 years ago
Victor Julien cf9678d926 detect: global registery for keyword thread data
Some keywords need a scratch space where they can do store the results
of expensive operations that remain valid for the time of a packets
journey through the detection engine.

An example is the reconstructed 'http_header' field, that is needed
in MPM, and then for each rule that manually inspects it. Storing this
data in the flow is a waste, and reconstructing multiple times on
demand as well.

This API allows for registering a keyword with an init and free function.

It it mean to be used an initialization time, when the keyword is
registered.
9 years ago
Victor Julien 8bd1422948 detect: detect engine registration cleanup 9 years ago
Victor Julien 49fbd28ceb detect: cleanup built-in list id's 9 years ago
Victor Julien 40851eecf0 template: dynamic buffer 9 years ago
Victor Julien 815120896b app-layer-events: dynamic list 9 years ago
Victor Julien b68343e372 files: use dynamic list 9 years ago
Victor Julien e4bfdd53c2 cip/enip: dynamic buffer 9 years ago
Victor Julien f2393c1ae0 modbus: dynamic buffer 9 years ago
Victor Julien cfdd934aba dnp3: dynamic buffers/lists 9 years ago
Victor Julien 9ba386a141 tls: dynamic buffers 9 years ago
Victor Julien d9b3ae6cd6 dns: use dynamic buffers 9 years ago
Victor Julien 6f867c3c73 http_raw_uri: dynamic buffer 9 years ago
Victor Julien ee55aefa1c http_client_body: dynamic buffer 9 years ago
Victor Julien e7d5e845c7 http_header / http_raw_header: dynamic buffers 9 years ago
Victor Julien b694d96e22 http_stat_msg: dynamic buffer 9 years ago
Victor Julien 7e3ab4f5ea http_stat_code: dynamic buffer 9 years ago
Victor Julien 128b59d4f6 http_raw_host: dynamic buffer 9 years ago
Victor Julien 296c275e23 http_host: dynamic buffer 9 years ago
Victor Julien 67b7d9734e http_cookie: dynamic buffer 9 years ago
Victor Julien 54604c7bf2 http_user_agent: dynamic buffer 9 years ago
Victor Julien 9262fa3dcf http_response_line: dynamic buffer 9 years ago
Victor Julien 6346a074a7 http_uri: dynamic buffer
Clean up tests
9 years ago
Victor Julien e34102d67a http_method: make list dynamic 9 years ago
Victor Julien 6bd37611ee file_data: dynamic buffer 9 years ago
Victor Julien 7052f9b933 http_request_line: dynamic buffer 9 years ago
Victor Julien 779d40cedf detect: remove hardcoded sm_list logic from setup
Introduce utility functions to aid this.
9 years ago
Victor Julien 04592efb76 detect: buffer type API
To replace the hardcoded SigMatch list id's, use this API to register
and query lists by name.

Also allow for registering descriptions and whether mpm is supported.

Registration is only allowed at startup.
9 years ago
Victor Julien 58e1180efe detect: inspect engine setup cleanup 9 years ago
Victor Julien 0a5ae415b8 detect: shrink Signature::sm_arrays
Signature::sm_arrays now only contains 'built-in' lists, and so is
sized appropriately.
9 years ago
Victor Julien 59303d1fbb threshold: fix and redo tests 9 years ago
Victor Julien 6f7e4adbe8 detect: improve memory handling & comments 9 years ago
Victor Julien 8edc954e82 detect: get rid of Signature::sm_lists
Instead use the lists in init_data during setup and the SigMatchData
arrays during runtime.
9 years ago
Victor Julien f281481b67 detect: use detect list passed to generic funcs
Until now the GenericList users used hardcoded list id's.
9 years ago
Victor Julien bd456076a8 detect: pass SigMatchData to inspect functions 9 years ago
Victor Julien a0fe67a3c0 detect: template list in engine 9 years ago
Victor Julien da7c816c7c detect: enip/cip list in engine 9 years ago
Victor Julien e94a7bddb3 detect: modbus list in engine 9 years ago
Victor Julien 7f7d4296da detect: file list in engine 9 years ago
Victor Julien f5adccba1d detect: app-event list in engine 9 years ago
Victor Julien 747dbf92ce detect: dns & tls lists in engine 9 years ago
Victor Julien 5a2e568385 detect: http lists in engine 9 years ago
Victor Julien 1ee32da2ca detect-engine: memory handling of sm_lists
For lists that are registered multiple times, like http_header and
http_cookie, making the engines owner of the lists is complicated.
Multiple engines in a sig may be pointing to the same list. To
address this the 'free' code needs to be extra careful about not
double freeing, so it takes an approach to first fill an array
of the to-free pointers before freeing them.
9 years ago
Victor Julien cf42fbf51f detect: use InspectEngineFuncPtr in inspect engines
Replace explicit function pointer use by InspectEngineFuncPtr typedef
9 years ago
Victor Julien 5f7e096be4 detect: shrink inspect engine by using 'id' as state flag 9 years ago
Victor Julien bfd4bc8233 detect: constify Signature/SigMatch use at runtime 9 years ago
Victor Julien a44da9f5cb detect: simplify SIG_FLAG_STATE_MATCH set logic 9 years ago
Victor Julien 31a96d5a79 detect: make tenant loading less verbose 9 years ago
Victor Julien f2f0f84cca detect: fix -Wshadow warning 9 years ago
Victor Julien 070a6caaf3 app engines: fix -Wshadow warning 9 years ago
Jason Ish 1a31bded4a DNP3: dnp3_data, dnp3_func, dnp3_ind, dnp3_obj rule keywords 9 years ago
Victor Julien db1c47cb6e multi-tenant: make less verbose 9 years ago
Victor Julien 059b25b564 detect: suppress debug message for reloads 9 years ago
kwong a3ffebd835 Adding SCADA EtherNet/IP and CIP protocol support
Add support for the ENIP/CIP Industrial protocol

This is an app layer implementation which uses the "enip" protocol
and "cip_service" and "enip_command" keywords

Implements AFL entry points
9 years ago
Victor Julien 58ac4027ef detect: clean up inspect engine registration 9 years ago
Victor Julien a24870f29f detect app-layer-event: clean up registration
Move engine and registration into the keyword file.

Register as 'ALPROTO_UNKNOWN' instead of per alproto. The
registration will only apply it to those rules that have
events set.
9 years ago
Victor Julien 9e35fa7f41 detect: remove empty app registration table 9 years ago
Victor Julien 8a0bea872c template_buffer: register inspect engine from keyword 9 years ago
Victor Julien 6f253e1ea7 file detect: register inspect engines from keyword 9 years ago
Victor Julien 08d0fe0916 modbus detect: register inspect engine from keyword 9 years ago
Victor Julien 2db094ab7a dns detect: register inspect engine from keyword 9 years ago
Victor Julien c9bb762f64 tls_cert_issuer: register inspect engine from keyword 9 years ago
Victor Julien e28e98bcaa tls_cert_subject: register inspect engine from keyword 9 years ago
Victor Julien a87c196b60 tls_sni: register inspect engine from keyword 9 years ago
Victor Julien 200a4c1593 http_stat_code: register inspect engine from keyword 9 years ago
Victor Julien cd705752db http_stat_msg: register inspect engine from keyword 9 years ago
Victor Julien 20e93ba419 file_data: register inspect engine from keyword 9 years ago
Victor Julien 0496b3f6a5 http_raw_host: register inspect engine from keyword 9 years ago
Victor Julien a00629ab55 http_host: register inspect engine from keyword 9 years ago
Victor Julien edb2936998 http_user_agent: register inspect engine from keyword 9 years ago
Victor Julien fc857c5455 http_raw_uri: register inspect engine from keyword 9 years ago
Victor Julien b1adea6eee http_cookie: register inspect engine from keyword 9 years ago
Victor Julien cd8b1b0b4c http_method: register inspect engine from keyword 9 years ago
Victor Julien b314829614 http_raw_header: register inspect engine from keyword 9 years ago
Victor Julien eb19eb3fe4 http_header: register inspect engine from keyword 9 years ago
Victor Julien 4096f76b1b http_client_body: register inspect engine from keyword 9 years ago
Victor Julien b96c2c5db5 http_uri: register inspect engine from keyword 9 years ago
Victor Julien cc96fedb90 http_response_line: register inspect engine from keyword 9 years ago
Victor Julien 0feeb8d538 http_request_line: register inspect engine from keyword 9 years ago
Victor Julien 5bde86b0e8 detect-engine: new registration call
Make it more in line with MPM registration.
9 years ago
Victor Julien 5f994756e6 detect-engine: improved inspect engines
Inspect engines are called per signature per sigmatch list. Most
wrap around DetectEngineContentInspection, but it's more generic.

Until now, the inspect engines were setup in a large per ipproto,
per alproto, per direction table. For stateful inspection each
engine needed a global flag.

This approach had a number of issues:
1. inefficient: each inspection round walked the table and then
   checked if the inspect engine was even needed for the current
   rule.
2. clumsy registration with global flag registration.
3. global flag space was approaching the need for 64 bits
4. duplicate registration for alprotos supporting both TCP and
   TCP (DNS).

This patch introduces a new approach.

First, it does away with the per ipproto engines. This wasn't used.

Second, it adds a per signature list of inspect engine containing
only those engines that actually apply to the rule.

Third, it gets rid of the global flags and replaces it with flags
assigned per rule per engine.
9 years ago
Victor Julien f1e3840516 http_response_body: implement keyword with mpm
Implemented as 'stickybuffer'.
9 years ago
Victor Julien 4c98b6cef3 http_request_line: implement keyword and mpm
Implemented as 'stickybuffer'.

Move all logic into the keyword file and remove bad tests that tested
URI instead of request line.
9 years ago
Victor Julien 5b2e36a1b0 mpm: add App Layer MPM registery
Register keywords globally at start up.

Create a map of the registery per detection engine. This we need because
the sgh_mpm_context value is set per detect engine.

Remove APP_MPMS_MAX.
9 years ago
Victor Julien 125603871b detect: config opt to enable keyword prefilters 9 years ago
Victor Julien 17bc0299fe detect: rename non_mpm lists/vars to non_pf
Rename to non_pf: non prefilter.
9 years ago
Mats Klepsland dc8e0b3cf2 detect: add detect engine for tls validity keywords
Add detect engine for tls validity keywords (tls_cert_notbefore and
tls_cert_notafter).
9 years ago
Mats Klepsland d91664d67a detect-dns: move DetectEngineInspectGenericList to detect-engine.c
Move DetectEngineInspectGenericList from detect-engine-dns.c to
detect-engine.c to enable it to be used other places as well.
9 years ago
Mats Klepsland 4172c4c8ac tls: add (mpm) keyword tls_cert_subject
This keyword is a replacement for tls.subject.
9 years ago
Mats Klepsland 9b2717799c tls: add (mpm) keyword tls_cert_issuer
This keyword is a replacement for tls.issuerdn.
9 years ago
Victor Julien 215d0d54c7 detect: optimize rule address parsing
Many rules have the same address vars, so instead of parsing them
each time use a hash to store the string and the parsed result.

Rules now reference the stored result in the hash table.
9 years ago
Eric Leblond f2d1e93e65 unix-socket: add auto mode
When running in live mode, the new default 'auto' value of
unix-command.enabled causes unix-command to be activated. This
will allow users of live capture to benefit from the feature and
result in no side effect for user running in offline capture.
9 years ago
Victor Julien b3b78d4326 detect: log earlier that rule reload is happening 9 years ago
Victor Julien a88359dcf0 detect: get proper legacy custom values. Issue #1804 9 years ago
Victor Julien b3bf7a5729 output: introduce config and perf output levels
Goal is to reduce info output
9 years ago
Victor Julien 72d3ea6552 detect: make pattern matcher messages less verbose 9 years ago
Victor Julien 36535cbc61 yaml: remove conf_filename global
conf_filename was a global pointer to the filename of the yaml.

Move into SCInstance. This reduces it's scope and cleans up the code.
9 years ago
Justin Viiret d807bf4e8a detect-engine: log MPM/SPM matchers being used 9 years ago
Justin Viiret c9d0d6f698 mpm: add "auto" default for mpm-algo
Setting mpm-algo to "auto" will use "hs" if Suricata was built against
Hyperscan, and "ac" otherwise (or "ac-tile" on Tilera platforms).
9 years ago
Victor Julien e43ce0a9ec file: switch to streaming buffer API
Make the file storage use the streaming buffer API.

As the individual file chunks were not needed by themselves, this
approach uses a chunkless implementation.
9 years ago
Victor Julien 46e55f1e34 http body handling: use streaming buffer API
Convert HTTP body handling to use the Streaming Buffer API. This means
the HtpBodyChunks no longer maintain their own data segments, but
instead add their data to the StreamingBuffer instance in the HtpBody
structure.

In case the HtpBodyChunk needs to access it's data it can do so still
through the Streaming Buffer API.

Updates & simplifies the various users of the reassembled bodies:
multipart parsing and the detection engine.
9 years ago
Victor Julien 52d500c670 flowworker: initial support
Initial version of the 'FlowWorker' thread module. This module
combines Flow handling, TCP handling, App layer handling and
Detection in a single module. It does all flow related processing
under a single flow lock.
9 years ago
Mats Klepsland a13df67864 detect: add (mpm) keyword for tls_sni
Match on server name indication (SNI) extension in TLS using tls_sni
keyword, e.g:

alert tls any any -> any any (msg:"SNI test"; tls_sni;
        content:"example.com"; sid:12345;)
9 years ago
Justin Viiret cce2d114e8 spm: add and use new SPM API
This new API allows for different SPM implementations, using a function
pointer table like that used for MPM.

This change also switches over the paths that make use of
DetectContentData (which previously used BoyerMoore directly) to the new
API.
9 years ago