Commit Graph

5514 Commits (def2b58725e6876abecceccecb096ba005eb34bc)

Author SHA1 Message Date
Victor Julien 22142d9b8b threshold: cleanups
Use pcre_copy_substring in the most common part of threshold rule
parsing. Minor other cleanups.
10 years ago
Victor Julien e86e1e0282 reference: use pcre_copy_substring 10 years ago
Victor Julien cc38df8074 classification: cleanups
Reduce hash table size as regular classification files are usually
below 100 in size. It's not performance critical anyway.

Convert pcre_get_substring calls to pcre_copy_substring.
10 years ago
Victor Julien deb98fab0c packet pool: init pool for autofp workers as well
Introduce a new 'PacketPoolInitEmpty' as these pools will not often
need packets.

Also, don't double 'destroy' in the main thread.
10 years ago
Victor Julien 1884227019 autofp: reduce flow storage space requirement
Use int16_t instead of a regular int to safe 2 bytes per flow.
10 years ago
Victor Julien 643c30f67f threading: add missing locks to threading storage
Lock threading root lock when walking or updating the threading
storage for 'ThreadVars' objects.
10 years ago
maxtors e7658fd4f4 Added new WebDAV http methods.
Added more WebDAV functions. A complete list of what http
methods libhtp can handle can be found at:
https://github.com/OISF/libhtp/blob/0.5.x/htp/htp_core.h#L260.
So now the methods array reflects these available functions.

The comments have also been changed to reflect the desired style.
10 years ago
maxtors 3b26b07963 Reworked pattern registration structure and feedback issues.
1) Reworked pattern registration for http methods and versions.

Instead of being a manual and verbose action of adding one
and one http method with N-amount if prefix spacings and
the same for HTTP versions (eg. HTTP/1.1) i moved it all
to be loop based actions reading values from char arrays.

In the future all that is needed is to add new methods
to the arrays and they will be added as a pattern.

2) Modified pattern registration after feedback.

Changed variable used in snprintf for http method registration
Should have been size of dest buffer at not another var (catsize)
that i had created. Also removed this variable.

Fixed a typo in the comment for registering http versions.
TO_CIENT -> TO_CLIENT.
10 years ago
maxtors 7a9e963619 Added http methods for webdav (rfc4918).
Included the following methods to app-layer-htp.c:
- PROPFIND
- PROPPATCH
- MKCOL
- COPY
- MOVE
- LOCK
- UNLOCK
10 years ago
maxtors a7abaf538c Added DELETE and PATCH http methods to app-layer-htp.c
Added two new http methods to the http decoding done in
app-layer-htp.c so that these messages will be decoded as well.
10 years ago
Victor Julien 915eb992ca app-layer: update protocol detection tests
AppLayerTest09, AppLayerTest10 and AppLayerTest11 depended on a max
protocol detection pattern size of < 17. Update the tests to pass one
extra byte to the app layer. This makes the protocol detection code
flag the session as 'proto detection completed' again.
10 years ago
David Maciejak 48971afdfb Fix Unit ID check 10 years ago
Eric Leblond f35001a614 util-running-mode: setup config file
Without that we get warning message.
10 years ago
Eric Leblond 6f5c4c12e1 list keywords: fix regression on app layer name
It was not anymore displayed.
10 years ago
Eric Leblond d98060410c list keywords: restore 2.0 similar output
Unset was previously displayed if the application layer was not
set. Code before this patch was displaying '(null)' which is not
user friendly.
10 years ago
Alexander Gozman 568635c2e1 Reject ICMP traffic, not only TCP/UDP 10 years ago
Jason Ish 1f9d9256e7 spelling - dubbed -> duped (for duplicate) 10 years ago
Jason Ish 8911b04077 DetectGidSet - safer stripping of quotes.
Discovered by AFL when using a rule like:
    alert tcp any any -> any any (content:"ABC"; gid:";)
resulting a negative array index.
10 years ago
Jason Ish da88b3b787 DetectSidSetup - safer stripping of quotes.
Discovered by AFL when using a rule like:
    alert tcp any any -> any any (content:"ABC"; sid:";)
would result in a negative array index.
10 years ago
Jason Ish ca60d8c370 ParseSizeString - don't abort on unknown unit type. 10 years ago
Victor Julien 00d6298666 host: handle timeout
Set expiration for regular hostbits keyword.
10 years ago
Victor Julien ab7882fbf3 ippair: handle timeouts in the flow manager
Like host and defrag, handle timeouts of ippair in the first flow
manager thread.
10 years ago
Victor Julien e8a5925012 ippair: implement basic timeout check
The only user is the xbits subsys, so it's timeout controls all.
10 years ago
Victor Julien f2349e10ad hosts: consider hostbits/xbits status in timeout
Consider the host's xbits expiry status when checking the host for
timeout. If a single active non-expired bit is found, the host won't
be timeout just yet.
10 years ago
Victor Julien 67dd5c0430 host: implement hostbits/xbits expire 10 years ago
Victor Julien ca1f3e68d4 ippair: implement xbits expiration 10 years ago
Victor Julien fbdcffabc2 xbits: expire (first steps) 10 years ago
Victor Julien 7b79b9458d xbits: implement ip_src/ip_dst support
Just a wrapper for regular 'hostbits'.
10 years ago
Victor Julien 57d93cbcfe hostbits: prepare for xbits integration 10 years ago
Victor Julien 8e6453faac xbits: prepare for non-ippair support 10 years ago
Victor Julien dea8dda3c1 vars: redo var name <-> idx handling
Can't use sm type anymore as multiple var carriers (host/flow/etc) will
all have xbits.
10 years ago
Victor Julien f77c3d9a2c xbits: hostbits use xbits type
Make hostbits use xbits type.
10 years ago
Victor Julien a716dded04 xbit: move to util-var
We can reuse the type for hostbits and flowbits as well.
10 years ago
Victor Julien 61cb2abc8d ippair: xbit implementation
detect xbits for ippair: initial implementation
10 years ago
Victor Julien 4c6d564211 hostbits: unittest cleanups 10 years ago
Victor Julien 2cb0df7cf9 hostbits: allow for src/dst parameter
By default, the hostbit checks the 'src' host. By adding the dst option
the dst host is used instead.
10 years ago
Victor Julien be725a5ba2 hostbits: add more tests, include sigorder 10 years ago
Victor Julien d67fd306f8 hostbits: implement sigorder
Like with flowbits, make sure that 'set's are evaluated before
'isset's, etc.
10 years ago
Victor Julien 5c880377ae detect: hostbits keyword
Per host bit similar to 'flowbits'.

Initial code that uses just the 'src' ip for the operations.
10 years ago
Victor Julien 99ae643e4e ippair: track ippairs, enable tests 10 years ago
Alexander Gozman 45ba20bcb0 DetectAddressParseString(): fix IPv6 address handling 11 years ago
Eric Leblond 605ef33972 runmodes: improve listing output
This patch removes some multiple line displayed before the
information about unix socket mode.
11 years ago
Eric Leblond e5ae808b0f runmodes: fix typo in output 11 years ago
Eric Leblond ef3290bb2e flow-manager: don't set cpu affinity twice
It is already set in the thread creation function.
11 years ago
Victor Julien 8e83d0073e stream: fix bad last_ack update leading to gaps
A bad last_ack update where it would be set beyond next_seq could
lead to rejection of valid segments and thus stream gaps.

Update tests to reflect new last_ack/next_seq behaviour.
11 years ago
Victor Julien 4e177bc9d6 detect-state: cleanups and comments 11 years ago
Victor Julien 304c711a27 detect-state: use f->protomap instead of FlowGetProtoMapping(f->proto) 11 years ago
Victor Julien e1efa7a0b5 detect-flowbits: locking update
Make matches that can be in the POSTMATCH list aware of the lock hint.
11 years ago
Victor Julien 4609495c60 detect-flowint: conditional locking 11 years ago
Victor Julien 5111aa2ec0 detect-state: handle 'post match' locking
The post match list was called with an unlocked flow until now.
However, recent de_state handling updates changed this. The stateful
detection code can now call the post match functions while keeping
the flow locked. The normal detection code still calls it with an
unlocked flow.

This patch adds a hint to the DetectEngineThreadCtx called
'flow_locked' that is set to true if the caller has already locked
the flow.
11 years ago
Victor Julien 9bcb02119f detect-flowint: fix unlocked flow access
Some of the access to the flow and to structures retrieved from the
flow was unlocked.

This patch changes the logic to be wrapped in lock calls.
11 years ago
Victor Julien 18f0351d9b detect-state: fix profiling 11 years ago
Victor Julien 0b262cbfda dns: optimize tx list walks
The detection engine and log engines can walk the tx list indirectly,
by looping AppLayerParserGetTx. This would lead to new list walks in
the DNS tx list though. Leading to bad performance.

This patch stores the last returned tx and uses that to determine if
the next tx is what we need next. If so, we can return that w/o list
walk.
11 years ago
Victor Julien 0704ece4d7 detect-reload: enable unconditionally
Reloading is available unconditionally now.
11 years ago
Victor Julien 724c7044e1 detect-reload: 0 detect threads is no error
The reload code would consider 0 detect threads to be an error,
but it's not in case of unix socket mode.
11 years ago
Victor Julien 7c9e015748 unix-socket: implement reload-rules
Implement the reload-rules unix socket command. The unix command
thread signals the main thread to do the reload and it waits for
it to complete.
11 years ago
Victor Julien 71d01f06b9 detect reload: load config
Load the YAML into a prefix "detect-engine-reloads.N" where N is the
reload counter. This way we can load the updated config w/o overwriting
the current one.
11 years ago
Victor Julien b51075e804 detect: remove config at prefix
Remove config at prefix when freeing a detect engine.
11 years ago
Victor Julien 85e12f2bc6 rule vars: support prefix
Support the detection engine's prefix when retrieving rule vars.
11 years ago
Victor Julien 0bc27c7dc7 rule-vars: take detect engine as arg 11 years ago
Victor Julien 3083f51cc6 detect:pass DetectEngineCtx to port parsing
Preparation for prefix handling in port parsing.
11 years ago
Victor Julien 2be9ccfe9c detect: pass DetectEngineCtx to address parsing
Preparation for prefix handling in address parsing.
11 years ago
Victor Julien 7108085d33 detect: initialize detection engine by prefix
Initalize detection engine by configuration prefix.

    DetectEngineCtxInitWithPrefix(const char *prefix)

Takes the detection engine configuration from:
<prefix>.<config>

If prefix is NULL the regular config will be used.

Update sure that DetectLoadCompleteSigPath considers the prefix when
retrieving the configuration.
11 years ago
Victor Julien 97d77e3540 conf: add ConfYamlLoadFileWithPrefix
Add function to load a yaml file and insert it into the conf tree at
a specific prefix.

Example YAML:
somefile: myfile.txt

If loaded using ConfYamlLoadFileWithPrefix with prefix "myprefix", it
can be retrieved by the name of "myprefix.somefile".
11 years ago
Victor Julien a80cc696d7 detect: allow det_ctx->de_ctx to be NULL
When freeing det_ctx, allow de_ctx to be NULL.
11 years ago
Victor Julien c9a8262ccf detect: reload thread init cleanup
Rename the thread init function DetectEngineThreadCtxInitForLiveRuleSwap
to DetectEngineThreadCtxInitForReload and change it's logic to take the
new detection engine as argument and let it return the
DetectEngineThreadCtx or NULL on error.

The old approach used the thread init API format, but it wasn't used in
that way.
11 years ago
Victor Julien 55e7370fc5 detect reload: allow master update during reload
Add DetectEngineReference, which takes a reference to a detect engine,
and make DetectEngineThreadCtxInitForLiveRuleSwap use it. This way
reload will not depend on master staying the same. This allows master
to be updated in between w/o affecting the reload that is in progress.
11 years ago
Victor Julien b1c54a8673 detect: remove old live reload implementation
Remove code that ran the reload in it's own thread. Simplify the
signal handling.
11 years ago
Victor Julien 0c263105cd detect: move reload into main loop
Use new DetectEngineReload() function. It's called from the main loop
instead of it being spawned into it's own temporary thread. This greatly
simplifies the signal handling.

An added advantage is that this seems to improve the memory usage.

Related to bug #1358
11 years ago
Victor Julien e7882da178 detect: introduce 'minimal' detect engine
The minimal detect engine has only the minimal memory use and setup
time. It's to be used for 'delayed' detect where the first detection
engine is essentially empty.

The threads setup are also minimal.
11 years ago
Victor Julien f4617d5357 threading: remove 'dummy' slot logic
Now that delayed detect is not using it anymore, the dummy logic
can be removed.
11 years ago
Victor Julien b0cb4c17ec detect: unify delayed detect and reload
Instead of threading logic with dummy slots and all, use the regular
reload logic for delayed detect.

This means we pass a empty detect engine to the threads and then
reload (live swap) it as soon as the engine is running.
11 years ago
Victor Julien 38b349af1e runmodes: remove DetectEngineCtx passing from API
No longer pass a pointer to the current detection engine to the
runmode API calls.

Note: breaks delayed detect. Will be fixed in a future commit.
11 years ago
Victor Julien b038b6a2f8 unittests: add exception to detect engine setup
Add code to allow for unittests not following the complete api.

Update replace tests as they don't use the unittests runmode that
powers the workaround based on RunmodeIsUnittests().
11 years ago
Victor Julien d66fa1add1 detect: update detect engine management
Update detect engine management to make it easier to reload the detect
engine.

Core of the new approach is a 'master' ctx, that keeps a list of one or
more detect engines. The detect engines will not be passed to any thread
directly, but instead will only be accessed through the detect engine
thread contexts. As we can replace those atomically, replacing a detect
engine becomes easier.

Each thread keeps a reference to its detect context. When a detect engine
is replaced or removed, it's added to a free list. Once its reference
count reaches 0, it is freed.
11 years ago
Victor Julien 092ddc1853 detect: no exit on reference/classification errors
Don't exit on errors during classification and reference parsing.

Add some suppression of error messages when in unittest mode.
11 years ago
Victor Julien 49bad2cfba detect: consolidate more setup into DetectEngineCtxInit
Loading of classifications, references and action order was done
unconditionally, so can be done in one place.
11 years ago
Victor Julien 69f99245c5 unix-command: fix duplicate registration error msg 11 years ago
Victor Julien 60a49657df Bug 1401: on midstream pickup, fix packet direction
On midstream SYN/ACK pickups, we would flip the direction of packets
after the first. This meant the first (pickup) packet's direction
was wrong.

This patch fixes that.
11 years ago
Jason Ish ee7e813256 Bug 1417 - Record sequence nodes as sequences.
Nodes that are sequences weren't being recorded as such, causing
rules to fail to load.

Change sequence test name to reflect better what it tests, and
test that the sequence node is detected as a sequence.
11 years ago
Victor Julien e250040b72 detect-state: implement tx state reset for reload
In case of Detect Reload, we need to reset active tx' state.
11 years ago
Victor Julien da3e8ad8f6 detect-state: split flow and tx state
Use separate data structures for storing TX and FLOW (AMATCH) detect
state.

- move state storing into util funcs
- remove de_state_m
- simplify reset state logic on reload
11 years ago
Victor Julien 840efe17fe modbus: tx de_state 11 years ago
Victor Julien e984a57203 smtp: tx de_state 11 years ago
Victor Julien 2a23627a82 dns: implement tx de_state 11 years ago
Victor Julien 774bb90351 http: clean up tx destate at tx destroy 11 years ago
Victor Julien 6279da0fbd http: support per TX destate storage 11 years ago
Victor Julien 1cf02560c8 app-layer: per tx destate
Add API calls for storing detection state in the TX.
11 years ago
Victor Julien 866d9684ea detect-state: fix profiling 11 years ago
Victor Julien 7e75279977 detect-state: various cleanups 11 years ago
Victor Julien eec22ce19b detect-state: rip per sig detect out of ContinueDetect 11 years ago
Victor Julien bf818b8fb2 detect-state: remove DeStateResetFileInspection
It was effectively unused.
11 years ago
Victor Julien 206f9d4010 detect-state: remove redundant code 11 years ago
Victor Julien e390e24a7c detect-state: add helper to test state
Add little helper function StateIsValid() to test if the state
can be inspected safely.

Cleans up stateful detection loops.
11 years ago
Victor Julien 072ae12771 detect-state: add helper to indicate last tx
Add little helper to indicate current tx is that last we have.
11 years ago
Victor Julien b710f2dd59 detect-state: cleanup ContinueDetection
Only lock f->de_state->m when we start to access it. So after
declaration and initialization of local vars.
11 years ago
Victor Julien 54cb2b6877 detect-state: cleanup retvals
Use DETECT_ENGINE_INSPECT_SIG_* instead of 0, 1, 2 and 3.
11 years ago
Victor Julien 97cab030d7 modbus: shrink data structure 11 years ago
Victor Julien 9f1b417660 file: don't 'close' file if we need to track it 11 years ago
Victor Julien 6723d03c7e http: add inspection engine for http request line
No MPM though.
11 years ago
Victor Julien c0f265fd82 http: fix stat_msg and stat_code state tracking 11 years ago
Victor Julien 5b8c94db30 Remove spinning PacketPoolWait
PacketPoolWait in autofp can wait for considerable time. Until now
it was essentially spinning, keeping the CPU 100% busy.

This patch introduces a condition to wait in such cases.

Atomically flag pool that consumer is waiting, so that we can sync
the pending pool right away instead of waiting for the
MAX_PENDING_RETURN_PACKETS limit.
11 years ago
Victor Julien 2d7f79a62b drop json log: log out 'drop' signature
If no normal sig was logged as the 'drop' reason, try the stored
drop signature instead, this will also log out 'noalert' sigs.
11 years ago
Victor Julien 2e754ca6fa drop json: make alerts logging optional
Make logging out alerts that caused the drop optional.
11 years ago
Victor Julien 6b172bb010 drop json: add sids (if applicable)
If a drop is caused by a SID match, add it to the drop record.
11 years ago
Victor Julien 8a97bb0d04 alert json: move alert info into function
Move adding the alert info (sid,rev,gid,etc) into it's own function,
so it can be called from other outputs as well.
11 years ago
Victor Julien e9857200b3 detect: set action from utility function
Set actions that are set directly from Signatures using the new
utility function DetectSignatureApplyActions. This will apply
the actions and also store info about the 'drop' that first made
the rule drop.
11 years ago
Victor Julien c914f7bcdc detect: cleanup, remove unused order_id
No longer used, so remove.
11 years ago
Eric Leblond b8e7d3a259 flow-timeout: fix init of pseudo packet
The code was not checking if we had enough room in the direct
data. In case default_packet_size was set really small, this was
resulting in data being written over the data and causing a crash.

The patch fixes the issue by forcing an allocation if the direct
data size in the Packet is to small.
11 years ago
Eric Leblond e138a2ac1e decode: introduce PacketCallocExtPkt function
In flow timeout handling we need a function that allocate and blank
a place that will be used to put constructed packet data. This new
function has no other goal.
11 years ago
Jason Ish ab1d69fc4e When re-opening a log file on HUP, always append.
This will prevent log files that have not been rotated by some
external tool from being deleted, but log files that were
rotated (moved out of the way) will be re-opened.

This is a better default behaviour, especially when not all
log files are rotated at the same time.

Thanks to iro on IRC.
11 years ago
Jason Ish 6ed246c041 Don't attempt to load the rule files if the rule-files configuration
node is not a sequence.  Instead log a warning as this is usually
a configuration error.
11 years ago
Jason Ish a243a42bdf New function to test if a configuration node is a sequence or not. 11 years ago
Eric Leblond 290b01f95e af-packet: don't unlock twice the bpf mutex 11 years ago
Eric Leblond 47d9c7b211 json-alert: use getter for appstate 11 years ago
Eric Leblond 54b13851cc flow: constify getters param
Some potential callers are already using constified values so it
is good to do it.
11 years ago
Eric Leblond 881f32cc02 json-alert: add SSH fields in alert logging 11 years ago
Eric Leblond c1970a3655 json-ssh: export logging function
It will be use in alert logging to display SSH information.
11 years ago
Eric Leblond 180faece7c json-alert: log tls info in alert
This patch adds the capabilities to log the TLS information the
same way it is currently possible to do with HTTP. As it is
quite hard to read ASN.1 directly in the stream, this will help
people to understand why suricata is firing on alert relative
to TLS.
11 years ago
Eric Leblond 22182e7a84 json-tls: refactor to export logging function
To be able to add TLS data in alert we need to do the same as what
is done with HTTP ie export the logging functions.
11 years ago
Ken Steele fb0ecaba05 Inject pseudo packet periodically when there is not traffic in mPIPE.
To prevent pseudo packets from not being processed when there is no traffic,
inject a pseudo packet if no traffic is seen by a thread for ~100ms.
11 years ago
Victor Julien 20b9849999 util-magic: make unittests less specific
So they pass on CentOS 5.11 as well.
11 years ago
Victor Julien ec2be2643b app-layer: init flow in tests 11 years ago
Eric Leblond ee7422de0a pcap-file: add missing atomic init
It is mandatory to init all atomic to avoid problem on system
without atomic support.
11 years ago
Eric Leblond ed147d3b91 runmode-pcap-file: suppress useless include 11 years ago
Victor Julien ae6ef9acb4 flow-manager: init global atomics 11 years ago
Victor Julien ae8f2a4e62 stream: fix unittests wrt flow 11 years ago
Victor Julien 256388325d Fix compiler warning on CentOS 5.11
cc1: warnings being treated as errors
app-layer-smtp.c: In function ‘SMTPParseCommandBDAT’:
app-layer-smtp.c:908: warning: dereferencing type-punned pointer will break strict-aliasing rules
11 years ago
Eric Leblond bed5b28412 util-ioctl: don't build code RX ring on old system
If ETHTOOL_GRXRINGS is undefined we will not be able to build the
RX rings code. So we can make the build conditional to the
definition of ETHTOOL_GRXRINGS.
11 years ago
Victor Julien e78e33a428 http: add event for suspicious method delimeter
Add event and rule for suspicious delim(s) between method and uri.

Add unittests as well.
11 years ago
Victor Julien 5ad7198dc0 http: add libhtp uri warning event
Add event for libhtp warning added 0.5.17 for URI's with suspicious
delimeters.
11 years ago
Victor Julien 67d80c36b4 stream: init global config after flow engine
Stream depends on flow engine.
11 years ago
Victor Julien e90714afd6 http: remove unused and broken 'content-len' logic
The HTTP tracking code would parse the content lenght and store it
in the TX user data. It didn't take the possibility or errors into
account though, leading to a possible negative int being cases to
unsigned int. Luckily, the result was unused.

This patch simply removes the offending code.

Reported-by: The Yahoo pentest team
11 years ago
Victor Julien 647985d20d dcerpc: don't exit() on malloc failure
In 2 places we would exit() if malloc failed. We should never exit in
such cases. This patch silently handles it.
11 years ago
Victor Julien 5cd7bb2f14 dcerpc: fix error handling for alloc errors
Fix error handling of stub parsers. In case of SCRealloc error the
function would return a non-error code. This could possibly lead to
memory corruption.

Reported-By: The Yahoo pentest team
11 years ago
Victor Julien 7426a9c645 flow: make TCP reuse handling in flow engine optional
In case of autofp (or more general, when flow and stream engine
run in different threads) the flow engine should not trigger a flow
reuse as this can lead to race conditions between the flow and the
stream engine.

In such cases, the flow engine can be far ahead of the stream engine
as packets are in a queue between the threads.

Observed:

Flow engine tags packet 10 as start of new flow. Flow is tagged as
'reused'.

Stream engine evaluates packet 5 which belongs to the old flow. It
rejects the flow as it's tagged 'reused'. Attaches packet 5 to the
new flow which is wrong.

Solution:

This patch connects the flow engines handling of reuse cases to
the runmode. It hooks into the RunmodeSetFlowStreamAsync() call to
notify the flow engine that it shouldn't handle the reuse.
11 years ago
Victor Julien d78be75b9f tcp reuse: don't double 'reuse'
If the flow engine already reused a flow then the stream engine
won't have to do the same.
11 years ago
Victor Julien 6ad53627de flow: tag first packet in each direction
Set a flowflag for the first packet in each direction:

FLOW_PKT_TOSERVER_FIRST and FLOW_PKT_TOCLIENT_FIRST
11 years ago
Victor Julien c88cbb39fe tcp reuse: enable stream handling based on runmode
Add a way for runmodes to state that flow and stream run asynchorously.

In the stream engine, enable the TCP reuse handling only if that flag
is set.
11 years ago
Victor Julien eaae008aeb tcp reuse: support reuse on syn/ack
Support TCP reuse on SYN/ACK. Only effective if midstream and/or
async-oneside options are enabled.
11 years ago
Victor Julien 11e3f25de3 tcp reuse: unify autofp and single/workers check 11 years ago
Victor Julien 29f70bad34 tcp reuse: handle reuse in stream engine
For the autofp case, handling TCP reuse in the flow engine didn't work.

The problem is the mismatch between the moment the flow engine looks at
packets and the stream, and the moment the stream engine runs. Flow engine
is invoked in the packet capture thread(s), while the stream engine runs
as part of the stream/detect thread(s). Because of the queues between
those threads the flow manager may already inspect a new SYN while the
stream engine still has to process the previous session.

Moving the flow engine to the stream/detect thread(s) wasn't an option
as the 'autofp' load balancing depends on the flow already being
available in the packet.

The solution here is to add a check for this condition to the stream
engine. At this point the TCP state is up to date. If a TCP reuse case
is encountered, this is the global logic:

- detach packet for old flow
- get a new flow and attach it to the packet
- flag the old flow that it is now obsolete

Additional logic makes sure that the packets already in the queue
between the flow thread(s) and the stream thread are reassigned the
new flow.

Some special handling:

Apply previous 'reuse' before checking for a new reuse. Otherwise we're
tagging the wrong flow in some cases (multiple reuses in the same tuple).

When in a flow/ssn reuse condition, properly remove the packet from
the flow.

Don't 'reuse' if packet is a SYN retransmission.

The old flow is timed out normally by the flow manager.
11 years ago
Victor Julien 2fb9611223 flow: add util func to remove packet from flow
Unsets the p::flowflags that were previously set.
11 years ago
Victor Julien cf58ecb084 flow: overwrite p::flowflags on first set 11 years ago
Victor Julien 34e1de6970 flow: move flow/packet updates into util func
Move the code responsible for updating the flow and packet after
a new packet has come in to a util func "FlowHandlePacketUpdate"
11 years ago
Victor Julien db9f12ee51 flow: add flow.tcp_reuse counter
This replaces the tcp.reused_ssn counter. The flow engine now
enforces the TCP flow reuse logic.

The counter is incremented only when the flow is timed out, so
after the "tcp closed" timeout expired for a flow.
11 years ago
Victor Julien d2eea7fd44 tcp reuse: remove old code 11 years ago
Victor Julien fb1b03c471 flow: handle TCP session reuse in flow engine
Until now, TCP session reuse was handled in the TCP stream engine.
If the state was TCP_CLOSED, a new SYN packet was received and a few
other conditions were met, the flow was 'reset' and reused for the
'new' TCP session.

There are a number of problems with this approach:
- it breaks the normal flow lifecycle wrt timeout, detection, logging
- new TCP sessions could come in on different threads due to mismatches
  in timeouts between suricata and flow balancing hw/nic/drivers
- cleanup code was often causing problems
- it complicated locking because of the possible thread mismatch

This patch implements a different solution, where a new TCP session also
gets a new flow. To do this 2 main things needed to be done:

1. the flow engine needed to be aware of when the TCP reuse case was
   happening
2. the flow engine needs to be able to 'skip' the old flow once it was
   replaced by a new one

To handle (1), a new function TcpSessionPacketSsnReuse() is introduced
to check for the TCP reuse conditions. It's called from 'FlowCompare()'
for TCP packets / TCP flows that are candidates for reuse. FlowCompare
returns FALSE for the 'old' flow in the case of TCP reuse.

This in turn will lead to the flow engine not finding a flow for the TCP
SYN packet, resulting in the creation of a new flow.

To handle (2), FlowCompare flags the 'old' flow. This flag causes future
FlowCompare calls to always return FALSE on it. In other words, the flow
can't be found anymore. It can only be accessed by:

1. existing packets with a reference to it
2. flow timeout handling as this logic gets the flows from walking the
   hash directly
3. flow timeout pseudo packets, as they are set up by (2)

The old flow will time out normally, as governed by the "tcp closed"
flow timeout setting. At timeout, the normal detection, logging and
cleanup code will process it.

The flagging of a flow making it 'unfindable' in the flow hash is a bit
of a hack. The reason for this approach over for example putting the
old flow into a forced timeout queue where it could be timed out, is
that such a queue could easily become a contention point. The TCP
session reuse case can easily be created by an attacker. In case of
multiple packet handlers, this could lead to contention on such a flow
timeout queue.
11 years ago
Victor Julien 3b05b73592 tcp midstream: fix window scaling
If stream is picked by ACK, we can't know the wscale, so we assume it's
set to max. Howver, we didn't apply this to the initial window size we
set.
11 years ago
Victor Julien d834173bb8 detect-flow: use dedicated flags
The flow keyword used flag names that were shared with the
Packet::flowflags field. Some of the flags were'nt used by the packet
though. This lead to waste of some 'flag space'.

This patch defines dedicated flags for the flow keyword and removes
the otherwise unused flags from the FLOW_PKT_* space.
11 years ago
Victor Julien bc2c7f462e stats api: call thread deinit API functions
Thread deinit funcs weren't called. This meant the lua scripts 'deinit'
functions weren't called either.
11 years ago
gureedo 10104066e1 netmap support 11 years ago
Victor Julien cbe934267e file: register filedata log before file log
This way the file log can log the 'stored' info that the filedata
log sets.
11 years ago
Victor Julien c58b2b4b18 file: improve file pruning
Check if file has been logged/stored before considering it 'done'.
11 years ago
Victor Julien e58fd3cc6e runmodes: add funcs to check if file loggers enabled
Add functions to check if file/filedata loggers are enabled.
11 years ago
Victor Julien fbe6dac1ae file: optimize file pruning
FilePrune would clear the files, but not free them and remove them
from the list. This lead to ever growing lists in some cases.
Especially in HTTP sessions with many transactions, this could slow
us down.
11 years ago
Victor Julien 5251ea9ff5 flow: lockless flow manager checks
Until this point, the flow manager would check for timed out flows
by walking the flow hash, locking first the hash row and then each
individual flow to get it's state and timestamp. To not be too
intrusive trylocks were used so that a busy flow wouldn't cause the
flow manager to wait for a long time while holding the hash row lock.

Building on the changes in handling of the flow state and lastts
fields, this patch changes the flow managers behavior.

It can now get a flows state atomically and the lastts can be safely
read while holding just the flow hash row lock. This allows the flow
manager to do the basic time out check much more cheaply:

1. it doesn't have to wait for getting a lock
2. it doesn't interupt the packet path

As a consequence the trylock is now also gone. A flow that returns
'true' on timeout is pretty much certainly not going to be busy so
we can safely lock it unconditionally. This also means the flow
manager now walks the entire row unconditionally and is guaranteed
to inspect each flow in the row.

To make sure the functions called before the flow lock don't
accidentally change the flow (which would require a lock) the args
to these flows are changed to const pointers.
11 years ago
Victor Julien 5587372ce1 flow: modify lastts update logic
In the lastts timeval struct field in the flow the timestamp of the
last packet to update is recorded. This allows for tracking the timeout
of the flow. So far, this value was updated under the flow lock and also
read under the flow lock.

This patch moves the updating of this field to the FlowGetFlowFromHash
function, where it updated at the point where both the Flow and the
Flow Hash Row lock are held. This guarantees that the field is only
updated when both locks are held.

This makes reading the field safe when either lock is held, which is the
purpose of this patch.

The flow manager, while holding the flow hash row lock, can now safely
read the lastts value. This allows it to do the flow timeout check
without actually locking the flow.
11 years ago
Victor Julien a0732d3db2 flow: change flow state logic
A flow has 3 states: NEW, ESTABLISHED and CLOSED.

For all protocols except TCP, a flow is in state NEW as long as just one
side of the conversation has been seen. When both sides have been
observed the state is moved to ESTABLISHED.

TCP has a different logic, controlled by the stream engine. Here the TCP
state is leading.

Until now, when parts of the engine needed to know the flow state, it
would invoke a per protocol callback 'GetProtoState'. For TCP this would
return the state based on the TcpSession.

This patch changes this logic. It introduces an atomic variable in the
flow 'flow_state'. It defaults to NEW and is set to ESTABLISHED for non-
TCP protocols when we've seen both sides of the conversation.

For TCP, the state is updated from the TCP engine directly.

The goal is to allow for access to the state without holding the Flow's
main mutex lock. This will later allow the Flow Manager(s) to evaluate
the Flow w/o interupting it.
11 years ago
Victor Julien 9327b08ab1 tcp: add stream.reassembly.zero-copy-size option
The option sets in bytes the value at which segment data is passed to
the app layer API directly. Data sizes equal to and higher than the
value set are passed on directly.

Default is 128.
11 years ago
Victor Julien 37b56dca55 tcp: add debug stats about reassembly fast paths
Only shown if --enable-debug is passed to configure.
11 years ago
Victor Julien 2bba5eb704 tcp: zero copy fast path in app-layer reassembly
Create 2 'fast paths' for app layer reassembly. Both are about reducing
copying. In the cases described below, we pass the segment's data
directly to the app layer API, instead of first copying it into a buffer
than we then pass. This safes a copy.

The first is for the case when we have just one single segment that was
just ack'd. As we know that we won't use any other segment this round,
we can just use the segment data.

The second case is more aggressive. When the segment meets a certain
size limit (currently hardcoded at 128 bytes), we pass it to the
app-layer API directly. Thus invoking the app-layer somewhat more often
to safe some copies.
11 years ago
Victor Julien 8c1bc7cfb6 stream: move raw stream gap handling into util func 11 years ago
Victor Julien 6ca9c8eb32 stream: move raw reassembly into util func 11 years ago
Victor Julien ff2fecf590 stream: remove StreamTcpReassembleInlineAppLayer
Function is now unused.
11 years ago
Victor Julien 97908bcd2d stream: unify inline and non-inline applayer assembly
Unifiy inline and non-inline app layer stream reassembly to aid
maintainability of the code.
11 years ago
Victor Julien e1d134b027 stream: remove STREAM_SET_FLAGS
Use the unified StreamGetAppLayerFlags instead.
11 years ago
Victor Julien 29d2483efb stream: update inline tests
Make sure inline tests set the stream_inline flag.
11 years ago
Victor Julien e4cb8715de stream: replace STREAM_SET_INLINE_FLAGS macro
Replace it by a generic function StreamGetAppLayerFlags, that can
be used both by inline and non-inline.
11 years ago
Victor Julien ed791a562e stream: track data sent to app-layer 11 years ago
Victor Julien e494336453 stream: move reassembly loop into util funcs
Move IDS per segment reassembly and gap handling into utility functions.
11 years ago
Victor Julien 5b6f8bda1d detect: fix small memory leaks
Fix small memory leaks in option parsing. Move away from
pcre_get_substring in favor of pcre_copy_substring.

Related to #1046.
11 years ago
Victor Julien 5a8094136c Clean up Conf API memory on shutdown. 11 years ago
Victor Julien 04e49cea89 Fix live reload detect counter setup
When profiling was compiled in the detect counters were not setup
properly after a reload.
11 years ago
Victor Julien 844065bf58 conf api: use const pointers where possible
Use const pointers where possible in the Conf API.
11 years ago
Victor Julien ddce14360d Cosmetic fixes to main() 11 years ago
Victor Julien a3de4ecd97 Suppress debug statements 11 years ago
Victor Julien a8c16405fb detect: properly size det_ctx::non_mpm_id_array
Track which sgh has the higest non-mpm sig count and use that value
to size the det_ctx::non_mpm_id_array array.
11 years ago
Victor Julien 62751c8017 Fix live reload detect thread ctx setup
Code failed to setup non_mpm_id_array in case of a live reload.
11 years ago
Victor Julien 4e98a3e530 AC: fix memory leak 11 years ago
Victor Julien f88405c650 geoip: adapt to 'const' pointer passing 11 years ago
Victor Julien f1f5428faa detect: expand mask checking
Change mask to u16, and add checks for various protocol states
that need to be present for a rule to be considered.
11 years ago
Victor Julien ca59eabca3 detect: introduce DetectPrefilterBuildNonMpmList
Move building of non-mpm list into a separate function, that is inlined
for performance reasons.
11 years ago
Victor Julien cc4f7a4b96 detect: add profiling for non-mpm list build & filter 11 years ago
Victor Julien 4c10635dc1 detect: optimize non-mpm mask checking
Store id and mask in a single array of type SignatureNonMpmStore so
that both are loaded into the same cache line.
11 years ago
Victor Julien b5a3127151 detect: add mask check prefilter for non mpm list
Add mask array for non_mpm sigs, so that we can exclude many sigs before
we merge sort.

Shows 50% less non mpm sigs inspected on average.
11 years ago
Ken Steele 904441327c Conditionalize SigMatch performance counters.
Only include the counters when PROFILING.
11 years ago
Victor Julien 30b7fdcb49 Detect perf counters 11 years ago
Victor Julien ef6875d583 detect: Disable unused SignatureHeader code 11 years ago
Ken Steele 65af1f1c5e Remove sgh->mask_array
Not needed by new MPM opt.
11 years ago
Ken Steele 4bd280f196 Indentation clean up 11 years ago
Ken Steele 403b5a4645 Further optimize merging mpm and non-mpm rule ID lists.
When reaching the end of either list, merging is no longer required,
simply walk down the other list.

If the non-MPM list can't have duplicates, it would be worth removing
the duplicate check for the non-MPM list when it is the only non-empty list
remaining.
11 years ago
Ken Steele 86f4c6c47b Custom Quick Sort for Signature IDs
Use an in place Quick Sort instead of qsort(), which does merge sort and
calls memcpy().

Improves performance on my tests.
11 years ago
Ken Steele 736ac6a459 Use SigIntId as the type for storing signature IDs (Internal)
Previously using uint32_t, but SigIntId is currently uint16_t, so arrays
will take less memory.
11 years ago
Ken Steele d01d3324fc Increase max pattern ID allowed in MPM AC-tile to 28-bits 11 years ago
Victor Julien 6717c356e3 Clean up sm_array memory at SigFree 11 years ago
Ken Steele 1874784c10 Create optimized sig_arrays from sig_lists
Create a copy of the SigMatch data in the sig_lists linked-lists and store
it in an array for faster access and not next and previous pointers. The
array is then used when calling the Match() functions.

Gives a 7.7% speed up on one test.
11 years ago
Ken Steele 923a77e952 Change Match() function to take const SigMatchCtx*
The Match functions don't need a pointer to the SigMatch object, just the
context pointer contained inside, so pass the Context to the Match function
rather than the SigMatch object. This allows for further optimization.

Change SigMatch->ctx to have type SigMatchCtx* rather than void* for better
type checking. This requires adding type casts when using or assigning it.

The SigMatch contex should not be changed by the Match() funciton, so pass it
as a const SigMatchCtx*.
11 years ago
Ken Steele 900def5caf Create Specialized SCMemcmpNZ() when the length can't be zero. 11 years ago
Ken Steele 7835070385 Replace memcpy() in MpmAddSids with copy loop
For the short size of most sids lists, a straight copy loop is faster.
11 years ago
Ken Steele 83ed01a279 Fix compiler warnings in ac-tile.
Signed vs unsigned comparisons.
11 years ago
Ken Steele 1c76fa50b1 Prefetch the next signature pointer
Read one signature pointer ahead to prefetch the value.
Use a variable, sflags, for s->flags, since it is used many times and the
compiles doesn't know that the signatures structure doesn't change, so it
will reload s->flags.
11 years ago
Ken Steele fa51118dfe Move type first in SigMatch array since it is used more often. 11 years ago
Ken Steele 7a2095d851 In AC-Tile, convert from using pids for indexing to pattern index
Use an MPM specific pattern index, which is simply an index starting
at zero and incremented for each pattern added to the MPM, rather than
the externally provided Pattern ID (pid), since that can be much
larger than the number of patterns. The Pattern ID is shared across at
MPMs. For example, an MPM with one pattern with pid=8000 would result
in a max_pid of 8000, so the pid_pat_list would have 8000 entries.

The pid_pat_list[] is replaced by a array of pattern indexes. The PID is
moved to the SCACTilePatternList as a single value. The PatternList is
also indexed by the Pattern Index.

max_pat_id is no longer needed and mpm_ctx->pattern_cnt is used instead.

The local bitarray is then also indexed by pattern index instead of PID, making
it much smaller. The local bit array sets a bit for each pattern found
for this MPM. It is only kept during one MPM search (stack allocated).

One note, the local bit array is checked first and if the pattern has already
been found, it will stop checking, but count a match. This could result in
over counting matches of case-sensitve matches, since following case-insensitive
matches will also be counted. For example, finding "Foo" in "foo Foo foo" would
report finding "Foo" 2 times, mis-counting the third word as "Foo".
11 years ago
Ken Steele 77269fbb2c Fix missing use of MpmAddPid()
Found by Victor using ASAN. One place was not checking to resize the
pid array before adding a new PID.
11 years ago
Ken Steele eaac9c8d93 fix check in PmqMerge 11 years ago
Ken Steele 1c03eb56d0 Fix bug in MPM rule array handling
In PmqMerge() use MpmAddSids() instead of blindly copying the src
rule list onto the end of the dst rule list, since there might not
be enough room in the dst list. MpmAddSids() will resize the dst array
if needed.

Also add code to MpmAddSids() MpmAddPid() to better handle the case
that realloc fails to get more space. It first tries 2x the needed
space, but if that fails, it tries for just 1x. If that fails resize
returns 0. For MpmAddPid(), if resize fails, the new pid is lost. For
MpmAddSids(), as many SIDs as will fit are added, but some will be
lost.
11 years ago
Ken Steele ab8b1158b0 Dynamically resize pattern id array as needed
Rather than creating the array of size maxpatid, dynamically resize as needed.
This also handles the case where duplicate pid are added to the array.

Also fix error in bitarray allocation (local version) to always use bitarray_size.
11 years ago
Ken Steele 104a903478 Dynamically resize pmq->rule_id_array
Rather than statically allocate 64K entries in every rule_id_array,
increase the size only when needed. Created a new function MpmAddSids()
to check the size before adding the new sids. If the array is not large
enough, it calls MpmAddSidsResize() that calls realloc and does error
checking. If the realloc fails, it prints an error and drops the new sids
on the floor, which seems better than exiting Suricata.

The size is increased to (current_size + new_count) * 2. This handles the
case where new_count > current_size, which would not be handled by simply
using current_size * 2. It should also be faster than simply reallocing to
current_size + new_count, which would then require another realloc for each
new addition.
11 years ago
Ken Steele d31db4ed1c Fix clang warning
Clang doesn't seem to like defining a function within a function.
11 years ago
Ken Steele 23d2a1c422 Optimize DetectPrefilterMergeSort
Fixup rebase changes to remove debug code
11 years ago
Ken Steele f83022d818 Implement MPM opt for ac-bs and ac-gfbs
Copies sids changes from ac.
11 years ago
Ken Steele d03f124445 Implement MPM opt for b2g, b3g, wumanber
Found problems in b2gm and b2gc, so those are removed.
11 years ago
Ken Steele edaefe5af2 Fix AC-tile for new pattern ID array. 11 years ago
Victor Julien 29074af9a6 AC: use local bit array
Use a local pattern bit array to making sure we don't match more than
once, in addition to the pmq bitarray that is still used for results
validation higher up in the rule matching process.

Why: pmq->pattern_id_bitarray is currently sometimes used in a
'stateful' way, meaning that for a single packet we run multiple
MPM's on the same pmq w/o resetting it.

The new bitarray is used to determine wherther we need to append the
patterns associated 'sids' list to the pmq rule_id_array.

It has been observed that MPM1 matches for PAT1, and MPM2 matches for
PAT1 as well. However, in MPM1 PAT1 doesn't have the same sids list.
In this case MPM2 would not add it's sids to the list, leading to missed
detection.
11 years ago
Victor Julien 7876277119 detect: move checks from prefilter to rule detect
Move the prefilter checks to the main detect loop.
11 years ago
Victor Julien d1d895a884 Replace build match array with new filter logic
Use MPM and non-MPM lists to build our match array. Both lists are
sorted, and are merged and sorted into the match array.

This disables the old match array building code and thus also bypasses
the mask checking.
11 years ago
Victor Julien 1f57e25c03 detect: Add negated MPM to non-MPM array
Treat negated MPM sigs as if non-MPM, so we consider them always.

As MPM results and non-MPM rules lists are now merged and considered
for further inspection, rules that need to be considerd when a pattern
is absent are caught in the middle.

As a HACK/workaround this patch adds them to the non-MPM list. This
causes them to be inspected each time.
11 years ago
Victor Julien f5df526f9b Detect: create per sgh non-MPM rule array
Array of rule id's that are not using MPM prefiltering. These will be
merged with the MPM results array. Together these should lead to a
list of all the rules that can possibly match.
11 years ago
Victor Julien e49d0a5924 MPM: build sid list from MPM matches
Pmq add rule list: Array of uint32_t's to store (internal) sids from the MPM.

AC: store sids in the pattern list, append to Pmq::rule_id_array on match.

Detect: sort rule_id_array after it was set up by the MPM. Rule id's
(Signature::num) are ordered, and the rule's with the lowest id are to
be inspected first. As the MPM doesn't fill the array in order, but instead
'randomly' we need this sort step to assure proper inspection order.
11 years ago
Ken Steele b96645ded2 Create a wrapper around DetectFlowvarProcessList() to check for empty list
Creates an inline wrapper to check for flowvarlist == NULL before calling
DetectFlowvarProcessList() to remove the overhead of checking since the
list is usually empty.
11 years ago
Ken Steele 5008d0a58b Remove the b2gm and b2gc MPMs
These MPMs have code that looks like it won't work and updating them to
for the new MPM optimization wasn't working.
11 years ago
Victor Julien 227a7de351 Global define of MIN
Some OS' provide it automatically, so make sure we define it
conditionally in one place.
11 years ago
Victor Julien bcfd61416f Fix a fix: defrag OOM condition
** CID 1257764:  Dereference after null check  (FORWARD_NULL)
/src/defrag.c: 291 in Defrag4Reassemble()

** CID 1257763:  Dereference after null check  (FORWARD_NULL)
/src/defrag.c: 409 in Defrag6Reassemble()

In the error case 'rp' can be both NULL or non-NULL.
11 years ago
Victor Julien 43a1007788 detect: add test for memcmp issue 11 years ago
Victor Julien 0d910bed1d Add test for memcmp issue. 11 years ago
Victor Julien 17dfd59bc3 memcmp: compare the first byte as well
MemcmpLowercase would not compare the first byte of both input buffers
leading to two non-identical buffers to be considered the same.

Affects SSE_4_1 and SSE_4_2 implementations of SCMemcmpLowercase, as well
as the non-SIMD implementation. SSE_3 and Tile version are not affected.
11 years ago
Victor Julien c51ce4d2c0 Fix OS X 10.10 unittest failure
Work around OS X 10.10 Yosemite returning EDEADLK on a rwlock wrlocked
then tested by wrtrylock. All other OS' (and versions of OS X that I
tested) seem to return EBUSY instead.
11 years ago
Victor Julien baa55ba239 Fix Tilera compilation
Use proper initializer for a static mutex declaration.

Credits: Ken Steele
11 years ago
Victor Julien 8e946b92b7 Fix compilation on OS X Yosemite
Due to our unconditional declaration of the strlcat and strlcpy
functions, compilation failed on OS X Yosemite.

Bug #1192
11 years ago
Victor Julien 485f34134e unix socket: support profiling 11 years ago
Victor Julien f32d79dfe0 smtp: fix tx handling
Fix issue where SMTPStateGetTxCnt would return the actual active tx'.

The 'GetCnt' API call is not named correctly. It should be 'GetMaxId',
as this is actually the expected behavior.
11 years ago
Victor Julien 105b4340c2 thread local storage: add to build-info 11 years ago
Victor Julien 623c2e78fd packet pool: make pending pool use more robust
Don't leave pointers dangling.
11 years ago
Victor Julien 6e174128c8 packet pool: memory fixes for non-TLS
If the posix TLS implementation is used, the packet pool is memset to
0 before use.

Also use proper 'free' function.
11 years ago
Victor Julien 2745cd2ce9 packet pool: fix wrong free call 11 years ago
Eric Leblond ff8dae3b75 app-layer: fix 'detection-only' keyword
If we follow the description in the yaml file, we should disable
parsing if 'detection-only' keyword is used.
11 years ago
Eric Leblond 969abc2ccd output-json: fix duplicate logging
This patches is fixing a issue in the OutputJSONBuffer function. It
was writing to file the content of the buffer starting from the start
to the final offset. But as the writing is done for each JSON string
we are duplicating the previous events if we are reusing the same
buffer.

Duplication was for example triggered when we have multiple alerts
attached to a packet. In the case of two alerts, the first one was
logged twice more as the second one.
11 years ago
Victor Julien dc5e2a515c stream: improve inline mode GAP handling
Don't conclude a GAP is 'final' until the missing data is ack'd.

Further, cleanup and unify more with the non-inline code.
11 years ago
Victor Julien b69ca16553 stream: move utility functions
This way they can be used by the *Inline* functions as well.
11 years ago
Victor Julien a095694945 host: register unittests
Host unittests were not registered so they wouldn't run.
11 years ago
Victor Julien 60b50e1ca5 packet-pool: free pending packets 11 years ago
Victor Julien 8b2dd81628 stats: stats threads don't need packet pools 11 years ago
Victor Julien ffd2248459 flow manager: destroy packet pool on close 11 years ago
Victor Julien c4e1324690 flow-timeout: use packet pool
Use packet pool for pseudo packets on flow timeout. Wait for a packet
if necessary.

For shutdown, alloc a new pool as the 'main()' thread calls this.
11 years ago
Victor Julien cef609bb73 threading: lock TmThreadKillThreadsFamily 11 years ago
Victor Julien 3499d682c4 flow timeout: cleanups
Rename FlowForceReassemblyForFlowV2 to just FlowForceReassemblyForFlow
as there is no V1.
11 years ago
Victor Julien 6e69b51123 flow timeout: cleanup
Remove now unused old flow timeout code.
11 years ago
Victor Julien de4bda14e6 stream: handle flow timeout stream end packets
Handle flow timeout packets in the stream engine. Previously the flow
timeout code would call reassembly code directly.
11 years ago
Victor Julien 0ffaad66eb flow-time: disable remainder of the old timeout code
Disable registration code that was looking for threadvars
and slots as timeout handling is now done in a live engine.
11 years ago
Victor Julien 8e86f387a6 flow-time: use live threads at shutdown
Update pktacq loop to process flow timeouts in a running engine.

Add a new step to the shutdown phase of packet acquisition loop
threads (pktacqloop).

The shutdown code lets the pktacqloop break out of it's packet
acquisition loop. The thread then enters a flow timeout loop, where
it processes packets from it's tv->stream_pq queue until it's
empty _and_ the KILL flag is set.

Make sure receive threads are done before moving on to flow hash
cleanup (recycle all). Without this the flow recycler could start
it's unconditional hash clean up while detect threads are still
running on the flows.

Update unix socket to match live modes.
11 years ago
Victor Julien c6ec92d9b1 flow-timeout: use live threads
Use live threads. Disable old timeout code.
11 years ago
Victor Julien 48eccf7d91 Assign thread_id to flow on first packet stream engine 11 years ago
Victor Julien 8c51b23e94 Thread registration: id's start at 1
Start thread id's at 1, so that in flow's we can use 0 to indicate
a thread id hasn't been set in it yet.
11 years ago
Victor Julien 7f80516563 Introduce Flow timeout injection api
Add function TmThreadsInjectPacketById that is to be used to inject flow
timeout packets into the threads stream_pq queue.

TmThreadsInjectPacketById will also wake up listening threads if
applicable.

Packets are passed all packets together in an NULL terminated array
to reduce locking overhead.
11 years ago
Victor Julien 51a782fd8c Define FlowThreadId and add it to the flow
16 bits id should be enough for threads for a while.
11 years ago
Victor Julien a260cba32b Give easy access for thread stream packet queue
Access it from ThreadVars. This allows for easy injection of packets
into the stream engine.
11 years ago
Victor Julien 489ee20560 Thread Registration API for ID's
Create thread registration and unregistration API for assigning unique
thread id's.

Threadvars is static even if a thread restarts, so we can do the
registration before the threads start.

A thread is unregistered when the ThreadVars are freed.
11 years ago
Duarte Silva e586644c25 Fix and improvements
- Added/removed missing/superfluous util-memrchr.h include
- Improved the extraction of a IP from the XFF chain of IPs
11 years ago
Duarte Silva 68f43ffffb Implemented the diferent behaviour depending on the proxy deployment
- In forward deployment mode the first IP will be returned
- In reverse deployment mode the last IP will be retuned
11 years ago
Duarte Silva 496200dd08 Prepared everything for the proxy deployment configuration
- Added the suricata.yaml configurations and updated the comments
- Renamed the field in the configuration structure to something generic
- Added two new constants and the warning codes
11 years ago
Duarte Silva 4e04cd2d1b Adding XFF support to EVE alert output
- Created app-layer-htp-xff.c and app-layer-htp-xff.h
- Added entries in the Makefile.am
- Added the necessary configuration options to EVE alert section
- Updated Unified2 XFF configuration comments and removed unnecessary whitespace
- Created a generic function to parse the configuration
- Release the flow locks sooner and remove debug logging
- Added XFF support to EVE alert output
11 years ago
DIALLO David 0bdf494b54 fix Cygwin build fails: array subscript has type char 11 years ago
DIALLO David bfc871ce85 Update AppLayerProtoDetectPrintProbingParsers with Modbus protocol 11 years ago
DIALLO David 83d9834e77 fix CID 1257762: Logically dead code(DEADCODE) 11 years ago
Victor Julien 84e8217fd8 unix-socket: allow socked in custom locations
Allow the socket to be set in any location. This allows for easy
setting up of a socket as a non-root user.
11 years ago
Victor Julien b978730486 unix-socket: fix restart/shutdown cycle
When cleaning up after a pcap was processed, the stats api was cleaned
up before the stats threads were killed, leading to a BUG_ON triggering.
11 years ago
Ken Steele 3f3481e4d2 Fix indentation 11 years ago
Ken Steele 3f86c5a83f Fix memory leak in ac-tile
Incorrectly reallocing the goto table after it was freed by calling
SCACTileReallocState() when really only want to realloc the output table.
This was causing a large goto table to be allocated and never used or
freed.
11 years ago
Ken Steele b9e20ab4b8 Clean up memory leaks in ac-tile code
Free some memory at exit that was not getting freed.

Change pid_pat_list to store copy of case-strings in the same block
of memory as the array of pointers.
11 years ago
Ken Steele 1faa94c314 Make bad copy-mode be an error in runmode-tile. 11 years ago
Victor Julien c779065d35 Bug 1329: error out on invalid rule protocol
Due to a logic error in AppLayerProtoDetectGetProtoByName invalid
protocols would not be detected as such. Instead of ALPROTO_UNKNOWN
ALPROTO_MAX was returned.

Bug #1329
11 years ago
Eric Leblond 9f22c878e8 unix-manager: fix cppcheck errors
This patch fixes the following errors:
 [src/unix-manager.c:306]: (error) Memory pointed to by 'client' is freed twice.
 [src/unix-manager.c:313]: (error) Memory pointed to by 'client' is freed twice.
 [src/unix-manager.c:323]: (error) Memory pointed to by 'client' is freed twice.
 [src/unix-manager.c:334]: (error) Memory pointed to by 'client' is freed twice.

Unix manager was treating the packet after closing the socket if message was
too long.
11 years ago
Victor Julien 096b85ab68 stream: don't send EOF to AppLayer too soon
Sending EOF too soon results in the AppLayer cleaning up prematurely.
11 years ago
Victor Julien 0bb2b15491 ipv6: check for MLD messages with HL not 1
MLD messages should have a hop limit of 1 only. All others are invalid.

Written at MLD talk of Enno Rey, Antonios Atlasis & Jayson Salazar during
Deepsec 2014.
11 years ago
Ken Steele 68e6c4e94b Correct flow memory usage bookkeeping error
Fix bug 1321 where flow_memuse was incremented more on allocation than
free.
11 years ago
Victor Julien d951de2f19 Bug 977: -T / --init-errors-fatal to process all rules
Have -T / --init-errors-fatal process all rules so that it's easier
to debug problems in ruleset. Otherwise it can be a lengthy fix, test
error cycle if multiple rules have issues.

Convert empty rulefile error into a warning.

Bug #977
11 years ago
Victor Julien e951afb911 afpacket: only check offloading once per iface
Instead of once per thread per iface.
11 years ago
Victor Julien fa10811585 ioctl: make all string args const pointers 11 years ago
Victor Julien c3c144d504 http: don't crash when normalizing uri on low memory 11 years ago
Victor Julien f8f2ff49de defrag: don't crash when out of memory
Handle memory allocation errors in defrag better. Could lead to
crashes if malloc errors happened.
11 years ago
Eric Leblond 56373e5b34 af-packet: no more threads than RSS queues
If we manage to read the number of RSS queues from an interface,
this means that the optimal number of capture threads is equal
to the minimum of this number and of the number of cores on the
system.

This patch implements this logic thanks to the newly introduced
function GetIfaceRSSQueuesNum.
11 years ago
Eric Leblond 123c58af4b util-ioctl: add message in case of failure 11 years ago
Eric Leblond b4bb6e67ba util-ioctl: Add function to get number of RSS queues on iface
The number of RSS queues can be fetched via a standard ioctl which
is independant of hardware.
11 years ago
Victor Julien c174c9d779 af-packet: threads: auto, default to workers
Add a new default value for the 'threads:' setting in af-packet: "auto".
This will create as many capture threads as there are cores.

Default runmode of af-packet to workers.
11 years ago
Victor Julien 7b4987abc3 Runmode: handle value 'auto'
Auto now selects the default runmode for the capture method.
11 years ago
Victor Julien 234d18ab68 threading: remove '1slot' functions
No longer in use after the 'auto' runmode removal.

All runmodes now use either varslot or pktacqloop support.
11 years ago
Victor Julien 7025aabe75 Runmodes: remove 'auto' runmodes
Remove 'auto' runmodes from all capture methods. It wasn't reliable
enough, as it didn't enforce inspection order of packets.
11 years ago
Victor Julien 81c42f4916 log-stats: expand membuffer if necessary
Many threads could lead to a membuffer size requirement bigger than
64k. So use the expansion call to grow the buffer as needed.
11 years ago
Victor Julien 6277d2e0e4 MemBuffer: add expansion call
For some of the buffer users it's hard to predict how big the data
will be. In the stats.log case this depends on chosen runmode and
number of threads.

To deal with this case a 'MemBufferExpand' call is added. This realloc's
the buffer.
11 years ago
Victor Julien 75397ed750 stats: expose stats to Lua output
Register with type 'stats':

    function init (args)
        local needs = {}
        needs["type"] = "stats"
        return needs
    end

The stats are passed as an array of tables:

    { 1, { name=<name>, tmname=<tm_name>, value=<value>, pvalue=<pvalue>}}
    { 2, { name=<name>, tmname=<tm_name>, value=<value>, pvalue=<pvalue>}}
    etc

Name is the counter name (e.g. decoder.invalid), tm_name is the thread name
(e.g. AFPacketeth05), value is current value, and pvalue is the value of the
last time the script was invoked.
11 years ago
Victor Julien 5d95b08172 output streaming: cleanup at runmode destruction 11 years ago
Victor Julien 51a540c27e stats: disable stats if no loggers are enabled 11 years ago
Victor Julien 6252d24e0b stats: initialize after outputs
Initialize stats after outputs so that we can check if we need to
initialize the stats api at all.
11 years ago
Victor Julien a95c95f74c stats: introduce global config
As the stats api calls the loggers at a global interval, the global
interval should be configured globally.

 # global stats configuration
 stats:
   enabled: yes
   # The interval field (in seconds) controls at what interval
   # the loggers are invoked.
   interval: 8

If this config isn't found, the old config will be supported.
11 years ago
Victor Julien e98346b555 Introduce stats log API, convert existing output
Convert regular 'stats.log' output to this new API.

In addition to the current stats value, also give the last value. This
makes it easy to display the difference.
11 years ago
Victor Julien ee8da21e36 pcre: fix var capture for non relative matches
Var capture setup depended on the match being relative due to a logic
error.
11 years ago
Giuseppe Longo 1ad2a231fe pfring: fixes memleaks
This fixes some memory leaks
Bug #1184
11 years ago
Victor Julien 16941468ce lua: in streaming api, indicate open/close
The SCStreamingBuffer call now also returns two booleans:
    data, data_open, data_close = SCStreamingBuffer()

The first indicates this is the first data of this type for this
TCP session or HTTP transaction.

The second indicates this is the last data.

Ticket #1317.
11 years ago
Ken Steele 8dcd99209e Update copyright year in detect-flowbits files. 11 years ago