Commit Graph

322 Commits (c78f5ac3169e8b4539c49fde9d9aa8fa8465f40e)

Author SHA1 Message Date
Victor Julien f59f90331d Applayer: remove obsolete StateUpdateTransactionId
Also, update StateTransactionFree to take an u64 tx id, so it's
consistant with the rest of the engine.

To reflect these changes, AppLayerRegisterTransactionIdFuncs has
been renamed to AppLayerRegisterTxFreeFunc.

HTP, DNS, SMB, DCERPC parsers updated.
12 years ago
Victor Julien e8ad876b48 App layer: add 'StateHasEvents' API call
Per TX decoder events resulted in significant overhead to the
detection engine, as it walked all TX' all the time to check
if decoder events were available.

This commit introduces a new API call StateHasEvents, which speeds
up this process, at the expense of keeping a counter in the state.

Implement this for DNS as well.
12 years ago
Victor Julien 9dc04d9fab app layer: add support for per TX decoder events 12 years ago
Victor Julien 59780ca770 Hacks to enable alert dns even though we have dnstcp and dnsudp parsers. Needs proper solution later. 12 years ago
Victor Julien 8e01cba85d DNS TCP and UDP parser and DNS response logger 12 years ago
Anoop Saldanha a490176c8a More lock fixes for the transaction update. Issues reported by Coverity. 12 years ago
Anoop Saldanha 7cf4042337 Fix luajit compilation failure introduced by the transaction update.
Fix coverity lock issues reported by transaction update as well.
12 years ago
Anoop Saldanha d4d18e3136 Transaction engine redesigned.
Improved accuracy, improved performance.  Performance improvement
noticeable with http heavy traffic and ruleset.

A lot of other cosmetic changes carried out as well.  Wrappers introduced
for a lot of app layer functions.

Failing dce unittests disabled.  Will be reintroduced in the updated dce
engine.

Cross transaction matching taken care of.  FPs emanating from these
matches have now disappeared.  Double inspection of transactions taken
care of as well.
12 years ago
Victor Julien afb2d4eddf Fix stateful inspection not always inspecting at stream end. 13 years ago
Eric Leblond 1fd47cfb96 Remove useless code. 13 years ago
Eric Leblond b3d4285982 fix logic error in sanity check 13 years ago
Last G 8ae11f73b2 Added parentheses to fix Eclipse static code analysis
Fixed bug in action priority (REJECT_DST had lowest prio)
13 years ago
Eric Leblond 6842545331 Add documentation url in list-keyword output.
The output of the list-keyword is modified to include the url to
the keyword documentation when this is available. All documented
keywords should have their link set.

list-keyword can be used with an optional value:
 no option or short: display list of keywords
 csv: display a csv output on info an all keywords
 all: display a human readable output of keywords info
 $KWD: display the info about one keyword.
13 years ago
Eric Leblond e176be6fcc Use unlikely for error treatment.
When handling error case on SCMallog, SCCalloc or SCStrdup
we are in an unlikely case. This patch adds the unlikely()
expression to indicate this to gcc.

This patch has been obtained via coccinelle. The transformation
is the following:

@istested@
identifier x;
statement S1;
identifier func =~ "(SCMalloc|SCStrdup|SCCalloc)";
@@

x = func(...)
... when != x
- if (x == NULL) S1
+ if (unlikely(x == NULL)) S1
13 years ago
Victor Julien 20d2db085e reintroduce pool free func for cases where block alloc is not used. 13 years ago
Eric Leblond 619014a280 pool: rename Free function to Cleanup
This patch renames Free functions to Cleanup as the free is made
by the pool system.
13 years ago
Eric Leblond fa079c1da0 pool: realize a block allocation for preallocated item.
This patch required a evolution of Pool API as it is needed to
proceed to alloc or init separetely. The PoolInit has been changed
with a new Init function parameter.
13 years ago
Victor Julien f93c54136c stream/app layer: call new Truncate callback for data gap case as well. 13 years ago
Victor Julien 869109a6a0 stream/app layer: add Truncate app layer callback that is called if stream depth is reached. Use it to trunc open files in HTTP. 13 years ago
Anoop Saldanha 6c5b596ada coverity fixes 13 years ago
Anoop Saldanha 109662450d Add new command line option --list-app-layer-protocols to list supported app layer protocols in sigs 14 years ago
Victor Julien cdba2f50d1 Various fixes and improvements based on feedback by Coverity analyzer. 14 years ago
Victor Julien 8b1333a277 Add more flow lock assertions to the debug validation code. 14 years ago
Victor Julien 5ba41c7890 Fix locking error in filestore handling. Add debug validate check for asserting a flow is locked. 14 years ago
Victor Julien f084874998 Fix HTTP state and raw stream not being inspected at the same time. Adds an exception to transaction id handling for HTTP. 14 years ago
Victor Julien 16cfae2f51 Trigger raw stream reassembly on receiving a full HTTP request or response. 14 years ago
Victor Julien f773942ce0 Disable printing dreaded app layer error messages to the screen: app layer events are here to safe us. 14 years ago
Anoop Saldanha 5311cd4866 Support for smtp decoder events 14 years ago
Anoop Saldanha eea5ab4a7a Support for app layer decoder events added + app_layer_event keyword added 14 years ago
Victor Julien 3009429e3c HTTP transaction handling improvement
In some cases AppLayerTransactionGetInspectId can return -1, which is
now handled by all it's callers.

Improve logic of selecting which transactions are inspected by the various
HTTP keywords.
14 years ago
Victor Julien 45d86ff58a Stream reassembly / app layer: disable gap errors
Gap errors on the app layer are now silently handled. No longer printed
to the screen.
14 years ago
Victor Julien b74c73309b file handling: improve filestore keyword handling
In stateful detection only inspect the file portion of the rule after all
other conditions matched. This to prevent "filestore" from tagging files
for storage during a partial match.

Add a couple of unittests to test the behaviour change.
14 years ago
Victor Julien 2ccd35c6e4 Fix code after rebase. 14 years ago
Victor Julien d59ca75e46 file extract: split toserver and toclient tracking
Split toserver and toclient file tracking for the http state.
14 years ago
Victor Julien e1022ee5ae file-extraction: Disconnect file handling from flow and move into the app layer state. 14 years ago
Victor Julien 23e01d23d3 Implement filestore keyword, including a way for the stateful detection engine to conclude that a file will never have to be stored. 14 years ago
Anoop Saldanha 9a6aef459e modify all relevant app layer API calls to accomodate passing parser local storage argument 14 years ago
Anoop Saldanha 01a35bb604 introduce app layer local storage api support 14 years ago
Victor Julien 262a7300d7 flow: shrink Flow datatype
Introduce a separate FlowAddress structure for holding the ipv4 or ipv6 address
that doesn't have the family in it like the Address structure. Instead, the
family is stored in the flow as a flag: FLOW_IPV4 and FLOW_IPV6.

Add macro's to check the family, copy the address, etc.

Update many unittests to reflect these changes. Introduce unittest helper
functions for creating and initializing a flow and freeing it again.

On 64 bit this shrinks the flow with 8 bytes.
14 years ago
Victor Julien 06904c9024 App Layer cleanup
Removal of per flow 'aldata' array. It contained a ptr for each ALPROTO. Instead now we have 2 ptrs in the flow: alparser and alstate.
Various cleanups and dead code removal from the app layer API.
Should safe 100+ bytes memory per flow on 64 bit.
Updated lots of unittests to reflect these changes.
14 years ago
Victor Julien 7e3c15e54a stream: improve TCP ssn reuse cleanup. 14 years ago
Anoop Saldanha de9ad02b59 Remove leftover imap and msn toclient alproto PM contents 14 years ago
Victor Julien 8186565240 Fix a number of potential issues found by CLANG and cppcheck. 14 years ago
Anoop Saldanha 0edf053f31 if app layer inspection is disabled, immediately set the eof flag 14 years ago
Anoop Saldanha fe11e02f58 fix inspect id update bug. This should prevent unnecessary FPs for pipelined requests 14 years ago
Anoop Saldanha 4e44073c79 http logging module should log all txs in the list and not just the last complete tx available on EOF 14 years ago
Anoop Saldanha c13ad8c28a Provide a function to set the app layer tx eof flag. Use this in FFR code instead of diretly setting the flag. This cleans up the API as well 14 years ago
Anoop Saldanha b406af451b updates to http tx id vars. FFR now flags the app layer session for EOF when creating a pseudo packet for a flow 14 years ago
Anoop Saldanha d68775d47d introduce bitmasks instead of alproto_masks for use by the probing parser. Remove all alproto_masks we had previouslys for PP 14 years ago
Anoop Saldanha d3989e7cee probing parser updated to always accept u32 buflens. Update all probing parser functions to accomodate this change 14 years ago
Anoop Saldanha 432c3317d2 app layer probing parser updates 14 years ago
Eric Leblond 6b9d1012ff Transform inet_ntop call into PrintInet one. 14 years ago
Anoop Saldanha 576ec7da66 smtp parser support 14 years ago
Victor Julien 73efb4c70f Add a app layer state and stateful detection engine counter that makes sure the stateful inspection is only done when the state changes. 14 years ago
Anoop Saldanha 6e0d98d9c4 fix valgrind issue for SMB test. Small restructuring. probing_parsers global variable now part of AlpProtoDetectCtx 14 years ago
Anoop Saldanha 7f8fb0f00d fix bounds checking in smb probing parser 14 years ago
Anoop Saldanha a40fdc794e Added probing parser for nbss/smb on port 139 14 years ago
Anoop Saldanha 7c31a2327e Add support for port based probing parsers for alproto detection 14 years ago
Anoop Saldanha fe6e41e3ef Removed FLOW_AL_NO_APPLAYER_INSPECTION. Moved it as FLOW_NO_APPLAYER_INSPECTION in Flow->flags. Turned Flow->flags into uint32_t and removed Flow->alflags 14 years ago
Anoop Saldanha 38fe2b9070 Removed FLOW_AL_STREAM_START, EOF and GAP flags. We don't need these. Just use STREAM_* flags 14 years ago
Anoop Saldanha 000ce98cd1 push all proto detection code into their respective app parser register functions for every alproto 14 years ago
Victor Julien 8fa5a2c025 Split applayer and raw stream reassembly
Split stream reassembly in 2 parts: a part that sends ack'd data to the app
layer parsers as soon as it's available, and another part that queues up
data into larger chunks for raw inspection.
15 years ago
Victor Julien dda6d3e07b Add error counters. 15 years ago
Eric Leblond 49adc264bc Don't print message after SCMalloc failure.
This patch generated via coccinelle is getting rid of logging
message after a SCMalloc failure. They were useless as SCMalloc
already displays a message.
15 years ago
Victor Julien b8fec77f37 Fix tcp connections that are reset (RST packet) not always inspecting the reassembled stream. Update transaction id code to make sure both directions of a transaction are inspected before incrementing the inspect_id. 15 years ago
William Metcalf 0e4235cc94 FLOW_DESTROY added to clean-up UT's that init flow 15 years ago
Victor Julien 83b2c8abdb Improve stateful uri detection code. 15 years ago
Victor Julien e8fce5f7fa Convert uricontent scanning to use the detect engine state. 15 years ago
Victor Julien ba12f3c109 Applayer to flow fixes and cleanups. 15 years ago
Pablo Rincon 8cc525c939 UDP support at AppLayer message handling 15 years ago
William Metcalf cc76aa4bc6 properly init flows inside of unit-tests caused lock-up when falling back to using mutex locks 15 years ago
Victor Julien 9f95ab7441 Make sure a stream that has a failing app layer inspection module no longer stops reassembly, but only app layer inspection. This way we can continue to inspect the reassembled stream. 15 years ago
Victor Julien 70b32f7380 First stab at creating a stateful detection engine.
Stateful detection for app layer detection keywords, except uricontent. Stores it's partial results in the flow structure. Other modifications:

- Generalize transaction tracking, logging and inspection.
- Adapt http and dcerpc to use the new transaction handling.
- Stream engine now always notifies app layer of a stream eof.

This commit fixes bug #124.
15 years ago
Gerardo Iglesias Galvan 9f4fae5b1a Fix inconsistent use of dynamic memory allocation 15 years ago
William Metcalf ce01927515 Import of GPLv2 Header 050410 15 years ago
Victor Julien c3392b7c22 Fix checking for the stream GAP after the ssn ptr was initialized. 15 years ago
Victor Julien 9676273e6d Kick out streams with gaps in them in the app layer parser until we add proper support. 15 years ago
Victor Julien 13e6c8035d Make sure we don't leak memory on app layer protocols we detect, but don't parse. Fixed #132. Thanks to Gurvinder Singh for pointing out where the issue was. 15 years ago
Pablo Rincon f862de2ee6 Fixing some code reviews (Thanks to Steve Grubb) 15 years ago
Gurvinder Singh 8e444f1772 stream and application layer improvements 16 years ago
Victor Julien 3d7b882bde Make sure all smsgs are handled every time, even in case or error. The fuzzer found an issue where unhandled messages remained in the queue leading to threading issues. 16 years ago
Pablo Rincon 25a3a5c6d8 Adding mem wrapper to debug runtime alloc()/free() functions. Fixing some memory leaks. 16 years ago
Pablo Rincon d0404d8447 Renaming errors with naming conventions 16 years ago
Pablo Rincon ad2c136e8f Renaming errors (naming conventions) 16 years ago
Victor Julien 6a53ab9c5a Stream engine memory handling update
The stream engine memory handling needed updating as it didn't scale. Changes:

- pools can now be initialized to size 0, meaning unlimited
- stream engine uses a memcap setting. Sessions, segments and aldata is part
  of this, app layer state isn't.
- memory is accounted using a global int that is spinlocked.
- a counter for sessions that have not been picked up because of memcap was
  added.
- all reassembly errors are converted to debug msgs.
16 years ago
Gurvinder Singh 356a8bf385 applayer uri match and modified http handling 16 years ago
Victor Julien c352bff6fb Remove unused conditional locking code from the app layer parsing code. 16 years ago
Pablo Rincon 705471e4ee Adding single pattern matcher algorithms. If you cannot store a context for the patterns, use SpmSearch() macro. Adding unittests and stats 16 years ago
Victor Julien cae8e06cb9 Properly lock app layer result pool and add some debugging code for memory tracking. 16 years ago
Gurvinder Singh 66cc392177 init b46 16 years ago
Victor Julien 4824868766 Application layer detection improvements
- improve locking of application layer handling, making sure that the flow cannot be freed/cleared when the detection engine is still working with it.
- add a check to the app layer detection to make sure that a match function will only inspect an app layer state if it's of the right type.
16 years ago
Gurvinder Singh d8433c7255 fixed-pool-error-and-tcp-state-transition 16 years ago
Victor Julien ecf86f9c23 Rename to Suricata. 16 years ago
Victor Julien 18fe3818dc Remove need_lock from app layer parsers. 16 years ago
Victor Julien ba7e8012af Add some debugging and simplify locking for app layer slightly. 16 years ago
Gurvinder Singh fc2f7f29fa app layer htp error handling and fixes for memory leaks and segv 16 years ago
Victor Julien d388444ac3 Use updated mutex calls. 16 years ago
Gurvinder Singh ad3e463974 updated error info ouput 16 years ago
Gurvinder Singh 1b39e602d0 fixed port info 16 years ago
Pablo Rincon e26833be3f Changing mutex/spinlocks/conditions naming types 16 years ago
Pablo Rincon 769022f4be Adding support for Mac OS X, FreeBSD, centrailizing mutex/spins/conditions in a macro API, and some unittests 16 years ago
Gurvinder Singh c1e485cc44 app layer error handling 16 years ago
Victor Julien 574bcea09d initial version of better error checking/handling in the app layer code 16 years ago
Victor Julien f1f7df0766 First iteration of doing app layer detection. 16 years ago
Gurvinder Singh a16e7b7455 tls no reassembly support 16 years ago
William Metcalf 5fc3005103 added check for full al_parser_table 16 years ago
root f3e3d3873f fix smb and dcerpc unit tests 16 years ago
Victor Julien 4914d8d903 Small stream fixes. 16 years ago
Victor Julien bcc5bbef93 Yet more logging api usage changes. 16 years ago
Victor Julien 91bc83e5c6 More logging API usage changes. 16 years ago
Victor Julien b3cb29b758 Fix engine lockup due to mutex locking error. 16 years ago
Victor Julien 4170ec8955 Make locking of the flow optional in the app layer subsys so we can also pass locked flows to it. 16 years ago
Victor Julien 5ecd187b6f Tie app layer parsing to the stream engine. 16 years ago
Victor Julien b102ea2123 Big update:
- Implement "closing" state in flow.
- Add protocol specific timeouts.
- Lots of stream tracking updates, fixing a lot of out of window issues.
- Stream reassembly fixes.
- Implement a new IDS runmode with 4 stream and detect threads.
- Added a BUG_ON macro that aborts the engine if the expression is true.
- Better balance the flow queue handler for traffic that doesn't have flow (like icmp currently).
- Simplify application level protocol in the Tcp Session.
- Add some debugging memory counters.
16 years ago
Victor Julien 086ba5f49b Add 'BySize' field parser. Add stub tls parser. 16 years ago
Victor Julien 7715e8f0fc Work around some Tcp session free issues in the app layer parsers. 16 years ago
Victor Julien 4369816cdd Improvements to content keyword memory handling.
First version of a simple pattern based L7 proto detection engine. Currently just works by matching a single pattern in the initial data. Implemented HTTP, SSL, MSN, JABBER, SMTP and a few more.

Couple of pattern matcher cleanups.
16 years ago
Victor Julien 5b946443d8 Use finer grained locking for app layer parsing. 16 years ago
Brian Rectanus fa5939ca91 64 bit cleanup part2 16 years ago
Victor Julien 9f78d47c2a Further work on the stream L7 parser, it's api and the http stub implementation. 16 years ago
Victor Julien 5a9a23f9bb Update to the parsers. 16 years ago
Victor Julien 8e10844f95 Initial code of Application Layer parsing framework. Rename of L7* to AppLayer*. 16 years ago