Commit Graph

226 Commits (df79c1019f6315154b1527fb190d57f9a865598f)

Author SHA1 Message Date
Victor Julien 129b6a65ca http: add test for HTTP_DECODER_EVENT_UNKNOWN_ERROR event as a result of a too long request 12 years ago
Anoop Saldanha 9e4eec200f Update htp event handler to both warning and error events regardless of any conditions. 12 years ago
Anoop Saldanha 5e2d9dbdc3 Add and use EventGetInfo for getting info on an event.
Also update existing parsers and app-layer-event Setup to use this.
12 years ago
Anoop Saldanha 6cb0014287 Move app event module registration as a part of app layer proto table. 12 years ago
Anoop Saldanha 0d7159b525 App layer protocol detection updated and improved. We now use
confirmation from both directions and set events if there's a mismatch
between the 2 directions.

FPs from corrupt flows have disappeared with this.
12 years ago
Anoop Saldanha ddde572fba Introduce new options into the conf file to enable/disable -
1. Proto detection
2. Parsers

For app layer protocols.

libhtp has now been moved to the section under app-layer.protocols.http,
but we still provide backward compatibility with older conf files.
12 years ago
Victor Julien 7f8d256e7c Fix tests that didn't expect radix to be freed 12 years ago
Victor Julien 6f2cb141cf Http: improve tx data cleanup 12 years ago
Victor Julien 67c12c61d3 Http: fix memory leaks when cleaning up our per-tx storage 12 years ago
Anoop Saldanha 94e2527606 Introduce a saner way to validate the completion of request and
response bodies.

Also don't change app state for http from inside inspection.
12 years ago
Anoop Saldanha dcdcbd9721 Fix creating a backup of htp config. This is used by unittests that
changed htp config.
12 years ago
Anoop Saldanha 56143131da Fix unittests that use chunked encoding. 12 years ago
Ken Steele 5532af4621 Create SCMUTEX_INITIALIZER to abstract out PTHREAD_MUTEX_INITIALIZER
This allows replacing pthread mutexes with other types of mutex.
12 years ago
Eric Leblond cd3e32ce19 unittests: some functions needs a flow lock.
In debug validation mode, it is required to call application layer
parsing and other functions with a lock on flow. This patch updates
the code to do so.
12 years ago
Victor Julien 9faa4b740d Add --unittests-coverage option to list how many code modules have tests 12 years ago
Anoop Saldanha 48cf0585fb Suricata upgrade to libhtp 0.5.x.
Remove the support for now unsupported personalities from libhtp -
TOMCAT_6_0, APACHE and APACHE_2_2.  We instead use the APACHE_2
personality.
12 years ago
Victor Julien 080c15b3fc Enable libhtp 0.3.0 compilation and crash free UT run. Still see 5 failed tests. 12 years ago
Victor Julien 0fd9b0c4fa HTP: free TX from transaction free API call 12 years ago
Victor Julien f59f90331d Applayer: remove obsolete StateUpdateTransactionId
Also, update StateTransactionFree to take an u64 tx id, so it's
consistant with the rest of the engine.

To reflect these changes, AppLayerRegisterTransactionIdFuncs has
been renamed to AppLayerRegisterTxFreeFunc.

HTP, DNS, SMB, DCERPC parsers updated.
12 years ago
Anoop Saldanha 058e9278c5 Fix wrong casting of htp pointer. Fixed it back to (HTPState *) inside
htp utility functions.
12 years ago
Anoop Saldanha ab4b15c2e7 fix for #788.
Now depth is kept in mind when we inspect chunks in client/server body.
This takes care of FPs originating from inspecting subsequent chunks that
match with depth, but shouldn't.
12 years ago
Anoop Saldanha d4d18e3136 Transaction engine redesigned.
Improved accuracy, improved performance.  Performance improvement
noticeable with http heavy traffic and ruleset.

A lot of other cosmetic changes carried out as well.  Wrappers introduced
for a lot of app layer functions.

Failing dce unittests disabled.  Will be reintroduced in the updated dce
engine.

Cross transaction matching taken care of.  FPs emanating from these
matches have now disappeared.  Double inspection of transactions taken
care of as well.
12 years ago
Eric Leblond 6d225378e4 Workaround function missing in libhtp include
As reported in bug #688, htp_config_set_path_decode_u_encoding
function is not included in libhtp header before 0.3.0. Result
is that suricata compilation fail with an external htp library.
The following patch detect the issue and adds the missing
declaration.
13 years ago
Victor Julien cc51eec59d Use new libhtp query string normalization. Bug #739. 13 years ago
Victor Julien 9f519e95a2 http: add event for libhtp detection of request port not matching tcp port. 13 years ago
Last G 8ae11f73b2 Added parentheses to fix Eclipse static code analysis
Fixed bug in action priority (REJECT_DST had lowest prio)
13 years ago
Victor Julien 5cd46433d3 http: now that htp_state has a cfg reference, use it for body limits 13 years ago
Victor Julien 2763a61213 http: allow configuration of request and response body inspection limits. Issue #560. 13 years ago
Victor Julien aa4ae98d37 http: fix multipart parsing leading to missing chunks of files in file extraction. 13 years ago
Anoop Saldanha 028c6c1782 Make available custom features of libhtp.
The power of libhtp customisation now available to users.

Options available -

path-backslash-separators: yes
path-compress-separators: yes
path-control-char-handling: none
path-convert-utf8: yes
path-decode-separators: yes
path-decode-u-encoding: yes
path-invalid-encoding-handling: preserve_percent
path-invalid-utf8-handling: none
path-nul-encoded-handling: none
path-nul-raw-handling: none
set-path-replacement-char: ?
set-path-unicode-mapping: bestfit

You can use this for your libhtp customisation.  Options explained in our
wiki.

https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Advanced_libhtp_customization
13 years ago
Anoop Saldanha 340542c44e refactor htpconfigure() 13 years ago
Eric Leblond e176be6fcc Use unlikely for error treatment.
When handling error case on SCMallog, SCCalloc or SCStrdup
we are in an unlikely case. This patch adds the unlikely()
expression to indicate this to gcc.

This patch has been obtained via coccinelle. The transformation
is the following:

@istested@
identifier x;
statement S1;
identifier func =~ "(SCMalloc|SCStrdup|SCCalloc)";
@@

x = func(...)
... when != x
- if (x == NULL) S1
+ if (unlikely(x == NULL)) S1
13 years ago
Victor Julien fcc21ae4cc http: fix multipart parsing bug 13 years ago
Victor Julien 869109a6a0 stream/app layer: add Truncate app layer callback that is called if stream depth is reached. Use it to trunc open files in HTTP. 13 years ago
Eric Leblond 92679442ca Convert to atomic and disable check on HTP config change.
This patch converts the series of variable to an atomic.

Furthermore, as the callbacks are now always run, it is not
necessary anymore to refuse a ruleswap if HTP parameters are
changing.
13 years ago
Eric Leblond 66a083dafa Get rid of AppLayerHtpRegisterExtraCallbacks
This patch add a early exit condition to the body handling callback.
This permits to avoid to avoid a complex system to handle htp
object change.
13 years ago
Victor Julien e737e2dc56 http: after path double decoding, also normalize the path again. #504. 13 years ago
Victor Julien e839cea9e5 Http: don't double decode URI path and query by default. Instead add per server options to enable double decoding for both cases. #464 #504. 13 years ago
Victor Julien ad827ad030 http: add more decoding unittests. 13 years ago
Victor Julien 43c7fd7585 file inspection: improve logging when stream.depth limit is reached. #493. 13 years ago
Victor Julien 0c98980e21 http: add unittest to test \r in header line. 13 years ago
Anoop Saldanha 6fa46d7526 If new ruleset requires any htp callbacks that aren't already set, don't load new ruleset; request user to restart suricata + disable setting fileinsepection flags unconditionally in main 13 years ago
Victor Julien ab3fcb01f9 http: decode double decoded path and query string characters. Bug #464. 13 years ago
Victor Julien f2f8dfd8d6 http: add test to make sure a missing space between header name and value is not a problem (ref #474). 13 years ago
Victor Julien 00948c86d5 Add debug messages to HTTP error/warning handling. 13 years ago
Anoop Saldanha 64625675ce set stream_eof flag per stream, only when the stream initiates a close. Fix htp parser to close connection per direction based on this 13 years ago
Anoop Saldanha 608f4fe787 bug 452 - enable http extra callbacks for configs other than the default configs 13 years ago
Victor Julien d378b76c04 http: body inspection improvement
Improve http_client_body and file_data performance when request and
response body limits are set to high values.
13 years ago
Victor Julien 4354434522 Add htp error debug printing. 13 years ago
Victor Julien 18837dce92 http: improve multipart parsing, skip empty records. 14 years ago
Victor Julien 18e81b7ba9 Bail out early if we're in http tunnel mode. 14 years ago
Anoop Saldanha 109662450d Add new command line option --list-app-layer-protocols to list supported app layer protocols in sigs 14 years ago
Victor Julien fe9258f0fb Fix issue discovered by Anoop. Passing u32 ptr to a size_t can caused badness. 14 years ago
Victor Julien e3935a2af2 Improve http filename parsing. 14 years ago
Victor Julien e21d8cdf01 file extract: improve multipart parsing and set events on some error conditions. 14 years ago
Victor Julien cdba2f50d1 Various fixes and improvements based on feedback by Coverity analyzer. 14 years ago
Victor Julien 3702a33ae9 file-inspection: support POST requests that do not use multipart. 14 years ago
Victor Julien 64827e3864 file-inspection: use filename= value from Content-Disposition where available to determine the filename in GET requests. 14 years ago
Victor Julien 93d121bf21 Update app layer events for HTTP now that libhtp has fixes for some response errors. 14 years ago
Victor Julien 16cfae2f51 Trigger raw stream reassembly on receiving a full HTTP request or response. 14 years ago
Victor Julien f713b653ab Convert error logging for HTTP to use new app layer event API. Expose libhtp warnings to this as well. 14 years ago
Victor Julien a6e75aff21 file-extraction: improve handling of complex multipart bodies. 14 years ago
Anoop Saldanha 4b8ebb5c53 set default response body limit for specific http server conf 14 years ago
Anoop Saldanha 6240131a4e updates to accomodate master rebase 14 years ago
Anoop Saldanha 7c9d1b80fd Update size parsing API with new calls for returing u8, u16, u32 and u64 values. Make updates in the codebase to use these new calls 14 years ago
Anoop Saldanha e0c13434ef bug 333 - support new Size Parsing API. Update various conf params inside the engine to use this API to parse sizes in the format xxx <-just the no represents bytes, xxxkb <- kilobytes, xxxmb <- megabytes, xxxgb <- gigabytes, where xxx is a \d+ 14 years ago
Victor Julien 004b5dde88 Support libhtp's different handling of CONNECT requests. 14 years ago
Victor Julien 33848124d1 Fix a multipart body parsing issue. 14 years ago
Victor Julien d59ca75e46 file extract: split toserver and toclient tracking
Split toserver and toclient file tracking for the http state.
14 years ago
Victor Julien 1c934acc85 Don't store fd per file (too many fd's). Enable IPv6 storing. Close file on receiving stream end flag. 14 years ago
Victor Julien b402d97179 File carving -- enable reponse file extraction
- Enable response body tracking
- Enable file extraction for responses
- File store meta file includes magic, close reason.
- Option to force magic lookup for all stored files.
- Fix libmagic calls thead safety.
14 years ago
Victor Julien 66a3cd96a8 Prepare HTTP response body tracking. 14 years ago
Victor Julien e1022ee5ae file-extraction: Disconnect file handling from flow and move into the app layer state. 14 years ago
Victor Julien 23e01d23d3 Implement filestore keyword, including a way for the stateful detection engine to conclude that a file will never have to be stored. 14 years ago
Victor Julien 403b2788d6 Add support for extracting PUT files. 14 years ago
Victor Julien 59cda9a358 Fix not using new htp callback when using the bundled htp. Add indication to --build-info. Fix valgrind warning in test and further improve test. 14 years ago
Victor Julien ef0536794c Adding comments, some cleanups. 14 years ago
Victor Julien 21acd72adf Cleanups to the Multipart parsing code. Fixes to negation in filename and fileext. 14 years ago
Victor Julien 32fb9f375d log-file log-dir option added, meta file created, fixes. 14 years ago
Victor Julien a6b7a560f1 Fix a bug in the HTTP file closing. 14 years ago
Victor Julien 7e3d537338 Fix setting libhtp personality. 14 years ago
Victor Julien cd618e48df Allow for 0 (unlimited) HTTP request_body_limit, fix option parsing. 14 years ago
Victor Julien 4723f07254 Improve testing and fix some bugs. 14 years ago
Victor Julien 9d5d46c4bb Implement flow file storage API, create HTP wrappers for it, use it in HTTP parsing. 14 years ago
Victor Julien a0ee6ade3e Improve HTTP multipart parsing, add streaming parsing for files. 14 years ago
Victor Julien 4537f889ef Handle all strings as raw strings in HTTP content-type and content-disposition header parsing. 14 years ago
Pablo Rincon 6d60b3a747 filename and fileext keywords 14 years ago
Anoop Saldanha 9a6aef459e modify all relevant app layer API calls to accomodate passing parser local storage argument 14 years ago
Victor Julien 262a7300d7 flow: shrink Flow datatype
Introduce a separate FlowAddress structure for holding the ipv4 or ipv6 address
that doesn't have the family in it like the Address structure. Instead, the
family is stored in the flow as a flag: FLOW_IPV4 and FLOW_IPV6.

Add macro's to check the family, copy the address, etc.

Update many unittests to reflect these changes. Introduce unittest helper
functions for creating and initializing a flow and freeing it again.

On 64 bit this shrinks the flow with 8 bytes.
14 years ago
Victor Julien 06904c9024 App Layer cleanup
Removal of per flow 'aldata' array. It contained a ptr for each ALPROTO. Instead now we have 2 ptrs in the flow: alparser and alstate.
Various cleanups and dead code removal from the app layer API.
Should safe 100+ bytes memory per flow on 64 bit.
Updated lots of unittests to reflect these changes.
14 years ago
Anoop Saldanha 4650bf7170 minor code cleanup. remove commented out code 14 years ago
Victor Julien 2d16abcf8b Minor code cleanups fixing all GCC 4.6 compiler warnings for default, debug and unittests mode. 14 years ago
Victor Julien 5395071c11 Make http logging code more robust against cases where the htp state is incomplete (out of memory conditions). 14 years ago
Eric Leblond 60a99915c1 doc: create http support group
This patch create an httplayer group and adds related files to
it. It also fixes some typo in documentation string and format.
14 years ago
Eric Leblond a64eea9628 Fix minor error message. 14 years ago
Eric Leblond 92d74fd480 doc: Add missing params in func description. 14 years ago
Victor Julien bde55578d6 Override HTP IDS personality normalizing the query string to lowercase. Bug #362. 14 years ago
Victor Julien 3644e90a2c Don't set higher transaction id's in HTTP sessions than we have. 14 years ago
Anoop Saldanha 4e44073c79 http logging module should log all txs in the list and not just the last complete tx available on EOF 14 years ago
Anoop Saldanha b406af451b updates to http tx id vars. FFR now flags the app layer session for EOF when creating a pseudo packet for a flow 14 years ago