Anoop Saldanha
058e9278c5
Fix wrong casting of htp pointer. Fixed it back to (HTPState *) inside
...
htp utility functions.
12 years ago
Anoop Saldanha
ab4b15c2e7
fix for #788 .
...
Now depth is kept in mind when we inspect chunks in client/server body.
This takes care of FPs originating from inspecting subsequent chunks that
match with depth, but shouldn't.
12 years ago
Anoop Saldanha
d4d18e3136
Transaction engine redesigned.
...
Improved accuracy, improved performance. Performance improvement
noticeable with http heavy traffic and ruleset.
A lot of other cosmetic changes carried out as well. Wrappers introduced
for a lot of app layer functions.
Failing dce unittests disabled. Will be reintroduced in the updated dce
engine.
Cross transaction matching taken care of. FPs emanating from these
matches have now disappeared. Double inspection of transactions taken
care of as well.
12 years ago
Eric Leblond
6d225378e4
Workaround function missing in libhtp include
...
As reported in bug #688 , htp_config_set_path_decode_u_encoding
function is not included in libhtp header before 0.3.0. Result
is that suricata compilation fail with an external htp library.
The following patch detect the issue and adds the missing
declaration.
13 years ago
Victor Julien
cc51eec59d
Use new libhtp query string normalization. Bug #739 .
13 years ago
Victor Julien
9f519e95a2
http: add event for libhtp detection of request port not matching tcp port.
13 years ago
Last G
8ae11f73b2
Added parentheses to fix Eclipse static code analysis
...
Fixed bug in action priority (REJECT_DST had lowest prio)
13 years ago
Victor Julien
5cd46433d3
http: now that htp_state has a cfg reference, use it for body limits
13 years ago
Victor Julien
2763a61213
http: allow configuration of request and response body inspection limits. Issue #560 .
13 years ago
Victor Julien
aa4ae98d37
http: fix multipart parsing leading to missing chunks of files in file extraction.
13 years ago
Anoop Saldanha
028c6c1782
Make available custom features of libhtp.
...
The power of libhtp customisation now available to users.
Options available -
path-backslash-separators: yes
path-compress-separators: yes
path-control-char-handling: none
path-convert-utf8: yes
path-decode-separators: yes
path-decode-u-encoding: yes
path-invalid-encoding-handling: preserve_percent
path-invalid-utf8-handling: none
path-nul-encoded-handling: none
path-nul-raw-handling: none
set-path-replacement-char: ?
set-path-unicode-mapping: bestfit
You can use this for your libhtp customisation. Options explained in our
wiki.
https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Advanced_libhtp_customization
13 years ago
Anoop Saldanha
340542c44e
refactor htpconfigure()
13 years ago
Eric Leblond
e176be6fcc
Use unlikely for error treatment.
...
When handling error case on SCMallog, SCCalloc or SCStrdup
we are in an unlikely case. This patch adds the unlikely()
expression to indicate this to gcc.
This patch has been obtained via coccinelle. The transformation
is the following:
@istested@
identifier x;
statement S1;
identifier func =~ "(SCMalloc|SCStrdup|SCCalloc)";
@@
x = func(...)
... when != x
- if (x == NULL) S1
+ if (unlikely(x == NULL)) S1
13 years ago
Victor Julien
fcc21ae4cc
http: fix multipart parsing bug
13 years ago
Victor Julien
869109a6a0
stream/app layer: add Truncate app layer callback that is called if stream depth is reached. Use it to trunc open files in HTTP.
13 years ago
Eric Leblond
92679442ca
Convert to atomic and disable check on HTP config change.
...
This patch converts the series of variable to an atomic.
Furthermore, as the callbacks are now always run, it is not
necessary anymore to refuse a ruleswap if HTP parameters are
changing.
13 years ago
Eric Leblond
66a083dafa
Get rid of AppLayerHtpRegisterExtraCallbacks
...
This patch add a early exit condition to the body handling callback.
This permits to avoid to avoid a complex system to handle htp
object change.
13 years ago
Victor Julien
e737e2dc56
http: after path double decoding, also normalize the path again. #504 .
13 years ago
Victor Julien
e839cea9e5
Http: don't double decode URI path and query by default. Instead add per server options to enable double decoding for both cases. #464 #504 .
13 years ago
Victor Julien
ad827ad030
http: add more decoding unittests.
13 years ago
Victor Julien
43c7fd7585
file inspection: improve logging when stream.depth limit is reached. #493 .
13 years ago
Victor Julien
0c98980e21
http: add unittest to test \r in header line.
13 years ago
Anoop Saldanha
6fa46d7526
If new ruleset requires any htp callbacks that aren't already set, don't load new ruleset; request user to restart suricata + disable setting fileinsepection flags unconditionally in main
13 years ago
Victor Julien
ab3fcb01f9
http: decode double decoded path and query string characters. Bug #464 .
13 years ago
Victor Julien
f2f8dfd8d6
http: add test to make sure a missing space between header name and value is not a problem (ref #474 ).
13 years ago
Victor Julien
00948c86d5
Add debug messages to HTTP error/warning handling.
13 years ago
Anoop Saldanha
64625675ce
set stream_eof flag per stream, only when the stream initiates a close. Fix htp parser to close connection per direction based on this
13 years ago
Anoop Saldanha
608f4fe787
bug 452 - enable http extra callbacks for configs other than the default configs
14 years ago
Victor Julien
d378b76c04
http: body inspection improvement
...
Improve http_client_body and file_data performance when request and
response body limits are set to high values.
14 years ago
Victor Julien
4354434522
Add htp error debug printing.
14 years ago
Victor Julien
18837dce92
http: improve multipart parsing, skip empty records.
14 years ago
Victor Julien
18e81b7ba9
Bail out early if we're in http tunnel mode.
14 years ago
Anoop Saldanha
109662450d
Add new command line option --list-app-layer-protocols to list supported app layer protocols in sigs
14 years ago
Victor Julien
fe9258f0fb
Fix issue discovered by Anoop. Passing u32 ptr to a size_t can caused badness.
14 years ago
Victor Julien
e3935a2af2
Improve http filename parsing.
14 years ago
Victor Julien
e21d8cdf01
file extract: improve multipart parsing and set events on some error conditions.
14 years ago
Victor Julien
cdba2f50d1
Various fixes and improvements based on feedback by Coverity analyzer.
14 years ago
Victor Julien
3702a33ae9
file-inspection: support POST requests that do not use multipart.
14 years ago
Victor Julien
64827e3864
file-inspection: use filename= value from Content-Disposition where available to determine the filename in GET requests.
14 years ago
Victor Julien
93d121bf21
Update app layer events for HTTP now that libhtp has fixes for some response errors.
14 years ago
Victor Julien
16cfae2f51
Trigger raw stream reassembly on receiving a full HTTP request or response.
14 years ago
Victor Julien
f713b653ab
Convert error logging for HTTP to use new app layer event API. Expose libhtp warnings to this as well.
14 years ago
Victor Julien
a6e75aff21
file-extraction: improve handling of complex multipart bodies.
14 years ago
Anoop Saldanha
4b8ebb5c53
set default response body limit for specific http server conf
14 years ago
Anoop Saldanha
6240131a4e
updates to accomodate master rebase
14 years ago
Anoop Saldanha
7c9d1b80fd
Update size parsing API with new calls for returing u8, u16, u32 and u64 values. Make updates in the codebase to use these new calls
14 years ago
Anoop Saldanha
e0c13434ef
bug 333 - support new Size Parsing API. Update various conf params inside the engine to use this API to parse sizes in the format xxx <-just the no represents bytes, xxxkb <- kilobytes, xxxmb <- megabytes, xxxgb <- gigabytes, where xxx is a \d+
14 years ago
Victor Julien
004b5dde88
Support libhtp's different handling of CONNECT requests.
14 years ago
Victor Julien
33848124d1
Fix a multipart body parsing issue.
14 years ago
Victor Julien
d59ca75e46
file extract: split toserver and toclient tracking
...
Split toserver and toclient file tracking for the http state.
14 years ago
Victor Julien
1c934acc85
Don't store fd per file (too many fd's). Enable IPv6 storing. Close file on receiving stream end flag.
14 years ago
Victor Julien
b402d97179
File carving -- enable reponse file extraction
...
- Enable response body tracking
- Enable file extraction for responses
- File store meta file includes magic, close reason.
- Option to force magic lookup for all stored files.
- Fix libmagic calls thead safety.
14 years ago
Victor Julien
66a3cd96a8
Prepare HTTP response body tracking.
14 years ago
Victor Julien
e1022ee5ae
file-extraction: Disconnect file handling from flow and move into the app layer state.
14 years ago
Victor Julien
23e01d23d3
Implement filestore keyword, including a way for the stateful detection engine to conclude that a file will never have to be stored.
14 years ago
Victor Julien
403b2788d6
Add support for extracting PUT files.
14 years ago
Victor Julien
59cda9a358
Fix not using new htp callback when using the bundled htp. Add indication to --build-info. Fix valgrind warning in test and further improve test.
14 years ago
Victor Julien
ef0536794c
Adding comments, some cleanups.
14 years ago
Victor Julien
21acd72adf
Cleanups to the Multipart parsing code. Fixes to negation in filename and fileext.
14 years ago
Victor Julien
32fb9f375d
log-file log-dir option added, meta file created, fixes.
14 years ago
Victor Julien
a6b7a560f1
Fix a bug in the HTTP file closing.
14 years ago
Victor Julien
7e3d537338
Fix setting libhtp personality.
14 years ago
Victor Julien
cd618e48df
Allow for 0 (unlimited) HTTP request_body_limit, fix option parsing.
14 years ago
Victor Julien
4723f07254
Improve testing and fix some bugs.
14 years ago
Victor Julien
9d5d46c4bb
Implement flow file storage API, create HTP wrappers for it, use it in HTTP parsing.
14 years ago
Victor Julien
a0ee6ade3e
Improve HTTP multipart parsing, add streaming parsing for files.
14 years ago
Victor Julien
4537f889ef
Handle all strings as raw strings in HTTP content-type and content-disposition header parsing.
14 years ago
Pablo Rincon
6d60b3a747
filename and fileext keywords
14 years ago
Anoop Saldanha
9a6aef459e
modify all relevant app layer API calls to accomodate passing parser local storage argument
14 years ago
Victor Julien
262a7300d7
flow: shrink Flow datatype
...
Introduce a separate FlowAddress structure for holding the ipv4 or ipv6 address
that doesn't have the family in it like the Address structure. Instead, the
family is stored in the flow as a flag: FLOW_IPV4 and FLOW_IPV6.
Add macro's to check the family, copy the address, etc.
Update many unittests to reflect these changes. Introduce unittest helper
functions for creating and initializing a flow and freeing it again.
On 64 bit this shrinks the flow with 8 bytes.
14 years ago
Victor Julien
06904c9024
App Layer cleanup
...
Removal of per flow 'aldata' array. It contained a ptr for each ALPROTO. Instead now we have 2 ptrs in the flow: alparser and alstate.
Various cleanups and dead code removal from the app layer API.
Should safe 100+ bytes memory per flow on 64 bit.
Updated lots of unittests to reflect these changes.
14 years ago
Anoop Saldanha
4650bf7170
minor code cleanup. remove commented out code
14 years ago
Victor Julien
2d16abcf8b
Minor code cleanups fixing all GCC 4.6 compiler warnings for default, debug and unittests mode.
14 years ago
Victor Julien
5395071c11
Make http logging code more robust against cases where the htp state is incomplete (out of memory conditions).
14 years ago
Eric Leblond
60a99915c1
doc: create http support group
...
This patch create an httplayer group and adds related files to
it. It also fixes some typo in documentation string and format.
14 years ago
Eric Leblond
a64eea9628
Fix minor error message.
14 years ago
Eric Leblond
92d74fd480
doc: Add missing params in func description.
14 years ago
Victor Julien
bde55578d6
Override HTP IDS personality normalizing the query string to lowercase. Bug #362 .
14 years ago
Victor Julien
3644e90a2c
Don't set higher transaction id's in HTTP sessions than we have.
14 years ago
Anoop Saldanha
4e44073c79
http logging module should log all txs in the list and not just the last complete tx available on EOF
14 years ago
Anoop Saldanha
b406af451b
updates to http tx id vars. FFR now flags the app layer session for EOF when creating a pseudo packet for a flow
14 years ago
Anoop Saldanha
41d71a6d70
fix http http transaction id update. Update transactions as soon as we receive a callback on new request
14 years ago
Victor Julien
0625d54267
Improve HTPParserTest07 test to be more helpful if it fails.
14 years ago
deltay
170efc8d38
Register http parser callbacks in the right place.
14 years ago
Gerardo Iglesias Galvan
a56592e556
Make sure we do all after the null check in HTPStateFree
14 years ago
Gerardo Iglesias Galvan
c4832814b4
Prevent a memory leak on low memory conditions in http client body handling
14 years ago
Victor Julien
63f6de58cb
Fix HTP unittests that test pre 0.2.6 libhtp issue. HTP config wasn't restored properly.
14 years ago
Victor Julien
326047eec1
Add unittests for debugging a libhtp issue.
14 years ago
Victor Julien
6384b39f18
Remove unused and broken htp code.
14 years ago
Victor Julien
36917c7d66
Fix not using new htp callback when using the bundled htp. Add indication to --build-info. Fix valgrind warning in test and further improve test.
14 years ago
Victor Julien
a3e2b35536
Add configure check for new htp 0.2.5 uri normalize hook.
14 years ago
Victor Julien
15ce850387
Add support for new libhtp htp_config_register_request_uri_normalize callback.
14 years ago
Victor Julien
149ee6b648
Disable to_client http detection. Libhtp expects to_server data first.
14 years ago
Victor Julien
9a58a02559
Wrap HTP code that is only used in debug mode in debug ifdefs.
14 years ago
Anoop Saldanha
000ce98cd1
push all proto detection code into their respective app parser register functions for every alproto
14 years ago
Victor Julien
c88630639e
Fix setting libhtp personality.
15 years ago
Victor Julien
dbe291bc50
Allow for 0 (unlimited) HTTP request_body_limit, fix option parsing.
15 years ago
Victor Julien
99fca03810
Move unittest code into UNITTESTS ifdefs in the HTP parser. Fixes a compiler warning.
15 years ago
Victor Julien
edeec290f6
Fix missing rename for request-body-limit to request_body_limit.
15 years ago
Victor Julien
a3303fcf9d
Rename request-body-limit to request_body_limit to remain consistant with other options. Keep old notation around for compatibility.
15 years ago
Victor Julien
6fca55e068
Add some debug output to app-layer-htp.
15 years ago
Victor Julien
743ed7626c
Fix potential null deref (introduced a few commits ago) found by clang.
15 years ago
Victor Julien
16e4e3fe50
Fix request-body-limit option for libhtp config.
15 years ago
Victor Julien
39a5348d2b
Remove dead pcre code.
15 years ago
Victor Julien
6ebe7b7cd3
Change the way the request body limit is enforced.
15 years ago
Anoop Saldanha
5c6a65dc58
support relative modifiers for http_client_body. Introduce body processing engine in detect-engine-hcbd.[ch]
15 years ago
Victor Julien
bc55fb27dc
Compiler warning fix for memory macro's. Small layout changes.
15 years ago
Pablo Rincon
06a65cb460
moving http_client_body logic to use it per transactions. Adding unittests
15 years ago
Victor Julien
b8fec77f37
Fix tcp connections that are reset (RST packet) not always inspecting the reassembled stream. Update transaction id code to make sure both directions of a transaction are inspected before incrementing the inspect_id.
15 years ago
Pablo Rincon
8cc525c939
UDP support at AppLayer message handling
15 years ago
Gurvinder Singh
a0fa924c15
fixed the memory leaks in htp and radix tree
15 years ago
Victor Julien
70b32f7380
First stab at creating a stateful detection engine.
...
Stateful detection for app layer detection keywords, except uricontent. Stores it's partial results in the flow structure. Other modifications:
- Generalize transaction tracking, logging and inspection.
- Adapt http and dcerpc to use the new transaction handling.
- Stream engine now always notifies app layer of a stream eof.
This commit fixes bug #124 .
15 years ago
Gerardo Iglesias Galvan
9f4fae5b1a
Fix inconsistent use of dynamic memory allocation
15 years ago
William Metcalf
ce01927515
Import of GPLv2 Header 050410
16 years ago
Gurvinder Singh
5b802f6099
fixed type in htp (bug 138)
16 years ago
Victor Julien
f81fccd66b
Reintroduce usage of the SC_RADIX_NODE_USERDATA marco into the htp code. Rewrite the macro slightly, add unittests for the macro.
16 years ago
Gurvinder Singh
69a4fee757
fixed the API and logic error reported by clang tool
16 years ago
Victor Julien
78e15ea7fa
Explicitly test for ipv6 in the htp personalities code. Update all affected unittests to set addr family to the flow.
16 years ago
Victor Julien
47a47e8ad4
Fix invalid free in HTP config deinit.
16 years ago
Victor Julien
44b6380a70
Improve http body chunk memory handling robustness.
16 years ago
Pablo Rincon
fe7948a7ae
Modifications on http body request handling
16 years ago
Pablo Rincon
f862de2ee6
Fixing some code reviews (Thanks to Steve Grubb)
16 years ago
Victor Julien
4129146a71
Because the HTP personalities code changes how the htp state's connp is initialized, we need to check for it in more places.
16 years ago
Victor Julien
ead13bda4a
Small cleanup and comment update to htp code.
16 years ago
Brian Rectanus
a9cdd2bbae
Add htp personality configuration.
16 years ago
Victor Julien
ffd85ac4a6
Use correct datatype in HTPCallbackResponse fixing possible endless looping issues.
16 years ago
Anoop Saldanha
97d49d8f5e
support for http_client_body keyword
16 years ago
Gurvinder Singh
8e444f1772
stream and application layer improvements
16 years ago
Victor Julien
7a8cd61fdf
Cleanups.
16 years ago
Pablo Rincon
25a3a5c6d8
Adding mem wrapper to debug runtime alloc()/free() functions. Fixing some memory leaks.
16 years ago
Gurvinder Singh
50f7d0a887
app layer htp logging and better htp request handling. removed recent_in_tx.
16 years ago
Victor Julien
4f3a04a410
Disable htp cleanup code as I'm not yet convinced it does what it should.
16 years ago
Gurvinder Singh
bf236e4567
better htp memory handling & flow valgrind error fixed
16 years ago
Pablo Rincon
0165b3f0d8
pcre P modifier support (pcre match over http body requests)
16 years ago
Victor Julien
187949b9ad
Make urilen inspect the normalized uri, cleanup uri (error) handling.
16 years ago
Pablo Rincon
d0404d8447
Renaming errors with naming conventions
16 years ago
Pablo Rincon
ad2c136e8f
Renaming errors (naming conventions)
16 years ago
Victor Julien
148883cedf
Work around for unsupported CONNECT support handling.
16 years ago
Victor Julien
6a53ab9c5a
Stream engine memory handling update
...
The stream engine memory handling needed updating as it didn't scale. Changes:
- pools can now be initialized to size 0, meaning unlimited
- stream engine uses a memcap setting. Sessions, segments and aldata is part
of this, app layer state isn't.
- memory is accounted using a global int that is spinlocked.
- a counter for sessions that have not been picked up because of memcap was
added.
- all reassembly errors are converted to debug msgs.
16 years ago
Gurvinder Singh
356a8bf385
applayer uri match and modified http handling
16 years ago
Victor Julien
c352bff6fb
Remove unused conditional locking code from the app layer parsing code.
16 years ago
Pablo Rincon
705471e4ee
Adding single pattern matcher algorithms. If you cannot store a context for the patterns, use SpmSearch() macro. Adding unittests and stats
16 years ago
Victor Julien
c3269dbcb4
Fix compiler warning in http method code
16 years ago
Brian Rectanus
c22d42693a
Added http_method rule keyword.
16 years ago
Gurvinder Singh
6814ea1a0f
some more stream fixes
16 years ago
Victor Julien
4824868766
Application layer detection improvements
...
- improve locking of application layer handling, making sure that the flow cannot be freed/cleared when the detection engine is still working with it.
- add a check to the app layer detection to make sure that a match function will only inspect an app layer state if it's of the right type.
16 years ago
Gurvinder Singh
0a85fd6787
htp error fixed
16 years ago
Victor Julien
ecf86f9c23
Rename to Suricata.
16 years ago
Victor Julien
18fe3818dc
Remove need_lock from app layer parsers.
16 years ago
Gurvinder Singh
7ce586bc77
updated htp error info
16 years ago