Commit Graph

257 Commits (8f15e30a58e2f619555f2bd5ab7eaee7b42e88e2)

Author SHA1 Message Date
Victor Julien 579d6d3ff7 http: allow lower/mixed case in proto detect
In HTTP detection registered patterns were upper case only. Since the
detection is based on both sides this would still work for sessions
where one of the talkers misbehaved. If both sides misbehave this
would fail however, so this patch introduces case insensive matching.
9 years ago
Victor Julien 44022743f2 http: track if request/response have trailers 9 years ago
Giuseppe Longo 675fa56497 app-layer: add ThreadVars to AppLayerParserParse
To be able to add a transaction counter we will need a ThreadVars
in the AppLayerParserParse function.
This function is massively used in unittests
and this result in an long commit.
9 years ago
Victor Julien 6530c3d0d8 unittests: replace SCMutex* calls by FLOWLOCK_* 9 years ago
Victor Julien 5ec885e451 http: set of response body decompress limit
This is a per personality setting.
9 years ago
Victor Julien b3bf7a5729 output: introduce config and perf output levels
Goal is to reduce info output
9 years ago
Victor Julien feafc838db http: make htpstate cfg ptr const 9 years ago
Victor Julien 24a2f51569 http: move body settings into per dir struct 9 years ago
Victor Julien 6fb808fc1a http: add per direction config for body parsing
The HTPCfgDir structure is meant to contain config for per direction
body parsing parameters.

This patch stores the streaming API config.
9 years ago
Victor Julien 46e55f1e34 http body handling: use streaming buffer API
Convert HTTP body handling to use the Streaming Buffer API. This means
the HtpBodyChunks no longer maintain their own data segments, but
instead add their data to the StreamingBuffer instance in the HtpBody
structure.

In case the HtpBodyChunk needs to access it's data it can do so still
through the Streaming Buffer API.

Updates & simplifies the various users of the reassembled bodies:
multipart parsing and the detection engine.
9 years ago
Mats Klepsland c52fe9a5fa http: register logger functions 9 years ago
Mats Klepsland c4b918b6c4 tx: do not store ProgressCompletionStatus per ipproto
Change AppLayerParserRegisterGetStateProgressCompletionStatus to
only store one ProgressCompletionStatus callback function for each
alproto, instead of storing one for each ipproto.

This enables us to use AppLayerParserGetStateProgressCompletionStatus
in functions where we do not know the ipproto used.
9 years ago
Victor Julien 83e0529b2b http: flag destate about new files
The stateful detection engine needs some assistance when inspecting
transactions with multiple files. This patch flags the detect state
(if any) about the availability of new files in http. For http it
should only apply to multipart bodies although the flag is set for
all files.
9 years ago
Victor Julien 077ac81688 QA: direct access from commandline to AppLayer API
This patch introduces a new set of commandline options meant for
assisting in fuzz testing the app layer implementations.

Per protocol, 2 commandline options are added:

--afl-http-request=<filename>
--afl-http=<filename>

In the former case, the contents of the file are passed directly to
the HTTP parser as request data.

In the latter case, the data is devided between request and responses.
First 64 bytes are request, then next 64 are response, next 64 are
request, etc, etc.
9 years ago
Jason Ish 796dd5223b tests: no longer necessary to provide successful return code
1 pass, 0 is fail.
9 years ago
Victor Julien 444c4b546e htp: optimize multipart parsing
Store boundary in a single stack buffer instead of 2 heap allocated
buffers.
10 years ago
Victor Julien 665cafbe3b htp: fix boundry error handling (CID 1153934) 10 years ago
Victor Julien bfc4be230c htp: fix dead code warning (CID 1153933) 10 years ago
Victor Julien 83f400dd60 http: remove unused param from HtpBodyAppendChunk 10 years ago
Giuseppe Longo 7a29aa116e http: close file when http body limit is reached
In some conditions, if stream.reassembly.depth is greater than
request/response-body-limit size, the logging output is wrong
if filestore keyword is used with http.

For example, we get:
{... "app_proto":"http","fileinfo":{"filename":"\/file.pdf","state":"CLOSED","stored":false,"size":1049292,"tx_id":0}}

"state":"CLOSED","stored":false should be "state":"TRUNCATED","stored":true.

This happens because the file state and file flags,
which is the information that determine a correct output,
are not set properly since a file is logged before and then closed (HTPFileClose).

The logic of this patch is to close a file when we are above
the limits, such that the proper state and flags can be set
and the file will be logged correctly.
10 years ago
Victor Julien b25bd2e180 http: fix NULL deref on certain out of memory conditions 10 years ago
Victor Julien 0a22ba7e23 http: fix multipart body tracking slowdown
Optimize HTTP multipart body parsing. Big records that were not files
could slow down Suricata. The reason was that the body tracker was not
moved forward. This lead to growing body buffers, which were expensive
wrt memory and inspection.

This patch add logic to move the tracker forward in this case.
10 years ago
Victor Julien efdd9e08f2 http: improve body pruning
In case the body wasn't inspected the body_inspected variable wouldn't
get updated leading to the body not getting pruned at all.

This patch adds support for this case.
10 years ago
Victor Julien a1075ee2f2 http: add test for plain http over connect 10 years ago
Victor Julien e86e27bab7 http: test cleanups 10 years ago
Victor Julien 77624701e7 htp: cleanup, remove unused declaration 10 years ago
Victor Julien 1235c578b2 xff: support ports and more ipv6 notations
It's not uncommon to see an header like:
X-Forwarded-For: 1.2.3.4:56789

This patch recognizes this case and ignores the port. It also supports
this for IPv6 if the address has the following notation:
X-Forwarded-For: [12::34]:1234

This patch also adds unittests.
10 years ago
Victor Julien bbc9874b81 http: destroy htp_tx_t even if incomplete 10 years ago
Victor Julien b8211e8c04 htp: hide BUG_ON's behind DEBUG_VALIDATION 10 years ago
Victor Julien b3b7625be5 htp: fix test 10 years ago
Victor Julien cf9ff6adbd app-layer: improve EOF handling
On receiving TCP end of stream packets (e.g. RST, but also sometimes FIN
packets), in some cases the AppLayer parser would not be notified. This
could happen in IDS mode, but would especially be an issue in IPS mode.

This patch changes the logic of the AppLayer API to handle this. When no
new data is available, and the stream ends, the AppLayer API now gets
called with a NULL/0 input, but with the EOF flag set.

This allows the AppLayer parser to call it's final routines still in the
context of a real packet.
10 years ago
Victor Julien 79d34f4575 http: body pruning update
Take inspect limits into account. Consider per direction inspect settings.
10 years ago
Giuseppe Longo a459376d2e app-layer-htp: add http_body_inline setting 10 years ago
Victor Julien 9fa2f85cc7 http: improve body pruning
Take inspect window into account.
10 years ago
Victor Julien 0bbc818b2d http: fix body tracking
In HTTP body tracking for response bodies, pruning body chunks was broken
as the body parsing code wouldn't update HtpBody::body_parsed.
10 years ago
Victor Julien cb5aa8f8d5 counters: work around unix-socket init issues 10 years ago
Victor Julien cddbb0f606 http: make http.memuse a global counter
http.memcap as well.
10 years ago
Victor Julien f536099a67 app-layer: de_state optimization
Add API to bypass expensive TX list walks. This API call is optional.

Implement it for HTTP and DNS.
10 years ago
Victor Julien 52195a4192 http: add event for leading spaces on request line
Libhtp will issue a warning in this case, so we can match on this.
This patch adds event, rule and unittest.
10 years ago
maxtors e7658fd4f4 Added new WebDAV http methods.
Added more WebDAV functions. A complete list of what http
methods libhtp can handle can be found at:
https://github.com/OISF/libhtp/blob/0.5.x/htp/htp_core.h#L260.
So now the methods array reflects these available functions.

The comments have also been changed to reflect the desired style.
10 years ago
maxtors 3b26b07963 Reworked pattern registration structure and feedback issues.
1) Reworked pattern registration for http methods and versions.

Instead of being a manual and verbose action of adding one
and one http method with N-amount if prefix spacings and
the same for HTTP versions (eg. HTTP/1.1) i moved it all
to be loop based actions reading values from char arrays.

In the future all that is needed is to add new methods
to the arrays and they will be added as a pattern.

2) Modified pattern registration after feedback.

Changed variable used in snprintf for http method registration
Should have been size of dest buffer at not another var (catsize)
that i had created. Also removed this variable.

Fixed a typo in the comment for registering http versions.
TO_CIENT -> TO_CLIENT.
10 years ago
maxtors 7a9e963619 Added http methods for webdav (rfc4918).
Included the following methods to app-layer-htp.c:
- PROPFIND
- PROPPATCH
- MKCOL
- COPY
- MOVE
- LOCK
- UNLOCK
10 years ago
maxtors a7abaf538c Added DELETE and PATCH http methods to app-layer-htp.c
Added two new http methods to the http decoding done in
app-layer-htp.c so that these messages will be decoded as well.
10 years ago
Victor Julien 774bb90351 http: clean up tx destate at tx destroy 11 years ago
Victor Julien 6279da0fbd http: support per TX destate storage 11 years ago
Victor Julien e78e33a428 http: add event for suspicious method delimeter
Add event and rule for suspicious delim(s) between method and uri.

Add unittests as well.
11 years ago
Victor Julien 5ad7198dc0 http: add libhtp uri warning event
Add event for libhtp warning added 0.5.17 for URI's with suspicious
delimeters.
11 years ago
Victor Julien e90714afd6 http: remove unused and broken 'content-len' logic
The HTTP tracking code would parse the content lenght and store it
in the TX user data. It didn't take the possibility or errors into
account though, leading to a possible negative int being cases to
unsigned int. Luckily, the result was unused.

This patch simply removes the offending code.

Reported-by: The Yahoo pentest team
11 years ago
Ken Steele 8f1d75039a Enforce function coding standard
Functions should be defined as:

int foo(void)
{
}

Rather than:
int food(void) {
}

All functions where changed by a script to match this standard.
11 years ago
sxhlinux c85674b0a6 Update app-layer-htp.c
When keyword "boundary=" doesn't exist in the http_header, the value of r is 0 and this condition shouldn't return 0 instead 1;
11 years ago