Added more WebDAV functions. A complete list of what http
methods libhtp can handle can be found at:
https://github.com/OISF/libhtp/blob/0.5.x/htp/htp_core.h#L260.
So now the methods array reflects these available functions.
The comments have also been changed to reflect the desired style.
1) Reworked pattern registration for http methods and versions.
Instead of being a manual and verbose action of adding one
and one http method with N-amount if prefix spacings and
the same for HTTP versions (eg. HTTP/1.1) i moved it all
to be loop based actions reading values from char arrays.
In the future all that is needed is to add new methods
to the arrays and they will be added as a pattern.
2) Modified pattern registration after feedback.
Changed variable used in snprintf for http method registration
Should have been size of dest buffer at not another var (catsize)
that i had created. Also removed this variable.
Fixed a typo in the comment for registering http versions.
TO_CIENT -> TO_CLIENT.
The HTTP tracking code would parse the content lenght and store it
in the TX user data. It didn't take the possibility or errors into
account though, leading to a possible negative int being cases to
unsigned int. Luckily, the result was unused.
This patch simply removes the offending code.
Reported-by: The Yahoo pentest team
By assuming that HTPCallbackRequestLine would always be run first,
an memory leak was introduced. It would not check if user data already
existed in the tx, causing it to overwrite the user data pointer is
it already existed.
Bug #1092.
The HTP config tree is a radix. The lookups are updated to the new API.
The return of user_data is treated as a succesful lookup, instead of
the node itself.
This patch introduces wrapper functions around allocation functions
to be able to have a global HTP memcap. A simple subsitution of
function was not enough because allocated size needed to be known
during freeing and reallocation.
The value of the memcap can be set in the YAML and is left by default
to unlimited (0) to avoid any surprise to users.
For AppLayerThreadCtx, AppLayerParserState, AppLayerParserThreadCtx
and AppLayerProtoDetectThreadCtx, use opaque pointers instead of
void pointers.
AppLayerParserState is declared in flow.h as it's part of the Flow
structure.
AppLayerThreadCtx is declared in decode.h, as it's part of the
DecodeThreadVars structure.
app-layer.[ch], app-layer-detect-proto.[ch] and app-layer-parser.[ch].
Things addressed in this commit:
- Brings out a proper separation between protocol detection phase and the
parser phase.
- The dns app layer now is registered such that we don't use "dnstcp" and
"dnsudp" in the rules. A user who previously wrote a rule like this -
"alert dnstcp....." or
"alert dnsudp....."
would now have to use,
alert dns (ipproto:tcp;) or
alert udp (app-layer-protocol:dns;) or
alert ip (ipproto:udp; app-layer-protocol:dns;)
The same rules extend to other another such protocol, dcerpc.
- The app layer parser api now takes in the ipproto while registering
callbacks.
- The app inspection/detection engine also takes an ipproto.
- All app layer parser functions now take direction as STREAM_TOSERVER or
STREAM_TOCLIENT, as opposed to 0 or 1, which was taken by some of the
functions.
- FlowInitialize() and FlowRecycle() now resets proto to 0. This is
needed by unittests, which would try to clean the flow, and that would
call the api, AppLayerParserCleanupParserState(), which would try to
clean the app state, but the app layer now needs an ipproto to figure
out which api to internally call to clean the state, and if the ipproto
is 0, it would return without trying to clean the state.
- A lot of unittests are now updated where if they are using a flow and
they need to use the app layer, we would set a flow ipproto.
- The "app-layer" section in the yaml conf has also been updated as well.
[src/app-layer-htp.c:1967] -> [src/app-layer-htp.c:1978]: (warning) \
Possible null pointer dereference: tx - otherwise it is redundant \
to check it against null.
The meta-field-option allows for setting the hard limit of request
and response fields in HTTP. In requests this applies to the request
line and headers, not the body. In responses, this applies to the
response line and headers, not the body.
Libhtp uses a default limit of 18k. If this is reached an event is
raised.
Ticket 986.
Strip the 'proxy' parts from the normalized uri as inspected by http_uri,
urilen, pcre /U and others.
In a request line like:
GET http://suricata-ids.org/blah/ HTTP/1.1
the normalized URI will now be:
/blah/
This doesn't affect http_raw_uri. So matching the hostname, etc is still
possible through this keyword.
Additionally, a new per HTTP 'personality' option was added to change
this behavior: "uri-include-all":
uri-include-all: <true|false>
Include all parts of the URI. By default the
'scheme', username/password, hostname and port
are excluded. Setting this option to true adds
all of them to the normalized uri as inspected
by http_uri, urilen, pcre with /U and the other
keywords that inspect the normalized uri.
Note that this does not affect http_raw_uri.
So adding uri-include-all:true to all personalities in the yaml will
restore the old default behavior.
Ticket 1008.
Libhtp decodes the + character in the query string to a space by default.
Suricata rules (e.g. etpro sid 2806767) are expecting to see the space in
the http_uri buffer.
Added an option per htp config to reenable this default behavior:
query-plusspace-decode: yes
Bug #1035.
This patch is a result of applying the following coccinelle
transformation to suricata sources:
@istested@
identifier x;
statement S1;
identifier func =~ "(SCMalloc|SCStrdup|SCCalloc|SCMallocAligned|SCRealloc)";
@@
x = func(...)
... when != x
- if (x == NULL) S1
+ if (unlikely(x == NULL)) S1
In case of recursive call to protocol detection from within protocol
detection, and the recursively invoked stream still hasn't been ack'ed
yet, protocol detection doesn't take place. In such cases we will end up
still calling the app layer with the wrong direction data. Introduce a
check to not call app layer with wrong direction data.
When sockets are re-used reset all relevant vars correctly.
This commit fixes a bug where we were not reseting app proto detection
vars.
While fixing #989, we discovered some other bugs which have also been
fixed, or rather some features which are now updated. One of the feature
update being if we recieve wrong direction data first, we don't reset the
protocol values for the flow. We let the flow retain the detected
values.
Unittests have been modified to accomodate the above change.
Some events we retrieved from error messages are flag now, so check
those. Not all can be converted though. These are no longer set:
HTTP_DECODER_EVENT_INVALID_TRANSFER_ENCODING_VALUE_IN_RESPONSE
HTTP_DECODER_EVENT_INVALID_AUTHORITY_PORT
Part of Bug #982.
Victor Julien
Giuseppe Longo
maxtors
Ken Steele
sxhlinux
Eric Leblond
Anoop Saldanha