It uses the new type field in the LogFileCtx instead.
This fixes the problem of not being able to use two eve-json
instance with different logging methods.
The cast of data to AlertJsonThread was not correct as the real
type of the void pointer is a OutputJsonCtx. This was working by
luck because they both have a file_ctx as first element.
Without support for OPT RR from RFC6891 (Extension mechanisms for DNS)
values of RCODE above 15 are not possible. Remove dead code which will
never match.
An IPv6 entry specified before an IPv4 entry on the host-os-policy
table can cause the stream byte array to be access one byte after
the end of the allocated memory at util-radix-tree.c:578.
Remove all strtok uses and replace them by strtok_r.
Do the same for Windows builds. Cygwin builds fine with strtok_r.
Add strtok to banned function list.
Due to an error at initialization, the stream engine would not disable
'raw' reassembly automatically when --disable-detection was used.
This lead to segments not getting cleared from the segment lists.
Instead, intruduce StreamTcpDisableAppLayer to disable app layer
tracking and reassembly. StreamTcpAppLayerIsDisabled can be used
to check it.
Replace all uses of FlowSetSessionNoApplayerInspectionFlag and
the FLOW_NO_APPLAYER_INSPECTION.
When the session/flow was flagged as 'no applayer inspect', which
could happen as a result various reasons, packets would still be
considered by the app layer reassembly.
When ACK'd, they would be removed again. Depending also on the raw
reassembly.
In very long sessions however, this meganism could fail leading to
virtually endlessly growing segment lists.
This patch makes sure that segments that come in on a 'no app layer'
session are tagged properly or even not added at all.
Use a new ssn flag instead of flow flag for no app tracking.
If for a packet we have a TX N that has detect state and a TX N+1 that
has no detect state, but does have 'progress', we have a corner case
in stateful detection.
ContinueDetection inspects TX N, but cannot flag the rule in the
de_state_sig_array as the next (TX N+1) has already started and needs
to be inspected. 'StartDetection' however, is then unaware of the fact
that ContinueDetection already inspected the rule. It uses the per
session 'inspect_id' that is only moved forward at the end of the
detection run.
This patch adds a workaround. It uses the DetectEngineThreadCtx::
de_state_sig_array to store an offset between the 'base' inspect_id
and the inspect_id that StartDetection should use. The data type is
limited, so if the offset would be too big, a search based fall back
is implemented as well.
Fix storing state and bypassing detection. Previously we'd store
on a match only, meaning that StartDetection would rerun often.
Make sure StartDetection only stores if there is something to store.
Previously we could never be calling DetectEngineHHDGetBufferForTX
for TX N and then afterwards for TX N - 1. Due to changes in the
stateful detection code this is now possible.
This patch changes the buffer logic to take the 'inspect_id' as it's
base, instead of the first transaction that we are called with.
When using engine analysis for print fast_pattern stats, print a
short summary at the end containing per buffer:
- smallest fp
- biggest fp
- number of patterns
- avg fp len
If TmThreadsUnregisterThread was called with out of range 'id', a lock
would not be cleared after returning from the function.
** CID 1264421: Missing unlock (LOCK)
/src/tm-threads.c: 2186 in TmThreadsUnregisterThread()
The code was assuming that the dictionnary containing the parameter
of a interface was ordered. But for YAML, the order is not assumed
so in case the configuration is generated we may not be able to
parse correctly the configuration file.
By iterating on child on main node and then iterating on subchild
and doing a match on the name, we are able to find the interface
list. In term of code, this algorithm was obtained by simply
removing the test on the name of the first child.
This checks if the signature's protocol is http
when setup the content keyword.
Also sets the proper flags based by protocol
since the flag SIG_FLAG_TOSERVER has to be set
if the proto is smtp, otherwise SIG_FLAG_TOCLIENT
is it's http.
This commit do a find and replace of the following:
- DETECT_SM_LIST_HSBDMATCH by DETECT_SM_LIST_FILEDATA
sed -i 's/DETECT_SM_LIST_HSBDMATCH/DETECT_SM_LIST_FILEDATA/g' src/*
- HSBD by FILEDATA:
sed -i 's/HSBDMATCH/FILEDATA/g' src/*
Wrap lines longer than 80 characters
Add "static" for unit tests.
Use (void) for () for function arguments.
Add space after "while(" -> "while ("
Remove space after function names.
Put open bracket of function on a new line.
I see two problems:
1) If allocating a newlist fails, the function returns NULL, which then
leaks any existing list elements.
2) The code to add the new value to the list works for the first two, but
for not the third. For example, replist=A, A->next=B, B->next=NULL, then
adding C results in replist=A, A->next=C, C->next=NULL, B is lost.
The fix pushes new values onto the head of the list, which might not be
what is needed, but there are no comments on what the function does, so I
made an assumption.
Allow next_seq updating to recover from cases where last_ack has been
moved beyond it. This can happen if ACK's have been accepted for missing
data that is later retransmitted.
This undoes some of the previous last_ack update changes
packets. Where rcode isn't "no error" this is displayed in both DNS and
JSON logs.
Note that this changes the current "No such domain" to "NXDOMAIN" in DNS
logs. This could be fixed if desired to maintain compatibility with
anybody crazy enough to parse the DNS log.
When the rcode is not "no error" (for example NXDOMAIN or SERVFAIL) it
is unlikely that there will be answer RRs. Therefore the rname from the
query is used.
Because the rcode applies to a whole answer packet (not individual
queries) it is impossible to determine which query RR caused the error.
Because of this most DNS servers currently reject multiple queries per
packet. Therefore each query RR is output instead with the relevant
error code, likely to be FORMERR if queries > 1.
If a boundary was longer than 254 bytes a stack overflow would result
in mime decoding.
Ticket #1449
Reported-by: Kostya Kortchinsky of the Google Security Team
A bad timestamp would lead to SCLocalTime returning NULL. This case
wasn't checked, leading to a NULL deref.
Reported-by: Kostya Kortchinsky of the Google Security Team
Implement LINKTYPE_NULL for pcap live and pcap file.
From: http://www.tcpdump.org/linktypes.html
"BSD loopback encapsulation; the link layer header is a 4-byte field,
in host byte order, containing a PF_ value from socket.h for the
network-layer protocol of the packet.
Note that ``host byte order'' is the byte order of the machine on
which the packets are captured, and the PF_ values are for the OS
of the machine on which the packets are captured; if a live capture
is being done, ``host byte order'' is the byte order of the machine
capturing the packets, and the PF_ values are those of the OS of
the machine capturing the packets, but if a ``savefile'' is being
read, the byte order and PF_ values are not necessarily those of
the machine reading the capture file."
Feature ticket #1445
Coverity:
** CID 1296115: Program hangs (ORDER_REVERSAL)
/src/tm-threads.c: 1670 in TmThreadClearThreadsFamily()
The problem is with the by default unused '%m' output parameter.
To get the thread vars it takes the tv_root_lock. This may already
be locked by the calling thread. Also, it could lead to a case of
wrong lock order between the tv_root_lock and the thread_store_lock.
Very unlikely to happen though.
As the %m param isn't really used (by default) this patch just
disables it.
Don't kill flow manager and recyclers before the rest of the threads. The
packet threads may still have packets from their pools. As the flow threads
would destroy their pools the packets would be lost.
This patch doesn't kill the threads, it just pulls them out of their run
loop and into a wait loop. The packet pools won't be cleared until all
threads are killed.
Wait for flow management threads to close before moving on to the
next steps in the shutdown process.
Don't destroy flow force reassembly packet pool too early. Worker
threads may still want to return packets to it.
Reduce hash table size as regular classification files are usually
below 100 in size. It's not performance critical anyway.
Convert pcre_get_substring calls to pcre_copy_substring.
Added more WebDAV functions. A complete list of what http
methods libhtp can handle can be found at:
https://github.com/OISF/libhtp/blob/0.5.x/htp/htp_core.h#L260.
So now the methods array reflects these available functions.
The comments have also been changed to reflect the desired style.
1) Reworked pattern registration for http methods and versions.
Instead of being a manual and verbose action of adding one
and one http method with N-amount if prefix spacings and
the same for HTTP versions (eg. HTTP/1.1) i moved it all
to be loop based actions reading values from char arrays.
In the future all that is needed is to add new methods
to the arrays and they will be added as a pattern.
2) Modified pattern registration after feedback.
Changed variable used in snprintf for http method registration
Should have been size of dest buffer at not another var (catsize)
that i had created. Also removed this variable.
Fixed a typo in the comment for registering http versions.
TO_CIENT -> TO_CLIENT.
AppLayerTest09, AppLayerTest10 and AppLayerTest11 depended on a max
protocol detection pattern size of < 17. Update the tests to pass one
extra byte to the app layer. This makes the protocol detection code
flag the session as 'proto detection completed' again.
Consider the host's xbits expiry status when checking the host for
timeout. If a single active non-expired bit is found, the host won't
be timeout just yet.
A bad last_ack update where it would be set beyond next_seq could
lead to rejection of valid segments and thus stream gaps.
Update tests to reflect new last_ack/next_seq behaviour.
The post match list was called with an unlocked flow until now.
However, recent de_state handling updates changed this. The stateful
detection code can now call the post match functions while keeping
the flow locked. The normal detection code still calls it with an
unlocked flow.
This patch adds a hint to the DetectEngineThreadCtx called
'flow_locked' that is set to true if the caller has already locked
the flow.
The detection engine and log engines can walk the tx list indirectly,
by looping AppLayerParserGetTx. This would lead to new list walks in
the DNS tx list though. Leading to bad performance.
This patch stores the last returned tx and uses that to determine if
the next tx is what we need next. If so, we can return that w/o list
walk.
Load the YAML into a prefix "detect-engine-reloads.N" where N is the
reload counter. This way we can load the updated config w/o overwriting
the current one.
Initalize detection engine by configuration prefix.
DetectEngineCtxInitWithPrefix(const char *prefix)
Takes the detection engine configuration from:
<prefix>.<config>
If prefix is NULL the regular config will be used.
Update sure that DetectLoadCompleteSigPath considers the prefix when
retrieving the configuration.
Add function to load a yaml file and insert it into the conf tree at
a specific prefix.
Example YAML:
somefile: myfile.txt
If loaded using ConfYamlLoadFileWithPrefix with prefix "myprefix", it
can be retrieved by the name of "myprefix.somefile".
Rename the thread init function DetectEngineThreadCtxInitForLiveRuleSwap
to DetectEngineThreadCtxInitForReload and change it's logic to take the
new detection engine as argument and let it return the
DetectEngineThreadCtx or NULL on error.
The old approach used the thread init API format, but it wasn't used in
that way.
Add DetectEngineReference, which takes a reference to a detect engine,
and make DetectEngineThreadCtxInitForLiveRuleSwap use it. This way
reload will not depend on master staying the same. This allows master
to be updated in between w/o affecting the reload that is in progress.
Victor Julien
Eric Leblond
Alexander Gozman
David Cannings
Jason Ish
Jason Ish
Giuseppe Longo
Ken Steele
Zachary Rasmor
DIALLO David
Pierre Chifflier
Tom DeCanio
maxtors
David Maciejak