Fix issue where SMTPStateGetTxCnt would return the actual active tx'.
The 'GetCnt' API call is not named correctly. It should be 'GetMaxId',
as this is actually the expected behavior.
This patches is fixing a issue in the OutputJSONBuffer function. It
was writing to file the content of the buffer starting from the start
to the final offset. But as the writing is done for each JSON string
we are duplicating the previous events if we are reusing the same
buffer.
Duplication was for example triggered when we have multiple alerts
attached to a packet. In the case of two alerts, the first one was
logged twice more as the second one.
Update pktacq loop to process flow timeouts in a running engine.
Add a new step to the shutdown phase of packet acquisition loop
threads (pktacqloop).
The shutdown code lets the pktacqloop break out of it's packet
acquisition loop. The thread then enters a flow timeout loop, where
it processes packets from it's tv->stream_pq queue until it's
empty _and_ the KILL flag is set.
Make sure receive threads are done before moving on to flow hash
cleanup (recycle all). Without this the flow recycler could start
it's unconditional hash clean up while detect threads are still
running on the flows.
Update unix socket to match live modes.
Add function TmThreadsInjectPacketById that is to be used to inject flow
timeout packets into the threads stream_pq queue.
TmThreadsInjectPacketById will also wake up listening threads if
applicable.
Packets are passed all packets together in an NULL terminated array
to reduce locking overhead.
Create thread registration and unregistration API for assigning unique
thread id's.
Threadvars is static even if a thread restarts, so we can do the
registration before the threads start.
A thread is unregistered when the ThreadVars are freed.
- Added the suricata.yaml configurations and updated the comments
- Renamed the field in the configuration structure to something generic
- Added two new constants and the warning codes
- Created app-layer-htp-xff.c and app-layer-htp-xff.h
- Added entries in the Makefile.am
- Added the necessary configuration options to EVE alert section
- Updated Unified2 XFF configuration comments and removed unnecessary whitespace
- Created a generic function to parse the configuration
- Release the flow locks sooner and remove debug logging
- Added XFF support to EVE alert output
Incorrectly reallocing the goto table after it was freed by calling
SCACTileReallocState() when really only want to realloc the output table.
This was causing a large goto table to be allocated and never used or
freed.
Free some memory at exit that was not getting freed.
Change pid_pat_list to store copy of case-strings in the same block
of memory as the array of pointers.
Due to a logic error in AppLayerProtoDetectGetProtoByName invalid
protocols would not be detected as such. Instead of ALPROTO_UNKNOWN
ALPROTO_MAX was returned.
Bug #1329
This patch fixes the following errors:
[src/unix-manager.c:306]: (error) Memory pointed to by 'client' is freed twice.
[src/unix-manager.c:313]: (error) Memory pointed to by 'client' is freed twice.
[src/unix-manager.c:323]: (error) Memory pointed to by 'client' is freed twice.
[src/unix-manager.c:334]: (error) Memory pointed to by 'client' is freed twice.
Unix manager was treating the packet after closing the socket if message was
too long.
MLD messages should have a hop limit of 1 only. All others are invalid.
Written at MLD talk of Enno Rey, Antonios Atlasis & Jayson Salazar during
Deepsec 2014.
Have -T / --init-errors-fatal process all rules so that it's easier
to debug problems in ruleset. Otherwise it can be a lengthy fix, test
error cycle if multiple rules have issues.
Convert empty rulefile error into a warning.
Bug #977
If we manage to read the number of RSS queues from an interface,
this means that the optimal number of capture threads is equal
to the minimum of this number and of the number of cores on the
system.
This patch implements this logic thanks to the newly introduced
function GetIfaceRSSQueuesNum.
Add a new default value for the 'threads:' setting in af-packet: "auto".
This will create as many capture threads as there are cores.
Default runmode of af-packet to workers.
For some of the buffer users it's hard to predict how big the data
will be. In the stats.log case this depends on chosen runmode and
number of threads.
To deal with this case a 'MemBufferExpand' call is added. This realloc's
the buffer.
Register with type 'stats':
function init (args)
local needs = {}
needs["type"] = "stats"
return needs
end
The stats are passed as an array of tables:
{ 1, { name=<name>, tmname=<tm_name>, value=<value>, pvalue=<pvalue>}}
{ 2, { name=<name>, tmname=<tm_name>, value=<value>, pvalue=<pvalue>}}
etc
Name is the counter name (e.g. decoder.invalid), tm_name is the thread name
(e.g. AFPacketeth05), value is current value, and pvalue is the value of the
last time the script was invoked.
As the stats api calls the loggers at a global interval, the global
interval should be configured globally.
# global stats configuration
stats:
enabled: yes
# The interval field (in seconds) controls at what interval
# the loggers are invoked.
interval: 8
If this config isn't found, the old config will be supported.
Convert regular 'stats.log' output to this new API.
In addition to the current stats value, also give the last value. This
makes it easy to display the difference.
The SCStreamingBuffer call now also returns two booleans:
data, data_open, data_close = SCStreamingBuffer()
The first indicates this is the first data of this type for this
TCP session or HTTP transaction.
The second indicates this is the last data.
Ticket #1317.
sfd->target.value was always being set, even if the targettype was
not FLOWINT_TARGET_VAL. This would cause the tvar to be overwritten
with garbage data.
Add the modbus.function and subfunction) keywords for public function match in rules (Modbus layer).
Matching based on code function, and if necessary, sub-function code
or based on category (assigned, unassigned, public, user or reserved)
and negation is permitted.
Add the modbus.access keyword for read/write Modbus function match in rules (Modbus layer).
Matching based on access type (read or write),
and/or function type (discretes, coils, input or holding)
and, if necessary, read or write address access,
and, if necessary, value to write.
For address and value matching, "<", ">" and "<>" is permitted.
Based on TLS source code and file size source code (address and value matching).
Signed-off-by: David DIALLO <diallo@et.esia.fr>
Decode Modbus request and response messages, and extracts
MODBUS Application Protocol header and the code function.
In case of read/write function, extracts message contents
(read/write address, quantity, count, data to write).
Links request and response messages in a transaction according to
Transaction Identifier (transaction management based on DNS source code).
MODBUS Messaging on TCP/IP Implementation Guide V1.0b
(http://www.modbus.org/docs/Modbus_Messaging_Implementation_Guide_V1_0b.pdf)
MODBUS Application Protocol Specification V1.1b3
(http://www.modbus.org/docs/Modbus_Application_Protocol_V1_1b3.pdf)
Based on DNS source code.
Signed-off-by: David DIALLO <diallo@et.esia.fr>
A tx is considered complete after the data command completed. However,
this would lead to RSET and QUIT commands setting up a new tx.
This patch simply adds a check that refuses to setup a new tx when these
commands are encountered after the data portion is complete.
SigMatch would be added to list, then the alproto check failed, leading
to freeing of sm. But as it was still in the list, the list now contained
a dangling pointer.
When multiple email addresses were in the 'to' field, sometimes
they would be logged as "\r\n \"Name\" <email>".
The \r\n was added by GetFullValue in the mime decoder, for unknown
reasons. Disabling this seems to have no drawbacks.
Turn all buffers into uint8_t (from char) and no longer use the
string functions like strncpy/strncasecmp on them.
Store url and field names as lowercase, and also search/compare
them as lowercase. This allows us to use SCMemcmp.
The global variable suricata_ctl_flags needs to volatile, otherwise the
compiler might not cause the variable to be read every time because it
doesn't know other threads might write the variable.
This was causing Suricata to not exit under some conditions.
For these 2 cases:
1. Missing SYN:
-> syn <= missing
<- syn/ack
-> ack
-> data
2. Missing SYN and 3whs ACK:
-> syn <= missing
<- syn/ack
-> ack <= missing
-> data
Fix session pickup. The next_win settings weren't correctly set, so that
packets were rejected.
Bug 1190.
If 3whs SYN/ACK and ACK are missing we can still pick up the session if
in async-oneside mode.
-> syn
<- syn/ack <= missing
-> ack <= missing
-> data
Bug 1190.
The entire body of these functions are protected by ifdef PROFILING.
If the functions are inlined, then this check removes the need for the
function entirely.
Previously, the empty function was still called, even when not built
for profiling. The functions showed as being 0.25% of total CPU time
without being built for profiling.
If vlan is disabled the cluster_flow mode will still take VLAN tags
into account due to using pf_ring's 6-tuple mode.
So this forces to use pf_ring's 5-tuple mode.
Bug #1292
Implements new API to expand the IP reputation
to netblocks with CIDR notation
A new object 'srepCIDRTree' is kept in the DetectionEngineCtx,
which contains two tree (one for ipv4 and one for ipv6)
where the reputation values are stored.
ACK packets completing a valid FIN shutdown could be flagged as
'bad window update' if they would shrink the window.
This patch detects this case before doing the bad window update
check.
If the 3whs ACK and some data after this is lost, we would get stuck
in the 'SYN_RECV' state, where from there each packet might be
considered invalid.
This patch improves the handling of this case.
The keyword would not allow matching on "OpenSSH_5.5p1 Debian-6+squeeze5"
as the + and space characters were not allowed.
This patch adds support for them.
Stream pseudo packets are taken from the packet pool, which can be empty.
In this case a pseudo packet will not be created and processed.
This patch adds a counter "tcp.pseudo_failed" to track this.
doesn't need the gpu results and to release the packet for the next run.
Previously the inspection engine wouldn't inform the cuda module, if it
didn't need the results. As a consequence, when the packet is next taken
for re-use, and if the packet is still being processed by the cuda module,
the engine would wait till the cuda module frees the packet.
This commits updates this functionality to inform the cuda module to
release the packet for the afore-mentioned case.
Add option (disabled by default) to honor pass rules. This means that
when a pass rule matches in a flow, it's packets are no longer stored
by the pcap-log module.
SigMatchGetLastSMFromLists() is finding the sm with the largest
index among all of the values returned from SigMatchGetLastSM() on
the set of (list and type) tuples passed as arguments.
The function was creating an array of the types, then creating an array
of the results of SigMatchGetLastSM(), sorting that list completely, then
only returning the first values from the list.
The new code, gets one set of arguments from the variable arguments, calls
SigMatchGetLastSM() and if the returned sm has a larger index, keeps that
as the last sm.
Allow a script to set the 'stream' buffer type. This will add the
script to the PMATCH list.
Example script:
alert tcp any any -> any any (content:"html"; lua:stream.lua; sid:1;)
function init (args)
local needs = {}
needs["stream"] = tostring(true)
return needs
end
-- return match via table
function match(args)
local result = {}
b = tostring(args["stream"])
o = tostring(args["offset"])
bo = string.sub(b, o);
print (bo)
return result
end
return 0
To prevent accidental writes into the orignal packet buffer, use
const pointers for the extension header pointers used by IPv6. This
will cause compiler warnings in case of writes.
A logic error in the IPv6 Routing header parsing caused accidental
updating of the original packet buffer. The calculated extension
header lenght was set to the length field of the routing header,
causing it to be wrong.
This has 2 consequences:
1. defrag failure. As the now modified payload was used in defrag,
the decoding of the reassembled packet now contained a broken length
field for the routing header. This would lead to decoding failure.
The potential here is evasion, although it would trigger:
[1:2200014:1] SURICATA IPv6 truncated extension header
2. in IPS mode, especially the AF_PACKET mode, the modified and now
broken packet would be transmitted on the wire. It's likely that
end hosts and/or routers would reject this packet.
NFQ based IPS mode would be less affected, as it 'verdicts' based on
the packet handle. In case of replacing the packet (replace keyword
or stream normalization) it could broadcast the bad packet.
Additionally, the RH Type 0 address parsing was also broken. It too
would modify the original packet. As the result of this code was not
used anywhere else in the engine, this code is now disabled.
Reported-By: Rafael Schaefer <rschaefer@ernw.de>
AF_PACKET is not setting the engine mode to IPS when some
interfaces are peered and use IPS mode. This is due to the
fact, it is possible to peer 2 interfaces and run an IPS on
them and have a third one that is running in normal IDS mode.
In fact this choice is the bad one as unwanted side effect is
that there is no drop log and that stream inline is not used.
To fix that, this patch puts suricata in IPS mode as soon as
there is two interfaces in IPS mode. And it displays a error
message to warn user that the accuracy of detection on IDS only
interfaces will be low.
When using AMATCH, continue detection would fail if the tx part
had already run. This lead to start detection rerunning, causing
multiple alerts for the same issue.
As there is no inspection engine for request_line, the sigmatch was
added to the AMATCH list. However, no AppLayerMatch function for
lua scripts was defined.
This patch defines a AppLayerMatch function.
Bug #1273.
I found three somewhat serious IPv6 address bugs within the Suricata 2.0.x source code. Two are in the source module "detect-engine-address.c", and the third is in "util-radix-tree.c".
The first bug occurs within the function DetectAddressParse2(). When parsing an address string and a negated block is encountered (such as when parsing !$HOME_NET, for example), any corresponding IPv6 addresses were not getting added to the Group Heads in the DetectAddressList. Only IPv4 addresses were being added.
I discovered another bug related to IPv6 address ranges in the Signature Match Address Array comparison code for IPv6 addresses. The function DetectAddressMatchIPv6() walks a signature's source or destination match address list comparing each to the current packet's corresponding address value. The match address list consists of value pairs representing a lower and upper IP address range. If the packet's address is within that range (including equal to either the lower or upper bound), then a signature match flag is returned.
The original test of each signature match address to the packet was performed using a set of four compounded AND comparisons looking at each of the four 32-bit blocks that comprise an IPv6 address. The problem with the old comparison is that if ANY of the four 32-bit blocks failed the test, then a "no-match" was returned. This is incorrect. If one or more of the more significant 32-bit blocks met the condition, then it is a match no matter if some of the less significant 32-bit blocks did not meet the condition. Consider this example where Packet represents the packet address being checked, and Target represents the upper bound of a match address pair. We are testing if Packet is less than Target.
Packet -- 2001:0470 : 1f07:00e2 : 1988:01f1 : d468:27ab
Target -- 2001:0470 : 1f07:00e2 : a48c:2e52 : d121:101e
In this example the Packet's address is less than the target and it should give a match. However, the old code would compare each 32-bit block (shown spaced out above for clarity) and logically AND the result with the next least significant block comparison. If any of the four blocks failed the comparison, that kicked out the whole address. The flaw is illustrated above. The first two blocks are 2001:0470 and 1f07:00e2 and yield TRUE; the next less significant block is 1988:01f1 and a48c:2e52, and also yields TRUE (that is, Packet is less than Target); but the last block compare is FALSE (d468:27ab is not less than d121:101e). That last block is the least significant block, though, so its FALSE determination should not invalidate a TRUE from any of the more significant blocks. However, in the previous code using the compound logical AND block, that last least significant block would invalidate the tests done with the more significant blocks.
The other bug I found for IPv6 occurs when trying to parse and insert an IPv6 address into a Radix Tree using the function SCRadixAddKeyIPV6String(). The test for min and max values for an IPv6 CIDR mask incorrectly tests the upper limit as 32 when it should be 128 for an IPv6 address. I think this perhaps is an old copy-paste error if the IPv6 version of this function was initially copied from the corresponding IPv4 version directly above it in the code. Without this patch, the function will return null when you attempt to add an IPv6 network whose CIDR mask is larger than 32 (for example, the popular /64 mask will cause the function to return the NULL error condition).
(amended by Victor Julien)
When using the inspection engines, track the current tx_id in the
thread storage the detect thread uses. As 0 is a valid tx_id, add
a simple bool that indicates if the tx_id field is set.
Allow use of the Flow Logging API through Lua scripts.
Minimal script:
function init (args)
local needs = {}
needs["type"] = "flow"
return needs
end
function setup (args)
end
function log(args)
startts = SCFlowTimeString()
ipver, srcip, dstip, proto, sp, dp = SCFlowTuple()
print ("Flow IPv" .. ipver .. " src " .. srcip .. " dst " .. dstip ..
" proto " .. proto .. " sp " .. sp .. " dp " .. dp)
end
function deinit (args)
end
Add SCStreamingBuffer lua function to retrieve the data passed
to the script per streaming API invocation.
Example:
function log(args)
data = SCStreamingBuffer()
hex_dump(data)
end
Make normalized body data available to the script through
HttpGetRequestBody and HttpGetResponseBody.
There no guarantees that all of the body will be availble.
Example:
function log(args)
a, o, e = HttpGetResponseBody();
--print("offset " .. o .. " end " .. e)
for n, v in ipairs(a) do
print(v)
end
end
Get the host from libhtp's tx->request_hostname, which can either be
the host portion of the url or the host portion of the Host header.
Example:
http_host = HttpGetRequestHost()
if http_host == nil then
http_host = "<hostname unknown>"
end
SCFlowAppLayerProto: get alproto as string from the flow. If alproto
is not (yet) known, it returns "unknown".
function log(args)
alproto = SCFlowAppLayerProto()
if alproto ~= nil then
print (alproto)
end
end
A new callback to give access to thread id, name and group name:
SCThreadInfo. It gives: tid (integer), tname (string), tgroup (string)
function log(args)
tid, tname, tgroup = SCThreadInfo()
SCFlowTimeString: returns string form of start time of a flow
Example:
function log(args)
startts = SCFlowTimeString()
ts = SCPacketTimeString()
if ts == startts then
print("new flow")
end
Add SCPacketTimeString to get the packets time string in the format:
11/24/2009-18:57:25.179869
Example use:
function log(args)
ts = SCPacketTimeString()
SCRuleIds(): returns sid, rev, gid:
function log(args)
sid, rev, gid = SCRuleIds()
SCRuleMsg(): returns msg
function log(args)
msg = SCRuleMsg()
SCRuleClass(): returns class msg and prio:
function log(args)
class, prio = SCRuleClass()
if class == nil then
class = "unknown"
end
Add flow store and retrieval wrappers for accessing the flow through
Lua's lightuserdata method.
The flow functions store/retrieve a lock hint as well.
If the script needing a packet doesn't specify a filter, it will
be run against all packets. This patch adds the support for this
mode. It is a packet logger with a condition function that always
returns true.
Add a lua callback for getting Suricata's log path, so that lua scripts
can easily get the logging directory Suricata uses.
Update the Setup logic to register callbacks before the scripts 'setup'
is called.
Example:
name = "fast_lua.log"
function setup (args)
filename = SCLogPath() .. "/" .. name
file = assert(io.open(filename, "a"))
end
Add file logger support. The script uses:
function init (args)
local needs = {}
needs['type'] = 'file'
return needs
end
The type is set to file to make it a file logger.
Add utility functions for placing things on the stack for use
by the scripts. Functions for numbers, strings and byte arrays.
Add callback for returing IP header info: ip version, src ip,
dst ip, proto, sp, dp (or type and code for icmp and icmpv6):
SCPacketTuple
Through 'needs' the script init function can indicate it wants to
see packets and select a condition function. Currently only alerts
is an option:
function init (args)
local needs = {}
needs["type"] = "packet"
needs["filter"] = "alerts"
return needs
end
Add an argument to the registration to indicate which iterator
needs to be used: Stream or HttpBody
Add HttpBody Iterator, calling the logger(s) for each Http body chunk.
StreamIterator implementation for iterating over ACKed segments.
Flag each segment as logged when the log function has been called for it.
Set a 'OPEN' flag for the first segment in both directions.
Set a 'CLOSE' flag when the stream ends. If the last segment was already
logged, a empty CLOSE call is performed with NULL data.
This patch adds a new Log API for streaming data such as TCP reassembled
data and HTTP body data. It could also replace Filedata API.
Each time a new chunk of data is available, the callback will be called.
- Removed unnecessary assignment of the data field
- Removed else condition (same function called for IPv4 and IPV6)
- Fixed constants to be a power of two (used in bitwise operations)
The field ext_pkt was cleaned before calling the release function.
The result was that IPS mode such as the one of AF_PACKET were not
working anymore because they were not able to send the data which
were initially pointed by ext_pkt.
This patch moves the ext_pkt cleaning to the cleaning macro. This
ensures that the cleaning is done for allocated and pool packets.
Call PACKET_RELEASE_REFS from PacketPoolGetPacket() so that
we only access the large packet structure just before actually
using it. Should give better cache behaviour.
The Source Routing Header had routing defined as a char* for a field
of variable size. Since that field was not being used in the code, I
removed the pointer and added a comment.
Structures that are used to cast packet data into fields need to be packed
so that the compiler doesn't add any padding to these fields. This also helps
Tile-Gx to avoid unaligned loads because the compiler will insert code to
handle the possible unaligned load.
Victor Julien
Eric Leblond
Duarte Silva
DIALLO David
Ken Steele
Giuseppe Longo
Christophe M
Tom DeCanio
David Abarbanel
Jason Ish
Anoop Saldanha
Mats Klepsland
bmeeks8