This patch adds a check that was missing when specifying BPF filter
from a file. Suricata behavior should have been the same as when
BPF filter is specified on command line.
Fix warning spotted by clang on FreeBSD:
source-ipfw.c:241:49: warning: use of logical '||' with constant operand [-Wconstant-logical-operand]
if (suricata_ctl_flags & (SURICATA_STOP || SURICATA_KILL)) {
^ ~~~~~~~~~~~~~
source-ipfw.c:241:49: note: use '|' for a bitwise operation
if (suricata_ctl_flags & (SURICATA_STOP || SURICATA_KILL)) {
^~
|
Use same logic as the one used in other capture mode.
In the case of running mode like NFQ there is no need possibility
to compute the statistics as it is done in LiveDevice (drop and
checksum count are meaningless).
This patch adds a function that allow running mode to disable the
display of the counters at exit.
Remove local copies from each MPM file and use include file instead.
Might be better to also add util-memcpy.c rather than inlining it each time,
to get smaller code, since only seems to be used at initialization.
This is required because SCMemcmpLowercase() expects it first argument
to be already lowercase for the comparison. This is done by using
memcpy_tolower() for NO_CASE patterns.
This addresses code review comments from Victor.
The case_state in MPMs was just to track when a pid could have no-case and
case-sensitive matches for the same PID. Now that can't happen after fixing
bug 1110, so remove the code and storage for case_state.
This fixes bug 1110. When assigning PIDs, use the NO_CASE flag when comparing
for duplicates. The state of the flag must be the same, but also use the same
type of comparisons when checking for duplicates.
Previously, "foo":CS would match with "foo":CI when it should not.
and "foo":CI would not match "FoO":CI when it should. Both of those
cases are fixed with this change.
This then allows simplifying the use of pid in MPMs because now if they
pids match, then so do the flags, so checking the flags is not required.
In the stream engine, Init() can fail if the memcap is reached. In this
case the segment was not freed by PoolGet:
==8600== Thread 1:
==8600== 70,480 bytes in 1,762 blocks are definitely lost in loss record 611 of 612
==8600== at 0x4C2A2DB: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==8600== by 0x914CC8: TcpSegmentPoolAlloc (stream-tcp-reassemble.c:166)
==8600== by 0xA0D315: PoolGet (util-pool.c:297)
==8600== by 0x9302CD: StreamTcpGetSegment (stream-tcp-reassemble.c:3768)
==8600== by 0x921FE8: StreamTcpReassembleHandleSegmentHandleData (stream-tcp-reassemble.c:1873)
==8600== by 0x92EEDA: StreamTcpReassembleHandleSegment (stream-tcp-reassemble.c:3584)
==8600== by 0x8D3BB1: HandleEstablishedPacketToServer (stream-tcp.c:1969)
==8600== by 0x8D7F98: StreamTcpPacketStateEstablished (stream-tcp.c:2323)
==8600== by 0x8F13B8: StreamTcpPacket (stream-tcp.c:4243)
==8600== by 0x8F2537: StreamTcp (stream-tcp.c:4485)
==8600== by 0x95DFBB: TmThreadsSlotVarRun (tm-threads.c:559)
==8600== by 0x8BE60D: TmThreadsSlotProcessPkt (tm-threads.h:142)
tcp.segment_memcap_drop | PcapFile | 1762
This patch fixes PoolGet to both Cleanup and Free the Alloc'd data in
case Init fails.
==15745== 3 bytes in 1 blocks are definitely lost in loss record 5 of 615
==15745== at 0x4C2A2DB: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==15745== by 0x71858C1: strdup (strdup.c:42)
==15745== by 0xA20814: SCProtoNameInit (util-proto-name.c:75)
==15745== by 0x952D1B: PostConfLoadedSetup (suricata.c:1983)
==15745== by 0x9537CD: main (suricata.c:2112)
Also, clean up and add a check to make sure it's initialized only once.
The full path of the script names is stored in a buffer that wasn't
freed at exit.
==24195== 41 bytes in 1 blocks are definitely lost in loss record 300 of 613
==24195== at 0x4C2A2DB: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==24195== by 0x565D06: DetectLoadCompleteSigPath (detect.c:251)
==24195== by 0x7CABE8: DetectLuajitParse (detect-luajit.c:595)
==24195== by 0x7CD2AE: DetectLuajitSetup (detect-luajit.c:827)
==24195== by 0x7DC273: SigParseOptions (detect-parse.c:547)
==24195== by 0x7DDC75: SigParse (detect-parse.c:856)
==24195== by 0x7E1C2B: SigInitHelper (detect-parse.c:1336)
==24195== by 0x7E2968: SigInit (detect-parse.c:1559)
==24195== by 0x7E37B1: DetectEngineAppendSig (detect-parse.c:1831)
==24195== by 0x566D17: DetectLoadSigFile (detect.c:335)
==24195== by 0x567636: SigLoadSignatures (detect.c:423)
==24195== by 0x951A97: LoadSignatures (suricata.c:1816)
This patch frees the buffer.
For packets that were freed, not recycled, profiling memory wasn't
freed:
==15745== 13,312 bytes in 8 blocks are definitely lost in loss record 611 of 615
==15745== at 0x4C2C494: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==15745== by 0xA190D5: SCProfilePacketStart (util-profiling.c:963)
==15745== by 0x4E4345: PacketGetFromAlloc (decode.c:134)
==15745== by 0x83FE75: FlowForceReassemblyPseudoPacketGet (flow-timeout.c:276)
==15745== by 0x8413BF: FlowForceReassemblyForHash (flow-timeout.c:588)
==15745== by 0x841897: FlowForceReassembly (flow-timeout.c:716)
==15745== by 0x9540F6: main (suricata.c:2296)
==15745==
==15745== 14,976 bytes in 9 blocks are definitely lost in loss record 612 of 615
==15745== at 0x4C2C494: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==15745== by 0xA190D5: SCProfilePacketStart (util-profiling.c:963)
==15745== by 0x4E4345: PacketGetFromAlloc (decode.c:134)
==15745== by 0x83FE75: FlowForceReassemblyPseudoPacketGet (flow-timeout.c:276)
==15745== by 0x841508: FlowForceReassemblyForHash (flow-timeout.c:620)
==15745== by 0x841897: FlowForceReassembly (flow-timeout.c:716)
==15745== by 0x9540F6: main (suricata.c:2296)
This patch addresses that.
If lock profiling was compiled in, but disabled in the config a
serious memory leak condition was triggered.
Valgrind output:
==11169== 9,091,248 bytes in 189,401 blocks are definitely lost in loss record 564 of 564
==11169== at 0x4C2A2DB: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==11169== by 0xABC44C: LockRecordAdd (util-profiling-locks.c:112)
==11169== by 0xABC950: SCProfilingAddPacketLocks (util-profiling-locks.c:141)
==11169== by 0xA04CD5: TmThreadsSlotVarRun (tm-threads.c:562)
==11169== by 0x958793: TmThreadsSlotProcessPkt (tm-threads.h:142)
==11169== by 0x9599C3: PcapFileCallbackLoop (source-pcap-file.c:172)
==11169== by 0x56FC130: ??? (in /usr/lib/x86_64-linux-gnu/libpcap.so.1.4.0)
==11169== by 0x959D24: ReceivePcapFileLoop (source-pcap-file.c:210)
==11169== by 0xA05B9E: TmThreadsSlotPktAcqLoop (tm-threads.c:703)
==11169== by 0x6155F6D: start_thread (pthread_create.c:311)
==11169== by 0x6E399CC: clone (clone.S:113)
Some of the packets counters were using a 32bit integer. Given the
bandwidth that is often seen, this is not a good idea. This patch
switches to 64bit counter.
Packets counter is incremented in AFPDumpCounters and it was
also incremented during packet reading. The result was a value
that is twice the expected result.
Spotted-by: Victor Julien <victor@inliniac.net>
Until now a PoolInit failure for the segment pools would result in
an abort() through BUG_ON(). This patch adds a proper error message,
then exits.
Bug #1108.
When TcpSegmentPoolInit fails (e.g. because of a too low memcap),
it would free the segment. However, the segment memory is managed
by the Pool API, which would also free the same memory location.
This patch fixes that.
Also, memset the structure before any checks are done, as the segment
memory is passed to TcpSegmentPoolCleanup in case of error as well.
Bug #1108
Fixes:
** CID 1075221: Dereference after null check (FORWARD_NULL)
/src/suricata.c: 1344 in ParseCommandLine()
The reason it gave this warning is that in other paths using optarg
there was a check, so the checker assumed optarg can be NULL.
This patch fixes:
** CID 1187544: Missing break in switch (MISSING_BREAK)
/src/decode-icmpv6.c: 268 in DecodeICMPV6()
** CID 1187545: Missing break in switch (MISSING_BREAK)
/src/decode-icmpv6.c: 270 in DecodeICMPV6()
** CID 1187546: Missing break in switch (MISSING_BREAK)
/src/decode-icmpv6.c: 272 in DecodeICMPV6()
** CID 1187547: Missing break in switch (MISSING_BREAK)
/src/decode-icmpv6.c: 274 in DecodeICMPV6()
It duplicates the logic instead of adding 'fall through' statements
as the debug statements were wrong and confusing. For ND_REDIRECT
all 5 ND_* types would have been printed.
By assuming that HTPCallbackRequestLine would always be run first,
an memory leak was introduced. It would not check if user data already
existed in the tx, causing it to overwrite the user data pointer is
it already existed.
Bug #1092.
In really short Suricata runtimes the capture counters would not
be updated. This patch does a force update at the end of the
capture loops in pcap and af-packet.
The probing parser registration function
AppLayerProtoDetectPPParseConfPorts was a void, meaning it would
give no feedback to the registering protocol implementation. If a
config was missing, it would just give up.
This patch changes it to return a bool. 0 if no config was found,
1 if a config was found.
This allows the caller to setup a default case.
The probing parser detection ports yaml settings of the TCP part
of the DNS parser accidentally used udp as protocol string, causing
the wrong part of the YAML to be evaluated.
If the protocol is disabled, app-layer-event would print a cryptic
error message. This patch makes sure we inform the user the protocol
is in fact disabled.
This patch introduces a new counter "decoder.vlan_qinq". It counts
packets that have more than two stacked vlan layers.
Packets with 2 vlan layers will both increment "decoder.vlan" and
"decoder.vlan_qinq".
A node isn't known to be a sequence node until the YAML is parsed.
If a node sequence node was set on the command line, promote
it to a sequence node when it is discovered by YAML to be
a sequence node.
Fixes comment #18 in issue 921.
When creating a pseudo packet with the reassembled IP packet, the
parent's vlan id or id's are also needed. The defrag packet is run
through decode and the flow engine, where the vlan id is necessary
for connecting the packet to the correct flow.
Some old distribution don't ship recent enough linux header. This
result in TP_STATUS_VLAN_VALID being undefined. This patch defines
the constant and use it as it is used in backward compatible method
in the code: the flag is not set by kernel and a test on vci value
will be made.
This should fix https://redmine.openinfosecfoundation.org/issues/1106
Flow-timeout code injects pseudo packets into the decoders, leading
to various issues. For a full explanation, see:
https://redmine.openinfosecfoundation.org/issues/1107
This patch works around the issues with a hack. It adds a check to
each of the decoder entry points to bail out as soon as a pseudo
packet from the flow timeout is encountered.
Ticket #1107.
FlowReference stores the flow in the destination pointer and increases
the flow reference counter (use_cnt). This should only be called once
per destination pointer. The reference counter is decremented when
FlowDereference is called. Multiple FlowReference calls would lead to
multiple use_cnt bumps, while there would be only one FlowRereference.
This lead to a use_cnt that would never become 0, meaning the flow
would stay in the hash for the entire lifetime of the process.
The fix here is to check if the destination pointer is already set to
the flow. If so, we don't increase the reference counter.
As this is really a bug, this condition will lead to a BUG_ON if the
DEBUG_VALIDATION checking is enabled.
The HTP config tree is a radix. The lookups are updated to the new API.
The return of user_data is treated as a succesful lookup, instead of
the node itself.
This patch updates all the radix tests to the new API. In most cases
it just passes a NULL user data return pointer.
It also removes the tests related to SC_RADIX_NODE_USERDATA, as this
macro is removed.
Bug #1073
The radix tree stores user data. However, it had no function to return
this data to the consumers of the API. Instead, on lookup, it would
set a field "user_data_result" in the nodes prefix structure which
could then be read by the caller.
Apart for this not being a very nice design as it exposes API internals
to the caller, it is not thread safe. By updating the global data
structure without any form (or suggestion) of locking, threads could
overwrite the same field unexpectedly.
This patch modifies the lookup logic to get rid of this stored
user_data_result. Instead, all the lookup functions how take an
addition argument: void **user_data_result.
Through this pointer the user data is returned. It's allowed to be
NULL, in this case the user data is ignored.
This is a significant API change, that affects a lot of tests and
callers. These will be updated in follow up patches.
Bug #1073.
Handles ND_ROUTER_SOLICIT, ND_ROUTER_ADVERT, ND_NEIGHBOUR_ADMIN,
ND_NEIGHBOUR_SOLICIT and ND_REDIRECT. Don't set ICMPV6_UNKONWN_CODE
if code is the expected value of 0.
This patch uses the new function SCKernelVersionIsAtLeast to know
that we've got a old kernel that do not strip the VLAN header from
the message before sending it to userspace.
Since commit in kernel
commit a3bcc23e890a6d49d6763d9eb073d711de2e0469
Author: Ben Greear <greearb@candelatech.com>
Date: Wed Jun 1 06:49:10 2011 +0000
af-packet: Add flag to distinguish VID 0 from no-vlan.
a flag is set to indicate VLAN has been set in packet header.
As suggested in commit message, using a test of the flag followed
by a check on vci value ensure backward compatibility of the test.
AppLayerParserProtocolIsTxEventAware would check if a proto is tx
event aware by checking if it had registered a StateHasEvents function.
However, this is an optimization function. This patch changes it to
use the StateGetEvents function instead, which is a better indicator.
When running on a TILEncore-Gx PCIe card, setting the filetype of fast.log
to pcie, will open a connection over PCIe to a host application caleld
tile-pcie-logd, that receives the alert strings and writes them to a file
on the host. The file name to open is also passed over the PCIe link.
This allows running Suricata on the TILEncore-Gx PCIe card, but have the
alerts logged to the host system's file system efficiently. The PCIe API that
is used is the Tilera Packet Queue (PQ) API which can access PCIe from User
Space, thus avoiding system calls.
Created util-logopenfile-tile.c and util-logopen-tile.h for the TILE
specific PCIe logging functionality.
Using Write() and Close() function pointers in LogFileCtx, which
default to standard write and close for files and sockets, but are
changed to PCIe write and close functions when a PCIe channel is
openned for logging.
Moved Logging contex out of tm-modules.h into util-logopenfile.h,
where it makes more sense. This required including util-logopenfile.h
into a couple of alert-*.c files, which previously were getting the
definitions from tm-modules.h.
The source and Makefile for tile-pcie-logd are added in contrib/tile-pcie-logd.
By default, the file name for fast.log specified in suricata.yaml is used as
the filename on the host. An optional argument to tile-pcie-logd, --prefix=,
can be added to prepend the supplied file path. For example, is the file
in suricata.yaml is specified as "/var/log/fast.log" and --prefix="/tmp",
then the file will be written to "/tmp/var/log/fast.log".
Check for TILERA_ROOT environment variable before building tile_pcie_logd
Building tile_pcie_logd on x86 requires the Tilera MDE for its PCIe libraries
and API header files. Configure now checs for TILERA_ROOT before enabling
builing tile_pcie_logd in contrib/tile_pcie_logd
Generate the alert string into a temporary buffer before aquiring the
file lock. Only hold the file lock while writing the alert string to the
file.
In the case of multiple alerts, it would be better to generate all the
alerts, then aquire the lock once and write them all and then flush.
Changed PrintRawLineHexFp, which printed to a file, to PrintBufferRawLineHex,
that puts the same output into a string buffer. It was only used by fast.log.
Fix issue where negating a range containing a negation would fail.
E.g. HOME_NET: [192.168.0.0/16,!192.168.10.0], can be used in a rule
as !$HOME_NET.
Also, fix another parsing issue:
If the negation range would be bigger than the 'positive' range, parsing
wouldn't be correct. Now this case is rejected.
E.g. [192.168.1.3,!192.168.0.0/16] is now explicitly rejected
Ticket 1079.
End profiling inside the lock for a tunnel packet as otherwise another
thread may already free the packet while the profiling code runs.
SEGV's observed and now gone.
The TM_FLAG_LOGAPI_TM flag indicates that a module is run by the log
api, not by the 'regular' thread module call functions.
Set flag in all all Log API users' registration code.
Purpose of this flag is in profiling. In profiling output it will be
used to list log api thread modules separately.
The log API calls thread modules directly, so the TMM profiling logic
can be applied to it. This patch does so.
The "Thread Module" out now again lists the individual loggers. As the
module are normally called much less frequently the numbers are hard to
compare to pre-log-api numbers.
Add option "profiling.sample-rate":
# Run profiling for every xth packet. The default is 1, which means we
# profile every packet. If set to 1000, one packet is profiled for every
# 1000 received.
#sample-rate: 1000
This allows for configuration of the sample rate.
Instead of a large (6k+) structure in the Packet, make the profiling
storage dynamic. To do this the Packet->profile is now a pointer.
Initial support for selective sampling, e.g. only profile every
1000th packet.
A negated match is matching if the tested field is NULL. But as it
is not set, nor negated nor normal test must match.
Without this patch, a rule like:
alert tls any any -> any any (msg:"negated match"; tls.subject:!"CN=home.regit.org"; sid:1; rev:1;)
is alerting for all connections. Event if they are done on a certificate
with matching subject. This was due to the fact that tls protocol
is discovered before the handshake is complete. Thus the condition
on tls is true with a NULL tls.subject. And code was returning a
positive match in the case of a NULL subject and a signature with
a negated match.
Introduce a utility function to convert an array of bytes into a
null-terminated string:
char *BytesToString(const uint8_t *bytes, size_t nbytes);
All non-printables are copied over, except for '\0', which is
turned into literal '\' '0' in the string. So the resulting string
may be bigger than the input.
In various places SCStrndup was used to 'dup' a bstr string, however
libhtp provides bstr_util_strdup_to_c for this. As this is a cleaner
interface, it's preferred.
This patch separates http keys from file to have a different value
list:
{
"time":"01\/31\/2014-12:04:52.837245","event_type":"file","src_ip":"5.3.1.1","src_port":80,"dest_ip":"1.8.1.9","dest_port":9539,"proto":"TCP",
"http":{"url":"/foo/","hostname":"bar.com","http_refer":"http:\/\/bar.org","http_user_agent":"Mozilla\/5.0"},
"file":{"filename":"bar","magic":"unknown","state":"CLOSED","stored":false,"size":21}
}
One interest of this modification is that it is possible to use the
same key as the one used in http events. Thus correlating both type
of events is trivial. On code side, this will permit to factorize
the code by simply asking the underlying protocol to output its
info in a json object.
Second interest is that adding file extraction for a new protocol
will result in only changing the protocol specific json list.
This patch adds an event_type key to the generated events. Current
value is one of "dns", "alert, "file", "tls", "http", "drop". It is
then easy to differentiate in log analysis tools the events based on
source inside Suricata.
Without this patch DNS answers for a single query are stored in a
single json event. The result is an array in the object like this one:
{"type":"answer","id":45084,"rrname":"s-static.ak.facebook.com","rrtype":"CNAME","ttl":734},
{"type":"answer","id":45084,"rrname":"s-static.ak.facebook.com.edgekey.net","rrtype":"CNAME","ttl":1710},
This type of output is not well supported in logstash. It is
displayed as it is written above and it is not possible to
query the fields.
I think the reason is that this is not logical if we consider search
query. For example if we search for "rrname" equal "s-static.ak.facebook.com"
we got one entry with two values in it. That's against the logic
of event. Furthermore, if we want to get a complete query, we can
used the id.
This patch splits the answer part in mulitple message. The result
is then accepted by logstash and fields can be queried easily.
This patch updates DNS field name to be in sync with RFC 2629:
https://github.com/adulau/pdns-qof
This will allow to easily use Suricata with other passive DNS tools.
This patch is synchronizing key name with Common Information Model.
It updates key name following what is proposed in:
http://docs.splunk.com/Documentation/PCI/2.0/DataSource/CommonInformationModelFieldReference
The interest of these modifications is that using the same key name
as other software will provide an easy to correlate and improve
data. For example, geoip setting in logstash can be applied on
all src_ip fields allowing geoip tagging of data.
Both the drop and tls logs are currently not designed to have multiple
instances running. So until that is changed, error out if more than one
instance is started.
To support the 'eve-log' idea, we need to be able to force all log
modules to be enabled by the master eve-log module, and need to be
able to make all logs go into a single file. This didn't fit the
API so far, so added the sub-module concept.
A sub-module is a regular module, that registers itself as a sub-
module of another module:
OutputRegisterTxSubModule("eve-log", "JsonHttpLog", "http",
OutputHttpLogInitSub, ALPROTO_HTTP, JsonHttpLogger);
The first argument is the name of the parent. The 4th argument is
the OutputCtx init function. It differs slightly from the non-sub
one. The different is that in addition to it's ConfNode, it gets
the OutputCtx from the parent. This way it can set the parents
LogFileCtx in it's own OutputCtx.
The runmode setup code will take care of all the extra setup. It's
possible to register a module both as a normal module and as a sub-
module, which can operate at the same time.
Only the TxLogger API is handled in this patch, the rest will be
updated later.
- restore json output-file.[ch] as output-json-file.[ch] after rebase conflict
- fix Makefile.am after merge conflict
- some dev-log-api-v4.0 rebase json fallout cleanup
Victor Julien
Eric Leblond
Ken Steele
Jason Ish
Anoop Saldanha
Tom DeCanio