Add keyword to do "raw" matching on each of the certificates in the
TLS certificate sticky buffer.
Example:
alert tls any any -> any any (msg:"tls.certs test"; tls.certs; \
content:"|01 02 03 04|"; sid:1;)
This change ensures that each anomaly is tagged with an
event type to support querying.
Each anomalous event will include `"event_type": "anomaly"`
in the log record.
This changeset adds anomaly logging to suricata for issue 2282.
Anomaly logging is controlled via the `anomaly` section within eve-log.
There is a single option -- `packethdr` -- for including the packet header
in the anomaly.
When killed with SIGHUP, suricata reopens the log files. If filemode
is set in the config, it needs pledge promise "fattr" to allow the
chmod(2) on OpenBSD.
Make sure all file sig mismatches indicate this in their return
code, not just the ones with filestore enabled. This is needed
to tell the stateful detect engine that it is dealing with a file
sig, so it can make sure these are inspected correctly even if
there are possibly multiple files per tx.
If the filename inspection function is returning nomatch this will
trigger iterative inspections with same content (aka filename) being
inspected. To avoid this we change the return as the buffer inspection
has not to be inspected anymore.
This patch implement an option named 'buffer' that can be used in the
init function of a lua signature:
function init (args)
local needs = {}
needs["buffer"] = tostring(true)
return needs
end
With this, the lua script will get access to the sticky buffer
content.
This fixes redmine bug #2057 by setting pseudopacket iface and vlan from
flow values, solving the problem of missing vlan/iface when psuedopacket
gets logged/alerted on.
Adds SSH_FLAG_VERSION_PARSED to flags before each return
This way, we are sure SSHParseBanner does not get called again
And proto_version does not get leaked
This changeset addresses 2 issues:
- 2459
- 2527
and improves handling for FTP active mode over IPv4 and IPv6.
Active mode is triggered when the FTP client conveys the port
that should be used for a data connection (PORT, EPRT).
When this occurs, the FTP state is marked as "active".
Instead of casting the packet buffer to a uint32, memcpy it to
avoid misaligned read error, as caught by the undefined behavior
detector (ubsan).
Redmine issue:
https://redmine.openinfosecfoundation.org/issues/2903
- There is now an option to automatically create streams on the
correct NUMA node when using cpu affinity.
- When not using cpu affinity the user can specify streams to be
created in the suricata.yaml file. It is no longer required to
use NTPL to create streams before running suricata.
- The legacy usage model of running NTPL to create streams is still
available. This can be used for legacy configurations and complex
configurations that cannot be satisfied by the auto-config option.
pledge(2) can be used on OpenBSD to restrict suricata possible
operation on the system once initialization is completed.
The process promises to only make use of:
- "stdio" to allow read(2) on IPS rules and write(2) on log file
- "rpath wpath cpath" to allow log rotation
- "unix" to operate the control unix socket and log unix sockets
- "dns" to retrieve DNS from recvfrom(2)/sento(2) in IPFW mode
- "bpf" as suricata uses libpcap, which uses the BIOCGSTATS operation
Signed-off-by: Emmanuel Roullit <emmanuel.roullit@cognitix.de>
The mode input in chmod is an octal integer. However when the warning is logged,
the file mode is printed in decimal which is confusing.
Signed-off-by: Emmanuel Roullit <emmanuel.roullit@cognitix.de>
The return value from the options decoder in TCP and IPv4 is ignored.
This commit changes the return type of the function to `void` and
modifies existing return points to return without a value.
When an error occurs, the packet state is being set to indicate whether
it's valid or not and the existing return value is never used.
This fixes two memleaks found with ASAN.
Direct leak of 96 byte(s) in 1 object(s) allocated from:
#0 0x7f59cf4a4d28 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1d28)
#1 0xd7f92f in ReceiveNFLOGThreadInit /home/glongo/suricata/src/source-nflog.c:221
#2 0xe9c8eb in TmThreadsSlotPktAcqLoop /home/glongo/suricata/src/tm-threads.c:293
#3 0x7f59cd7aa4a3 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3)
Indirect leak of 70000 byte(s) in 1 object(s) allocated from:
#0 0x7f59cf4a4d28 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1d28)
#1 0xd814ea in ReceiveNFLOGThreadInit /home/glongo/suricata/src/source-nflog.c:324
#2 0xe9c8eb in TmThreadsSlotPktAcqLoop /home/glongo/suricata/src/tm-threads.c:293
#3 0x7f59cd7aa4a3 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3)
SUMMARY: AddressSanitizer: 70096 byte(s) leaked in 2 allocation(s).
When Suricata picks up a flow it assumes the first packet is
toserver. In a perfect world without packet loss and where all
sessions neatly start after Suricata itself started, this would be
true. However, in reality we have to account for packet loss and
Suricata starting to get packets for flows already active be for
Suricata is (re)started.
The protocol records on the wire would often be able to tell us more
though. For example in SMB1 and SMB2 records there is a flag that
indicates whether the record is a request or a response. This patch
is enabling the procotol detection engine to utilize this information
to 'reverse' the flow.
There are three ways in which this is supported in this patch:
1. patterns for detection are registered per direction. If the proto
was not recognized in the traffic direction, and midstream is
enabled, the pattern set for the opposing direction is also
evaluated. If that matches, the flow is considered to be in the
wrong direction and is reversed.
2. probing parsers now have a way to feed back their understanding
of the flow direction. They are now passed the direction as
Suricata sees the traffic when calling the probing parsers. The
parser can then see if its own observation matches that, and
pass back it's own view to the caller.
3. a new pattern + probing parser set up: probing parsers can now
be registered with a pattern, so that when the pattern matches
the probing parser is called as well. The probing parser can
then provide the protocol detection engine with the direction
of the traffic.
The process of reversing takes a multi step approach as well:
a. reverse the current packets direction
b. reverse most of the flows direction sensitive flags
c. tag the flow as 'reversed'. This is because the 5 tuple is
*not* reversed, since it is immutable after the flows creation.
Most of the currently registered parsers benefit already:
- HTTP/SMTP/FTP/TLS patterns are registered per direction already
so they will benefit from the pattern midstream logic in (1)
above.
- the Rust based SMB parser uses a mix of pattern + probing parser
as described in (3) above.
- the NFS detection is purely done by probing parser and is updated
to consider the direction in that parser.
Other protocols, such as DNS, are still to do.
Ticket: #2572
Better handle case where pcap file receive thread fails to initialize. Allow
initialize to complete, but terminate the thread quickly. Delay exiting
unix socket runmode as late as possible.
Global NFQ contexts were not freed properly causing
'use-after-free' error. Moving contexts cleanup to a
separate NFQContextsCleanup() and calling it from
GlobalsDestroy(), like it's done for AFPacket, solves
the problem.
In case the pid file is given in the config file, the file name is
stored in volatile memory. Removal of the pid file happens after
cleanup of config memory. Create a copy of the name which will be
freed after the pid file has been removed.
Flow direction doesn't need explicit checking as the rule groups (sgh)
are already per direction. So if a rule sets only flow:to_server or
flow:to_client, we can avoid adding a sigmatch to the signature.
Fix match array reset depending on prefilter matches for the
current run. If there were none, the match array of the previous
packet was used. This could lead to inspection of rules from the
wrong rule group.
Loading rules with iprep keyword cause
memory leaks due to missing frees.
Direct leak of 8 byte(s) in 4 object(s) allocated from:
#0 0x7f81c862bd28 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1d28)
#1 0x7f81c6afea69 in pcre_get_substring (/lib/x86_64-linux-gnu/libpcre.so.3+0x27a69)
#2 0x43206f7420676e68 (<unknown module>)
SUMMARY: AddressSanitizer: 8 byte(s) leaked in 4 allocation(s).
The regular expression was accepting something like
"flowbits:!isset,isma;" without complaining even if it is not
correct and don't have the expected result.
In low memory or memcap reached conditions a crash could happen in
inline stream detection.
The crash had the following path:
A packet would come in and it's data was added to the stream. Due
to earlier packet loss, the stream buffer uses a stream buffer block
tree to track the data blocks. When trying to add the current packets
block to the tree, the memory limit was reached and the add fails.
A bit later in the pipeline for the same packet, the inline stream
mpm inspection function gets the data to inspect. For inline mode
this is the current packet + stream data before and after the packet,
if available.
The code looking up the packets data in the stream would not
consider the possibility that the stream block returned wasn't
the right one. The tree search returns either the correct or the
next block. In adjusting the returned block to add the extra stream
data it would miscalculate offsets leading to a corrupt pointer to the
data.
This patch more carefully checks the result of the lookup, and
falls back to simply inspecting the packet payload if the lookup
didn't produce the expected result.
Bug 2842.
Reported-by: Ad Schellevis <ad@opnsense.org>
Process multiple packets at nm_dispatch. Use zero copy for workers
recv mode.
Add configure check netmap check for API 11+ and find netmap api version.
Add netmap guide to the userguide.
Cipher suites length should always be divisible by two. If it is a
odd number, which should not happen with normal traffic, it ends up
reading one byte too much.
No resizing is done in Ja3BufferResizeIfFull() when the buffer is
empty. This leads to a potential overflow when this happens, since
a ',' is appended even when the buffer is empty.
Bug #2762
No longer set stream events after a gap or wrong thread. We know
we lost sync and are now in 'lets make the best of it'-mode. No
point in flooding the system with stream events.
Ticket #2484
Add a keyword configuration dump-all-headers, with allowed values
{both, request, response}, dumping all HTTP headers in the eve-log http
object. Each header is a single object in the list request_headers
(response_headers) with the following notation:
{
"name": <header name>,
"value": <header value>
}
To avoid forged malicious headers, the header name size is capped at 256
bytes, the header value size at 2048.
By default, dump-all-headers is disabled.
Add a raw-extraction option for smtp. When enabled, this feature will
store the raw e-mail inside a file, including headers, e-mail content,
attachments (base64 encoded). This content is stored in a normal File *,
allowing for normal file detection.
It'd also allow for all-emails extraction if a rule has
detect-filename:"rawmsg" matcher (and filestore).
Note that this feature is in contrast with decode-mime.
This feature is disabled by default, and will be disabled automatically
if decode-mime is enabled.
Previously this was limited to 16, however Netfilter allows
up to 65535 queues. Suricata now is able to create as many
queues as possible, but at the same time warns user if one
specifies more queues than available CPU cores.
This change involves dynamic (de)allocation of NFQ contexts
instead of on-stack arrays to use less memory.
If one needs to use multiple sequential Netfilter queues,
it can be done with a new '-q' option's syntax: "start:end"
(just like it's done with iptables '--queue-balance' option).
If a bad RST was received the stream inspection would not happen
for that packet, but it would still move the 'raw progress' tracker
forward. Following good packets would then fail to detect anything
before the 'raw progress' position.
Bug #2770
Reported-by: Alexey Vishnyakov
In the eve log the decoder events are added as optional counters. This
behaviour is enabled by default. However, lots of the counters are
missing, as the names colide with other counters.
E.g.
decoder.ipv6 counts ipv6 packets
decoder.ipv6.unknown_next_header counts how often an unknown next
header is encountered.
In this example 'ipv6' would be both a json integer and a json object.
It appears that jansson favours the first that is generated, so the
event counters are mostly missing.
This patch registers them as 'decoder.events.<event>' instead. As
these names are generated on the fly, a hash table to contain the
allocated strings was added as well.
Synchronize start was disabled for v2 when v3 was introduced, without
a reason being given.
Re-enable as v2 will otherwise also start reading packets before the
other threads are set up. This will lead to hashing issues.
Part of bug #2788.
The tpacket-v3 implementation of the synchonize start logic would
not correctly consider the timestamp parameter, leading to threads
starting before synchronization between threads was complete.
Bug #2788
Fixes nfqueue and delayed-detect.
On systems with small amount of traffic (or with no traffic at all)
nfqueue with 'delayed-detect' enabled hanged in 'workers' mode.
Bug #2362.
The DCERPC UDP detection would misfire on DNS with transaction
ID 0x0400. This would happen as the protocol detection engine
gives preference to pattern based detection over probing parsers for
performance reasons.
This hack/workaround fixes this specific case by still running the
probing parser if DCERPC has been detected on UDP. The probing
parser result will take precedence.
Bug #2736.
Invalid Teredo can lead to valid DNS traffic (or other UDP traffic)
being misdetected as Teredo. This leads to false negatives in the
UDP payload inspection.
Make the teredo code only consider a packet teredo if the encapsulated
data was decoded without any 'invalid' events being set.
Bug #2736.
In the stats loop sleep for a time period more closely matching
the stats.interval setting. Fix an off by one that would make
the loop wake up ~1 second early.
Bug #2716
The app layers with a custom iterator would skip a tx if during
the ..Cleanup() pass a transaction was removed.
Address this by storing the current index instead of the next
index. Also pass in the next "min_tx_id" to be incremented from
the last TX. Update loops to do this increment.
Also make sure that the min_id is properly updated if the last
TX is removed when out of order.
Finally add a SMB unittest to test this.
Reported by: Ilya Bakhtin
Stress condition in Suricata could lead to interface to disconnect
when it is not necessary. This patch updates the error handling
code to try to continue reading when such a case occurs.
The fast pattern selection logic would truncate a patterns len to
255 leading to assigning the same pid to different patterns.
This in turn would be caught by the hyperscan setup code which would
abort.
Bug #2714.
Fix and Optimize cleanup. For the simple single inspect buffer optimize
the cleanup by keeping track of the actually used buffers. This avoid
looping over unused buffers.
Fix the case of cleaning not being done after a tx if the next tx is
also inspected in the context of the same packet.
Fix cleanup of the multi-inspect buffers. Optimize in 2 ways. First
like with single keep track of which multi-inspect buffers have been
used. Second, keep a max of the buffers within a multi-inspect buffer.
Use this max to limit (nested) looping.
Due to missing error handling, a bad mime message could put the
mime parser in an error state, without the SMTP layer taking this
into account. So the SMTP layer would continue to pass data to the
mime parser, even though it was in an error state.
When the parser would be fed a very long line while in this state,
it would try to set an error flag in the state. However, due to
the error state, this setting of the flag would dereference a null
pointer.
This patch fixes this issue by updating the mime parser to check
the state it is in when receiving new input. It will refuse to
process futher data while in the error state. It will also return
a new error code to indicate to the SMTP layer that the parser
was in an error state.