Since enabling custom logging will replace the extended logging, thus
possibly leading to certain fields disappearing from the logs, mention
this aspect.
Related to
Bug #7333
Add EVE documentation for QUIC and Pgsql to their respective sections of
the userguide.
Also add a complete EVE reference as an appendix.
Other protocols can be done, but its a manual process to document in the
schema, then add the glue to pull them into the documentation.
The documentation is generated during "make dist", or if it doesn't
exist, "conf.py" will attempt to generate the eve documentation for
building on Readthedocs.
Ticket: 7191
So as to avoid quadratic complexity in libhtp.
Make the limit configurable from suricata.yaml,
and have an event when network traffic goes over the limit.
Use sed + regex to replace all occurrences of suricata.yaml terms that
used underscore for their up-to-date dash version.
Also search for such terms in the eve-log.yaml partials file, as that
is referenced in the configuration section.
commands used:
sed -i 's/\(^ *[a-z]*\)_\([a-z]*:\)/\1-\2/g'
sed -i 's/\(^ *[a-z]*\)_\([a-z]*\)_\([a-z]*:\)/\1-\2-\3/g'
Some other instances were found manually.
Task #7260
If a stream-only rule matches, and we find a tx where we
want to log the app-layer data, store into the tx data that
we already logged, so that we do not log again the app-layer metadata
Ticket: 7085
Although the `prune-flows` option was removed with a5587fec2e,
when documentation for the suricata.yaml config file was added with
b252b0d, this option was also included - as has remained until now.
They have updated their docs domain, leading to the link we had
returning a 404.
Also checked the other links. Although some seem to only contain old
traffic, they all still work.
This ensures that the correct spelling of the `security_result` EVE
field for RFB (as opposed to `security-result`) is also reflected in the
documentation.
Ticket: #7210
While not incorrect, the previous wording made the sentence almost
paradoxical. While at it, also highlight a side effect that might not be
so clear to users.
Related to
Bug #6976
Ticket: 5734
Adds frames for SSH records, that come after banner, and before
the data is encrypted.
These records may contain cipher lists for instance.
Ticket: 4863
On the way, convert some keywords to use the first-class integer
support.
And helpers for pure rust the support for multi-buffer.
Move the C unit tests about keyword mqtt.protocol_version
to unit tests for generic integer parsing, and test version 5
instead of testing twice version 3.
Also iterate all tx's messages for reason code as is done for other
keywords.
And allow detection on empty topics.
Ticket: 4863
On the way, convert unit test DetectSNMPCommunityTest to a SV test.
And also, make snmp.pdu_type use a generic uint32 for detection,
allowing operators, instead of just equality.
With triggering stream reassembly early, since for certain types of
rules there may be more alerts triggered - even in IPS mode, make this
clear in the upgrading section.
Bug #7026
Ticket: 3958
- transactions are now bidirectional
- there is a logger
- gap support is improved with probing for resync
- frames support
- app-layer events
- enip_command keyword accepts now string enumeration as values.
- add enip.status keyword
- add keywords :
enip.product_name, enip.protocol_version, enip.revision,
enip.identity_status, enip.state, enip.serial, enip.product_code,
enip.device_type, enip.vendor_id, enip.capabilities,
enip.cip_attribute, enip.cip_class, enip.cip_instance,
enip.cip_status, enip.cip_extendedstatus
Although we have the example for a commit message in our Code Submission
Process sub-chapter, seems that people still oversee it a lot. It was
suggested that we put it in a note-box, to make it more visible.
When we added feature #5976 (72146b969), we overlook that we also have
a config stats option for the human-readable stats logs to output
0 counters.
Due to not seeing this before, we now have two different setting names
for basically the same thing, but in different logs:
- zero-valued-counters for EVE
- null-values for stats.log
This ensures we use the same terminology, and change the recently added
one to `null-values`, as this one has been around for longer.
Task #6962
Some stats can be quite verbose if logging all zero valued-counters.
This allows users to disable logging such counters. Default is still
true, as that's the expected behavior for the engine.
Task #5976
Some exception policies can only be applied to entire flows or
individual packets, for some exception scenarios. Make this easier to
read, in the documentation.
Related to
Task #5816
One of the lessons of the XZ backdoor story was that just linking to
libsystemd to call sd_notify is discouraged by the systemd project:
Lennart Poettering:
"PSA: In context of the xzpocalypse we now added an example reimplementation
of sd_notify() to our man page:
https://www.freedesktop.org/software/systemd/man/devel/sd_notify.html#Notes
It's pretty comprehensive (i.e. uses it for reload notification too), but
still relatively short.
In the past, I have been telling anyone who wanted to listen that if all you
want is sd_notify() then don't bother linking to libsystemd, since the
protocol is stable and should be considered the API, not our C wrapper
around it. After all, the protocol is so trivial"
From: https://mastodon.social/@pid_eins/112202687764571433
This commit takes the example code and uses it to reimplement the notify
logic.
The code is enabled if Linux is detected in configure. Since the code
won't do anything if the NOTIFY_SOCKET env var isn't set, this should
also work fine on systems w/o systemd.
Ticket: #6913.
By default, when Sphinx generates the man pages, the current date will
be embedded in them. This can be set to a specific date with the
"today" variable. Typically the date embedded in manpages in the
release date.
To achieve this, attempt to use the environment variable, RELEASE_DATE
to set the "today" variable, reverting back to the empty string if not
set. It is up to our build system to properly set this date.
Ticket: #6911
dns.rcode matches the rcode header field in DNS messages
It's an unsigned integer
valid ranges = [0-15]
Does not support prefilter
Supports matches in both flow directions
Task #6621
It matches the rrtype field in DNS
It's an unsigned integer match
valid ranges = [0-65535]
Does not support prefilter
Supports flow in both directions
Feature #6666
Our code handles Uint ranges as exclusive, but for bsize, our
documentation stated that they're inclusive.
Cf. from uint.rs:
DetectUintMode::DetectUintModeRange => {
if val > x.arg1 && val < x.arg2 {
return true;
}
}
Task #6708
Remove references that are mentioning Suricata 3 or less
As a note - only one Suricata 4 reference found:
(suricata-yaml.rst:"In 4.1.x")
Fast pattern selection criteria can be internally found by inspecting
SupportFastPatternForSigMatchList and SigTableSetup functions.
Ticket: #6570
When the packet load is low, Suricata can run in interrupt
mode. This more resembles the classic approach of processing
packets - CPU cores run low and only fetch packets
on interrupt.
Ticket: #5839
Previous integration of hugepage analysis only fetched data
from /proc/meminfo. However this proved to be often
deceiving mainly for providing only global information and
not taking into account different hugepage sizes (e.g. 1GB
hugepages) and different NUMA nodes.
Ticket: #6419
If a commit introduces code that changes Suricata behavior, the related
documentation changes should go in a separate commit, but refer to the
same ticket number.
This reduces the chances of said changes being lost if there are backports
while still keeping the backporting process a bit less bulky, for each
commit.
Related to
Task #6568
This section seemed to aim both at PR reviewers and PR authors at the
same time, even though some info is probably of low value for
contributors.
Created new section for PR reviewers and maintainers, and kept the info
for PR authors separated. Also highlighted information on requested
changes and stale PRs.
This could be justified from a semantic point of view, and also can help
in bringing more attention to where this information is, as it is less
hidden, now.
Also add Dev Guide as one of our resources in our Readme.
Add a new rule keyword "requires" that allows a rule to require specific
Suricata versions and/or Suricata features to be enabled.
Example:
requires: feature geoip, version >= 7.0.0, version < 8;
requires: version >= 7.0.3 < 8
requires: version >= 7.0.3 < 8 | >= 8.0.3
Feature: #5972
Co-authored-by: Philippe Antoine <pantoine@oisf.net>
Add a more visible explanation of that requests, responses, frontend and
and backend are, in Pgsql context, to avoid having to repeat that over
different portions of the docs.
A CanceldRequest can occur after any query request, and is sent over a
new connection, leading to a new flow. It won't take any reply, but, if
processed by the backend, will lead to an ErrorResponse.
Task #6577
Rename DetectAppLayerInspectEngineRegister2 to
DetectAppLayerInspectEngineRegister as there is no other variant of
this function, and the versioning with lack of supporting
documentation can lead to confusion.
The old DetectAppLayerMpmRegister has not been around since 4.1.x.
Rename the v2 of this function to a versionless function as there is no
documentation referring to what the 2 means.
The term "whitelist" is actually used to store a list of DetectPort type
items for tcp and udp in detect.h. Using the same term for also keeping
the score that affects the grouping of rules is confusing. So, rename
the variable to "score".
Our tls fields not_after and not_before are actually logged as
`notafter` and `notbefore`, but were documented with the underscore.
Update the documentation, since updating the log format itself would be
a breaking change.
Task #5494
Although we have an updated version of instructions for installation
from git, our install guide was only referring to RedMine, which is less
up-to-date.
Kept that reference, since it might still be useful for non-Ubuntu
cases.
- Use SPHINX_BUILD instead of HAVE_SPHINX_BUILD, as here we're
actually using the path of the program.
- Wrap some elements in [] as is done in modern idiomatic autoconf
When viewing the docs online at Readthedocs, or similar it might be
immediately apparent what version of the documentation is being
displayed. Display the version on the first line before the table of
contents to make it clear.
With the release of 7, people are starting to have issues with traffic
being blocked. While we don't add a more expansive documentation for
this, add a link to the FAQ covering possible fixes for drops caused by
the fail closed default behavior of the exception policies.
Add CSS to avoid horizontal scroll in tables on ReadTheDocs. This will
wrap the text instead.
Also, vertically align to top so if a cell does wrap, other cells that
do not wrap don't place the text in the middle of the cell.
The `field action` portion seemed to be comprised of a more generic
section that followed it. Also formatted the section for lines to be
within the character limit.
If an exception policy wasn't set up individually, use the GetDefault
function to pick one. This will check for the master switch option and
handle 'auto' cases.
Instead of deciding what the auto value should be when we are parsing
the master switch, leave that for when some of the other policies is to
be set via the master switch, when since this can change for specific
exception policies - like for midstream, for instance.
Update exceptions policies documentation to clarify that the default
configuration in IPS when midstream is enabled is `ignore`, not
`drop-flow`.
Bug #6169
DPDK apps can specify multiple arguments of the same
type. YAML format only allows unique keys within a single
node. This commit adds support for multiple EAL arguments
of the same type to be used within suricata.yaml.
Ticket: #5964
Some exception policies can only be applied to the triggering packet or
only make sense considering the whole flow. Highlight such cases in the
table showing each exception policy.
Related to
Bug #5825
The different interactions between midstream pick-up sessions and the
exception policy can be quite difficult to visualize. Add a section for
that in the userguide.
Related to
Bug #5825