This is to prepare for JsonBuilder conversion where we can't
overwrite an already set value. Here we prepare the addresses
to be logged in a struct, overwite with XFF if needed, then
log.
Convert alert Eve logging JsonBuilder. Currently
makes heavy use of JsonBuilder being able to log Jansson's json_t
which is a temporary measure until all protocols loggers can be
converted to JsonBuilder.
New functions that replace Jansson versions with JsonBuilder
variations use "Eve" instead of "JSON".
JsonBuilder is a Rust module for creating JSON output. Unlike
Jansson, the final JSON string is built up as items are added,
instead of building up an object tree and rendering it when
done.
The idea is to create a more efficient JSON serializer instead
of a flexible one.
Update the source/target logging to use the cached address info
instead of fetching it from the constructed json_t object.
This is required for migration to JsonBuilder which does not
have the ability to retrieve already set fields.
Currently the flow logger also logs app_proto information,
but not to the flow object, but instead to the root object
of the log record.
Refactor into 2 separate methods, one for the app_proto
and one for the flow, to make this more clear, as well
as make it easier to refactor for JsonBuilder as JsonBuilder
can only write to the currently open object.
Currently alert logging relies on the ability to change existing
values in the json_t structure to overwrite addresses with xff
data. This feature is also used for the "target" logging.
As we can't do this with JsonBuilder, create a new struct to
hold the 5 tuple, with the values swapped as needed, and
overwritten with XFF data if needed. This struct will now
be used to write out the 5 tuple, as well as cache the information
for log fields to be written out later on in the log path.
Remove the Jansson specific feature of being able to delete
an object from json_t, in prep for refactors to JsonBuilder.
Instead create a new header for each alert to be logged.
Move the logging of the rule text to where the alert object
is being logged to remove the usage of json_object_get...
Getting previously logged objects will not be possible with
JsonBuilder.
To signal incomplete data, we must return the number of
consumed bytes. When we get a banner and some records, we have
to take into account the number of bytes already consumed by
the banner parsing before reaching an incomplete record.
When PCRE `jit` is available, store the JIT stack in the keyword context
instead of on a global id. This ensures proper cleanup and
re-initialization over a rule reload.
Create a "prep" build that parses libhtp, suricata-update and
suricata-verify repo and branch information from the pull
request message and turn these into artifacts that are
used by the builders
Examples:
libhtp-repo: https://github.com/OISF/libhtp
libhtp-branch: 0.5.x
suricata-verify-repo: http://github.com/OISF/suricata-verify
suricata-verify-branch: master
suricata-update-repo: http://github.com/OISF/suricata-update
suricata-update-branch: master
Other changes:
- checkout@v2 (faster)
- working directory cleanup
File store v1 has been deprecated and was scheduled for removal
by June 2020.
Log an error if a file-store configuration is loaded without
version set to 2.
All the dead code in C after the Rust implementation is hereby removed.
Invalid/migrated tests have also been deleted.
All the function calls in C have been replaced with appropriate calls to
Rust functions. Same has been done for smb/detect.rs as a part of this
migration.
This parser rewrites the DCE/RPC protocol implementation of Suricata
in Rust. More tests have been added to improve the coverage and some
fixes have been made to the tests already written in C. Most of the
valid tests from C have been imported to Rust.
File anatomy
src/dcerpc.rs
This file contains the implementation of single transactions in DCE/RPC
over TCP. It takes care of REQUEST, RESPONSE, BIND and BINDACK business
logic before and after the data parsing. DCERPCState holds the state
corresponding to a particular transaction and handles all important
aspects. It also defines any common structures and constants required
for DCE/RPC parsing irrespective of the carrier protocol.
src/dcerpc_udp.rs
This file contains the implementation of single transactions in DCE/RPC
over UDP. It takes care of REQUEST and RESPONSE parsing. It borrows the
Request and Response structs from src/dcerpc.rs.
src/detect.rs
This file contains the implementation of dce_iface and opnum detect
keywords. Both the parsing and the matching is taken care of by
functions in this file. Tests have been rewritten with the test data
from C.
src/parser.rs
This file contains all the nom parsers written for DCERPCRequest,
DCERPCResponse, DCERPCBind, DCERPCBindAck, DCERPCHeader, DCERPCHdrUdp.
It also implements functions to assemble and convert UUIDs. All the
fields have their endianness defined unless its an 8bit field or an
unusable one, then it's little endian but it won't make any difference.
src/mod.rs
This file contains all the modules of dcerpc folder which should be
taken into account during compilation.
Function calls
This is a State-wise implementation of the protocol for single
transaction only i.e. a valid state object is required to parse any
record. Function calls start with the app layer parser in C which
detects the application layer protocol to be DCE/RPC and calls the
appropriate functions in C which in turn make a call to these functions
in Rust using FFI. All the necessary information is passed from C to the
parsers and handlers in Rust.
Implementation
When a batch of input comes in, there is an analysis of whether the
input header and the direction is appropriate. Next check is about the
size of fragment. If it is as defined by the header, process goes
through else the data is buffered and more data is awaited. After this,
type of record as indicated by the header is checked. A call to the
appropriate handler is made. After the handling, State is updated with
the latest information about whatever record came in.
AppLayerResult::ok() is returned in case all went well else
AppLayerResult::err() is returned indicating something went wrong.
Jason Ish
Victor Julien
Philippe Antoine
Eric Leblond
Jeff Lucovsky
Shivani Bhardwaj