Commit Graph

10040 Commits (6fe8bce3b0c653df8e72bdb81bc235672386b8b1)

Author SHA1 Message Date
Philippe Antoine 5d676c5998 http2: http.uri.raw keyword now works for HTTP2 6 years ago
Philippe Antoine 47928babfc http2: http.user_agent keyword now works for HTTP2 6 years ago
Philippe Antoine a98d0fe6ed http2: http.uri keyword now works for HTTP2
cf #4067
6 years ago
Philippe Antoine 707f027231 protos: renaming ALPROTO_HTTP* constants
Having now ALPROTO_HTTP1, ALPROTO_HTTP2 and ALPROTO_HTTP

Run with 3 sed commands
git grep ALPROTO_HTTP | cut -d: -f1 | uniq |
 xargs sed -i -e 's/ALPROTO_HTTP/ALPROTO_HTTP1/g'
git grep ALPROTO_HTTP12 | cut -d: -f1 | uniq |
 xargs sed -i -e 's/ALPROTO_HTTP12/ALPROTO_HTTP2/g'
git grep ALPROTO_HTTP1_ANY | cut -d: -f1 | uniq |
 xargs sed -i -e 's/ALPROTO_HTTP1_ANY/ALPROTO_HTTP/g'

and then running clang-format
6 years ago
Philippe Antoine 93e6401ce0 http: introduces ALPROTO_HTTP_ANY
For any versions of HTTP, both ALPROTO_HTTP and ALPROTO_HTTP2
6 years ago
Philippe Antoine c8dbe24fb6 proto: introduce signature protocol, as extension to flow protocol
AppProtoEquals function allows to check if a flow protocol
matches a signature protocol
6 years ago
Jason Ish 02218a8a42 Makefile: break headers and source into 2 vars
Split the headers and source into 2 variables. Headers are
marked noinst so they don't get automatically installed on
"make install". Instead they will be installed by a custom
Makefile target, "make install-headers".
6 years ago
Jason Ish 2c5e1d6a6d rust: separate the rust lib from RUST_LDADD
Fix another issue with library ordering when breaking apart
LDFLAGS from LIBS for outputting usable command lines for
users of a Suricata library.

RUST_LDADD should just contain the extra libs required by
Rust, not the actual Suricata Rust library.
6 years ago
Jason Ish dbae17dbc0 install: makefile target to install libraries
As we don't install the libraries by default, provide a make target,
"install-library" to install the libsuricata library files.

If shared library support exists, both the static and shared
libraries will be installed, otherwise only the static libraries
will be installed.
6 years ago
Jason Ish e227d97e5e lib: build shared library on Linux
Building the shared library on Linux is not something by default.
Instead a user must opt-in to building by running the
"make libsuricata.so" target in the src/ directory.

Currently shared library support is only available on Linux. More
OSs will be supported as we can test them.
6 years ago
Jason Ish e99dde0078 build: use a static convenience library for C code
With the circular reference gone, we can now make use
of a convenience library for the Suricata program
as well as any other programs that depend on the same
source such as the fuzzer.

While its not a libtool convenience library, it serves
the same purpose and is a common idiom in Make and CMake
projects whereas the COMMON_SOURCES approach was more
of a hack we had to resort to until the circular
reference was resolved.
6 years ago
Victor Julien 6bfc5afa23 host: improve compare logic
The old compare macro would compare all bytes of an address, even
when for IPv4 addresses the additional bytes were not in use. This
made the logic vulnerable to mistakes like in issue #4280.
6 years ago
Victor Julien 7b03e6837e detect/iprep: fix loading of mixed ipv4/ipv6 lists
Improper reuse of the address data structure between loading
different lines in the iprep file would lead to the host using
a malformed address.
6 years ago
Jason Ish 3ada5e1480 rust/ffi: provide AppLayerRegisterParser in context
AppLayerRegisterParser was creating a link error when attempting
to use a convenience library for the Suricata C code, then linking
the library of C code with the library of Rust code into a final
Suricata executable, or use with fuzz targets.

By moving AppLayerRegisterParser to the context structure and
calling it like a callback the circular reference is removed
allowing the convenience libraries to work again.

This is also a stepping block to proving a Suricata library
as a single .a or .so file.
6 years ago
Victor Julien 3ce05a3583 fuzz: run OSS-Fuzz corpus and track coverage 6 years ago
Philippe Antoine 2b043150ed detect: initializes memory in bytemath parsing 6 years ago
Philippe Antoine b5d24a9a57 fuzz: driver running directories as well as single files 6 years ago
Eric Leblond 0dba1b09de suricata: improve list keywords
Exit with error if a keyword is not supported or not existing
and display a message.
6 years ago
Eric Leblond 2e4af5a091 suricata: return error value of custom run modes 6 years ago
Eric Leblond 44460f1945 util/running-modes: don't exit in running mode 6 years ago
Eric Leblond 921d44b262 log/pcap: exit on invalid filename
If the filename has to % sign and if pcap logging is using multi
mode, then the pcap capture will fail. So let's exit if ever this
is the case.
6 years ago
Eric Leblond 6a45064d4c suricata: unix-socket mode and -l are compatible
Commit 93642a0d1d did prevent to
specify the logging directory on command line and use the unix
socket.

It looks like the implementation has evolved and the arbitrary
limitation can be removed allowing a user to start unix socket
without editing the configuration file.
6 years ago
Eric Leblond 7304389438 eve: only output ja3 and ja3s if present
This will prevent JSON entries like the following that occur
with the dedault configuration (ja3 deactivated and extended
tls ouput activated):

  "tls": {
    "subject": "C=GB, ST=London, L=London, O=Global Security, OU=IT Department, CN=example.com",
    "issuerdn": "C=GB, ST=London, L=London, O=Global Security, OU=IT Department, CN=example.com",
    "serial": "00:9C:FC:DA:1D:A4:70:87:5D",
    "fingerprint": "b8:18:2d:cb:c9:f8:1a:66:75:13:18:31:24:e0:92:35:42🆎96:89",
    "version": "TLSv1",
    "notbefore": "2020-05-03T11:07:28",
    "notafter": "2021-05-03T11:07:28",
    "ja3": {},
    "ja3s": {}
  }
6 years ago
Jeff Lucovsky cbb03dbb39 detect/pcre: Test capture group/var mismatch 6 years ago
Jeff Lucovsky 469d5bb214 detct/pcre: Correct capture group count check
This commit corrects the validation check between the number of
variables used and the number of specified capture groups.
6 years ago
Victor Julien ed05c51d99 detect/state: optimize state keeping 6 years ago
Victor Julien 13cebb1857 detect: fix heap overflow issue with buffer setup
In some cases, the InspectionBufferGet function would be followed by
a failure to set the buffer up, for example due to a HTTP body limit
not yet being reached. Yet each call to InspectionBufferGet would lead
to the matching list_id to be added to the
DetectEngineThreadCtx::inspect.to_clear_queue. This array is sized to
add each list only once, but in this case the same id could be added
multiple times, potentially overflowing the array.
6 years ago
Victor Julien 17a38f1823 flow/manager: (u)sleep slightly longer
Sleep 250 microseconds instead of 100 as running in KVM cause the
old value to use 100% CPU for these threads.

Perf testing suggests no measurable impact for the non-KVM case.

Ticket: #4096
6 years ago
Victor Julien 8baef60d60 app-layer: fix transaction cleanup
Fix a 'skipped' transaction early in the list leading to all further
transactions getting skipped, even if they were fully processed and
ready to be cleaned up.
6 years ago
Philippe Antoine 62e665c848 fuzz: rightly uses PacketFreeOrRelease in target
instead of PacketFree because packets
may belong to the pool
6 years ago
Philippe Antoine e586d8526b fuzz: use some value for max_pending_packets
so as not to timeout waiting forever for the condition
in PacketPoolWait
6 years ago
Philippe Antoine a6bbb608f7 fuzz: makes target sigpcap more reproducible
By removing the temporary rules file if it existed
before the first run
6 years ago
Philippe Antoine b869ac01ee http: enables request decompression 6 years ago
Eric Leblond 85327890f5 suricata: avoid at exit crash in nfq mode
When Suricata was build with ebpf support and when it was started
in NFQ mode, it was crashing at exit because it was trying to free
the device extension.

This patch fixes the issue by only trigger the eBPF related code
when Suricata is running in AFP_PACKET mode.
6 years ago
Eric Leblond e6cfcb704c storage: fix a variable name 6 years ago
Eric Leblond 628458e7d3 detect: fix link to documentation 6 years ago
Philippe Antoine 43f25f127f ftp: ftp-data recognized by StringToAppProto 6 years ago
Philippe Antoine d861228214 http2: decompression for files
gzip and brotli decompression for files
6 years ago
Philippe Antoine 76db6e34a1 protocol detection: fix failure case
as reached by CIFuzz even if unreachable from Suricata
6 years ago
Philippe Antoine f5d8e953a8 protodetect: debug validation when multiple patterns match 6 years ago
Victor Julien 00d7c9034b stream: remove debug assert
In cases of large windows in the past the check would tigger.
6 years ago
Victor Julien b66d013294 detect/http_client_body: minor test cleanups 6 years ago
Eric Leblond 64f994f753 dataset: fix dataset string lookup
The data was unlocked but the use_cnt was not decreased resulting
in the data entry not being removable.
6 years ago
Victor Julien 191461a028 detect/file_data: cleanup tests 6 years ago
Victor Julien 116c089de0 stream/tests: minor cleanups 6 years ago
Victor Julien ee6d792b02 stream: move tests into tests/ 6 years ago
Victor Julien 226a82bade detect/fast_pattern: redo unittests 6 years ago
Victor Julien 66d7f5941a detect/fast_pattern: remove dead code 6 years ago
Victor Julien bc9e7743f3 detect/http-ua: cleanup tests 6 years ago
Emmanuel Thompson f12daa710f decode/flow/esp: Add ESP decoder & flow
- Adds an ESP (Encapsulating Security Payload) header decoder
- Tracks ESP flows via the SPI field
6 years ago
Victor Julien 9adeae07b1 decode: reformat REINIT macro 6 years ago
Victor Julien 3f4398cc90 decode: minor unittest cleanups 6 years ago
Victor Julien bf00285d0a proto/names: add SCTP if not defined in system
If SCTP is missing from /etc/protocols, add it manually.
6 years ago
Victor Julien c25afbccc1 json: remove unused jansson wrappers 6 years ago
Victor Julien b6b317cae6 http: enable and fix content range tests 6 years ago
Victor Julien a7cd765f20 app-layer/nfs: dead code removal 6 years ago
Juliana Fajardini 97350d9a2c detect/rpc: clean up unittests
- detect-rpc: convert unit tests to new FAIL/PASS API.
- detect-rpc: replace SigInit with DetectEngineAppendSig for more
  concise code.
6 years ago
Jason Ish 512b0350a0 lua: fix coverity issue with out of scope variable
Fix usage of out-of-scope variables. Introduced with the hashing
and adding the guard of g_disable_hashing.

To fix, just remove the guard so all variables are in scope. Hashes
are not initialized here so there is no need for the guard.
6 years ago
Jeff Lucovsky 1c68f4aed6 lua/test: Test cases using SC prefix
This commit adds paired test cases to ensure that the SC variant of the
entry points are tested.
6 years ago
Jeff Lucovsky c845974639 general: Correct typo 6 years ago
Jeff Lucovsky 431018d6f7 lua: Use SC prefix for Lua functions
This commit adds additional Lua API interfaces to bring consistency to
functions such that the `SC` prefix is available consistently across
flow int and flow var functions.
6 years ago
Victor Julien 3a8ba663a9 email/md5: optimize md5 handling 6 years ago
Jason Ish 6299222c4e email/eve: use Rust function to hash buffer to hex
Use SCMd5HashBufferToHex to hash the subject to a hex string.
Removes snprintf loop.
6 years ago
Jason Ish 0a3b9e0220 rust/hashing: add function to finalize md5 to hex string
New function, SCMd5FinalizeToHex to finalize an md5 hash
to a hex string.
6 years ago
Jason Ish 3a82153866 ja3: use SCMd5HashBufferToHex to print hash as hex
Replace snprintf loop with new function that hashes a single
buffer to an MD5 hex string.
6 years ago
Jason Ish e00d21a5cb filestore: respect g_disable_hashing
If g_disable_hashing is set, behave like libnss wasn't compiled
in.
6 years ago
Jason Ish 7525295e63 hashing: remove remaining HAVE_NSS guards
For features, we pretend to HAVE_NSS so scripts, external tests
continue to work.
6 years ago
Jason Ish 9b314bebe0 output-json-email: use Rust md5 bindings instead of libnss 6 years ago
Jason Ish 815396263b util/mime: use Rust md5 bindings instead of libnss
As the new Md5 hashing consumes its context on finalize, an bool
has_md5 flag has been added to let the logger know there is an
md5 hash available.
6 years ago
Jason Ish 9e14c00298 transform-sha256: use Rust sha256 bindings
Use new Rust sha256 bindings insead of libnss.
6 years ago
Jason Ish d98e34045b transform-sha1: use Rust sha1 bindings
Removes dependence on NSS.
6 years ago
Jason Ish 3d1a294bc7 util-crypt: remove ComputeSHA1
There is now a Rust replacement for ComputeSHA1 that is exposed
to C as SCSha1HashBuffer.
6 years ago
Jason Ish 90dcaff41d ssl: replace ComputeSHA1 with Rust SCSha1HashBuffer
Removes the final use of internal ComputeSHA1 function.
6 years ago
Jason Ish e0a2d10e7e transform/md5: use Rust md5 bindings 6 years ago
Jason Ish c2dd2c25b6 error: new disabled hashing error code
Add new error code to be used when something fails due to hashing being
disabled.
6 years ago
Jason Ish f723dc1309 suricata: add --disable-hashing option
The idea with a flag to disable hashing is to provide a way to
get the effect of building Suricata without libnss, which is
sometimes done for performance reasons.
6 years ago
Jason Ish 245a1bd4ce suricata.c: clang-format required after new option added 6 years ago
Jason Ish 7bb917b835 suricata.c: disable clang-format on long_opts init
clang-format will reformat this to have multiple options on a
single line which is hard to read and I don't see a specific
clang-format option to format this better. So for now,
disable clang-format on this section of code.
6 years ago
Jason Ish 0321f60ebf community-id: use Rust SHA1 to compute hash
Removes use of our internal ComputeSHA1 for calculating the
community ID.
6 years ago
Jason Ish d5bf748c90 ja3: remove requirement on NSS 6 years ago
Jason Ish e50ee7eb62 filestore: remove requirement of nss for filestore
Required including NSS header in places that depended on
util-file.h including it.

All filestore suricata-verify tests now pass without libnss.

Also enabled detect-file{md5,sha1,sha256} without NSS support.
6 years ago
Jason Ish e4acbcbb5e filestore: use Rust bindings for sha256/sha1/md5 6 years ago
Jason Ish f77fd0c0cb rust: include file cleanup
The cbindgen generated header should not include rust.h as
rust.h already includes the generated binding.

Fixup C source code that only pulled the generated include, it
should instead pull in "rust.h" which includes the generated
binding plus other misc. stuff.
6 years ago
Philippe Antoine 47dd9a5ebc signature: Fix leak in urilen parsing
cf #4254
6 years ago
Victor Julien 423a5e2f6c detect/file.data: support ftp and ftp-data 6 years ago
Victor Julien 45eddde573 detect/file.name: register inspect engine for ftp-data 6 years ago
Jeff Lucovsky f8fef0dd05 decode/tcp: Improved handling of TFO options
This commit improves handling of TCP fast open options
- Option length must be in [6, 18]
- Option length must be an even value
6 years ago
Gianni Tedesco 10ea60a237 detect: Validate that NOOPT options don't have optvals
Without this, a simple typo between : and ; is able to hide actual bugs
in rules.

I discovered 2 bugs in ET open ruleset this way.
6 years ago
Gianni Tedesco cebe15c23b detect-fast-pattern: Mark as OPTIONAL_OPT, instead of NOOPT
Also update the erroneous comment about it.
6 years ago
Eric Leblond 8d034b4163 eve/dhcp: avoid to call common logging twice 6 years ago
Joshua Lumb 9d432bbab3 threads: More descriptive startup output 6 years ago
Carl Smith 9b840104bd lua: Make the rule action available to output scripts
Useful for those that want to do custom logging from lua
6 years ago
Justin Ossevoort 320de5f43d eve: Log tenant_id for all eve-json messages 6 years ago
Carl Smith 660b68a083 nsh: Parsing of Network Services Header and payload
Support for EtherType 0x894F and basic header
6 years ago
Victor Julien db2dbaaf40 stream/midstream: handle packet loss after SYN/ACK 6 years ago
Victor Julien 895938080f stream/tcp: fix invalid ack events in timewait state 6 years ago
Victor Julien 49bd1f85b9 flow/timeout: fix TCP seq/ack for reversed flows
When a flow is swapped it also swaps the stream trackers, so it does
not make sense to reverse them during pseudo packet creation.
6 years ago
Victor Julien 3c7c361b60 detect/stream: fix async stream inspection
Move raw progress forward only if detect uses stream data, indicated
by the PKT_DETECT_HAS_STREAMDATA flag.
6 years ago
Victor Julien 8aa02c6d15 stream/tcp: fix async mode ACK validation 6 years ago
Philippe Antoine 8d659c6500 fuzz: improves sigpcap target with PacketPoolInit 6 years ago
Shivani Bhardwaj 6c19e7d0d4 counters: convert to FAIL/PASS API 6 years ago
Shivani Bhardwaj 7eef4e83e5 confyaml: convert to FAIL/PASS API 6 years ago
Shivani Bhardwaj fb311e3d9d applayer: convert to FAIL/PASS API 6 years ago
Shivani Bhardwaj bfd956ce67 applayer/htp/xff: convert to FAIL/PASS API 6 years ago
Shivani Bhardwaj b195ffbe18 applayer/ftp: convert to FAIL/PASS API 6 years ago
Shivani Bhardwaj 2e3d408e6f fastlog: convert to FAIL/PASS API 6 years ago
Janani Ramjee 2d3cf11906 detect/mark: convert unittests to use PASS/FAIL API 6 years ago
Juliana Fajardini 2765bff0c5 detect/msg: convert to FAIL/PASS API
Issue 4053. Adjust code formatting style (wrap long lines).
Replace SigInit with DetectEngineAppendSig.
6 years ago
Sumera Priyadarsini d4264748d8 detect: filesha1: convert unittests to use PASS/FAIL API
Currently, unit tests use integer values 1 and 0 to denote pass
and fail status of tests respectively. Modify the unit test
detect-filesha1 to use the PASS/FAIL API instead.
6 years ago
Juliana Fajardini d2144c0dfb detect-sid: fail/pass api
- convert unittests to new FAIL/PASS API.
6 years ago
Jason Ish f2ab5803fb dns: initialize log flags as an unsigned long long
On 64 bit all 64 bits were being initialized, but on 32 bit
only 32 bits were as it was being initialized as a long.

Redmine issue:
https://redmine.openinfosecfoundation.org/issues/4206
6 years ago
Victor Julien efc9a7a398 app-layer: remove callback for completion status
Since the completion status was a constant for all parsers, remove the
callback logic and instead register the values themselves. This should
avoid a lot of unnecessary callback calls.

Update all parsers to take advantage of this.
6 years ago
Victor Julien 84385549fe detect: remove unused arg from generic list inspect 6 years ago
Victor Julien d694dac5a8 detect: remove inspect v1 API 6 years ago
Victor Julien 92a8d8b4f4 detect/template: convert to v2 inspect API 6 years ago
Victor Julien 494f8f2700 detect/tls: convert to v2 inspect API 6 years ago
Victor Julien 71a508000b detect/ssh: convert to v2 inspect API 6 years ago
Victor Julien 86686eff17 detect/snmp: convert to v2 inspect API 6 years ago
Victor Julien 962230040a detect/rfb: convert to v2 inspect API 6 years ago
Victor Julien b11cc0fbcd detect/nfs: convert to v2 inspect API 6 years ago
Victor Julien 15c84bf3c6 detect/lua: convert to v2 inspect API 6 years ago
Victor Julien 5062023ffe detect/ftp: convert to v2 inspect API 6 years ago
Victor Julien 2320d62872 detect/mqtt: convert to v2 inspect API 6 years ago
Victor Julien 5611f9ddbd detect/modbus: convert to v2 inspect API 6 years ago
Victor Julien ce6f8605ea detect/enip/cip: convert to v2 inspect API 6 years ago
Victor Julien 35310313de detect/http2: inspect API v2 6 years ago
Victor Julien 67d45f7e3c detect/krb5: use inspect api v2 6 years ago
Victor Julien 197f99a134 detect/ftpbounce: switch to inspect api v2 6 years ago
Victor Julien 0f2c94ffea detect/files: remove unused prototypes 6 years ago
Victor Julien e0e4454db7 detect/files: inspect api v2 6 years ago
Victor Julien 48094b05c8 detect/file: minor cleanup 6 years ago
Victor Julien cd146adef0 detect/dnp3: convert to v2 inspect API 6 years ago
Victor Julien 9f8a8f368a detect/dcerpc: convert to v2 inspect API 6 years ago
Victor Julien db0665bccc detect/dns: convert to v2 inspect API 6 years ago
Victor Julien 8c7423fcb8 detect/app-layer-event: convert to v2 inspect API 6 years ago
Victor Julien c6c93d1d12 detect/stream: convert to inspect API v2 6 years ago
Victor Julien 51f38f6453 detect/payload: minor formatting fixup 6 years ago
Philippe Antoine 8c4e53e803 dnp3: regenerates C code with script 6 years ago
Shivani Bhardwaj ea0936199d output: use BASE64_BUFFER_SIZE macro
Base64Encode function requires the maximum length of the output string
as its last parameter. Use the macro BASE64_BUFFER_SIZE to calculate it
correctly.
6 years ago
Shivani Bhardwaj b0a6ed1e2a datasets/string: fix buffer overflow
The size of encoded_data array and the maximum output length parameter
to Base64Encode function were incorrect leading to buffer overflow for
certain cases. The algorithm requires at least 5 bytes of space to even
convert a string of length 1.

Use BASE64_BUFFER_SIZE macro to correctly calculate this output length.
Set size of encoded_data array to the calculated output length.
6 years ago
Shivani Bhardwaj 02942a123a util/crypt: Add macro for max base64encode len
Maximum length of a base64 encoded string can be 33% over the actual
length of the input string. The formula to best cover all the edge cases
is mathematically
(4 * (input_length + 2) / 3) + 1

Add a macro to calculate this for a given input length.
6 years ago
Philippe Antoine 057c4b34c8 ftp: optimize FTPGetOldestTx by starting from last handled tx
Avoids DOS by quadratic complexity algorithm.
Attack is
1 stack many requests/transactions (like cwd commands on a line)
2 get many answers
6 years ago
Victor Julien a8fc481cd3 http2: avoid null pointer deref in alert output
Bug #4120.
6 years ago
Philippe Antoine 096dce4bba http2: allow filestore to work with HTTP2 6 years ago
Philippe Antoine d5ac77ee8b packet: set length of 0 for too big copy 6 years ago
Philippe Antoine 6d2247391e ssl: improves keyword ssl_version parsing
Removes the use of PCRE for performance
Forbids empty negations after a valid pattern
Forbids mixing negative and positive forms as it is irrelevant
Forbids useless repetition of a version
6 years ago
Philippe Antoine 48f0c612e3 ssl: upgarde to uint32 for bytes_processed
as it can overflow, and is compared with uint32 such as
message_length and message_start and
6 years ago
Philippe Antoine 445e03d752 detect: null sanity checks for pkthdr
Even when the rules are only applied on traffic with the protocol
the structure for the protocol header can be set to NULL if there
was an error parsing the header
6 years ago
Philippe Antoine 33905b0910 fuzz: check PacketCopyData return value before processing packet 6 years ago
Philippe Antoine 18d8164a27 fuzz: limit memcap for datasets when fuzzing 6 years ago