diff --git a/doc/userguide/configuration/suricata-yaml.rst b/doc/userguide/configuration/suricata-yaml.rst index e79c3576ca..a1567fb252 100644 --- a/doc/userguide/configuration/suricata-yaml.rst +++ b/doc/userguide/configuration/suricata-yaml.rst @@ -265,6 +265,8 @@ appearance of a single fast.log-file line: append: yes/no #If this option is set to yes, the last filled fast.log-file will not be #overwritten while restarting Suricata. +.. _suricata-yaml-outputs-eve: + Eve (Extensible Event Format) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -679,6 +681,52 @@ a Netfilter log format. append: yes #If this option is set to yes, the last filled drop.log-file will not be #overwritten while restarting Suricata. If set to 'no' the last filled drop.log file will be overwritten. +.. _suricata-yaml-file-store: + +File-store (File Extraction) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The `file-store` output enables storing of extracted files to disk and +configures where they are stored. + +The following shows the configuration options for version 2 of the +`file-store` output. + +.. code-block:: yaml + + - file-store: + # This configures version 2 of the file-store. + version: 2 + + enabled: no + + # Set the directory for the filestore. If the path is not + # absolute will be be relative to the default-log-dir. + #dir: filestore + + # Write out a fileinfo record for each occurrence of a + # file. Disabled by default as each occurrence is already logged + # as a fileinfo record to the main eve-log. + #write-fileinfo: yes + + # Force storing of all files. Default: no. + #force-filestore: yes + + # Override the global stream-depth for sessions in which we want + # to perform file extraction. Set to 0 for unlimited. + #stream-depth: 0 + + # Uncomment the following variable to define how many files can + # remain open for filestore by Suricata. Default value is 0 which + # means files get closed after each write + #max-open-files: 1000 + + # Force logging of checksums, available hash functions are md5, + # sha1 and sha256. Note that SHA256 is automatically forced by + # the use of this output module as it uses the SHA256 as the + # file naming scheme. + #force-hash: [sha1, md5] + Detection engine ---------------- diff --git a/doc/userguide/file-extraction/file-extraction.rst b/doc/userguide/file-extraction/file-extraction.rst index db40bc5010..ab31d802d4 100644 --- a/doc/userguide/file-extraction/file-extraction.rst +++ b/doc/userguide/file-extraction/file-extraction.rst @@ -32,14 +32,70 @@ Settings Output ~~~~~~ -For file extraction two separate output modules were created: -"file-log" and "file-store". They need to be enabled in the -:doc:`../configuration/suricata-yaml`. For "file-store", the "files" -drop dir must be configured. +File-Store and Eve Fileinfo +--------------------------- +There are two output modules for logging information about files +extracted. The first is ``eve.files`` which is an ``eve`` sub-logger +that logs ``fileinfo`` records. These ``fileinfo`` records provide +metadata about the file, but not the actual file contents. -:: +This must be enabled in the ``eve`` output:: + + - outputs: + - eve-log: + types: + - files: + force-magic: no + force-hash: [md5,sha256] + +See :ref:`suricata-yaml-outputs-eve` for more details on working +with the `eve` output. + +The other output module, ``file-store`` stores the actual files to +disk. + +The ``file-store`` uses its own log directory (default: `filestore` in +the default logging directory) and logs files using the SHA256 of the +contents as the filename. Each file is then placed in a directory +named `00` to `ff` where the directory shares the first 2 characters +of the filename. For example, if the SHA256 hex string of an extracted +file starts with "f9bc6d..." the file we be placed in the directory +`filestore/f9`. + +Using the SHA256 for file names allows for automatic de-duplication of +extracted files. However, the timestamp of a pre-existing file will be +updated if the same files is extracted again, similar to the `touch` +command. + +Optionally a ``fileinfo`` record can be written to its own file +sharing the same SHA256 as the file it references. To handle recording +the metadata of each occurrence of an extracted file, these filenames +include some extra fields to ensure uniqueness. Currently the format +is:: + + ...json + +where ```` is the seconds from the packet that triggered the +stored file to be closed and ```` is a unique ID for the runtime +of the Suricata instance. These values should not be depended on, and +are simply used to ensure uniqueness. +These ``fileinfo`` records are idential to the ``fileinfo`` records +logged to the ``eve`` output. + +See :ref:`suricata-yaml-file-store` for more information on +configuring the file-store output. + +.. note:: This section documents version 2 of the ``file-store``. + +File-Store (Version 1) +---------------------- + +File-store version 1 has been replaced by version 2 and is no longer +recommended. + +:: - file-store: enabled: yes # set to yes to enable @@ -54,9 +110,15 @@ drop dir must be configured. Each file that is stored will have a name "file.". The id will be reset and files will be overwritten unless the waldo option is used. A "file..meta" file is generated containing file metadata if write-meta is set to yes (default). If the include-pid option is set, the files will instead have a name "file..", and metafiles will be "file...meta". Files will additionally have the suffix ".tmp" while they are open, which is only removed when they are finalized. +File-Log (Deprecated) +--------------------- -:: +The file-log is a standalone logger that logs information about files +extracted. It is now deprecated and should not be used. It has been +replaced by ``eve.files`` in the :ref:`suricata-yaml-outputs-eve` +output. +:: - file-log: enabled: yes