diff --git a/doc/userguide/command-line-options.rst b/doc/userguide/command-line-options.rst index 15f9c61651..ed522eff5d 100644 --- a/doc/userguide/command-line-options.rst +++ b/doc/userguide/command-line-options.rst @@ -79,15 +79,13 @@ For more information about runmodes see: :doc:`performance/runmodes` Capture Options ~~~~~~~~~~~~~~~ -.. option:: --af-packet -.. option:: --af-packet= +.. option:: --af-packet[=] Enable capture of packet using AF_PACKET on Linux. If no device is supplied, the list of devices from the af-packet section in the yaml is used. -.. option:: --netmap -.. option:: --netmap= +.. option:: --netmap[=] Enable capture of packet using NETMAP on FreeBSD or Linux. If no device is supplied, the list of devices from the netmap section diff --git a/doc/userguide/file-extraction/file-extraction.rst b/doc/userguide/file-extraction/file-extraction.rst index f4bc439ff4..cb7d10259c 100644 --- a/doc/userguide/file-extraction/file-extraction.rst +++ b/doc/userguide/file-extraction/file-extraction.rst @@ -99,4 +99,3 @@ Suricata can calculate MD5 checksums of files on the fly and log them. See :doc: md5 public-sha1-md5-data-sets - diff --git a/doc/userguide/file-extraction/filemd5-and-whiteblacklisting-with-md5.rst b/doc/userguide/file-extraction/filemd5-and-whiteblacklisting-with-md5.rst deleted file mode 100644 index 1e24c2a591..0000000000 --- a/doc/userguide/file-extraction/filemd5-and-whiteblacklisting-with-md5.rst +++ /dev/null @@ -1,256 +0,0 @@ -.. _filemd5-listing: - -Filemd5 and white or black listing with MD5 hashes -================================================== - - - -This guide will show you how to set up a white/black MD5 listing using latest Suricata on Ubuntu LTS 12.04 64 bit -and a file containing 87 million MD5s - white list. - - -In this set up we had/did the following : - -1. A Suricata installation with :doc:`md5` enabled. -2. We start Suricata with about 4.5K rules and the MD5 white list file containing 87 million MD5 entries. -3. We have a 9.5Gb of traffic. -4. We have the following set up: - - -:: - - - CPU: One Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz (16 cores counting Hyperthreading) - Memory: 32Gb - capture NIC: Intel 82599EB 10-Gigabit SFI/SFP+ - - - -You need to get the white list file containing MD5s from here - http://www.nsrl.nist.gov/Downloads.htm -This is an official database containing SHA-1 and MD5s for files that are " traceable ". -For example after you download and unzip (I used the Combo DVD link) you would get a file like so: - - -:: - - - - "SHA-1","MD5","CRC32","FileName","FileSize","ProductCode","OpSystemCode","SpecialCode" - "000000206738748EDD92C4E3D2E823896700F849","392126E756571EBF112CB1C1CDEDF926","EBD105A0","I05002T2.PFB",98865,3095,"WIN","" - "0000004DA6391F7F5D2F7FCCF36CEBDA60C6EA02","0E53C14A3E48D94FF596A2824307B492","AA6A7B16","00br2026.gif",2226,228,"WIN","" - "000000A9E47BD385A0A3685AA12C2DB6FD727A20","176308F27DD52890F013A3FD80F92E51","D749B562","femvo523.wav",42748,4887,"MacOSX","" - "00000142988AFA836117B1B572FAE4713F200567","9B3702B0E788C6D62996392FE3C9786A","05E566DF","J0180794.JPG",32768,16848,"358","" - "00000142988AFA836117B1B572FAE4713F200567","9B3702B0E788C6D62996392FE3C9786A","05E566DF","J0180794.JPG",32768,18266,"358","" - "00000142988AFA836117B1B572FAE4713F200567","9B3702B0E788C6D62996392FE3C9786A","05E566DF","J0180794.JPG",32768,20316,"358","" - "00000142988AFA836117B1B572FAE4713F200567","9B3702B0E788C6D62996392FE3C9786A","05E566DF","J0180794.JPG",32768,20401,"358","" - "00000142988AFA836117B1B572FAE4713F200567","9B3702B0E788C6D62996392FE3C9786A","05E566DF","J0180794.JPG",32768,2322,"WIN","" - "00000142988AFA836117B1B572FAE4713F200567","9B3702B0E788C6D62996392FE3C9786A","05E566DF","J0180794.JPG",32768,23822,"358","" - - -The file contains 87 million entries/lines, each line having -"SHA-1","MD5","CRC32","FileName","FileSize","ProductCode","OpSystemCode","SpecialCode" -of a traceable file. - -You can use the Linux "cut" utility to make this file into only a MD5 file - have only the MD5 check sums (second row) - which is what we need. -Here is what you can do: - -Download the ComboDVD - http://www.nsrl.nist.gov/RDS/rds_2.47/RDS_247.iso - -unzip/tar, then - -:: - - - cat NSRFile.txt | more - -would be able to see the beginning of the file and get an idea of what does it contain.(shown above) -**NOTE – this is a 10GB file, make sure you have the space!** - -the MD5 column ... should start with "MD5" - - -:: - - - sudo cut -d'"' -f4 NSRFile.txt >> MD5.txt - -and you have only MD5s ...but still with "MD5" on top - - -:: - - - sed -i '1d' MD5.txt - -wait for a while -check the file - - -:: - - - wc -l MD5.txt - -wait for about 2-3 min - -should be about 87Mil something md5s , one per line -**NOTE:** -You can also import the file into a MySQL database, if you would like: - - -:: - - - mysql> create user 'filejson'@'localhost' IDENTIFIED BY 'PASSWORD123'; - Query OK, 0 rows affected (0.00 sec) - - mysql>create database filejsondb; - mysql> grant all privileges on filejsondb.* to 'filejson'@'localhost' with grant option; - mysql> flush privileges; - mysql> use filejsondb; - -Then: - -:: - - - CREATE TABLE MD5(md5_whitelist VARCHAR(32)); - -The trick here for the import is that **the table name and the file name MUST be the same** - aka MySQL table "MD5" and the file is called MD5.txt! - -:: - - - sudo mysqlimport -u root -p --local filejsondb MD5.txt - -Where "filejsondb" is the name of the database. - - - -So , here is how we did it .... - - - -:: - - - pevman@suricata:~$ ls -lh /etc/suricata/et-config/MD5_NSRFile.txt - -rw-r--r-- 1 root root 2.7G Aug 29 00:35 /etc/suricata/et-config/MD5_NSRFile.txt - - pevman@suricata:~$ wc -l /etc/suricata/et-config/MD5_NSRFile.txt - 87345542 /etc/suricata/et-config/MD5_NSRFile.txt - pevman@suricata:~$ - - - - - - -:: - - - pevman@suricata:~$ sudo tcpstat -i eth3 - Time:1346241952 n=6664547 avg=898.01 stddev=757.68 bps=9575769952.00 - Time:1346241957 n=6670750 avg=897.22 stddev=754.07 bps=9576254160.00 - Time:1346241962 n=6626520 avg=903.62 stddev=747.26 bps=9580577822.40 - Time:1346241967 n=6685956 avg=895.15 stddev=749.34 bps=9575883715.20 - Time:1346241972 n=6712481 avg=891.53 stddev=747.34 bps=9575027134.40 - Time:1346241977 n=6696189 avg=893.82 stddev=746.62 bps=9576294273.60 - Time:1346241982 n=6681826 avg=895.75 stddev=749.67 bps=9576376033.60 - - - - - -:: - - - pevman@suricata:~$ suricata --build-info - [2746] 29/8/2012 -- 15:07:25 - (suricata.c:539) (SCPrintBuildInfo) -- This is Suricata version 1.4dev (rev 75af345) - [2746] 29/8/2012 -- 15:07:25 - (suricata.c:612) (SCPrintBuildInfo) -- Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK HAVE_HTP_TX_GET_RESPONSE_HEADERS_RAW HAVE_NSS - [2746] 29/8/2012 -- 15:07:25 - (suricata.c:626) (SCPrintBuildInfo) -- 64-bits, Little-endian architecture - [2746] 29/8/2012 -- 15:07:25 - (suricata.c:628) (SCPrintBuildInfo) -- GCC version 4.6.3, C version 199901 - [2746] 29/8/2012 -- 15:07:25 - (suricata.c:634) (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_1 - [2746] 29/8/2012 -- 15:07:25 - (suricata.c:637) (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_2 - [2746] 29/8/2012 -- 15:07:25 - (suricata.c:640) (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4 - [2746] 29/8/2012 -- 15:07:25 - (suricata.c:643) (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8 - [2746] 29/8/2012 -- 15:07:25 - (suricata.c:646) (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_16 - [2746] 29/8/2012 -- 15:07:25 - (suricata.c:650) (SCPrintBuildInfo) -- compiled with -fstack-protector - [2746] 29/8/2012 -- 15:07:25 - (suricata.c:656) (SCPrintBuildInfo) -- compiled with _FORTIFY_SOURCE=2 - -During Suricata start up : -... - -:: - - - [3071] 29/8/2012 -- 15:23:45 - (detect.c:670) (SigLoadSignatures) -- Loading rule file: /var/data/peter/md5test.rules - [3071] 29/8/2012 -- 15:23:45 - (detect-filemd5.c:105) (Md5ReadString) -- [ERRCODE: SC_ERR_INVALID_MD5(214)] - md5 string not 32 bytes - [3071] 29/8/2012 -- 15:24:25 - (detect-filemd5.c:277) (DetectFileMd5Parse) -- MD5 hash size 1399625840 bytes, negated match - [3071] 29/8/2012 -- 15:24:25 - (detect.c:701) (SigLoadSignatures) -- 5 rule files processed. 4641 rules succesfully loaded, 0 rules failed - - -You will get the - -:: - - [ERRCODE: SC_ERR_INVALID_MD5(214)] - md5 string not 32 bytes - -if a line of the file is not containing a proper MD5 hash. - -For example above we get the err message (which is more of a warning) because in the file , the first line was containing: -"SHA-1","MD5","CRC32","FileName","FileSize","ProductCode","OpSystemCode","SpecialCode" - -However , nonetheless Suricata will continue loading the rest of the MD5 hashes from the file. REMEMBER - you would need put only the MD5 hashes in the file. - - -Then we just created the following test rules: - -:: - - - root@suricata:/etc/suricata/peter-yaml# cat /var/data/peter/md5test.rules - alert http any any -> any any (msg:"FILE MD5 Check PDF against a white list"; filemagic:"pdf"; filemd5:!MD5_NSRFile.txt; sid:9966699; rev:1;) - alert http any any -> any any (msg:"FILE MD5 Check EXE against a white list"; filemagic:"exe"; filemd5:!MD5_NSRFile.txt; sid:9977799; rev:2;) - -Make sure the **MD5_NSRFile.txt** file (containing the MD5 hashes) is in your "rules directory" (where you load the rules from). - - -Basically the two rules above are telling Suricata to do the following: -1. If you see a PDF document that has a MD5 hash NOT in the MD5_NSRFile.txt - generate an alert -2. If you see an EXE file that has a MD5 hash NOT in the MD5_NSRFile.txt - generate an alert - -all that on the fly, while inspecting traffic. - - -Then all that is left is to start Suricata: - -:: - - - sudo /usr/local/bin/suricata -c /etc/suricata/peter-yaml/suricata-af-packet-mmap.yaml -s /var/data/peter/md5test.rules --af-packet=eth3 - -and we get the alerts: - - -:: - - - 08/29/2012-15:38:43.165038 [**] [1:9977799:2] FILE MD5 Check EXE against a white list [**] [Classification: (null)] [Priority: 3] {TCP} y.y.y.y:80 -> x.x.x.x:23836 - 08/29/2012-15:39:32.551950 [**] [1:9977799:2] FILE MD5 Check EXE against a white list [**] [Classification: (null)] [Priority: 3] {TCP} y.y.y.y:2091 -> x.x.x.x:80 - - - -That's it. - -You can reverse and use the above rules with a "blacklisting" of MD5 hashes, for example: - - -:: - - - alert http any any -> any any (msg:"FILE MD5 Check PDF against a black list"; filemagic:"pdf"; filemd5:BlackMD5s.txt; sid:9966699; rev:1;) - -You can also use the filestore keyword to store the file on disk and do further analysis on the particular file - or blend it in with other :doc:`../rules/file-keywords`. - -Peter Manev diff --git a/doc/userguide/output/files-json/elk.rst b/doc/userguide/output/files-json/elk.rst deleted file mode 100644 index 234cf203ca..0000000000 --- a/doc/userguide/output/files-json/elk.rst +++ /dev/null @@ -1,239 +0,0 @@ -Logstash Kibana and Suricata JSON output -======================================== - -With the release of Suricata 2.0rc1 , Suricata introduces all JSON output capability. - -What is JSON - http://en.wikipedia.org/wiki/JSON - -One way to handle easily Suricata's JSON log outputs is through Kibana - http://kibana.org/ : - -:: - - Kibana is a highly scalable interface for Logstash (http://logstash.net/) and ElasticSearch (http://www.elasticsearch.org/) that allows you to efficiently search, graph, analyze and otherwise make sense of a mountain of logs. - -The installation is very simple/basic start up with minor specifics for ubuntu. You can be up and running, looking through the logs in under 5 min. - -The downloads can be found here - http://www.elasticsearch.org/overview/elkdownloads/ - -This is what yo need to do. - -Suricata ---------- - -Make sure your Suricata is compiled/installed with libjansson support enabled: - -:: - - - $ suricata --build-info - This is Suricata version 2.0 RELEASE - Features: NFQ PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK HAVE_NSS HAVE_LIBJANSSON - ... - libnss support: yes - libnspr support: yes - libjansson support: --> yes <-- - Prelude support: no - PCRE jit: no - libluajit: no - libgeoip: yes - Non-bundled htp: yes - Old barnyard2 support: no - CUDA enabled: no - ... - -If it isn't check out the `Suricata Installation `_ page to install or compile Suricata for your distribution. -**NOTE:** you will need these packages installed -> **libjansson4** and *libjansson-dev* before compilation. - -Configure suricata ------------------- - -In your suricata.yaml - -:: - - - # "United" event log in JSON format - - eve-log: - enabled: yes - type: file #file|syslog|unix_dgram|unix_stream - filename: eve.json - # the following are valid when type: syslog above - #identity: "suricata" - #facility: local5 - #level: Info ## possible levels: Emergency, Alert, Critical, - ## Error, Warning, Notice, Info, Debug - types: - - alert - - http: - extended: yes # enable this for extended logging information - - dns - - tls: - extended: yes # enable this for extended logging information - - files: - force-magic: yes # force logging magic on all logged files - force-md5: yes # force logging of md5 checksums - #- drop - - ssh - - smtp - - flow - -Install ELK (elasticsearch, logstash, kibana) ---------------------------------------------- - -First install the dependencies - -**NOTE:** ELK recommends running with Oracle Java - how to: - -* http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-service.html#_installing_the_oracle_jdk - -Otherwise you can install the openjdk: - -:: - - - apt-get install apache2 openjdk-7-jdk openjdk-7-jre-headless - -Then download and install the software. - -Make sure you download the latest versions - - -* http://www.elasticsearch.org/overview/elkdownloads/ - -The installation process is simple (for example): - -:: - - - wget https://download.elasticsearch.org/kibana/kibana/kibana-3.0.0.tar.gz - wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.6.1.deb - wget https://download.elastic.co/logstash/logstash/packages/debian/logstash_1.5.3-1_all.deb - - tar -C /var/www/ -xzf kibana-3.0.0.tar.gz - dpkg -i elasticsearch-1.6.1.deb - dpkg -i logstash_1.5.3-1_all.deb - -Logstash configuration ----------------------- - -Create and save a **logstash.conf** file with the following content in the /etc/logstash/conf.d/ directory : - -:: - - - touch /etc/logstash/conf.d/logstash.conf - -Insert the following(make sure the directory path is correct): - -:: - - - input { - file { - path => ["/var/log/suricata/eve.json"] - sincedb_path => ["/var/lib/logstash/"] - codec => json - type => "SuricataIDPS" - } - - } - - filter { - if [type] == "SuricataIDPS" { - date { - match => [ "timestamp", "ISO8601" ] - } - ruby { - code => "if event['event_type'] == 'fileinfo'; event['fileinfo']['type']=event['fileinfo']['magic'].to_s.split(',')[0]; end;" - } - } - - if [src_ip] { - geoip { - source => "src_ip" - target => "geoip" - #database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat" - add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ] - add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ] - } - mutate { - convert => [ "[geoip][coordinates]", "float" ] - } - if ![geoip.ip] { - if [dest_ip] { - geoip { - source => "dest_ip" - target => "geoip" - #database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat" - add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ] - add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ] - } - mutate { - convert => [ "[geoip][coordinates]", "float" ] - } - } - } - } - } - - output { - elasticsearch { - host => localhost - #protocol => http - } - } - - -Configure the start-up services -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - -:: - - - update-rc.d elasticsearch defaults 95 10 - update-rc.d logstash defaults - - service apache2 restart - service elasticsearch start - service logstash start - -Enjoy ------ - -That's all. Now make sure Suricata is running and you have logs written in your JSON log files and you point your browser towards - -:: - - - http://localhost/kibana-3.0.0 - -**NOTE:** -Some ready to use templates can be found here: - -* https://github.com/pevma/Suricata-Logstash-Templates - -From here on if you would like to customize and familiarize yourself more with the interface you should read the documentation about Kibana and Logstash. -Please have in mind that this is a very quick(under 5 min) tutorial. You should customize and review the proper way for you of using it as a service and/or consider using **httpS web interface and reversy proxy with some authentication**. - -Some possible customization of the output of Logstash and Kibana - - - -.. image:: elk/Logstash1.png - - -.. image:: elk/Logstash2.png - - -.. image:: elk/Logstash3.png - - -.. image:: elk/Logstash4.png - - -.. image:: elk/Logstash5.png - - -.. image:: elk/Logstash6.png - -Peter Manev diff --git a/doc/userguide/output/files-json/files-json.rst b/doc/userguide/output/files-json/files-json.rst deleted file mode 100644 index 472ae3bbd1..0000000000 --- a/doc/userguide/output/files-json/files-json.rst +++ /dev/null @@ -1,62 +0,0 @@ -What to do with files-json.log output -===================================== - -.. toctree:: - - script-follow-json - mysql - postgresql - useful-queries-for-mysql-and-postgresql - mongodb - elk - -Suricata has the ability to produce the files-json.log output. -Basically this is a JSON style format output logfile with entries like this: - -:: - - { - "timestamp": "10\/01\/2012-16:52:59.217616", - "ipver": 4, - "srcip": "80.239.217.171", - "dstip": "192.168.42.197", - "protocol": 6, - "sp": 80, - "dp": 32982, - "http_uri": "\/frameworks\/barlesque\/2.11.0\/desktop\/3.5\/style\/main.css", "http_host": "static.bbci.co.uk", "http_referer": "http:\/\/www.bbc.com\/", "filename": "\/frameworks\/barl - esque\/2.11.0\/desktop\/3.5\/style\/main.css", - "magic": "ASCII text, with very long lines, with no line terminators", - "state": "CLOSED", - "md5": "be7db5e9a4416a4123d556f389b7f4b8", - "stored": false, - "size": 29261 - } - -for every single file that crossed your http pipe. -This in general is very helpful and informative. -In this section we are going to try to explore/suggest approaches for putting it to actual use, since it could aggregate millions of entries in just a week. -There are a god few options in general since the JSON style format is pretty common. -http://www.json.org/ - - -This guide offers a couple of approaches - -use of custom created script with MySQL or PostgreSQL import (bulk or continuous) -or importing it directly to MongoDB(native import of JSON files). - -Please read the all the pages before you jump into executing scripts and/or installing/configuring things. -Te guide is written using Ubuntu LTS server 12.04 - -Thee are 3 options in general that we suggest, that we are going to explain here: - -1. import JSON into MySQL -2. import JSON into PostgreSQL -3. import JSON into MongoDB - -The suggested approach is -configure Suricata.yaml -configure your Database -run the script (not applicable to MongoDB) -and then execute queries against the DB to get the big picture. - - -Peter Manev diff --git a/doc/userguide/output/files-json/mongodb.rst b/doc/userguide/output/files-json/mongodb.rst deleted file mode 100644 index 0143ab6dff..0000000000 --- a/doc/userguide/output/files-json/mongodb.rst +++ /dev/null @@ -1,97 +0,0 @@ -MongoDB -======= - -If you do not have it installed, follow the istructions here: -http://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/ - -Basically you do: - - -:: - - - sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10 - deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen - sudo apt-get update && sudo apt-get install mongodb-10gen - - -The bigest benefit of MongoDB is that it can natively import json.log files: -if you have MongoDB installed - all you have to do is: - - -:: - - - mongoimport --db filejsondb --collection filejson --file files-json.log - -here: - -* --db filejsondb is the database, -* --collections filejson is the equivalent of SQL "table" -* --file files-json.log - is the json log created and logged into from Suricata. - -last but not least - it would automatically create the db and tables for you. - -this would import a 5 Gb (15 million entries) json log file in about 5-10 minutes - default configuration, without tuning MongoDB for high performance. (your set up and HW will definitely have effect on the import time ) - - - -MongoDB Example queries (once you have imported the files-json.log - described above - just go ahead with these queries): - - -:: - - - db.files.group( { cond : {"magic":/.*PDF*./ }, key: {"srcip":true,"http_host":true,"magic":true} ,initial: {count: 0},reduce: function(value, total) {total+=value.count;} } ); - - -:: - - - db.filejson.find({magic:/.*PDF.*/},{srcip:1,http_host:1,magic:1}).sort({srcip:1,http_host:1,magic:1}).limit(20) - - -Get a sorted table biggest to smallest number hosts of file downloads: - - -:: - - - > map = function () { emit({srcip:this.srcip,http_host:this.http_host,magic:this.magic}, {count:1}); } - function () { - emit({srcip:this.srcip, http_host:this.http_host, magic:this.magic}, {count:1}); - } - > reduce = function(k, values) {var result = {count: 0}; values.forEach(function(value) { result.count += value.count; }); return result; } - function (k, values) { - var result = {count:0}; - values.forEach(function (value) {result.count += value.count;}); - return result; - } - > db.filejson.mapReduce(map,reduce,{out: "myoutput" }); - { - "result" : "myoutput", - "timeMillis" : 578806, - "counts" : { - "input" : 3110871, - "emit" : 3110871, - "reduce" : 673186, - "output" : 219840 - }, - "ok" : 1, - } - > db.myoutput.find().sort({'value.count':-1}).limit(10) - { "_id" : { "srcip" : "184.107.x.x", "http_host" : "arexx.x", "magic" : "very short file (no magic)" }, "value" : { "count" : 42560 } } - { "_id" : { "srcip" : "66.135.210.182", "http_host" : "www.ebay.co.uk", "magic" : "XML document text" }, "value" : { "count" : 30896 } } - { "_id" : { "srcip" : "66.135.210.62", "http_host" : "www.ebay.co.uk", "magic" : "XML document text" }, "value" : { "count" : 27812 } } - { "_id" : { "srcip" : "213.91.x.x", "http_host" : "www.focxxxx.x", "magic" : "HTML document, ISO-8859 text" }, "value" : { "count" : 26301 } } - { "_id" : { "srcip" : "195.168.x.x", "http_host" : "search.etaxxx.x", "magic" : "JPEG image data, JFIF standard 1.01, comment: \"CREATOR: gd-jpeg v1.0 (using IJG JPEG v80), quality = 100\"" }, "value" : { "count" : 16131 } } - { "_id" : { "srcip" : "184.107.x.x", "http_host" : "p2p.arxx.x:2710", "magic" : "ASCII text, with no line terminators" }, "value" : { "count" : 15829 } } - { "_id" : { "srcip" : "213.91.x.x", "http_host" : "www.focxx.x", "magic" : "HTML document, ISO-8859 text" }, "value" : { "count" : 14472 } } - { "_id" : { "srcip" : "64.111.199.222", "http_host" : "syndication.exoclick.com", "magic" : "HTML document, ASCII text, with very long lines, with no line terminators" }, "value" : { "count" : 14009 } } - { "_id" : { "srcip" : "69.171.242.70", "http_host" : "www.facebook.com", "magic" : "ASCII text, with no line terminators" }, "value" : { "count" : 13098 } } - { "_id" : { "srcip" : "69.171.242.74", "http_host" : "www.facebook.com", "magic" : "ASCII text, with no line terminators" }, "value" : { "count" : 12801 } } - > - - - -Peter Manev diff --git a/doc/userguide/output/files-json/mysql.rst b/doc/userguide/output/files-json/mysql.rst deleted file mode 100644 index 0e0c328670..0000000000 --- a/doc/userguide/output/files-json/mysql.rst +++ /dev/null @@ -1,36 +0,0 @@ -MySQL -===== - -If you do not have MySQL installed - go ahead and do so: - -:: - - - sudo apt-get update && sudo apt-get upgrade - sudo apt-get install mysql-server mysql-client - - -For MySQL make sure you create a db and a table: - - -:: - - - mysql>create database filejsondb; - mysql> create user 'filejson'@'localhost' IDENTIFIED BY 'PASSWORD123'; - Query OK, 0 rows affected (0.00 sec) - mysql> grant all privileges on filejsondb.* to 'filejson'@'localhost' with grant option; - mysql> flush privileges; - mysql> use filejsondb; - - mysql> CREATE TABLE filejson( time_received VARCHAR(64), ipver VARCHAR(4), srcip VARCHAR(40), dstip VARCHAR(40), protocol SMALLINT UNSIGNED, sp SMALLINT UNSIGNED, dp SMALLINT UNSIGNED, http_uri TEXT, http_host TEXT, http_referer TEXT, filename TEXT, magic TEXT, state VARCHAR(32), md5 VARCHAR(32), stored VARCHAR(32), size BIGINT UNSIGNED); - - mysql> show columns from filejson; - - - -OPTIONALLY - if you would like you can add in the MD5 whitelist table and import the data as described here :ref:`FileMD5 and white/black listing with md5 ` - -now you can go ahead and execute the script - :ref:`Script FollowJSON ` - -Peter Manev diff --git a/doc/userguide/output/files-json/postgresql.rst b/doc/userguide/output/files-json/postgresql.rst deleted file mode 100644 index 256fea4d4d..0000000000 --- a/doc/userguide/output/files-json/postgresql.rst +++ /dev/null @@ -1,79 +0,0 @@ -PostgreSQL -========== - -If you do not have PostgreSQL installed: - - -:: - - - sudo apt-get update && sudo apt-get upgrade - sudo apt-get install postgresql - - - -:: - - - sudo vim /etc/postgresql/9.1/main/pg_hba.conf - -change the line: - - -:: - - - local all all trust - -to - - -:: - - - local all all md5 - - -login and change passwords - -:: - - - sudo -u postgres psql postgres - \password postgres - - -Then - - - - -:: - - - create database filejsondb; - \c filejsondb; - create user filejson with password 'PASSWORD123'; - CREATE TABLE filejson( time_received VARCHAR(64), ipver VARCHAR(4), srcip VARCHAR(40), dstip VARCHAR(40), protocol INTEGER, sp INTEGER, dp INTEGER, http_uri TEXT, http_host TEXT, http_referer TEXT, filename TEXT, magic TEXT, state VARCHAR(32), md5 VARCHAR(32), stored VARCHAR(32), size BIGINT); - grant all privileges on database filejsondb to filejson; - -Log out and log in again (with the "filejson" user) to test if everything is ok: - - -:: - - - psql -d filejson -U filejson - - - - -Optionally you could create and import the MD5 white list data if you wish - generally the same guidance as described in :ref:`FileMD5 and white/black listing with md5 ` - -Some more general info and basic commands/queries: -http://jazstudios.blogspot.se/2010/06/postgresql-login-commands.html -http://www.thegeekstuff.com/2009/05/15-advanced-postgresql-commands-with-examples/ - - -now you can go ahead and execute the script - :ref:`Script FollowJSON ` - -Peter Manev diff --git a/doc/userguide/output/files-json/script-follow-json.rst b/doc/userguide/output/files-json/script-follow-json.rst deleted file mode 100644 index a914fa4082..0000000000 --- a/doc/userguide/output/files-json/script-follow-json.rst +++ /dev/null @@ -1,100 +0,0 @@ -.. _script-follow-json: - -Script FollowJSON -================= - -BEFORE you run the script - make sure you have set up suricata.yaml and your database correctly !! - -Suricata.yaml: - -1. make sure json-log is enabled -2. and append is set to yes -3. optionally - you have compilled in Suricata with MD5's enabled - -MD5's are enabled and forced in the suricata yaml config ( :ref:`MD5 ` ) -bottom of the page "Log all MD5s without any rules" . - - -:: - - - - file-log: - enabled: yes - filename: files-json.log - append: yes - #filetype: regular # 'regular', 'unix_stream' or 'unix_dgram' - force-magic: yes # force logging magic on all logged files - force-md5: yes # force logging of md5 checksums - - -**Append is set to yes** - this is very important if you "follow" , json.log - if you use the tool to constantly parse and insert logs from files-json.log as they are being written onto the log file. - - -There is a python script (in BETA now and) available here: - -* https://redmine.openinfosecfoundation.org/attachments/download/843/FollowJSON.tar.gz - -that you can use for helping out in importing files-json.log entries into a MSQL or PostgreSQL database. - -The script would allow you to do the following: - - -* it contains 2 files -* one python executable -* one yaml config file -* one LICENSE (GPLv2) - -This is what the script does: - -1. Multi-threaded - spawns multiple processes if itself -2. uses yaml as configuration -3. Can: - - 3.1. Read files-json.log file - - 3.1.1. - Continuously - as logs are being written in the log file - 3.1.2. - mass import a stand alone files-json.log into a database - - 3.2. Into (your choice) - - 3.2.1. MySQL DB (locally/remotely,ip) - 3.2.2. PostgreSQL DB (locally/remotely,ip) - -4. Customizable number of processes (default is number of cores - if you have more then 16 - suggested value is NumCores/2) -5. Customizable "chunk" lines to read at once by every process - suggested (default) value is 10 (16 cores = 16 processes * 10 = 160 entries per second) - -**Please look into the configurational yaml file** for more information. - -The script is in BETA state - it has been tested , it works - but still, you should test it and adjust the configuration accordingly and run it on your test environment first before you put it in production. - -After you have made: - -#. your choices of database type (MySQL or PostgreSQL and installed/configured tables for it), -#. created the appropriate database structure and tables (explained in the next tutorial(s) ), -#. adjusted the yaml configuration accordingly, -#. started Suricata, - -you would need: - -:: - - - sudo apt-get install python-yaml python-mysqldb python-psycopg2 - -Then you just run the script, after you have started Suricata: - - -:: - - - sudo python Follow_JSON_Multi.py - -if you would like to execute the script in the background: - - -:: - - - sudo python Follow_JSON_Multi.py & - -Peter Manev diff --git a/doc/userguide/output/files-json/useful-queries-for-mysql-and-postgresql.rst b/doc/userguide/output/files-json/useful-queries-for-mysql-and-postgresql.rst deleted file mode 100644 index d3d81ec13a..0000000000 --- a/doc/userguide/output/files-json/useful-queries-for-mysql-and-postgresql.rst +++ /dev/null @@ -1,138 +0,0 @@ -Useful queries - for MySQL and PostgreSQL -========================================= - - -General Purpose and Useful Queries (MySQL - 99% the same for PostgreSQL) for the files-json.log databases and tables: - - -:: - - - mysql>select srcip,http_host,count(*) as total from filejson where magic like "%PDF document%" group by srcip,http_host order by total DESC limit 10; - -above top 10 source ip from which PDF's where downloaded -change srcip with dstip to get top 10 IPs downloading PDFs - - -:: - - - mysql>select srcip,http_host,count(*) as total from filejson where magic like "%executable%" group by srcip,http_host order by total DESC limit 10; - -above top 10 source ip from which executables where downloaded from, -change srcip with dstip to get top 10 IPs downloading executables - - - -:: - - - mysql> SELECT srcip,http_host,count(*) AS Total , (COUNT(*) / (SELECT COUNT(*) FROM filejson where magic like "%executable%")) * 100 AS 'Percentage to all items' FROM filejson WHERE magic like "%executable%" GROUP BY srcip,http_host order by total DESC limit 10; - -:: - - - +----------------+----------------------+-------+-------------------------+ - | srcip | http_host | Total | Percentage to all items | - +----------------+----------------------+-------+-------------------------+ - | 149.5.130.7 | ws.livepcsupport.com | 225 | 9.1167 | - .............................. - ............................. - -This would give you a sorted table depicting source ip and host name, number of executable downloads from that host/source ip and what percentage is that of the total executable downloads. -Note: the term executable means - dll, exe, com, msi, java ... and so on , NOT just .exe files - - - -:: - - - mysql>select count(magic) as totalPDF from filejson where magic like "%PDF%" - -This will give you the total number of PDFs out of all files - - -:: - - - mysql>SELECT ( select count(magic) from filejson where magic like "%PDF%" ) as "PDF Total" , (select count(magic) from filejson where magic like "%executable%") as "Executables Total" , (select count(magic) from filejson where filename like "%.xls") as "Excel Total"; - -This will give you: - -:: - - - +-----------+-------------------+-------------+ - | PDF Total | Executables Total | Excel Total | - +-----------+-------------------+-------------+ - | 391 | 2468 | 7 | - +-----------+-------------------+-------------+ - - -:: - - - mysql> SELECT ( select count(magic) from filejson where magic like "%PDF%" ) as "PDF Total" , (select count(magic) from filejson where magic like "%executable%") as "Executables Total" , (select count(magic) from filejson where filename like "%.xls") as "Excel Total", (select count(magic) from filejson) as "TOTAL NUMER OF FILES"; - -:: - - - +-----------+-------------------+-------------+----------------------+ - | PDF Total | Executables Total | Excel Total | TOTAL NUMER OF FILES | - +-----------+-------------------+-------------+----------------------+ - | 391 | 2468 | 7 | 3743925 | - +-----------+-------------------+-------------+----------------------+ - -the above query - a breakdown for PDF, executables and files hat have extension .xls - - - -:: - - - mysql>select srcip,filename,http_host,count(*) as total from filejson where filename like "%.xls" group by srcip,filename,http_host order by total DESC limit 10; - -the above will select top 10 source ip and document NAMES where excel files (files with extension .xls) were downloaded form - - -:: - - - mysql>select srcip,http_host,count(*) as total from filejson where filename like "%.exe" group by srcip,http_host order by total DESC limit 10; - -the above will select the top 10 source ips from where ".exe" files where downloaded from - - -:: - - - mysql>select srcip,http_host,count(*) as total from filejson where filename like "%.doc" group by srcip,http_host order by total DESC limit 10; - -the above for ".doc" files - - -:: - - - mysql>select magic,http_host,count(*) as count from filejson group by magic,http_host order by count DESC limit 20; - -select top 20 hosts grouped and ordered by count - - -:: - - - mysql>select dstip,size,count(*) as total from filejson group by dstip,size order by total DESC limit 10; - -the above query will show you he top 10 downloading ips by size of downloads - - -:: - - - mysql>select dstip,http_host,count(*) as total from filejson where filename like "%.exe" group by dstip order by total DESC limit 5; - -the above query will show you the top 5 downloading ips (and the hosts they downloaded from) that downloaded files with .exe extensions. - - -Peter Manev diff --git a/doc/userguide/performance/tuning-considerations.rst b/doc/userguide/performance/tuning-considerations.rst index 28ebd54d85..6549c1c5b6 100644 --- a/doc/userguide/performance/tuning-considerations.rst +++ b/doc/userguide/performance/tuning-considerations.rst @@ -15,7 +15,7 @@ Suggested setting: 1000 or higher. Max is ~65000. mpm-algo: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Controls the pattern matcher algorithm. AC is the default. On supported platforms, :doc:`performance/Hyperscan` is the best option. +Controls the pattern matcher algorithm. AC is the default. On supported platforms, :doc:`hyperscan` is the best option. detect.profile: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/doc/userguide/rules/enip-keyword.rst b/doc/userguide/rules/enip-keyword.rst index fdcc6eec4b..d833e0923f 100644 --- a/doc/userguide/rules/enip-keyword.rst +++ b/doc/userguide/rules/enip-keyword.rst @@ -37,4 +37,4 @@ Examples:: (cf. http://read.pudn.com/downloads166/ebook/763211/EIP-CIP-V1-1.0.pdf) Information on the protocol can be found here: -http://literature.rockwellautomation.com/idc/groups/literature/documents/wp/enet-wp001_-en-p.pdf +``_