fuzzybunny/moloch repository overview

⁠moloch.compose.yml:

version: '2' services: moloch-capture: image: "moloch:bluefin" restart: "always" network_mode: "host" restart: "always" volumes: - "/data/moloch/pcap:/data/moloch/raw" - "/data/moloch/logs:/data/moloch/logs" - "/operator/dip/sensor/moloch/etc/config.ini:/data/moloch/etc/config.ini" command: > /bin/bash -c "groupmod -g1000 daemon; bin/moloch-capture -c etc/config.ini"

moloch-viewer: image: "moloch:bluefin" restart: "always" # Port-forwarding is not working, so host-network mode is being used # The viewer will be reachable on port 8005 network_mode: "host" restart: "always" volumes: - "/data/moloch/pcap:/data/moloch/raw" - "/data/moloch/logs:/data/moloch/logs" - "/operator/dip/sensor/moloch/etc/config.ini:/data/moloch/etc/config.ini" command: > /bin/bash -c "groupmod -g1000 daemon; cd viewer; ../bin/node viewer.js -c ../etc/config.ini"

⁠moloch.service:

[Unit] Description = CVAH Moloch Service Module

Wants = network.target Requires = docker.service PartOf = cvah.sensor.service

[Service] Type = simple

ExecStartPre = /sbin/iptables -I INPUT 1 -p tcp --dport 8005 -j ACCEPT

ExecStart = /usr/bin/docker-compose -p cvah -f /operator/dip/compose_files/cvah.moloch.compose.yml up ExecStop = /usr/bin/docker-compose -p cvah -f /operator/dip/compose_files/cvah.moloch.compose.yml stop ExecStop = /sbin/iptables -D INPUT -p tcp --dport 8005 -j ACCEPT ExecStopPost = /usr/bin/docker-compose -p cvah -f /operator/dip/compose_files/cvah.moloch.compose.yml down

Restart=always RestartSec=3

[Install] WantedBy = multi-user.target

⁠config.ini:

⁠Latest settings documentation: https://github.com/aol/moloch/wiki/Settings⁠

⁠Moloch uses a tiered system for configuration variables. This allows Moloch

⁠to share one config file for many machines. The ordering of sections in this

⁠file doesn't matter.

⁠

⁠Order of config variables:

⁠1st) [optional] The section titled with the node name is used first.

⁠Moloch will always tag sessions with node:

⁠2nd) [optional] If a node has a nodeClass variable, the section titled with

⁠the nodeClass name is used next. Sessions will be tagged with

⁠node: which is useful if watching different

⁠network classes.

⁠3rd) The section titled "default" is used last.

[default]

⁠Comma seperated list of elasticsearch host:port combinations. If not using a

⁠elasticsearch VIP, a different elasticsearch node in the cluster can be specified

⁠for each Moloch node to help spread load on high volume clusters

elasticsearch=http://elastic_url:9200,http://elastic_url:9200

⁠How often to create a new elasticsearch index. hourly,daily,weekly,monthly

⁠Changing the value will cause previous sessions to be unreachable

rotateIndex=daily

⁠Cert file to use, comment out to use http instead

⁠certFile=/data/moloch/etc/moloch.cert

⁠File with trusted roots/certs. WARNING! this replaces default roots

⁠Useful with self signed certs and can be set per node.

⁠caTrustFile=/data/moloch/etc/roots.cert

⁠Private key file to use, comment out to use http instead

⁠keyFile=/data/moloch/etc/moloch.key

⁠S2S and Password Hash secret - Must be in default section. Since elasticsearch

⁠is wide open by default, we encrypt the stored password hashes with this

⁠so a malicous person can't insert a working new account. It is also used

⁠for secure S2S communication. Comment out for no user authentication.

⁠Changing the value will make all previously stored passwords no longer work.

passwordSecret = putapasswordhere

⁠HTTP Digest Realm - Must be in default section. Changing the value

⁠will make all previously stored passwords no longer work

httpRealm = Moloch

⁠The base path for Moloch web access. Must end with a / or bad things will happen

⁠Default: "/"

⁠webBasePath = /moloch/

⁠Semicolon ';' seperated list of interfaces to listen on for traffic

interface=ens1

⁠The bpf filter

#bpf=not port 9200

⁠The yara file name

#yara=

⁠Start wiseService configuration

⁠Host to connect to for wiseService

#wiseHost=127.0.0.1

⁠Number of seconds to cache results before asking wiseService again

#wiseCacheSecs=600

⁠Max number of items to store in the wise cache that is local to each moloch-capture node

#wiseMaxCache=100000

⁠Number of connections to wiseService, this is also the number of concurrent wise queries.

#wiseMaxConns=10

⁠Number of oustanding requests to the wiseService

#wiseMaxRequests=100

⁠End wiseService configuration

⁠Uncomment to log access requests to a different log file

#accessLogFile = /data/moloch/logs/access.log

⁠The directory to save raw pcap files to

pcapDir = /data/moloch/raw

⁠The max raw pcap file size in gigabytes, with a max value of 36G.

⁠The disk should have room for at least 10*maxFileSizeG

maxFileSizeG = 12

⁠The max time in minutes between rotating pcap files. Default is 0, which means

⁠only rotate based on current file size and the maxFileSizeG variable

#maxFileTimeM = 60

⁠TCP timeout value. Moloch writes a session record after this many seconds

⁠of inactivity.

tcpTimeout = 600

⁠Moloch writes a session record after this many seconds, no matter if

⁠active or inactive

tcpSaveTimeout = 720

⁠UDP timeout value. Moloch assumes the UDP session is ended after this

⁠many seconds of inactivity.

udpTimeout = 30

⁠ICMP timeout value. Moloch assumes the ICMP session is ended after this

⁠many seconds of inactivity.

icmpTimeout = 10

⁠An aproximiate maximum number of active sessions Moloch/libnids will try

⁠and monitor

maxStreams = 1000000

⁠Moloch writes a session record after this many packets

maxPackets = 10000

⁠Delete pcap files when free space is lower then this in gigabytes OR it can be

⁠expressed as a percentage (ex: 5%). This does NOT delete the session records in

⁠the database. It is recommended this value is between 5% and 10% of the disk.

⁠Database deletes are done by the db.pl expire script

freeSpaceG = 5%

⁠The port to listen on, by default 8005

viewPort = 8005

⁠The host/ip to listen on, by default 0.0.0.0 which is ALL

#viewHost = localhost

⁠By default the viewer process is https://hostname⁠: for each node.

#viewUrl = https://HOSTNAME:8005⁠

⁠Path of the maxmind geoip country file. Download free version from:

⁠http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz⁠

#geoipFile = /data/moloch/etc/GeoIP.dat

⁠Path of the maxmind geoip ASN file. Download free version from:

⁠http://www.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz⁠

#geoipASNFile = /data/moloch/etc/GeoIPASNum.dat

⁠Path of the rir assignments file

⁠https://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.csv⁠

rirFile = /data/moloch/etc/ipv4-address-space.csv

⁠User to drop privileges to. The pcapDir must be writable by this user or group below

dropUser=nobody

⁠Group to drop privileges to. The pcapDir must be writable by this group or user above

dropGroup=daemon

⁠Semicolon ';' seperated list of tags which once capture sets for a session causes the

⁠remaining pcap from being saved for the session. It is likely that the initial packets

⁠WILL be saved for the session since tags usually aren't set until after several packets

⁠Each tag can optionally be followed by a : which specifies how many total packets to save

#dontSaveTags=

⁠Header to use for determining the username to check in the database for instead of

⁠using http digest. Use this if apache or something else is doing the auth.

⁠Set viewHost to localhost or use iptables

⁠Might need something like this in the httpd.conf

⁠RewriteRule .* - [E=ENV_RU:%{REMOTE_USER}]

⁠RequestHeader set MOLOCH_USER %{ENV_RU}e

#userNameHeader=moloch_user

⁠Should we parse extra smtp traffic info

parseSMTP=true

⁠Should we parse extra smb traffic info

parseSMB=true

⁠Should we parse HTTP QS Values

parseQSValue=false

⁠Semicolon ';' seperated list of SMTP Headers that have ips, need to have the terminating colon ':'

smtpIpHeaders=X-Originating-IP:;X-Barracuda-Apparent-Source-IP:

⁠Semicolon ';' seperated list of directories to load parsers from

parsersDir=/data/moloch/parsers

⁠Semicolon ';' seperated list of directories to load plugins from

pluginsDir=/data/moloch/plugins

⁠Semicolon ';' seperated list of plugins to load and the order to load in

⁠plugins=tagger.so; netflow.so

⁠Plugins to load as root, usually just readers

#rootPlugins=reader-pfring; reader-daq.so

⁠Semicolon ';' seperated list of viewer plugins to load and the order to load in

⁠viewerPlugins=wise.js

⁠NetFlowPlugin

⁠Input device id, 0 by default

#netflowSNMPInput=1

⁠Outout device id, 0 by default

#netflowSNMPOutput=2

⁠Netflow version 1,5,7 supported, 7 by default

#netflowVersion=1

⁠Semicolon ';' seperated list of netflow destinations

#netflowDestinations=localhost:9993

⁠Specify the max number of indices we calculate spidata for.

⁠ES will blow up if we allow the spiData to search too many indices.

spiDataMaxIndices=3

⁠Uncomment the following to allow direct uploads. This is experimental

#uploadCommand=/data/moloch/bin/moloch-capture --copy -n {NODE} -r {TMPFILE} -c {CONFIG} {TAGS}

⁠Title Template

⁠cluster = ES cluster name

⁠userId = logged in User Id

⁠userName = logged in User Name

⁠page = internal page name

⁠expression = current search expression if set, otherwise blank

⁠-expression = " - " + current search expression if set, otherwise blank, prior spaces removed

⁠view = current view if set, otherwise blank

⁠-view = " - " + current view if set, otherwise blank, prior spaces removed

#titleTemplate=cluster - page -view -expression

⁠Number of threads processing packets

packetThreads=2

⁠ADVANCED - Semicolon ';' seperated list of files to load for config. Files are loaded

⁠in order and can replace values set in this file or previous files.

#includes=

⁠ADVANCED - How is pcap written to disk

⁠simple = use O_DIRECT if available, writes in pcapWriteSize chunks,

⁠a file per packet thread.

pcapWriteMethod=simple

⁠ADVANCED - Buffer size when writing pcap files. Should be a multiple of the raid 5 or xfs

⁠stripe size. Defaults to 256k

pcapWriteSize = 262143

⁠ADVANCED - value for pcap_set_buffer_size, may not be used depending on kernel etc

pcapBufferSize = 30000000

⁠ADVANCED - Number of bytes to bulk index at a time

dbBulkSize = 300000

⁠ADVANCED - Number of seconds before we force a flush to ES

dbFlushTimeout = 5

⁠ADVANCED - Compress requests to ES, reduces ES bandwidth by ~80% at the cost

⁠of increased CPU. MUST have "http.compression: true" in elasticsearch.yml file

compressES = false

⁠ADVANCED - Max number of connections to elastic search

maxESConns = 30

⁠ADVANCED - Max number of es requests outstanding in q

maxESRequests = 500

⁠ADVANCED - Number of packets to ask libnids/libpcap to read per poll/spin

⁠Increasing may hurt stats and ES performance

⁠Decreasing may cause more dropped packets

packetsPerPoll = 50000

⁠ADVANCED - Moloch will try to compensate for SYN packet drops by swapping

⁠the source and destination addresses when a SYN-acK packet was captured first.

⁠Probably useful to set it false, when running Moloch in wild due to SYN floods.

antiSynDrop = true

⁠DEBUG - Write to stdout info every X packets.

⁠Set to -1 to never log status

logEveryXPackets = 100000

⁠DEBUG - Write to stdout unknown protocols

logUnknownProtocols = false

⁠DEBUG - Write to stdout elastic search requests

logESRequests = true

⁠DEBUG - Write to stdout file creation information

logFileCreation = true

##############################################################################

⁠Classes of nodes

⁠Can override most default values, and create a tag call node:

#[class1] #freeSpaceG = 10%

##############################################################################

⁠Nodes

⁠Usually just use the hostname before the first dot as the node name

⁠Can override most default values

#[node1] #nodeClass = class1

⁠Might use a different elasticsearch node

#elasticsearch=elasticsearchhost1

⁠Uncomment if this node should process the cron queries, only ONE node should process cron queries

⁠cronQueries = true

#[node2] #nodeClass = class2

⁠Might use a different elasticsearch node

#elasticsearch=elasticsearchhost2

⁠Uses a different interface

#interface = eno1

##############################################################################

⁠override-ips is a special section that overrides the MaxMind databases for

⁠the fields set, but fields not set will still use MaxMind (example if you set

⁠tags but not country it will use MaxMind for the country)

⁠Spaces and capitalization is very important.

⁠IP Can be a single IP or a CIDR

⁠Up to 10 tags can be added

⁠

⁠ip=tag:TAGNAME1;tag:TAGNAME2;country:3LetterUpperCaseCountry;asn:ASN STRING

#[override-ips] #10.1.0.0/16=tag:ny-office;country:USA;asn:AS0000 This is an ASN

##############################################################################

⁠It is now possible to define in the config file extra http/email headers

⁠to index. They are accessed using the expression http. and

⁠email. with optional .cnt expressions

⁠

⁠Possible config atributes for all headers

⁠type: (string|integer|ip) = data type (default string)

⁠count: = index count of items (default false)

⁠unique: = only record unique items (default true)

⁠headers-http-request is used to configure request headers to index

#[headers-http-request] #referer=type:string;count:true;unique:true

⁠headers-http-response is used to configure http response headers to index

#[headers-http-response] #location=type:string;count:true

⁠headers-email is used to configure email headers to index

#[headers-email] #x-priority=type:integer

##############################################################################

⁠If you have multiple clusters and you want the ability to send sessions

⁠from one cluster to another either manually or with the cron feature fill out

⁠this section

#[moloch-clusters] #forensics=url:https://viewer1.host.domain:8005;passwordSecret:password4moloch;name:Forensics⁠ Cluster #shortname2=url:http://viewer2.host.domain:8123;passwordSecret:password4moloch;name:Testing⁠ Cluster