This post was written in 2018. Tools and versions mentioned may be outdated, though the underlying ideas still hold.

I first started using Elasticsearch back around November 2016 — storing crawler data in it, and later playing with it again for log analysis. Then I lived through that whole wave of ES instances getting exposed to the public internet. Versions were jumping like crazy. And now I look up and it’s got machine learning built in. Reminded me of how I tried to build a search engine with Lucene in my junior year of college and completely failed.

Logstash

Here’s my log file:

2009-12-31 00:00:04 W3SVC1 22x.xxxx.7.21 GET /Newmsg/rccp_news.asp classid=233&siteid=2910 80 - 124.115.1.59 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1) 200 0 0
2009-12-31 00:00:27 W3SVC1 22x.xxxx.7.21 GET /luntan/topic_stats.asp sort_order=T_VIEW_COUNT&whichpage=3 80 - 124.115.1.59 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1) 200 0 0
2009-12-31 00:01:03 W3SVC1 22x.xxxx.7.21 GET /rdxw.asp Page=60 80 - 67.195.114.58 Mozilla/5.0+(compatible;+Yahoo!+Slurp/3.0;+http://help.yahoo.com/help/us/ysearch/slurp) 200 0 0
2009-12-31 00:01:13 W3SVC1 22x.xxxx.7.21 GET /rsdt.asp Page=24 80 - 124.115.4.205 Sosospider+(+http://help.soso.com/webspider.htm) 200 0 0

Write a grok pattern to match all this:

%{TIMESTAMP_ISO8601:YYYY-MM-dd HH:mm:ss} %{WORD:serviceName} %{IP:serverIP} %{WORD:method} %{URIPATH:uriStem} %{NOTSPACE:uriQuery} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clientIP} %{NOTSPACE:userAgent} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:win32response}"

Here’s my full isslog.conf:

input {  
  file {
    type => "iis-w3c"
    path => "/home/I/biglog/log/2010/*.log"
    start_position => "beginning"
  }

}


filter {
 #ignore log comments
 if [fusion_builder_container hundred_percent="yes" overflow="visible"][fusion_builder_row][fusion_builder_column type="1_1" background_position="left top" background_color="" border_size="" border_color="" border_style="solid" spacing="yes" background_image="" background_repeat="no-repeat" padding="" margin_top="0px" margin_bottom="0px" class="" id="" animation_type="" animation_speed="0.3" animation_direction="left" hide_on_mobile="no" center_content="no" min_height="none"][message] =~ "^#" {
 drop {}
 }
 
 
 grok {
 # check that fields match your IIS log settings
 match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:serviceName} %{IP:serverIP} %{WORD:method} %{URIPATH:uriStem} %{NOTSPACE:uriQuery} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clientIP} %{NOTSPACE:userAgent} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:win32response}"]
 }
 #Set the Event Timesteamp from the log
 date {
 match => [ "log_timestamp", "YYYY-MM-dd HH:mm:ss" ]
 timezone => "Etc/UTC"
 }
 useragent {
 source=> "useragent"
 prefix=> "browser"
 }
 mutate {
 remove_field => [ "log_timestamp"]
 }
}
 
output {
stdout {}
elasticsearch {
hosts => ["127.0.0.1:9200"]
}
stdout { codec => rubydebug }
}

After writing the config, just run logstash -f isslog.conf. But after importing, I couldn’t see anything in Kibana. Checked ES directly and the data was definitely in there. Couldn’t figure it out — then it hit me: the timestamps were wrong. The logs I was analyzing spanned 2009 to 2014, so once I set the time range in Kibana to that period, everything showed up.

grokdebug online

X-pack

Versions must match exactly — not just major versions, minor versions too. You can install directly with xxx-plugin install x-pack (where xxx is any one of the ELK components), but the download is painfully slow. Better to download it locally through a proxy first, then install from local with file://. I was using 6.0, so:
/kibana-6.0.0-linux-x86_64/bin/kibana-plugin install file:///home/mour/devops/elk/x-pack-6.0.0.zip
After installing, remember to restart Kibana.
X-pack includes machine learning (which is why I installed it), but it’s only a 30-day trial. Check the docs for how to activate it. I’m still loading data.
curl -XGET -u root:root 'http://localhost:9200/_xpack/license/trial_status'
Or just drop this in the config:
xpack.license.self_generated.type: trial

Machine Learning

Looking at the results, performance is rock solid and genuinely fast — even running ML on hundreds of millions of records, it returns results in seconds, scanning the entire dataset at a steady pace. You could totally feed new data streams in real-time. That said, I still think importing data from ES into Spark and using Spark ML is probably the more solid approach. (ES itself does provide some unsupervised learning algorithms and time-series anomaly detection.)

Other Notes

All config files live under the config directory in each component’s folder. Pretty readable. When in doubt, check the docs — it’s not complicated.
uniq must be used with sort or you’ll get wrong results. Honestly just use awk. And when searching with find, using xargs is faster than running separate exec calls.
There’s elasticsearchdump for backing up ES data — really handy, Node.js-based, but the speed leaves something to be desired (tested on a remote server; haven’t tried locally yet).

Run ./elasticsearch-6.0.0/bin/x-pack/setup-passwords auto to auto-generate passwords, or use interactive to set them yourself. After setting up, go into Kibana config and fill in the ES username/password (find the commented-out lines, uncomment, fill in the generated credentials). Then disable x-pack security by adding xpack.security.enabled: false to the config.

After generating passwords, Logstash needs to be configured too before it can keep pushing data into Elasticsearch:
Can go inside or outside the input block

elasticsearch {
  hosts => ["127.0.0.1:9200"]
  username => "elastic"
  password => "epPz+Oxxxxxxxx&VWnrc"
}

Config docs

To use ML, you need to enable it in both Kibana and Elasticsearch configs:
kibana

xpack.ml.enabled : true

elastic

xpack.ml.enabled : true
node.ml : true

Why are some ES indices red?
bash

#!/bin/bash

for index in $(curl  -s 'http://localhost:9200/_cat/shards' | grep UNASSIGNED | awk '{print $1}' | sort | uniq); do
    for shard in $(curl  -s 'http://localhost:9200/_cat/shards' | grep UNASSIGNED | grep $index | awk '{print $2}' | sort | uniq); do
        echo  $index $shard

        curl -XPOST 'localhost:9200/_cluster/reroute' -d "{
            'commands' : [ {
                  'allocate' : {
                      'index' : $index,
                      'shard' : $shard,
                      'node' : 'Master',
                      'allow_primary' : true
                  }
                }
            ]
        }"

        sleep 5
    done
done

Fix referenced from here
Find errors

overhead error
No idea why, but it just went away on its own after a while — once everything settled into a normal state, the overhead error disappeared.
Why out of memory when there’s still 20GB of RAM?
It’s a Java heap issue — you have to set it manually. Edit jvm.options in the config directory and bump -Xms1g up, like to -Xms8g. After that, no more out of memory, no heap map issues, no cluster block issues either — things just work. One caveat from the docs though: even if you have 1TB of RAM, don’t allocate more than 32GB to it.
Out of memory

Todo:

Use Spark ML for analysis