Playing with logstash

I wanted to play with logstash, ideally using my current favourite tools of Vagrant and Chef.

I googled around, but the projects I found that use these tools were too complex, for my taste, so I rolled my own.

First, I wanted to start from a very simple Vagrant + Chef Solo + Ubuntu 12.04 configuration. Here’s one I made, earlier;


git clone https://github.com/digitalronin/chef_project_template.git logstash
cd logstash
rm -rf .git
./bootstrap_vagrantvm

This will take a few minutes.

Now we have a Vagrant VM, based on Ubuntu 12.04, with Ruby 2.0 as the system ruby, and a basic configuration using Chef Solo.

For more information, checkout this post.

Now to add logstash.

We’re going to install logstash via the apt package manager, from the Elasticsearch package repository.

mkdir -p chef/cookbooks/logstash/recipes

vi chef/cookbooks/logstash/recipes/default.rb

Here’s the content we need;

execute "add-logstash-repo-key" do
command "wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | apt-key add -"
not_if "apt-key list | grep Elasticsearch"
end

execute "add-logstash-repo" do
command "echo 'deb http://packages.elasticsearch.org/logstash/1.4/debian stable main' >> /etc/apt/sources.list"
not_if "grep packages.elasticsearch.org.logstash /etc/apt/sources.list"
end

execute "apt-get update"

package "logstash"

Now add “recipe[logstash]” to the runlist in chef/server.json;

{
"run_list": [
"recipe[ntp]",
"recipe[logstash]"
]
}

…and apply our new configuration;


cd chef
./deploy.sh root@192.168.11.11

That’s it. We now have a Vagrant VM with logstash 1.4.0 installed.

Let’s see what files logstash created;

dpkg -L logstash

Among others, you’ll see this file;

/opt/logstash/bin/logstash

Let’s try it out;

root@myserver:~# /opt/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'

Run that command, and then type something. You’ll have to wait a little to see the output, presumably because logstash is batching things up;

root@myserver:~# /opt/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'
Hello, world
2014-03-29T13:42:38.437+0000 myserver Hello, world

Press Ctrl-D to exit.

Now let’s add elasticsearch. The installation recipe is very similar to that for logstash;

vi chef/cookbooks/elasticsearch/recipes/default.rb

execute “add-elasticsearch-repo-key” do

command “wget -O – http://packages.elasticsearch.org/GPG-KEY-elasticsearch | apt-key add -”
not_if “apt-key list | grep Elasticsearch”
end

execute “add-elasticsearch-repo” do
command “echo ‘deb http://packages.elasticsearch.org/elasticsearch/0.90/debian stable main’ >> /etc/apt/sources.list”
not_if “grep packages.elasticsearch.org.elasticsearch /etc/apt/sources.list”
end

execute “apt-get update”

package “elasticsearch”

Don’t forget to add it to our server.json runlist;

{
“run_list”: [
“recipe[ntp]”,
“recipe[logstash]”,
“recipe[elasticsearch]”
]
}

…and apply the new configuration;

cd chef

./deploy.sh root@192.168.11.11

Elasticsearch should now be running. You can confirm that by logging onto the VM via ssh and running this;

wget -O - 'http://localhost:9200/_search?pretty'

…which should produce output something like this;

{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 0,
"successful" : 0,
"failed" : 0
},
"hits" : {
"total" : 0,
"max_score" : 0.0,
"hits" : [ ]
}
}

You can also fire up a web browser on your host machine and visit http://192.168.11.11:9200/

That’s as far as I’m going to go in this blog post, mainly because I don’t know much about Logstash and Elasticsearch (yet). More information is available here;

http://logstash.net/docs/1.4.0/tutorials/getting-started-with-logstash

Just remember that our logstash executable is /opt/logstash/bin/logstash when you work through their examples.

All the code for this blog post is available on github;

https://github.com/digitalronin/chef-logstash-elasticsearch

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: