Archive for March, 2014

Moving to Github Pages

March 29, 2014

I’ve decided that WordPress.com isn’t the right place for this blog.

I’m a hacker, so I should blog like one.

So, I’m moving this blog to here. I hope you’ll come visit.

Playing with logstash

March 29, 2014

I wanted to play with logstash, ideally using my current favourite tools of Vagrant and Chef.

I googled around, but the projects I found that use these tools were too complex, for my taste, so I rolled my own.

First, I wanted to start from a very simple Vagrant + Chef Solo + Ubuntu 12.04 configuration. Here’s one I made, earlier;


git clone https://github.com/digitalronin/chef_project_template.git logstash
cd logstash
rm -rf .git
./bootstrap_vagrantvm

This will take a few minutes.

Now we have a Vagrant VM, based on Ubuntu 12.04, with Ruby 2.0 as the system ruby, and a basic configuration using Chef Solo.

For more information, checkout this post.

Now to add logstash.

We’re going to install logstash via the apt package manager, from the Elasticsearch package repository.

mkdir -p chef/cookbooks/logstash/recipes

vi chef/cookbooks/logstash/recipes/default.rb

Here’s the content we need;

execute "add-logstash-repo-key" do
command "wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | apt-key add -"
not_if "apt-key list | grep Elasticsearch"
end

execute "add-logstash-repo" do
command "echo 'deb http://packages.elasticsearch.org/logstash/1.4/debian stable main' >> /etc/apt/sources.list"
not_if "grep packages.elasticsearch.org.logstash /etc/apt/sources.list"
end

execute "apt-get update"

package "logstash"

Now add “recipe[logstash]” to the runlist in chef/server.json;

{
"run_list": [
"recipe[ntp]",
"recipe[logstash]"
]
}

…and apply our new configuration;


cd chef
./deploy.sh root@192.168.11.11

That’s it. We now have a Vagrant VM with logstash 1.4.0 installed.

Let’s see what files logstash created;

dpkg -L logstash

Among others, you’ll see this file;

/opt/logstash/bin/logstash

Let’s try it out;

root@myserver:~# /opt/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'

Run that command, and then type something. You’ll have to wait a little to see the output, presumably because logstash is batching things up;

root@myserver:~# /opt/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'
Hello, world
2014-03-29T13:42:38.437+0000 myserver Hello, world

Press Ctrl-D to exit.

Now let’s add elasticsearch. The installation recipe is very similar to that for logstash;

vi chef/cookbooks/elasticsearch/recipes/default.rb

execute “add-elasticsearch-repo-key” do

command “wget -O – http://packages.elasticsearch.org/GPG-KEY-elasticsearch | apt-key add -”
not_if “apt-key list | grep Elasticsearch”
end

execute “add-elasticsearch-repo” do
command “echo ‘deb http://packages.elasticsearch.org/elasticsearch/0.90/debian stable main’ >> /etc/apt/sources.list”
not_if “grep packages.elasticsearch.org.elasticsearch /etc/apt/sources.list”
end

execute “apt-get update”

package “elasticsearch”

Don’t forget to add it to our server.json runlist;

{
“run_list”: [
“recipe[ntp]”,
“recipe[logstash]”,
“recipe[elasticsearch]”
]
}

…and apply the new configuration;

cd chef

./deploy.sh root@192.168.11.11

Elasticsearch should now be running. You can confirm that by logging onto the VM via ssh and running this;

wget -O - 'http://localhost:9200/_search?pretty'

…which should produce output something like this;

{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 0,
"successful" : 0,
"failed" : 0
},
"hits" : {
"total" : 0,
"max_score" : 0.0,
"hits" : [ ]
}
}

You can also fire up a web browser on your host machine and visit http://192.168.11.11:9200/

That’s as far as I’m going to go in this blog post, mainly because I don’t know much about Logstash and Elasticsearch (yet). More information is available here;

http://logstash.net/docs/1.4.0/tutorials/getting-started-with-logstash

Just remember that our logstash executable is /opt/logstash/bin/logstash when you work through their examples.

All the code for this blog post is available on github;

https://github.com/digitalronin/chef-logstash-elasticsearch

Getting started with Chef (and Vagrant)

March 21, 2014

I’m a big fan of configuration management, and the whole “infrastructure as code” approach. Currently, I’m managing hundreds of machines with Puppet, but I decided to take a look at Chef…just because.

I find a lot of the “getting started with Chef” articles and blog posts dive too deep into tool magic before giving a real understanding of the basics. When I learn a new programming language, I start with “Hello, World,” and I want the same kind of thing for learning Chef. I just googled “Getting started with Chef,” and the first three entries all have “now install knife and use this magic command to go get a bunch of cookbooks that other people have written.”

Maybe it’s just me, but I really don’t like typing “do this magic thing” and ending up with a ton of configuration files I don’t fully understand. That’s one of the reasons I don’t use IDEs when I develop software. I prefer to start from the bottom and work upwards, only adding a layer when I’m confident I understand the layers I’ve already built. That’s a slower approach, but we’re talking about the code that builds crucial pieces of my systems’ infrastructures, and there’s no way I’m just downloading a long, complex recipe and running it without fully understanding what it’s doing to my machine, and why (I have another rant about the complexity of most of the publicly available puppet/chef recipes you find on directories, but I’ll save that for another post).

From that point of view, the very best article I found about starting out with Chef was this one, from Jo Liss;

Chef Solo tutorial: Managing a single server with Chef

I also really like Vagrant as a way to iterate puppet/chef recipes quickly, and create throwaway VMs to play around with stuff. So, I’ve created the simplest possible Chef project I could build, which also has a Vagrantfile and a bootstrap script. This can be used as the basis for any server which you plan to manage using Chef. It also installs ruby2.0 as the system ruby.

Here’s how it works;

Clone the repo;

git clone https://github.com/digitalronin/chef_project_template.git myproject

Make it a new git project of your own;

cd myproject
rm -rf .git
git init

Edit the Vagrantfile and replace my SSH public key at the top, with one of your own. This public key will be installed in your Vagrant VM so that you have passwordless root access, which enables the script to do everything else without pausing to prompt you for a password.

Now for the fun part. This assumes you already have Vagrant, and that you already have the Ubuntu 12.04 “box.” If you don’t, there are instructions in the readme. It also assumes you’re running ruby 1.9 or greater. If you’re not, you’ll need to tweak the Vagrantfile to change “foo: bar” to “:foo => bar”;

./bootstrap_vagrantvm

Here’s what that does;

  1. Start an Ubuntu 12.04 vagrant VM (creating it, if it doesn’t already exist), with IP number 192.168.11.11
  2. Install your public key into the root account (/root/.ssh/authorized_keys)
  3. cd into the local “chef” directory and run deploy.sh, targeting the new VM
  4. The deploy.sh uploads the contents of the chef directory to the VM, unpacks it (removing any chef directory that was there before), and executes the “install.sh” script.
  5. install.sh bootraps the box to the point where it has ruby2.0 and chef-solo, and then runs the chef run list defined in server.json

This should take somewhere around 10 minutes or so, assuming a reasonably fast machine with a good internet connection.

I’ve left the run list almost empty. It just installs NTP and sets the server timezone to GMT, for the sake of having Chef do something.

If you don’t want to use Vagrant, you can use exactly the same process with any Ubuntu 12.04 server to which you can ssh as root, either by providing a password or by using an ssh key. Just start at step 3.

Whenever you make changes to your chef recipes, just run from step 3 again. After the first setup is completed, any subsequent runs should be quite quick, depending on your chef changes, so you should be able to iterate very quickly.

Anytime you want to go back and start from scratch again, just run;

vagrant destroy
./bootstrap_vagrantvm

Being able to use the same code to deploy a local development VM or a live server is very handy (although I would never recommend leaving root access open on your live servers), and it’s a big help in getting to the point where you have a Walking Skeleton of the system you’re building.

 

 

Habit-forming

March 10, 2014

I’ve been thinking a lot about habits, recently. In particular, how to create good habits in myself. I remember reading somewhere that it takes two or three weeks of daily repetitions to form a new habit. However, it seems this might be an underestimate.

The advantage of a habit, or a good habit, anyway, is that it seems to require so much less effort to do whatever it is once it’s habitual. In many ways, not doing an habitual task seems more effortful than doing it. It becomes harder to break the habit than it is to perform the task. Cory Doctorow puts it well when he says that “Habits are things you get for free.”

Regardless of the exact number, it’s clear that it takes some time, and some number of regular repetitions before something can be considered truly habitual.

Being a geek, I wanted to keep a record so I could monitor my progress. For a while, I tried recording in a text file the days on which I had or hadn’t done something I wanted to make it into a habit – in this case, doing daily Spanish practice using duolingo.com. But then I ended up having either multiple lists for multiple daily tasks, or one list that was difficult to read and annoying and time-consuming to maintain.

So, I created a little web application to help me manage the tasks I wanted to do every day, and to keep track of the days when I had completed those tasks, or not.

They say that naming things is one of only two hard problems in computer science (the second being cache-expiry, and the third off-by-one errors), so I don’t feel too bad about the uninspired name I chose for it. If you’re interested, go take a look;

http://mydailytodolist.org

Now I get to tick off “Blog” on today’s list.

Let me know what you think in the comments.

 

It’s been a while

March 7, 2014

I can’t believe I haven’t posted anything on this blog since 2011.

Actually, I can, but it’s still pretty shocking.

I’m going to try to make an effort to start posting again, partially inspired by this post;

http://timkastelle.org/blog/2014/03/you-should-start-a-blog-right-now/

Will I manage to stick to it, this time? Who knows?

Actually, if you’re reading this much after March 2014, then you already know.

How did I do?

David