In this blog post I will show you how to install and setup Graylog2 to manage logs on a Ubuntu 11.04 64bit server.

I am installing most of the applications in /opt. All of the commands are run on the root account.

Server Installation

As always, on a new installation, you should update

sudo apt-get update
sudo apt-get upgrade

Install OpenJDK and its dependencies. This can take a while, there is a lot to install.

sudo apt-get install python-software-properties
sudo add-apt-repository ppa:ferramroberto/java
sudo apt-get update
sudo apt-get install sun-java6-jre sun-java6-plugin openjdk-6-jre

Install ElasticSearch and the service wrapper.

cd /opt
curl http://cloud.github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.19.2.tar.gz | tar zxv
ln -s elasticsearch-0.19.2/ elasticsearch

curl -k -L http://github.com/elasticsearch/elasticsearch-servicewrapper/tarball/master | tar -xz
mv *servicewrapper*/service elasticsearch/bin/
rm -Rf *servicewrapper*
/opt/elasticsearch/bin/service/elasticsearch install

Configure ElasticSearch and test that it works ok.

ln -s `readlink -f elasticsearch/bin/service/elasticsearch` /usr/bin/elasticsearch_ctl
sed -i -e 's|# cluster.name: elasticsearch|cluster.name: graylog2|' /opt/elasticsearch/config/elasticsearch.yml
/etc/init.d/elasticsearch start

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'

If all is working then you should get something similar to this:

[email protected]:~# curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
  "cluster_name" : "graylog2",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0
}

Next up is the installation of MongoDB, the database server which Graylog2 uses. To do this run

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10

This will ensure that the database software you download is actually the software published by 10gen. Now we need to add the 10gen repository on the system. Do this by adding the following to the end of /etc/apt/sources.list

deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen

Depending on your system, you may need deb http://downloads-distro.mongodb.org/repo/debian-sysvinit dist 10gen. This is for “if you’re on a Debianoid with SysV style init scripts (e.g., Debian Lenny or older Ubuntus)”. If you’re using the latest version of Ubuntu, like this tutorial, then the ubuntu-upstart repo should do fine.

Update the repository list and install MongoDB

sudo apt-get update
sudo apt-get install mongodb-10gen

When MongoDB has finished installing, run mongo just to check whether MongoDB is running correctly. I got an Error: couldn't connect to server 127.0.0.1 shell/mongo.js:84 error to start with but after waiting a few minutes I was able to run mongo and it showed this:

[email protected]:~# mongo
MongoDB shell version: 2.0.4
connecting to: test
>

If you get the same as above it means MongoDB is successfully installed.

Now we need to create an admin user for mongo and add a user to mongo for the ‘graylog2’ collection. I like to use a Secure Password Generator to generate passwords.

mongo
use admin
db.addUser('admin', 'password123')
db.auth('admin', 'password123')
use graylog2
db.addUser('grayloguser', 'password123')
db.auth('grayloguser', 'password123')

Now time to install the actual Graylog2 Server.

cd /opt
curl  http://cloud.github.com/downloads/Graylog2/graylog2-server/graylog2-server-0.9.6.tar.gz | tar zxv
ln -s graylog2-server-0.9.6/ graylog2-server

cp /opt/graylog2-server/graylog2.conf{.example,}
cd /etc
ln -s /opt/graylog2-server/graylog2.conf graylog2.conf
cd -
sed -i -e 's|mongodb_useauth = true|mongodb_useauth = false|' /opt/graylog2-server/graylog2.conf

Now to create an init script for starting the Graylog2 server easily.

nano /etc/init.d/graylog2-server

Paste the following in:

#!/bin/sh
#
# graylog2-server:   graylog2 message collector
#
# chkconfig: - 98 02
# description:  This daemon listens for syslog and GELF messages and stores them in mongodb
#
CMD=$1
NOHUP=`which nohup`
JAVA_CMD=/usr/bin/java
GRAYLOG2_SERVER_HOME=/opt/graylog2-server

start() {
    echo "Starting graylog2-server ..."
    $NOHUP $JAVA_CMD -jar $GRAYLOG2_SERVER_HOME/graylog2-server.jar > /var/log/graylog2.log 2>&1 &
}

stop() {
        PID=`cat /tmp/graylog2.pid`
    echo "Stopping graylog2-server ($PID) ..."
        kill $PID
}

restart() {
    echo "Restarting graylog2-server ..."
        stop
        start
}

case "$CMD" in
    start)
        start
        ;;
    stop)
        stop
        ;;
    restart)
        restart
        ;;
    *)
        echo "Usage $0 {start|stop|restart}"
        RETVAL=1
esac

Change the permissions of the init script so that it is executable.

chmod +x /etc/init.d/graylog2-server

Set Graylog2 to start on bootup

update-rc.d graylog2-server defaults

Web Interface Installation

Download the latest version of the Graylog2 Web Interface (currently 0.9.6)

cd /opt
curl  http://cloud.github.com/downloads/Graylog2/graylog2-web-interface/graylog2-web-interface-0.9.6.tar.gz | tar zxv
ln -s graylog2-web-interface-0.9.6 graylog2-web-interface

Install Ruby 1.9.2. The Graylog2 Web Interface is powered by Ruby on Rails so Ruby is required. I have created a script which installs Ruby 1.9.2 and the latest version of Rubygems which is on my GitHub account. It can take a while to compile Ruby so grab a coffee and what a few minutes.

sudo apt-get -y install curl && curl https://raw.github.com/deanperry/onelineserver/master/ruby/192.sh >> 192.sh && chmod a+x 192.sh && sudo ./192.sh

Create a Graylog2 user

useradd graylog2 -d /opt/graylog2-web-interface
chown -R graylog2:graylog2 /opt/graylog2-web-interface*

Give root/sudo priviledges on the graylog2 user

usermod -G admin graylog2

Install the gems for the web interface

cd /opt/graylog2-web-interface
gem install bundler --no-ri --no-rdoc
bundle install

Check/set the web interface MongoDB settings

nano /opt/graylog2-web-interface/config/mongoid.yml

They should be:

production:
  host: localhost
  port: 27017
  username:
  password:
  database: graylog2

Change to the graylog2 user and run the server. This launches it on port 3000

su - graylog2
RAILS_ENV=production script/rails server

Browse to http://SERVERIP:3000 and setup your first user.

It is best to serve Graylog2 over either Apache or nginx instead of using the default server which comes with Rails, which is WEBrick (I say Rails because the Graylog2 web interface is powered by Ruby on Rails). We will be installing Apache2 and Passenger. Follow through the instructions.

sudo apt-get install apache2 libcurl4-openssl-dev apache2-prefork-dev libapr1-dev libcurl4-openssl-dev apache2-prefork-dev libapr1-dev
sudo gem install passenger
sudo passenger-install-apache2-module

Add the code which is says to add, into /etc/apache/httpd.conf.

sudo /etc/init.d/apache2 restart

Configure a virtual host

sudo nano /etc/apache2/sites-available/graylog2

Copy this and paste it into the graylog2 file you just created. Change what’s relevant for your system. The common things to change is the IP, ServerName and ServerAlias.

<VirtualHost *:80>
	ServerName graylog2.example.com
	ServerAlias graylog2.example.com
	DocumentRoot /opt/graylog2-web-interface/public

	<Directory /opt/graylog2-web-interface/public>
		Allow from all
		Options -MultiViews
	</Directory>

	ErrorLog /var/log/apache2/error.log
	LogLevel warn
	CustomLog /var/log/apache2/access.log combined
</VirtualHost>

Enable the virtual host you have just created

sudo a2ensite graylog2

Restart apache to activate the new configuration.

sudo /etc/init.d/apache2 restart

Now access the domain you set instead of graylog2.example.com and you should see the Graylog2 web interface which you saw earlier when setting up the first user.

Congratulations. You have now setup a Graylog2 server with the web interface! Next up, sending logs over to the Graylog2 server.

Sending logs to the Graylog2 server

There are many ways that logs can be sent to your newly created Graylog2 server.

rsyslog should be installed by default. To forward logs to your Graylog2 server do the following:

sudo nano /etc/rsyslog.d/graylog2.conf

Add the following to graylog2.conf. Change x.x.x.x to your Graylog2 server IP.

 Forward all logs to Graylog2
# Replace x.x.x.x with your own Graylog2 server IP
*.*     @x.x.x.x  #udp forwarding

Restart rsyslog and you should see logs being sent to your server straight away.

sudo service rsyslog restart

Rails Exception Logging

Ruby on Rails exceptions can also be caught by Graylog2.

Install the graylog2_exceptions gem and checkout the wiki on how to setup the gem with the version of Rails you are using.

Complete!

You’ve now completed the installation of Graylog2 along with the configuration of sending logs to your newly created Graylog2 server!