In this blog post I will show you how to install and setup Graylog2 to manage logs on a Ubuntu 11.04 64bit server.
I am installing most of the applications in /opt. All of the commands are run on the root account.
Server Installation
As always, on a new installation, you should update
sudo apt-get update
sudo apt-get upgrade
Install OpenJDK and its dependencies. This can take a while, there is a lot to install.
sudo apt-get install python-software-properties
sudo add-apt-repository ppa:ferramroberto/java
sudo apt-get update
sudo apt-get install sun-java6-jre sun-java6-plugin openjdk-6-jre
Install ElasticSearch and the service wrapper.
cd /opt
curl http://cloud.github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.19.2.tar.gz | tar zxv
ln -s elasticsearch-0.19.2/ elasticsearch
curl -k -L http://github.com/elasticsearch/elasticsearch-servicewrapper/tarball/master | tar -xz
mv *servicewrapper*/service elasticsearch/bin/
rm -Rf *servicewrapper*
/opt/elasticsearch/bin/service/elasticsearch install
Configure ElasticSearch and test that it works ok.
ln -s `readlink -f elasticsearch/bin/service/elasticsearch` /usr/bin/elasticsearch_ctl
sed -i -e 's|# cluster.name: elasticsearch|cluster.name: graylog2|' /opt/elasticsearch/config/elasticsearch.yml
/etc/init.d/elasticsearch start
curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
If all is working then you should get something similar to this:
root@ks35410:~# curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
"cluster_name" : "graylog2",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}
Next up is the installation of MongoDB, the database server which Graylog2 uses. To do this run
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10
This will ensure that the database software you download is actually the software published by 10gen.
Now we need to add the 10gen repository on the system. Do this by adding the following to the end of /etc/apt/sources.list
deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen
Depending on your system, you may need deb http://downloads-distro.mongodb.org/repo/debian-sysvinit dist 10gen. This is for “if you’re on a Debianoid with SysV style init scripts (e.g., Debian Lenny or older Ubuntus)”. If you’re using the latest version of Ubuntu, like this tutorial, then the ubuntu-upstart repo should do fine.
Update the repository list and install MongoDB
sudo apt-get update
sudo apt-get install mongodb-10gen
When MongoDB has finished installing, run mongo just to check whether MongoDB is running correctly. I got an Error: couldn't connect to server 127.0.0.1 shell/mongo.js:84 error to start with but after waiting a few minutes I was able to run mongo and it showed this:
root@ks35410:~# mongo
MongoDB shell version: 2.0.4
connecting to: test
>
If you get the same as above it means MongoDB is successfully installed.
Now we need to create an admin user for mongo and add a user to mongo for the ‘graylog2’ collection. I like to use a Secure Password Generator to generate passwords.
mongo
use admin
db.addUser('admin', 'password123')
db.auth('admin', 'password123')
use graylog2
db.addUser('grayloguser', 'password123')
db.auth('grayloguser', 'password123')
Now time to install the actual Graylog2 Server.
cd /opt
curl http://cloud.github.com/downloads/Graylog2/graylog2-server/graylog2-server-0.9.6.tar.gz | tar zxv
ln -s graylog2-server-0.9.6/ graylog2-server
cp /opt/graylog2-server/graylog2.conf{.example,}
cd /etc
ln -s /opt/graylog2-server/graylog2.conf graylog2.conf
cd -
sed -i -e 's|mongodb_useauth = true|mongodb_useauth = false|' /opt/graylog2-server/graylog2.conf
Now to create an init script for starting the Graylog2 server easily.
nano /etc/init.d/graylog2-server
Paste the following in:
#!/bin/sh
#
# graylog2-server: graylog2 message collector
#
# chkconfig: - 98 02
# description: This daemon listens for syslog and GELF messages and stores them in mongodb
#
CMD=$1
NOHUP=`which nohup`
JAVA_CMD=/usr/bin/java
GRAYLOG2_SERVER_HOME=/opt/graylog2-server
start() {
echo "Starting graylog2-server ..."
$NOHUP $JAVA_CMD -jar $GRAYLOG2_SERVER_HOME/graylog2-server.jar > /var/log/graylog2.log 2>&1 &
}
stop() {
PID=`cat /tmp/graylog2.pid`
echo "Stopping graylog2-server ($PID) ..."
kill $PID
}
restart() {
echo "Restarting graylog2-server ..."
stop
start
}
case "$CMD" in
start)
start
;;
stop)
stop
;;
restart)
restart
;;
*)
echo "Usage $0 {start|stop|restart}"
RETVAL=1
esac
Change the permissions of the init script so that it is executable.
chmod +x /etc/init.d/graylog2-server
Set Graylog2 to start on bootup
update-rc.d graylog2-server defaults
Web Interface Installation
Download the latest version of the Graylog2 Web Interface (currently 0.9.6)
cd /opt
curl http://cloud.github.com/downloads/Graylog2/graylog2-web-interface/graylog2-web-interface-0.9.6.tar.gz | tar zxv
ln -s graylog2-web-interface-0.9.6 graylog2-web-interface
Install Ruby 1.9.2. The Graylog2 Web Interface is powered by Ruby on Rails so Ruby is required. I have created a script which installs Ruby 1.9.2 and the latest version of Rubygems which is on my GitHub account. It can take a while to compile Ruby so grab a coffee and what a few minutes.
sudo apt-get -y install curl && curl https://raw.github.com/deanperry/onelineserver/master/ruby/192.sh >> 192.sh && chmod a+x 192.sh && sudo ./192.sh
Create a Graylog2 user
useradd graylog2 -d /opt/graylog2-web-interface
chown -R graylog2:graylog2 /opt/graylog2-web-interface*
Give root/sudo priviledges on the graylog2 user
usermod -G admin graylog2
Install the gems for the web interface
cd /opt/graylog2-web-interface
gem install bundler --no-ri --no-rdoc
bundle install
Check/set the web interface MongoDB settings
nano /opt/graylog2-web-interface/config/mongoid.yml
They should be:
production:
host: localhost
port: 27017
username:
password:
database: graylog2
Change to the graylog2 user and run the server. This launches it on port 3000
su - graylog2
RAILS_ENV=production script/rails server
Browse to http://SERVERIP:3000 and setup your first user.
It is best to serve Graylog2 over either Apache or nginx instead of using the default server which comes with Rails, which is WEBrick (I say Rails because the Graylog2 web interface is powered by Ruby on Rails). We will be installing Apache2 and Passenger. Follow through the instructions.
sudo apt-get install apache2 libcurl4-openssl-dev apache2-prefork-dev libapr1-dev libcurl4-openssl-dev apache2-prefork-dev libapr1-dev
sudo gem install passenger
sudo passenger-install-apache2-module
Add the code which is says to add, into /etc/apache/httpd.conf.
sudo /etc/init.d/apache2 restart
Configure a virtual host
sudo nano /etc/apache2/sites-available/graylog2
Copy this and paste it into the graylog2 file you just created. Change what’s relevant for your system. The common things to change is the IP, ServerName and ServerAlias.
<VirtualHost *:80>
ServerName graylog2.example.com
ServerAlias graylog2.example.com
DocumentRoot /opt/graylog2-web-interface/public
<Directory /opt/graylog2-web-interface/public>
Allow from all
Options -MultiViews
</Directory>
ErrorLog /var/log/apache2/error.log
LogLevel warn
CustomLog /var/log/apache2/access.log combined
</VirtualHost>
Enable the virtual host you have just created
sudo a2ensite graylog2
Restart apache to activate the new configuration.
sudo /etc/init.d/apache2 restart
Now access the domain you set instead of graylog2.example.com and you should see the Graylog2 web interface which you saw earlier when setting up the first user.
Congratulations. You have now setup a Graylog2 server with the web interface! Next up, sending logs over to the Graylog2 server.
Sending logs to the Graylog2 server
There are many ways that logs can be sent to your newly created Graylog2 server.
rsyslog should be installed by default. To forward logs to your Graylog2 server do the following:
sudo nano /etc/rsyslog.d/graylog2.conf
Add the following to graylog2.conf. Change x.x.x.x to your Graylog2 server IP.
Forward all logs to Graylog2
# Replace x.x.x.x with your own Graylog2 server IP
*.* @x.x.x.x #udp forwarding
Restart rsyslog and you should see logs being sent to your server straight away.
sudo service rsyslog restart
Rails Exception Logging
Ruby on Rails exceptions can also be caught by Graylog2.
Install the graylog2_exceptions gem and checkout the wiki on how to setup the gem with the version of Rails you are using.
Complete!
You’ve now completed the installation of Graylog2 along with the configuration of sending logs to your newly created Graylog2 server!