Setting up a RELK stack from scratch (Redis, Elasticsearch, Logstash and Kibana)

Screen Shot 2014-12-15 at 12.34.15

Recently I thought i’d re-do all of my ELK stack setup, as i didnt fully understand every facet of it and i was really interested in introducing Redis into the mix. I’ve also messed around with the existing Kibana and Logstash front-end to the point it was fairly bricked, so it was ripe for a change.

What I wanted to get to, was having my 2 servers and my main router having their logs and syslog data sent into my log box so I could view and correlate across multiple systems. Heres a pretty diagram to explain what i wanted:


To achieve this setup I used a stack of Redis, Elasticsearch, Logstash and Kibana. I used logstash forwarders on my servers to send the specified logs into a redis queue on my Kibana server. Once in the queue, Logstash would carve and process the logs and store them within Elasticsearch, from where Kibana would give me a nice front end to analyze the data. Simple right?


1. Redis

First, lets install Redis on our log monitoring server (Kibana.home, from herein). You can run all of the constituent parts of this setup on different boxes, just modify the IP’s/hostnames in the config files and remember to open up firewall ports if need be. On my small scale setup, running all of the parts on one VM was simply enough

To install redis, do the following:

You may need to install gcc / make (apt-get install make gcc) if your system doesnt have them. At this point it would be prudent to have 2 terminals (split vertically in iTerm or similar). Next, copy the redis.conf file from the extracted packages to the same location as the binary, i.e:

Open this file and modify it, if you wish to change the IP address its bound to, port, etc. Next, you need to startup redis using the command:

In a seperate window, run:

You should get a ‘pong’ reply, which tells you that redis is up and running. Finally, daemonize redis so that is set to run even when you kill the terminal. Open up /usr/local/bin/redis.conf and set ‘daemonize yes’, then restart redis.

Thats redis done...

2. Logstash forwarders

Next, on the client servers (devices we went to send logs FROM), run the following.

Create your logstash config file (where you will set WHAT is exported) in /etc/logstash/logstash-test.conf and put the following in it:

Basically, we are going to take whatever we type in the console, and output it to the screen to test logstash is indeed working:

As you can see, whatever we have typed (hi hi hi) is spat back out in a formatted fashion. So, that shows logstash is working (in a very limited way at least). Next, we need to test that logstash on this server can send data into our kibana.home server’s redis queue. To do this, create another config file in /etc/logstash called logstash-redis-test.conf, and in it add the following (obviously change my IP to the IP of your redis server!):

Next, start up logstash with this new config file (you may need to do ‘ps aux | grep java’ and then ‘kill -9 pid-of-the-java-instance‘), using the command:

Now, whatever we type should not only be spat back to us on the screen in a formatted fashion but should also appear in the redis-queue. So, on your 2nd terminal that is on the CLI of kibana.home (your server running redis), connect to redis so we can watch whats coming in:

Now, back to server.home – lets generate some traffic! Type some random rubbish in and hit enter:

On our kibana.home console, run the following 2 command – ‘LPOP logstash’ and ‘LLEN logstash'; the latter will tell you how many items are in the queue currently and the former will pop an item off the top of the queue / stack and display it to you, as below:

This shows that our logstash-forwarder can send events straight into the redis queue on our kibana.home server. This is where we are at the moment then:


Now, lets get some real data into redis instead of our testing! Create another file called /etc/logstash/logstash-shipper.conf which will be our ‘production config file’. In my example, I want to send my Apache log and Syslogs from /var/log into the queue, therefore i have a config as follows:

What you will notice, or should notice, is the ‘type’ line – this is VERY important for later on. Essentially, our redis queue will receive data and that data will be taged with a ‘type’. This type tells logstash later on, HOW to parse/process that log – i.e. which filters to apply. I’ve also got the IP address of my kibana.home in the output line; this config file essentially tells the logstash forwarder to send the 3+ log files to redis, using the type (tags) specified.

Note: The java process we are running will obviously die when the terminal is closed. To prevent this from happening, run the following command – which will daemonise it:

We're now shipping logs..

3. Elasticsearch

Now, firmly back on kibana.home, lets install Elasticsearch. This is where the log data will eventually live. To do this, install java and then download and install the Elasticsearch package (im running all of my boxes on Ubuntu):

Elasticsearch should have started after installation – to test that it is indeed running and accessible, use CURL as below:

Note: Elasticsearch at :9200 needs to be accessible from your browser – so if you have elasticsearch only available on or localhost it wont work and kibana will be upset.

We will also want to setup a ‘limit’ on the elasticsearch data, so we dont save logs for longer than we need (And thus run out of space!). To do this, we need to download and run a program called ‘curator’, via the method below:

Then in crontab, add the following line:

This essentially tells the curator program to delete any syslogs / data that is older than 60 days (you can make it longer / shorter depending).

Now that elasticsearch is installed, we now need to link the redis queue to it – i.e. take data off the queue (LPOP..), parse it, and store it within elasticsearch. To do this, we will use logstash.

Elasticsearch is now stretching..

4. Logstash indexer

To start, lets install logstash on kibana.home:

For all intents and purposes, you can ignore logstash-web, just ensure that logstash is running (the daemon).  Next, lets create the config file which this logstash instance will be using, at /etc/logstash/conf.d/logstash-indexer.conf:

Here we have a few things going on. we have an input section, and an output section – similar to the previous configurations. In this input section, we are taking 3 syslog files and tagging them with ‘syslog’, we are specifying port 5145 for udp/tcp to receive ‘syslog-network’ type data on, and we are also taking data from our redis-queue as an input also.  We are then outputting this data into elasticsearch to be stored. Simple right?

Note: Because you are reading /var/log/auth.log and others in /var/log, you will need to setup access control to allow the ‘logstash’ user to view these logs.

The best way to do this is to use setfacl/getfacl. You will need to install the package ‘acl’ to do this, and then run a command similar to:

You can test this quickly by editing /etc/passwd and giving the logstash user a shell, and then trying to ‘cd /var/log’. If it works, then logstash will be able to see these logs – if not, your setfacl command was wrong!

Now, back to that big config file. What you’ll notice is that we dont have here are any filters – we arent acting on the ‘type’ parameters we specified. The beauty of logstash is you can seperate your config out into seperate files – so instead of one god-awful long configuration file, you can have multiple little ones:

Here i have files for parsing different ‘types’ of traffic, for example anything that gets sent in with the type ‘syslog-network’ (i.e. logs from my draytek router), are pushed through rules in this config file:

This takes the raw data recieved from my router, and chops it into usable fields using Grok.  I have a seperate .conf file for Opsview log traffic, Syslog traffic and also Apache traffic (i will put the output of these at the bottom!).

Essentially, you are telling Logstash – “Hey, if you see a log that has this type, then prepare it for storage using this filter”.

Now we have a configuration file(s), we can restart logstash:

We now have logstash-forwarders sending data into redis, and logstash-indexer on kibana.home is taking that data and chomping it up and storing it in Elasticsearch, as below:


Note: If there are errors in any of your config files, logstash will die after around 10 seconds.

It is therefore recommend to run ‘watch /etc/init.d/logstash status’ for about 20 seconds to make sure it doesnt fall over. It is does (i.e. your missing a quote or parenthesis, etc) then tail the logstash log using:

This will tell you generally where you are going wrong. BUT, ideally you wont have made any errors! :)

We can test that logstash, redis and elasticsearch are playing nicely together by running ‘LLEN logstash’ in redis-cli (as we did earlier) and seeing it at 0 or reducing, i.e. 43 dropped to 2. This means that logstash is popping from the queue, parsing it through our filters, and storing it in elasticseach. Now, all we need to do is slap a front-end on it!

Almost there

5. Kibana and Nginx

As i’m running nginx as my front-end, I used a config file i found which worked a treat. Put this config file at /etc/nginx/sites-available:

This helps get around the problems with elasticsearch being exposed outside of, etc. Now, hit up ‘http://kibana.home/’ (the address/IP of your log server, obviously!) and you should see Kibana! Here is an example dashboard i have built using the apache logs, router logs, Opsview logs and a few others:

Screen Shot 2014-12-15 at 12.34.15
You did it

6. Wash-up and notes

So there you have it; logs being sent via logstash-forwarder’s into a central redis queue, which is watched and processed by a logstash-indexer and stored in elasticsearch – where it is interpreted using Kibana running on nginx. The following places are the items to mentally bookmark for your fingers:

On the Kibana/Elasticsearch/Logstash/Redis server:

  • Logstash directory (where all your configs are): /etc/logstash/conf.d/
  • Redis: /usr/local/bin/redis.conf
  • Elasticsearch: /etc/elasticsearch/elasticsearch.yml
  • Kibana: /var/www/kibana3

On the servers you are sending logs from:

  • Logstash: /etc/logstash/logstash-shipper.conf

One final hint / tip – To have named log all of its requests to syslog, run the command:

Grok filters
Apache logs filter:

 Draytek router logs filter:

 Opsview filter:

 Syslog filter:

Integrating Opsview with ELK Log Monitoring

Hello all!

This is a brief blog post to explain how I quickly integrated my existing Opsview server, with my existing ELK deployment. I basically wanted a way that within Opsview, i can see that a host has failed or is having problems and go “Hmm, lets have a look at the logs to see whats happening” without:

A) Having to SSH to the box and start tailing or

B) Have to fire up ELK and start filtering.

To step back just a moment, what is ELK? ELK stands for ElasticSearch / Logstash / Kibana, and essentially it uses ElasticSearch/Logstash to collect and handle the log data, and Kibana as the graphical front end through which users can create their own filters, graphs, etc.

So, onto the integration. My current setup is as follows:

  • Opsview is monitoring all of my servers, network devices, virtual machines (KVM) and so forth – for items such as load average, memory usage, LVM capacity, temperatures, processes/services running, response times and so forth.
  • ELK is collecting logs from all of the aforementioned devices.

What I wanted to be able to do off-the-bat with ELK was use URL-based syntax to filter the ELK view, however this isnt immediately possible out of the box it appears, so you will have to make slight modifications to the .json file (default.json or whatever your ELK view is saved as). Open up your json view (i.e. /var/www/kibana3/app/dashboards/default.json) and edit the top part to look similar to the following:

Essentially what we are doing, is allowing URL-based querying by parsing through the ‘?q=MYFILTER’ variable straight to the query box, which wasnt available by default. This allows us to open http://my-elk-server/default.json?q=opsview and ELK will be opened with a filter of ‘opsview’ by default. Neat huh!

So, now that is working – test it out as above, you should get something similar to the following:



Screen Shot 2014-11-25 at 10.52.55


If not then filtering we created/edited above isnt working. If it is working, then great – proceed to the next section!

Setting up in Opsview

In Opsview we are going to use the in-built ‘Management URLs’ functionality (docs here), which allows users to create a host template i.e. ‘My Linux Template’ and create a management URL of ‘ssh://$HOSTADDRESS$:22′ for example. This allows the user to dive straight into an SSH shell on that box from within the Opsview UI, when that template is applied to a host. Cool huh? You can use this for anything, wiki’s, confluence, service desks, you name it – i.e. create a ‘Wiki’ host template of ‘$HOSTADDRESS$’ – when this is applied to a series of hosts, you will be able to load the wiki and search it for the name of the server you are looking at, from one menu option.

For our purposes, we are planning on creating an ‘ELK’ host template, whom we will apply to all of the host’s whose logs we collecting with ELK.

Step 1: Create the host template

Fairly simply, go to ‘Settings > Host Templates > Add new’, and populate it with a name and description as below:

Screen Shot 2014-11-25 at 09.50.22

Step 2: Create the management URL

After clicking ‘Submit changes’, you will now be able to click on the previously-greyed-out ‘Management URLs’ button. In here we will need to create our ELK link, as below:

Screen Shot 2014-11-25 at 09.50.55


For reference, the syntax is ‘http://elk-log-server/index.html#/dashboard/file/default.json?query=$HOSTADDRESS$’. The important part here is $HOSTADDRESS$ – this variable or macro will be substituted out for the address of the host, i.e. if this template is applied to ‘’, when the management URL is clicked on that host the full URL will be http://elk-log-server/index.html#/dashboard/file/default.json?

Step 3: Apply the template to the hosts

Next, we will need to apply the template to the hosts whose logs we want to monitor. You can do this via ‘Host template’ tab using the hosts section, but because Im lazy I did it via the host itself (Settings > Hosts > and then clicked on my host in question), as below:

Screen Shot 2014-11-25 at 09.51.23

..and thats it, the template is applied.

Step 4: View the logs

After a quick reload, go to your Opsview monitoring screens and click on the contextual menu and you will now see an extra option there – ELK:

Screen Shot 2014-11-25 at 10.00.00

And thats pretty much it! Now, all you need to do is apply the ‘ELK’ host template to the hosts whose logs you are monitoring and this option will appear ^^. That way in the future, you can see ‘oo we have a host failure’ and dive straight into the logs at the click of a button, as below:

Screen Shot 2014-11-25 at 10.52.55





Collaboration and innovation in 2014

As those of you who know me personally can attest, i’m never particularly enamoured with the phrase – “Thats just what we’ve always used”, or “Thats just how its been since i started”. Im a firm advocate of the Winston Churchill quote “To improve is to change; to be perfect is to change often.” and yet I feel that a lot of companies fail to do this.

A lot of us in IT like to reminisce about “Oh wow, 2.5″ floppy drives – I bet you dont even know why there isnt a B drive in Windows do you?” (Or worse still with the older generation – “Oh VAX was the best..”), instead of looking at the drawbacks which led to the eventual demise of these technologies. I feel the same can be said for email; its so engrained in corporate culture that no one has actually thought to product analyze email itself – in terms of:

1. What problems does it solve?

2. Is it fit for purpose and easy to use?

3. Is it advanced enough to allow for further growth and evolution?

In 1971 when ARPANET first devised email, I imagine the above were true – but it was also true that it would be 4 years until Microsoft would be founded, and another year until the breakthrough Intel 8008 microprocessor would be released. Nowadays, email tends to be the cause of more problems that it is worth, especially when used inter-company.

Dont get me wrong, I dont think email is all bad – for starters, how else would Sales teams communicate with customers? How else would electronic invoices be sent? How about emails that need to be tracked for auditing purposes (i.e. formal notices from HR, legal, and so forth)?

I think email has the same purpose as the one a postal system holds; content delivery rather than collaboration and speed. However, in the last 10-15 years  email has been used more and more as a collaborative tool between teams and departments, leading to infuriatingly long email threads, reply-to-all’s, and a lack of dynamism that can really hurt companies in the fast moving society in which we currently inhabit.

Email is also prone to abuse – we have:

1. Spammers / Scammers

2. Automated junk from websites and services

3. Junk you might have once thought of as a useful idea (i.e. “Oh its great that Salesforce emails me everytime a lead is created”).

Along with a million other ‘Junk rule’ creating annoyances.

So, what is the answer?

Recently, I’ve been working more and more in the world of ‘integrations’, taking data and events from one tool and doing something in another – and this has exposed me to a world of possibilities in terms of – “Why cant this idea be taken and applied on a company wide scale?”.

What if instead of having hundreds of folders containing automated emails – and then having to forward these emails to colleagues, then skype them “Hi, about that email..” – you could both see the ‘event occur’, and then discuss it live, along with the rest of your team?

In my company alone, we use at least 10-15 platforms across departments, ranging from ServiceCloud for customer success (nee: technical support), Salesforce for CRM and Sales, Jenkins for builds, Git for storing magic, JIRA Agile for SCRUM/Agile, ProdPad for Product management, and so forth. All of these tools generate events and it would be awesome if we could quickly access and collaborate on these; yet currently we (like many others) are reticent to set up email alerts as they will inevitably annoy us and be tossed into junk via a rule.

A potential answer? A chatroom-like tool that allows feeds directly from the aforementioned tools, but at the same time allows for the dynamic creation of rooms/group chats, sharing of files (Docs, images, etc) and more importantly – dynamic discussion and resolutions. Namely, Slack or Atlassian Hipchat.



Hipchat is a free (for normal use) tool from Atlassian, the makers of JIRA, JIRA Agile, Confluence, BitBucket and more. It allows tools such as their aforementioned products to send data and messages into various ‘rooms’, such as ‘Engineering’, ‘Support’ and more – meaning that if a new customer incident is raised in JIRA, it will appear in the Support room within seconds, allowing support engineers to quickly discuss the incident and assign appropriately.

There are a number of Hipchat integrations available here, ranging from GitHub and Drupal through to email and even Opsview (i.e. ‘customers edge router has gone down, send a message into the Networking room to tell them).



Slack is like Hipchats better looking, cooler younger brother. Its much more user friendly, the UI design is excellent and I love how easy it is to setup. The main differences I found with Hipchat and Slack are:

  • Hipchat seems more corporate; you have an IRC style interface for a start – with a list of users on the right that you can double click on to either IM or video chat (are people STILL trying to flog that as an idea? Let it die!).
  • Slack is easier to use and is a hell of a lot prettier from a UX perspective, which ALWAYS helps adoption.


The issue I found with trying to setup Slack and Hipchat as my email-killer, is that whilst they offer integrations, they dont offer an all-encompassing range that fits my needs perfectly, and as im not a developer I’d have no idea where to start in terms of writing my own integrations.

For example, with Hipchat the integration with JIRA Agile is excellent, you can be notified for all sorts of things – which is what you’d expect from 2 tools from the same vendor, however they offered no integration with Salesforce or ServiceCloud. Likewise, Hipchat had some neat integrations via webhooks, but also fell down on Salesforce integration, and their JIRA integration is pretty poor.

In order to get messages from these vendors into the ‘chat tools’ i just mentioned, you have to use another product that has popped up in the market in recent years, an ‘event broker’.

Event brokers

An event broker is essentially a man in the middle, who takes data from one tool, formats it, and sends it to the other tool – for a nominal fee (or free, depending on your tool de jour).

The benefit of using event brokers is that as long as they can talk to your chosen client (Hipchat, Slack, etc) then its generally much easier to setup integrations as thats their bread and butter. A few examples of event brokers are:

  1. Zapier
  2. IFTTT (If This Then That)
  3. Cloudwork
  5. We Wired Web

Some of these tools are free-free, such as IFTTT, and some are free to a point, such as Zapier, WWW and more. The pricing tends to hinge around:

  • Number of integrations
  • Frequency with which integrations can be delivered
  • Total messages that can be sent per month
  • Users

In my testing, I have so far looked at IFTTT and Zapier.


Verdict? Its OK – probably more use for home users and script kiddies than to a business, judging from the list of integrations:



It doesnt have Salesforce support, JIRA support, or other items like Jenkins etc however it does have GitHub support.


Verdict? Pretty good! It is incredibly simple to configure and use, and linking your accounts (i.e. the Salesforce account you want to get information via) is VERY simple to do. The configuration of what you want to send is also very simple, as it sucks the fields in from Salesforce, Twitter, or any other tool you want – for example, I want to send a message into my marketing room whenever my company was mentioned on Twitter, so they can discuss if they need to respond, etc.

To do this, I set up a new ‘Zap’ as below:





Trust me, it IS as easy as that. And now, whenever anyone tweets with the words ‘Opsview’, it will appear in our Marketing teams Slack/Hipchat room. Observe:


We can now discuss the tweet, respond if its a query, pass it onto Sales if its an enquiry, and so forth! All done without a single email. Ah the bliss.

What next?

The end goal is to have a fully automated solution, that takes in inputs from all of our chosen tools, uses an event broker (if required – some tools can output directly into Slack/Hipchat using Webhooks), and puts them into the correct room, in the correct format – allowing for quick diagnosis, recognition – and most importantly, COLLABORATION.

Heres a little screenshot of what ive set up so far:


Here, we are sending a message into the #sales room in Slack when a new opportunity is created (via the website or an account manager). We are sending an alert into the marketing room when we are tweeted about, and when a customer raises a ticket in ServiceCloud – the support team gets an alert. The plan is to roll this out to our other functions, then start the most painful part – driving adoption!

Potential barriers to adoption

So – this things bloody marvellous, why wouldnt anyone want to use it? Well there are a few areas:

1. It doesnt have a calendar functionality, meaning people will still be using Outlook to send calendar invites, and view other users diaries.

2. People are familiar with Outlook – meaning that when driving adoption, highlighting the advantages of this disruptive move (Crossing the Chasm, Geoffrey Moore) is going to be key to its adoption throughout.

3. People use Outlook for task management – Simple, use something designed for that purpose like Wunderlist, my personal lord and saviour of time management. Again however, there is likely to be some pushback on this – given people are still using IE, Id be amazed if they jumped on something like Wunderlist without reticence.


To conlude, email is good for external communications – but in my opinion, stay away from outlook when thinking of internal communications – it might have been the way to do it 20 years ago, but with the rise of event brokers, tools like hipchat/Slack, and an onus from software vendors on creating an easy to use API to allow for automation, the days of “email for everything!” is nigh – collaborative tools will drive customer success, and eventually the success of your business as a whole.

I see event brokers becoming more and more prominent amongst a user base who wants integration without the development headache – forcing vendors to create ‘recipes’ for these platforms such as IFTTT, Zapier, and co. Overall – software will become better at talking to other tools, which can only be good for customers.

Slack and IRC

As part of my Slack evangelism I’ve been hellbent on trying to integrate as many products I/we use into the aforementioned tool, to test out where the limit is on what this awesome product can do. Next on my list of things is IRC – that old warhorse. Opsview has a large number of users on IRC, yet I continually forget to start my IRC client after reboots, etc meaning i miss out on all the useful questions, comments and other items that our community have to offer.

Therefore, it sounds like a perfect technology to stream into Slack; 24/7 coverage of ANY IRC room – pretty cool, huh?


To find the right integration, I scoured the internet and tested a few different solutions. Some where just OVERLY complex, some just didnt work, and some were just right (Codename Goldilocks). The one that I found worked perfectly for me was ‘slack-irc-plugin‘, by Jimmy Hillis. To install any of the IRC bots, it turns out you need ‘node’ installed on your server. But lets back-up – how does this actually works?

Basically, you need something that is always-0n, which the irc-bot can run on. This bot connects to the IRC channel(s) and monitors what is going on, s that when something changes like someone posting a new message, the bot notices this and sends the information to Slack using the given parameters.  Essentially, you need the following setup:

Screen Shot 2014-09-04 at 13.03.16


The IRC Bot runs on your server/PC, and updates Slack whenever anything happens. So, how do we install it?


This bot, and the others I tested, all run on node which is interesting, as i’ve never had the pleasure of using node before. After some testing, it turns out that the version of node that some OS’s have in their repositories (like Ubuntu) doesnt work, so in my installation instructions I will give you the correct repo to use.

Firstly, lets add that repo:

Next,  clone the slack-irc-plugin from git:

Of course, change the directory above – DONT blindly copy and paste. Next, go into that new directory and lets build the tool:

This npm command is inherited from node. There will be some output now, as node downloads the actual components of the bot (listed in the node_modules dir) and installs them. After this completes, lets copy the config file and then edit it:

In here, we will need to edit the following:

Let me explain the above:

Server = This is the IRC server you are going to connect to, i.e. Freenode is

Nick = This is the name you want the bot to show as in the channels it joins.

Username = The bots actual username.

Token = You will need to create a new INCOMING WEBHOOK via – which will give you a token as a result, i.e.:

webhook api inbound

Next, we have Channels: This is where you bind which channels you want to watch in IRC, and where that chat is dumped to, so i’m monitoring the #opsview room in IRC, and I want to mirror it into an #irc channel in Slack.

Users: This tripped me up for a WHILE, but this is where you bind IRC users to their slack usernames. This is very finicky, but you need to include the tilde, i.e. ~ircuser, and you need to ensure that it is the same username as what is shown when you do an ‘/whois ircuser’. Your slack username is shown on your account, if you dont know it:


Finally, you really want to set ‘silent’ to true, otherwise the bot randomly starts blabbering into the channel which is pretty embarrassing, as i found out.

NOTE: This is all you need to get it working, as per the instructions, but i found that when i was running this (3rd Sept 2014) my bot was constantly joining/leaving the channel. I found out it was being kicked due to ‘Flooding’, even though it was posting no messages. After digging around in the API of the actual bot module, (link here if your bored), I had to edit the following file and line:

Doing this solved all my problems (well, IRC ones at least…). Finally, lets start her up:


I have no idea what any of that means, but i’ve fed it back to the developer as they might have more of an insight! But thats it, your bot is now configured and is sending data back to your Slack room when items are entered into IRC:


Pretty fancy eh? Now, go forth and Slack!