Sending logs from NodeMCU/Lua to Graylog2

Graylog2 is a great product. It might not be the most sophisticated one when it comes to log aggregation and interrogation, but it does its job very well. After employing Graylog2 to collect logs from all my Docker containers I thought that it would be absolutely fantastic to be able to manage logs from the IoT devices as well.

Installing the right input plugin

Before we can push stuff into Graylog2 we need to teach it the protocol of IoT devices - MQTT. Luckily enough there is a ready-made and supported input plugin just for that. To install it simply download it to the plugin folder in your Graylog2 installation and restart Graylog2. Done.

The next thing is to add an input for MQTT. Select System/Inputs and then create an input by selecting MQTT TCP (GEFL) from the drop-down list of available connectors. You need to specify the MQTT server and queue which the connector will subscribe to. In the example below I used system/logs so if you would go for a different name you need to adjust the code accordingly. And last but not least you need to start the input by pressing the Start input button. Done.

Programming in Lua

Lua and the NodeMCU environment is a beautiful thing. It is simple, powerful and it just rocks. For our little project we'll need to build ourselves a version with bmp085, cjson, wifi and mqtt modules. Then we need to download it to the module and we're all set to start coding in Lua.

The code here initializes WiFi connection, once the connection is established connects to the MQTT server, initializes the BMP085 sensor (here we're using pins D2-sda and D1-scl as per the documentation) and sets up a recurring alarm every second to read the sensor and send GEFL-formatted logs if MQTT has been connected. Easy.

Transferring files to NodeMCU

There are many great ways to send Lua files to the module. I like to use the one written in NodeJS called nodemcu-tool

$ npm install -g nodemcu-tool
$ nodemcu-tool upload init.lua
$ nodemcu-tool terminal

Now press the reset button and your NodeMCU should inform you of 3 things:

1. That it connected to WiFi network and has an IP address
2. That it has connected to the MQTT server
3. That it is sending logs to MQTT server

There is not much more to it. Give it a try and if you stumble upon any challenges getting it to work feel free to leave a comment.

Happy logging!

Spring Boot application with Graylog2

I have been looking for an alternative to Splunk for a while and recently I have stumbled upon Graylog2. It is a very advanced tool well worth looking at. I decided to do a minimalist Spring Boot project to test its capabilities

The configuration

There are 2 steps required in order to get the logs to Graylog2:

• Adding an artifact with GEFL appender
• Adding the GEFL appender

I have tried 2 different GEFL appenders: Logback-gelf v0.10p1 which did work but didn't give me all the necessary details into Graylog2, and logback-gelf that worked for me very well.

I added the dependency to my pom.xml as follows

<dependency>
    <groupId>de.siegmar</groupId>
    <artifactId>logback-gelf</artifactId>
    <version>1.0.3</version>
</dependency>

And then added a logback configuration file to my resources

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <include resource="org/springframework/boot/logging/logback/base.xml" />

    <appender name="GELF" class="de.siegmar.logbackgelf.GelfUdpAppender">
        <graylogHost>localhost</graylogHost>
        <graylogPort>12201</graylogPort>
        <layout class="de.siegmar.logbackgelf.GelfLayout">
            <originHost>localhost</originHost>
            <includeRawMessage>false</includeRawMessage>
            <includeMarker>true</includeMarker>
            <includeMdcData>true</includeMdcData>
            <includeCallerData>true</includeCallerData>
            <includeRootCauseData>true</includeRootCauseData>
            <includeLevelName>true</includeLevelName>
            <shortPatternLayout class="ch.qos.logback.classic.PatternLayout">
                <pattern>%m%nopex</pattern>
            </shortPatternLayout>
            <fullPatternLayout class="ch.qos.logback.classic.PatternLayout">
                <pattern>%m</pattern>
            </fullPatternLayout>
            <staticField>app_name:backend</staticField>
            <staticField>os_arch:${os.arch}</staticField>
            <staticField>os_name:${os.name}</staticField>
            <staticField>os_version:${os.version}</staticField>
        </layout>
    </appender>

    <root level="trace">
        <appender-ref ref="GELF" />  
        <appender-ref ref="CONSOLE" />
    </root>
</configuration>

This gave me all the logs into a local instance of Graylog2 running on Docker.

version: '2'
services:
  mongo:
    image: "mongo:3"
  elasticsearch:
    image: "elasticsearch:2"
    command: "elasticsearch -Des.cluster.name='graylog'"
  graylog:
    image: graylog2/server:2.2.2-1
    environment:
      GRAYLOG_PASSWORD_SECRET: somepasswordpepper
      GRAYLOG_ROOT_PASSWORD_SHA2: 8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
      GRAYLOG_WEB_ENDPOINT_URI: http://127.0.0.1:9000/api
    depends_on:
      - mongo
      - elasticsearch
    ports:
      - "9000:9000"
      - "12201/udp:12201/udp"
      - "1514/udp:1514/udp"

Happy logging!