At Affectv we logs because they give us insights into our systems. Due to our current infrastructure, the log management system we are using to store, process and search log entries can only receive TCP connections, which can create a troublesome coupling between it and our services.

The solution was to set up a relay service as NXLog listening on a connectionless protocol such as UDP (User Datagram Protocol) locally, allowing the service to send the logs in a non-blocking way. NXLog also allowed us to improve the security and comprehension of our log messages, but that’s another story for another day.

In our Go systems we are using Logrus, a popular open source logger package. Logrus allows the creation of third-party hooks to connect with different services. Sadly, NXLog was missing from the list of available hooks so we decided to create our own. We also liked the idea of offering it as open source to contribute to the community.

A simple diagram to explain the functionality of our logrus-nxlog-hook would be:

What are exactly NXLog and Logrus?

NXLog is a simple backend log manager that allows log forwarding from an input to an output (cool, isn’t it?). It implements different input modules and it’s able to forward them through several output modules

It allowed us to decouple the services generating the logs from the log management system. Additionally, it brought some nice functionalities to the system like being able to understand different message formats, and transforming them into something our log management system would recognise.

Logrus is a logger for Golang. It’s widely used and we adopted it as our main logger package for Go long time ago.

Implementation details

The hook currently supports all the ways to connect with NXLog externally. This allows users to configure NXLog according to their needs without worrying if they will be able to use our Logrus hook.

The maximum size of UDP packets is not limited by our hook. We have tried several chunking implementations for UDP and seems that NXLog does not understand them, resulting in corrupted messages.

Further information can be found in the Golang Docs of the hook.

Installation

The hook is available as open source. The only dependency is Logrus

go get github.com/affectv/logrus-nxlog-hook

Configuration

To be able to start using it you have to import the package in our Golang code, altogether with the base Logrus logger.

import (
  logrus "github.com/sirupsen/logrus"
  nxlog "github.com/affectv/logrus-nxlog-hook"
)

First we must initialise the hook, adding it to the logger we want to use.

We can use the default logger:

hook, err := nxlog.NewNxlogHook("tcp", "ip:port", nil)
if err == nil {
  logrus.AddHook(hook)
  logrus.Info("NXLog hook added successfully")
}

We can use it with a specific logger:

var logger = logrus.New()
hook, err := nxlog.NewNxlogHook("tcp", "ip:port", nil)
if err == nil {
  logger.AddHook(hook)
  logger.Info("NXLog hook added successfully")
}

Formatting

Sometimes we want the logs in different formats depending on the output we are choosing. Currently Logrus only allows to change the log formatter globally, so our hook implements an extra layer to allow us to set a specific formatter.

NXLog is able to decode JSON, adding new properties or modifying them, and then forwarding them somewhere else. So by default our hook will try to send the entries in a JSON format to NXLog.

It can be changed in code by any formatter available in Logrus.

hook, _ := nxlog.NewNxlogHook("tcp", "ip:port", nil)
hook.Formatter = &logrus.TextFormatter{}

We can also let the hook use the default formatter used in the logger.

hook, _ := nxlog.NewNxlogHook("tcp", "ip:port", nil)
hook.Formatter = nil

Log Levels

The idea of having different log hooks is to be able to send some logs to specific locations depending on their level. The hook allows to se the minimum log entry level that will trigger the NXLog hook, so only those entries and above will be sent to NXLog.

In this case, only info, warn, error, fatal and panic levels will be sent to NXLog, whilst debug messages will be ignored.

var logger = logrus.New()
hook, err := nxlog.NewNxlogHook("tcp", "ip:port", nil)
if err == nil {
  hook.Level = logrus.InfoLevel
  logger.AddHook(hook)
  logger.Info("NXLog hook added successfully")
}

Any suggestions, bugs or general inquiries can be addressed directly into the issues section of the github repository.

That’s all folks!