In modern web applications, logging is an essential part of monitoring and debugging. Traditional log files stored on disk can be challenging to manage and analyze. Enterprising logging solutions provide real-time log monitoring and analysis, making it easier to identify and troubleshoot issues.

One such solution is Kibana, a powerful visualization and analytics platform designed to work with the Elasticsearch stack. In this blog post, we’ll explore how to implement real-time logging with Kibana in a Node.js application.

Prerequisites

Before we get started, make sure you have the following prerequisites set up:

1. Node.js installed on your system.
2. Elasticsearch and Kibana running locally or on a remote server.

Setting up the logging infrastructure

To implement real-time logging with Kibana, we use the Winston logging library in Node.js. Winston provides flexible and configurable logging capabilities, allowing us to send logs directly to Elasticsearch, which can then be visualized in Kibana.

To begin, let’s install Winston and its Elasticsearch transport:

npm install winston winston-elasticsearch

Next, create a logger.js file and add the following code to set up the logger with the Elasticsearch transport:

const winston = require('winston');
require('winston-elasticsearch');

const logger = winston.createLogger({
  level: 'info',
  format: winston.format.json(),
  transports: [
    new winston.transports.Elasticsearch({
      index: 'logs',
      node: 'http://localhost:9200'
    })
  ]
});

module.exports = logger;

In the above code, we create a logger instance using the winston.createLogger method and specify the log level and log format. We then add the Elasticsearch transport by instantiating winston.transports.Elasticsearch and providing the Elasticsearch endpoint.

Logging in your application

With the logging infrastructure in place, you can now start adding logs to your Node.js application. For example, if you have an Express.js application, you can add the logger middleware as follows:

const express = require('express');
const logger = require('./logger');

const app = express();

// Add the logger middleware
app.use((req, res, next) => {
  logger.info(`[${req.method}] ${req.path}`);
  next();
});

// Other routes and middleware

app.listen(3000, () => {
  console.log('Server started on port 3000');
});

In the above code, we require the logger module we defined earlier and use it in our middleware function to log the HTTP method and path for each incoming request. Modify the logger configuration and log format as per your requirements.

Visualizing logs in Kibana

Once your application is up and running with the logging infrastructure integrated, you can navigate to the Kibana web interface and start visualizing your logs.

1. Open your browser and go to http://localhost:5601 (or the URL of your remote Kibana instance).
2. Click on the Discover tab in the left-hand menu.
3. Create an index pattern by entering the index name you specified in the logger configuration (e.g., logs) and follow the prompts.
4. Once your index pattern is created, you can start exploring and visualizing your logs using the various features provided by Kibana, such as search queries, filters, and visualizations.

#Conclusion

In this blog post, we learned how to implement real-time logging with Kibana in a Node.js application using the Winston logging library. We set up the logger with the Elasticsearch transport to send logs directly to Elasticsearch. Finally, we explored how to visualize the logs in real-time using the Kibana web interface.

By leveraging real-time logging with Kibana, you can gain valuable insights into the behavior of your Node.js application, monitor its performance, and quickly identify and resolve issues.