Why monitoring is important?

Monitoring is a very important aspect in the present Information Technology age. They keep us updated with the status of a running application along with other factors that can impact the performance of the application. We are dealing with so many different components and it is impossible to predict how they are going to impact the performance of an application. Monitoring helps us to keep an eye on the performance of these components and get notified before it actually impacts the performance of a running application. This way we can keep a check on any issue which can impact the business continuity.

Now the question arises that what to monitor and whether it is sufficient to get the complete idea of a system. Sometimes it is very difficult to know the root cause of any issue as it can be because of the code, database query, server issue, or any external factor. So it is very important to get data from all sources into the monitoring setup. Please refer to the below image to understand how full-stack monitoring can be configured:

Now take an example of a microservice-based application, here we can think of the following components which can impact the performance:

First of all, we want to know if all API’s are running and they are fully functional.
In this case, we can use any tool (like Elastic Heartbeat) which can keep on checking the APIs and can provide that data to a central server where we can monitor them. This way we can have the current status of all running API’s and we can also know when service was down and for how long.

Database server monitoring to keep a check on the database performance
Using database performance monitoring we can get details about user connections, DB transactions, requests per second, command rates, running threads, etc. These are very important information using which we can have an idea of how the database is performing under various scenarios. We can use Metricbeat with the database module to fetch specific database metrics.

Monitoring of application, server, and database logs to know if they are giving any error which can impact.
It is very important to read data from application, server, and database logs as they can provide us the details if there is any error. This has to be configured on each server where the application or databases are running. We need to collect these data to a central server on which we can configure the monitoring.

Application performance monitoring to get details on database queries, response time for incoming requests, external HTTP requests, errors and exceptions, etc.
There are many tools like Elastic APM which can be configured with the application in order to get the internal application metrics. Using these tools we can get the details of database queries, response time for incoming requests, external HTTP requests, errors and exceptions, etc.

The infrastructure monitoring to check if it is working fine without any issue of processes, CPU, RAM, or disk space.
Infrastructure is very vital for any running application and the performance of any application depends on the infrastructure. If there is any bottleneck in the infrastructure, the application would create issues so it is very important to monitor the infrastructure. There are many tools like Metricbeats using which we can monitor the application infrastructure. They can be installed on the servers from where they collect the metrics and send them to a central server for monitoring.

Network monitoring to check the connectivity, error or packet loss, network latency, and bandwidth utilization, etc.
There are different network monitoring tools like t-shark, Packetbeat, etc using which we can capture the network packet details. Again these tools can be configured on individual servers from where they collect the metrics and send it to a central server for monitoring.

To monitor any external dependency on the application.
We also need to set up monitoring if there is any dependency on the external system. These systems can be any external APIs, or anything else, etc, we can plan their monitoring as well. Like in the case of external APIs we can configure Heartbeat to check their availability etc.

There can be other components also which we can monitor but mainly these are the main components that we should monitor to get the complete monitoring data for any running application. These metrics can be pushed into Elasticsearch and using Kibana we can configure the dashboard. Once the full-stack monitoring system is configured, we can get all the details on a single dashboard using which we get all the details. Alerts can also be configured on different thresholds so that we can react in a proactive way to any issues.

This blog is on introduction to monitoring and it covers how important is the monitoring. It helps us to figure out any issue in the system which is the most tedious task in debugging. I will provide details on the different monitoring components in future blogs.

In case of any doubt please leave your comments. You can also follow me on Twitter: https://twitter.com/anu4udilse

If you found this article interesting, then you can explore “Mastering Kibana 6.x”, “Kibana 7 Quick Start Guide”, “Learning Kibana 7”, and “Elasticsearch 7 Quick Start Guide” books to get more insight about Elastic Stack and how we can create dashboards for key performance indicators using Kibana.

Originally published at https://bqstack.com.