Centralized Logging Management with ELK Stack
ELK Stack is a widely adopted solution for centralized logging in DevOps environments. Comprising Elasticsearch, Logstash, and Kibana, it enables the collection, processing,...
ELK Stack is a widely adopted solution for centralized logging in DevOps environments. Comprising Elasticsearch, Logstash, and Kibana, it enables the collection, processing, storage, and visualization of log data from diverse sources. This guide details the deployment and configuration of ELK Stack on Cloud VPS or Dedicated Servers to establish an effective log monitoring system, supporting real-time application behavior analysis and troubleshooting.
Overview of ELK Stack
ELK Stack is an open-source suite developed by Elastic, consisting of three core components: Elasticsearch, Logstash, and Kibana. Elasticsearch serves as a real-time search and analytics engine, Logstash processes and transforms log data, and Kibana provides a visual interface for querying and presenting data. Deploying ELK Stack centralizes logs from servers, applications, and services, reducing data loss risks and improving monitoring efficiency.
ELK Stack is commonly deployed on high-performance platforms such as High-Performance Dedicated Servers for Production Clusters or NVMe Cloud VPS for Docker, Nginx & Databases. With appropriate configuration, the system can handle millions of log entries daily and enable fast query responses. Enterprises use ELK Stack to monitor system failures, analyze user behavior, ensure compliance, and optimize application performance.
Environment Requirements and Preparation
To deploy ELK Stack, a server with sufficient resources is required. Below are minimum requirements:
- OS: Ubuntu 20.04 LTS or CentOS 8
- RAM: At least 8GB (recommended 16GB for large log volumes)
- Storage: NVMe SSD, minimum 100GB (depending on log volume)
- Java 11 or higher (required for Elasticsearch)
Before installation, ensure the server has internet connectivity and update the system:
sudo apt update && sudo apt upgrade -y
sudo apt install openjdk-11-jdk -yFor High-Performance Dedicated Servers, use distributed storage or RAID to ensure data reliability. Additionally, open necessary network ports such as 9200 (Elasticsearch), 5044 (Logstash), and 5601 (Kibana).
Deploying ELK Stack
The deployment process includes installing each component and configuring their connections.
Installing Elasticsearch
Elasticsearch is the data processing core. Install it by adding the Elastic repository and installing the appropriate version:
curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elastic.gpg
echo "deb [signed-by=/usr/share/keyrings/elastic.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt update
sudo apt install elasticsearch -yAfter installation, edit the configuration file `/etc/elasticsearch/elasticsearch.yml`:
cluster.name: elasticsearch-cluster
node.name: elasticsearch-node-1
network.host: 0.0.0.0
http.port: 9200
discovery.type: single-nodeEnable and start the service:
sudo systemctl enable elasticsearch
sudo systemctl start elasticsearchInstalling Logstash
Logstash is used to collect and process logs. Install it as follows:
sudo apt install logstash -yConfigure `/etc/logstash/conf.d/01-inputs.conf` to receive logs from sources:
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "logs-%{+YYYY.MM.dd}"
}
}Start Logstash:
sudo systemctl enable logstash
sudo systemctl start logstashInstalling Kibana
Kibana provides the visualization interface. Install it with:
sudo apt install kibana -yModify `/etc/kibana/kibana.yml`:
server.port: 5601
server.hosts: ["0.0.0.0"]
elasticsearch.hosts: ["http://localhost:9200"]Enable and start the service:
sudo systemctl enable kibana
sudo systemctl start kibanaAnalyzing and Monitoring Logs
Once ELK Stack is operational, access Kibana via a browser at http://. Users can create dashboards, query data using KQL (Kibana Query Language), and set up alerts for incidents.
For performance monitoring, use plugins like Elastic APM or integrate with tools such as CrowdSec to detect suspicious activities from logs.
Verification and Troubleshooting
To verify ELK Stack is functioning correctly:
- Run Elasticsearch test:
curl -X GET "http://localhost:9200/" - Check Logstash status:
sudo systemctl status logstash - Access Kibana:
http://localhost:5601
If issues arise, check logs in /var/log/elasticsearch/, /var/log/logstash/, and /var/log/kibana/. Common errors include network configuration, insufficient RAM, or incompatible Java versions.
References
Recommended NVMe Cloud VPS Plans to Deploy This Setup
Enterprise RAID 10 NVMe SSDs, CMC & FPT District 9 uplinks with Anycast Anti-DDoS and 99.99% SLA.
VPS-3E
10% VAT included
- CPU: 2 vCPU
- RAM: 4 GB
- SSD: 50 GB NVMe
