Back to articles
AI & Cloud

Configuring Vector.dev: Solution for Collecting and Optimizing Gigabyte Docker Logs to Grafana Loki

Guide to configuring Vector.dev for collecting, processing, and forwarding Gigabyte-scale Docker logs to Grafana Loki with high performance, cost optimization, and stability. This...

DPTCloud Engineering4 min read
Configuring Vector.dev: Solution for Collecting and Optimizing Gigabyte Docker Logs to Grafana Loki

Guide to configuring Vector.dev for collecting, processing, and forwarding Gigabyte-scale Docker logs to Grafana Loki with high performance, cost optimization, and stability. This solution enables efficient log management in containerized environments, filtering unnecessary logs, enriching metadata, and reducing bandwidth through compression and batching.

Overall Solution: Vector.dev + Grafana Loki

Vector.dev is an open-source, high-performance tool for collecting, transforming, and forwarding logs, metrics, and events. It supports multiple input sources (Docker, file logs, syslog) and output destinations (Grafana Loki, Elasticsearch, Kafka, etc.). Grafana Loki is a distributed log aggregation system optimized for storing and querying logs based on metadata, ideal for containerized applications.

This solution enables:

  • Comprehensive log collection from Docker containers.
  • Log processing (filtering, transformation, metadata enrichment) before sending to Loki.
  • Bandwidth and storage optimization via compression, filtering, and log batching.
  • Low-latency, reliable log forwarding to Grafana Loki.
  • Reduced storage costs and improved log query performance.

Requirements and Prerequisites

To deploy this solution, you need:

Configuring Vector.dev

Vector.dev uses YAML-based configuration. Create a `vector.yaml` file with the following content:

# vector.yaml
sources:
  # Collect logs from Docker
  docker_logs:
    type: docker
    # Specify Docker socket path
    socket: /var/run/docker.sock
    # Filter containers to collect logs from
    container_names:
      - "your-app-container"
    # Optional: add metadata
    metadata:
      container_name: true
      image_name: true

transforms:
  # Filter out unnecessary logs (e.g., debug logs)
  filter_logs:
    type: filter
    inputs:
      - docker_logs
    condition: "!(log.contains('DEBUG') || log.contains('INFO'))"

  # Add metadata to logs (e.g., environment, tenant)
  add_metadata:
    type: add_fields
    inputs:
      - filter_logs
    fields:
      environment: "production"
      tenant: "your-tenant-id"

sinks:
  # Forward processed logs to Grafana Loki
  loki_sink:
    type: loki
    inputs:
      - add_metadata
    endpoint: http://loki:3100/loki/api/v1/push
    # Optional: compress logs
    compression: gzip
    # Optional: add labels to logs
    labels:
      job: "docker-logs"
      service: "your-app-service"
    # Optional: batch logs to optimize traffic
    batch:
      max_size: 1048576  # 1MB
      timeout_secs: 10

Deployment and Verification

1. Run Vector.dev with the configuration:

vector --config vector.yaml

2. Check Vector status and logs:

# Check Vector status
vector status

# View Vector logs
vector logs

3. Verify logs in Grafana Loki:

  • Open Grafana and navigate to the Logs panel.
  • Select Loki as the data source.
  • Query logs using the added labels (e.g., `job="docker-logs"`, `service="your-app-service"`).

Troubleshooting and Optimization

Common Issues:

  • Vector fails to collect Docker logs: Check permissions on `/var/run/docker.sock`. Run Vector as root or add the user to the `docker` group.
  • Logs not appearing in Loki: Verify the Loki endpoint (IP, port). Check label and batch configuration.
  • High resource usage by Vector: Optimize batch size and timeout. Use filters to remove unnecessary logs.

Performance Optimization:

  • Use compression (gzip) to reduce bandwidth usage.
  • Analyze logs to identify types that can be filtered (e.g., debug, info).
  • Adjust batch size and timeout based on log load.
  • Ensure Vector runs on a machine with sufficient RAM and CPU.

References

This solution enables efficient handling of Gigabyte-scale Docker logs, ensuring logs are collected comprehensively, processed efficiently, and stored optimally on Grafana Loki—ideal for modern containerized log monitoring and analysis.

DPTCloud Tier III Infrastructure

High-RAM Cloud VPS Plans Optimized for AI, Databases & Cache

Enterprise RAID 10 NVMe SSDs, CMC & FPT District 9 uplinks with Anycast Anti-DDoS and 99.99% SLA.

View 22 VPS Plans
ECO · vn-02Recommended

VPS-4E

297.000 ₫ / mo

10% VAT included

  • CPU: 4 vCPU
  • RAM: 8 GB
  • SSD: 50 GB NVMe
ECO · vn-02

VPS-5E

614.900 ₫ / mo

10% VAT included

  • CPU: 8 vCPU
  • RAM: 16 GB
  • SSD: 100 GB NVMe
ADVANCED · vn-03Recommended

VPS-4T

548.900 ₫ / mo

10% VAT included

  • CPU: 4 vCPU
  • RAM: 8 GB
  • SSD: 50 GB NVMe

Related Articles

View all
ZaloZalo SupportDiscord0384 753 478