Clearing Docker logs is important for maintaining the performance and reliability of your containerized applications. Docker logs can grow rapidly, consuming significant amounts of disk space. If not cleared regularly, this can lead to disk space exhaustion, causing applications to fail or the host system to become unresponsive.

This guide discusses Docker logs, methods for clearing them, and how they can be monitored.

Understanding Docker Logs

Docker logs are the output generated by applications running inside Docker containers, capturing both the standard output (stdout) and standard error (stderr). These logs provide visibility into application behavior, performance metrics, and error messages, essential for troubleshooting, monitoring, and operational insights.

Docker logs
Docker logs

Methods to Clear Docker Logs

There are different methods that can be used in clearing Docker logs which will be discussed in this section.

Note: It's recommended to stop running containers before clearing logs. This prevents potential issues and ensures no new logs are generated during the process.

Run the below command to stop your running container(s):

docker stop <container_id>

1. Truncating Log Files

Truncating log files is essential for reducing the storage space occupied by logs. This process becomes necessary when log files grow too large, potentially impacting the system or the application running on it. Truncation provides a quick fix for urgent disk space recovery or emergency maintenance.

To truncate log files from a Docker container, use the following command:

truncate -s 0 /var/lib/docker/containers/<container_id>/<container_id>-json.log

Replace <container_id> with the ID of the actual container.

2. Deleting Log Files

Deleting log files involves removing them entirely from the system. While this method helps manage disk space, it comes with inherent risks and considerations. Deleting a log file removes all stored log data, so it's crucial to ensure compliance with any regulatory or compliance requirements regarding log retention.

To delete a log file from a Docker container, use the following command:

rm /var/lib/docker/containers/<container_id>/<container_id>-json.log

Once the log file is deleted, restart Docker to ensure uninterrupted operation:

sudo systemctl restart docker

3. Using Docker's Built-In Logging Drivers

Docker’s built-in logging drivers allow you to configure how logs are collected, stored, and forwarded. By default, Docker uses the json-file logging driver, which stores logs in JSON format on the local disk.

Here are some other available logging drivers:

  • syslog: Sends logs to a syslog server.
  • journald: Integrates with the systemd journal.
  • awslogs: Sends logs to Amazon CloudWatch.

The json-file logging driver supports log rotation, automatically clearing old logs by limiting file size and count. You can configure the logging driver globally by editing the Docker daemon configuration file:

sudo nano /etc/docker/daemon.json

Add or update the logging driver configuration to include log rotation options:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

After making changes, restart the Docker daemon to apply them:

sudo systemctl restart docker

Alternatively, you can apply these settings at the container level using the docker run command:

docker run --log-driver=json-file --log-opt max-size=10m --log-opt max-file=3 -d <my-container-id>

4. Using Logrotate

Logrotate is a powerful tool used to manage the log files generated by various services, including Docker. It assists in deleting, rotating, and compressing old log files to keep them from taking up too much disk space. Logrotate is typically managed by a daily cron job that executes the log rotation based on the defined configurations.

To clear Docker logs using Logrotate, you need to create a Logrotate configuration file specifically for Docker logs in the /etc/logrotate.d/ directory. This file will define how Logrotate should handle the log rotation for Docker containers.

sudo nano /etc/logrotate.d/docker-container-logs

Add the below configuration in the file:

/var/lib/docker/containers/*/*.log {
    rotate 7
    daily
    compress
    missingok
    delaycompress
    copytruncate
    notifempty
    create 0640 root root
}

The above configuration instructs Logrotate on how to rotate Docker container logs. Here is a breakdown:

  • /var/lib/docker/containers/*/*.log: This is the file pattern that matches all Docker container log files. The asterisks (*) act as wildcards, meaning this rule applies to all log files within any subdirectory of /var/lib/docker/containers/.
  • rotate 7: Keep 7 rotations of the log files. Older logs will be removed after this number is reached.
  • daily: Rotate logs daily.
  • compress: Compress rotated logs using gzip.
  • missingok: Don't throw an error if a log file doesn't exist.
  • delaycompress: Delay compression until the next rotation cycle. This helps if an application is still writing to the original log file.
  • copytruncate: Create a copy of the current log file, then empty the original. This is useful to avoid interrupting applications that are actively writing to the log.
  • notifempty: Don't rotate the log if it's empty.
  • create 0640 root root: Create new log files with the specified permissions (owner: root, group: root, permissions: 640 – read/write for owner, read for group).

You can customize the Logrotate configuration further to suit your needs.

/var/lib/docker/containers/*/*.log {
    rotate 7
    daily
    compress
    missingok
    delaycompress
    copytruncate
    notifempty
    create 0640 root root
    size 100M
    maxage 30
    dateext
}
  • size: Rotate the log file if it grows beyond a specified size (e.g., size 100M).
  • maxage: Remove rotated logs older than a specified number of days (e.g., maxage 30).
  • dateext: Append the date to the rotated log file name.

5. Automating Log Rotation using Docker Compose

If you are running your Docker containers using Docker Compose, you can directly configure log rotation in your Docker Compose file. This requires a specification for the logging driver and options such as max-size and max-file to control log rotation in each container.

An example repository has been provided with a Docker Compose configuration. Clone the repository and open the folder in your code editor.

git clone https://github.com/SigNoz/opentelemetry-nodejs-example.git

Modify the docker-compose.yml file by replacing its current contents with the provided configuration:

version: "3.8"
services:
  mongodb:
    image: mongo:latest
    ports:
      - "27017:27017"
    volumes:
      - mongo-data:/data/db
    networks:
      - myapp-network
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "5"

  order:
    build: ./order-service
    ports:
      - "3001:3001"
    environment:
      - OTEL_TRACES_EXPORTER=otlp
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
      - OTEL_NODE_RESOURCE_DETECTORS=env,host,os
      - OTEL_SERVICE_NAME=order-service
      - NODE_OPTIONS=--require @opentelemetry/auto-instrumentations-node/register
    depends_on:
      - mongodb
      - otel-collector
    networks:
      - myapp-network
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "5"

  payment:
    build: ./payment-service
    ports:
      - "3002:3002"
    environment:
      - OTEL_TRACES_EXPORTER=otlp
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
      - OTEL_NODE_RESOURCE_DETECTORS=env,host,os
      - OTEL_SERVICE_NAME=payment-service
      - NODE_OPTIONS=--require @opentelemetry/auto-instrumentations-node/register
    depends_on:
      - mongodb
      - otel-collector
    networks:
      - myapp-network
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "5"

  product:
    build: ./product-service
    ports:
      - "3003:3003"
    environment:
      - OTEL_TRACES_EXPORTER=otlp
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
      - OTEL_NODE_RESOURCE_DETECTORS=env,host,os
      - OTEL_SERVICE_NAME=product-service
      - NODE_OPTIONS=--require @opentelemetry/auto-instrumentations-node/register
    depends_on:
      - mongodb
      - otel-collector
    networks:
      - myapp-network
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "5"

  user:
    build: ./user-service
    ports:
      - "3004:3004"
    environment:
      - OTEL_TRACES_EXPORTER=otlp
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
      - OTEL_NODE_RESOURCE_DETECTORS=env,host,os
      - OTEL_SERVICE_NAME=user-service
      - NODE_OPTIONS=--require @opentelemetry/auto-instrumentations-node/register
    depends_on:
      - mongodb
      - otel-collector
    networks:
      - myapp-network
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "5"

  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: ["--config", "/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml:ro
    ports:
      - "4317:4317"
      - "4318:4318"
      - "8888:8888"
      - "2255:2255"
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=${OTEL_EXPORTER_OTLP_ENDPOINT}
      - SIGNOZ_INGESTION_KEY=${SIGNOZ_INGESTION_KEY}
    networks:
      - myapp-network
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "5"

networks:
  myapp-network:

volumes:
  mongo-data:

From the above configuration, all the container services have been modified to use the json-file logging driver and are configured to rotate logs when each log file reaches 10MB, keeping a maximum of 5 rotated log files.

Start the containers using the below command:

docker compose up -d

If all the containers start without any error, log rotation is successfully implemented. To check the logs, you can use Docker commands to monitor logs and verify rotation:

docker logs <container-name>

Monitor your Docker Logs with SigNoz

To effectively monitor Docker logs, using an advanced observability platform like SigNoz can be highly beneficial. SigNoz is an open-source observability tool that provides end-to-end monitoring, troubleshooting, and alerting capabilities for your applications and infrastructure.

Here's how you can leverage SigNoz for monitoring Docker logs:

  1. Create a SigNoz Cloud Account

SigNoz Cloud provides a 30-day free trial period. This demo uses SigNoz Cloud, but you can choose to use the open-source version.

  1. Add .env File to the Root Project Folder

In the root directory of your project folder, create a new file named .env and paste the below into it:

OTEL_COLLECTOR_ENDPOINT=ingest.{region}.signoz.cloud:443
SIGNOZ_INGESTION_KEY=***
  • OTEL_COLLECTOR_ENDPOINT: Specifies the address of the SigNoz collector where your application will send its telemetry data.
  • SIGNOZ_INGESTION_KEY: Authenticates your application with the SigNoz collector.

Note: Replace {region} with your SigNoz region and SIGNOZ_INGESTION_KEY with your actual ingestion key.

To obtain the SigNoz ingestion key and region, navigate to the “Settings” page in your SigNoz dashboard. You will find the ingestion key and region under the “Ingestion Settings” tab.

SigNoz Ingestion Settings Page
SigNoz Ingestion Settings Page
  1. Set Up the OpenTelemetry Collector Config

An OTel Collector is a vendor-agnostic service that receives, processes, and exports telemetry data (metrics, logs, and traces) from various sources to various destinations.

In the root of your project folder, create a file otel-collector-config.yaml and paste the below contents in it:

receivers:
  tcplog/docker:
    listen_address: "0.0.0.0:2255"
    operators:
      - type: regex_parser
        regex: '^<([0-9]+)>[0-9]+ (?P<timestamp>[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}(\\.[0-9]+)?([zZ]|([\\+-])([01]\\d|2[0-3]):?([0-5]\\d)?)?) (?P<container_id>\\S+) (?P<container_name>\\S+) [0-9]+ - -( (?P<body>.*))?'
        timestamp:
          parse_from: attributes.timestamp
          layout: '%Y-%m-%dT%H:%M:%S.%LZ'
      - type: move
        from: attributes["body"]
        to: body
      - type: remove
        field: attributes.timestamp
      - type: filter
        id: signoz_logs_filter
        expr: 'attributes.container_name matches "^signoz-(logspout|frontend|alertmanager|query-service|otel-collector|otel-collector-metrics|clickhouse|zookeeper)"'

processors:
  batch:
    send_batch_size: 10000
    send_batch_max_size: 11000
    timeout: 10s

exporters:
  otlp:
    endpoint: ${env:OTEL_COLLECOTR_ENDPOINT}
    tls:
      insecure: false
    headers:
      "signoz-ingestion-key": ${env:SIGNOZ_INGESTION_KEY}

service:
  pipelines:
    logs:
      receivers: [tcplog/docker]
      processors: [batch]
      exporters: [otlp]

The above configuration sets up the OpenTelemetry Collector to listen on 0.0.0.0:2255 for incoming Docker logs over TCP and to parse and filter the logs using a regex parser. The logs are then processed in batches for optimized handling and exported securely via OTLP to the OTEL_COLLECTOR_ENDPOINT specified in your .env file.

  1. Add Services to the Docker Compose File

In your Docker Compose file, add the below configuration:

  otel-collector:
    image: signoz/signoz-otel-collector:${OTELCOL_TAG:-0.79.7}
    container_name: signoz-otel-collector
    command:
      [
        "--config=/etc/otel-collector-config.yaml",
        "--feature-gates=-pkg.translator.prometheus.NormalizeName"
      ]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    environment:
      - OTEL_RESOURCE_ATTRIBUTES=host.name=signoz-host,os.type=linux
      - DOCKER_MULTI_NODE_CLUSTER=false
      - LOW_CARDINAL_EXCEPTION_GROUPING=false
    env_file:
      - ./.env
    ports:
      - "4317:4317" # OTLP gRPC receiver
      - "4318:4318" # OTLP HTTP receiver
    restart: on-failure

  logspout:
    image: "gliderlabs/logspout:v3.2.14"
    container_name: signoz-logspout
    volumes:
      - /etc/hostname:/etc/host_hostname:ro
      - /var/run/docker.sock:/var/run/docker.sock
    command: syslog+tcp://otel-collector:2255
    depends_on:
      - otel-collector
    restart: on-failure

The above Docker Compose configuration sets up two services:

  • otel-collector:
    • Uses the Signoz OpenTelemetry Collector image.
    • Mounts the otel-collector-config.yaml file into the container.
    • Uses environment variables from the .env file for configuration.
    • Exposes ports 4317 (OTLP gRPC receiver) and 4318 (OTLP HTTP receiver).
  • logspout:
    • Uses the gliderlabs/logspout image to collect logs from other Docker containers.
    • Forwards logs to the otel-collector service on port 2255.
    • Depends on the otel-collector service, ensuring it starts first so it can forward the logs to it.
  1. Start the Docker Compose file

Run the below command in your terminal to start the containers:

docker compose up -d

If it runs without any error, navigate to your SigNoz Cloud. Under the “Logs” tab, you should see the incoming logs.

Docker logs sent to SigNoz
Docker logs sent to SigNoz

Conclusion

  • Clearing Docker logs is crucial for maintaining the performance and reliability of your containerized applications.
  • There are various methods to clear Docker logs like truncating log files, deleting log files, using Docker's built-in logging drivers, and employing tools like Logrotate.
  • Each method has its use cases and considerations, allowing you to choose the best approach based on your specific requirements.
  • Using Logrotate and adding logging drivers to Docker Compose file it becomes easy to automate the whole task.
  • To manage your logs efficiently, use a centralized monitoring tool like SigNoz.

FAQ

How to Clear Docker Logs for a Container?

To clear Docker logs for a specific container, you can truncate the log file using the following command:

truncate -s 0 /var/lib/docker/containers/<container_id>/<container_id>-json.log

Replace <container_id> with the actual container ID.

How to Clear Logs of Running Docker Containers?

It's recommended to stop the running container before clearing logs to avoid potential issues. Use the following command to stop the container:

docker stop <container_id>

Then, you can truncate or delete the log file as needed.

Clearing Docker Logs on a Schedule

You can use logrotate to schedule automatic log rotation for Docker logs. Create a logrotate configuration file in /etc/logrotate.d/ and specify the rotation rules for Docker container logs.

How to Safely Remove Docker Host Logs?

To safely remove Docker host logs, stop the Docker service and then delete the log files. Restart the Docker service after deletion to ensure uninterrupted operation:

sudo systemctl stop docker
sudo rm /var/lib/docker/containers/*/*.log
sudo systemctl start docker

Clearing Logs for a Specific Container

To clear logs for a specific container, use the truncate command to reset the log file:

truncate -s 0 /var/lib/docker/containers/<container_id>/<container_id>-json.log

Make sure to replace <container_id> with the actual container ID.

Clearing Logs for All Containers

To clear logs for all containers, iterate through the log files and truncate each one:

for log in /var/lib/docker/containers/*/*.log; do truncate -s 0 "$log"; done

This will reset the logs for all containers running on the host.

Was this page helpful?