Blog

Application Logging with Watchtower and CloudWatch Logs

Centralized logging systems are vital in cloud-based architectures due to their inherent ephemeral nature. Whether using auto-scaling instances or serverless computing resources ...

by Nick Whitt

Centralized logging systems are vital in cloud-based architectures due to their inherent ephemeral nature. Whether using auto-scaling instances or serverless computing resources, event streams must persist beyond the temporary lifecycle of the resource. Many third-party solutions exist to solve this problem, including Fuentd, Splunk, and Papertrail; but we’re going to focus on Watchtower. Watchtower is a tool to enable log collection and storage in AWS CloudWatch Logs.

Loggers

Python’s native Loggers allow for a variety of Handlers to support dispatching (aka shipping) events to a specific destination. Typically, these destinations are within the compute environment, such as a local file or a memory buffer, but they could be on a remote server using Syslog. With the Logging basicConfig factory, we can easily generate a root logger:

$ python
>>> import logging
>>> logging.basicConfig(format='%(asctime)s %(message)s', handlers=[logging.StreamHandler()])
>>> logging.error('foo')
2020-11-05 11:47:44,390 foo

Many aggregator services, such as Fluent Bit and Logstash, work by “watching” system files and forwarding events to an external document store. While this makes the migration from traditional to cloud-based servers easier, it requires additional memory resources and configuration beyond those of your application.

CloudWatch Logs

AWS CloudWatch is directly integrated with many AWS services to provide real-time monitors of system metrics. An additional CloudWatch Agent can be installed on EC2 instances to provide log aggregator services as described above. This solves the problem of data persistence, but still requires a lot of external configuration to ensure proper logging streams and filters exist.

Enter Watchtower, “a lightweight adapter between the Python logging system and CloudWatch Logs.” With the CloudWatchLogHandler() Handler, Watchtower will not just forward your events, but directly send them to CloudWatch Logs using the boto3 library you are likely already using.

$ python
>>> import logging, watchtower
>>> logging.basicConfig(format='%(asctime)s %(message)s', handlers=[watchtower.CloudWatchLogHandler()])
>>> logging.error('foo')
2020-11-05 11:47:44,390 foo

$ aws logs get-log-events --log-group-name watchtower --log-stream-name 20201105
{
    "nextForwardToken": "f/31961209122447488583055879464742346735121166569214640130",
    "events": [
        {
            "ingestionTime": 1604681353115,
            "timestamp": 1604681264390,
            "message": "foo"
        }
    ],
    "nextBackwardToken": "b/31961209122358285602261756944988674324553373268216709120"
}

To integrate Watchtower with the popular Django framework, you can define a new handler for your Loggers to use. This allows you to continue using Loggers exactly as you have, without the need to manage additional applications and/or configurations.

LOGGING = {
  ...
  "handlers": {
    ...
    "watchtower": {
      "level": "INFO",
      "class": "watchtower.CloudWatchLogHandler",
      "log_group": "AppName",
      "stream_name": "StreamName",
    },
  }
  ...
}

Lambda Functions

Lambda functions will automatically store events from stdout or stderr in CloudWatch Logs. This can be invaluable for function-specific details such as billed duration, memory usage, and X-Ray tracing. However, it does not easily integrate itself with Python’s logging.Handler objects. All events are reported to a single log group (by function), with an individual log stream for each function instance. Watchtower will not prevent any of this, but rather adds to this the ability to group events by application or module logic, per Handler definition.

With the following dictionary-based configuration, the root Logger will use both a console and watchtower Handler, ensuring all events are sent to both log groups: the default via stdout, and an application-specific stream defined below.

version: 1
formatters:
  simple:
    format: "%(asctime)s %(message)s"
handlers:
  console:
    (): logging.StreamHandler
    level: DEBUG
    formatter: simple
    stream: sys.stdout
  watchtower:
    (): watchtower.CloudWatchLogHandler
    formatter: simple
    level: INFO
    log_group: "AppName"
    stream_name: "StreamName"
loggers:
  root:
    handlers: [console, watchtower]

The JBS Quick Launch Lab

Free Qualified Assessment

Quantify what it will take to implement your next big idea!

Our assessment session will deliver tangible timelines, costs, high-level requirements, and recommend architectures that will work best. Let JBS prove to you and your team why over 24 years of experience matters.

Get Your Assessment