Exporting SpringBoot-application metrics to AWS CloudWatch

This is an English version of my article posted here: https://habr.com/ru/company/luxoft/blog/512628/

With no doubt, it's necessary to know how your application performs in production. It's metrics, that can provide this knowledge: either technical (CPU, Memory, Disk utilization) or business ones (the number of daily orders for instance).

Instant data slice is not so representative as a certain frame of data. Here is where gathering, storing and viewing those metrics come into play.

The need for metrics will only increase if your product, your application which might seem a monolithic one for the users, actually consists of several interacting services. Moreover, if those services are hosted in a cloud. Can have a bunch of Carolina Reapers with all that in place, feeling it?

The Subject

This article will go through the practical steps of gathering Spring Boot-based applications (services) metrics and exporting them to AWS CloudWatch. This would be a step-by-step guide with details and possible problems discussion.

When talking about some practical task, one needs to understand the context. This will help you decide whether you can implement it in your environment or some adoption is needed. Here is our context for today:

• Spring Boot application with Java 8+ and Maven as a build tool,
• Docker or not - doesn't matter. Running in Docker does not require any additional step
• AWS EC2 will be our running environment. In our case, it's nothing more than just a virtual server in AWS.
• AWS CloudWatch is a monitoring control panel system for your Amazon Cloud environment.

Spring Boot

Let review Spring Boot feature that can help as. The first and most important - Spring Boot Actuator. This module allows you to look into your running application and tune it in a certain manner. For instance:

• Health-check: your overall application and individual components state.
• Logging settings and runtime changes
• Beans in the context
• Read application metrics like CPU, Memory, GC, etc.
• ...

Like most other Spring modules, Actuator is extendable. Want to go deeper? Can start from here.

Currently, the metrics feature is what we'll focus on. Actuator and metrics feature are not only extendable but pre-configured as well: there are more than a couple of dozens of metrics coming out of the box. Custom metrics can be implemented and registered. If a web module is included, a /metrics endpoint is available to view them all.

The metrics feature is implemented using a micrometer library - a vendor-agnostic facade for collecting and exposing metrics in Java applications, developed by Pivotal (part of VMware currently).

To use Actuator the following starter should be included in your pom.xml:

<dependency>
   <groupid>org.springframework.boot</groupid>
   <artifactid>spring-boot-starter-actuator</artifactid>
</dependency>

Spring Cloud...

NOTE: In the initial article Spring Boot 2.2.x was used. In this section, I will cover 2.2.x as well as 2.4.x.

Spring Boot 2.2.x - Spring Cloud

Next, we'll need a Spring Cloud module (starter), called spring-cloud-starter-aws.

Every Spring Cloud module has its own versioning, thus you should be using a BOM spring-cloud-dependencies with a certain version (release train) to ensure modules compatibility.

Alongside auto-configuration features, spring-cloud-starter-aws also adds an aws-java-sdk as a transitive dependency.

In terms of our Maven configuration, need to add/modify the dependencyManagement section:

<dependencyManagement>
...
   <dependencies>
       <dependency>
          <groupId>org.springframework.cloud</groupId>
          <artifactId>spring-cloud-dependencies</artifactId>
          <version>Hoxton.RELEASE</version>
          <type>pom</type>
          <scope>import</scope>
       </dependency>
   </dependencies>
...
</dependencyManagement>

and a dependency:

<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-aws</artifactId>
</dependency>

Spring Boot 2.4.x - Spring Cloud AWS

Spring AWS modules were moved to a separate project (io.awspring.cloud) and to utilize it, you will need to include

<dependency>
   <groupId>io.awspring.cloud</groupId>
   <artifactId>spring-cloud-aws-autoconfigure</artifactId>
   <version>2.3.0</version>
</dependency>

in your pom.xml.

Micrometer registry

We now have a micrometer as a part of  Spring Boot Actuator and aws-java-sdk. The micrometer is not able to export metrics to AWS CloudWatch out of the box. We'll need a certain implementation of MeterRegistry - a micrometer abstraction for collecting metrics. By default, a SimpleMeterRegistry is used, which stores all the metrics in memory. Needed implementation is located in micrometer-registry-cloudwatch:

<dependency>
   <groupId>io.micrometer</groupId>
   <artifactId>micrometer-registry-cloudwatch</artifactId>
   <version>1.6.4</version>
</dependency>

POM.xml

The resulting pom.xml will be:

• https://github.com/MrArtemAA/blog-demos/blob/deab2064d6a76da76b817cdd81ea91ff854920cd/export-metrics-to-cloudwatch/pom.xml - for Spring Boot 2.2.x
• https://github.com/MrArtemAA/blog-demos/blob/master/export-metrics-to-cloudwatch/pom.xml - for Spring Boot 2.4.x

app.properties

Application properties do play a big role in our case.

1. management.metrics.export.cloudwatch.namespace: a namespace must be defined, for CloudWatch to save those metrics. As the metric does not contain any information regarding the application/service instance, the namespace is defining it. Otherwise, metrics from different instances will be mixed up,
2. management.metrics.export.cloudwatch.batch-size: you must set it to 20 explicitly. The metrics are sent to Amazon CloudWatch async in batches of 20 per request - it's an AWS limitation. This is NOT NEEDED to set it explicitly for Spring Boot 2.4.x and spring-cloud-aws-autoconfigure,
3. cloud.aws.stack.auto=false: if not using AWS CloudFormation stack, you'll need to switch auto-detection off, as it's true by default. This property is responsible for the service to auto-detect stack name for cloud configuration. While using EC2 as an ordinary virtual machine, there is no such information.

It's important to understand, that any information AWS SDK will try to auto-detect will be taken from EC2 metadata. There is a service endpoint for this.

Houston, we might have a problem here...

Let's take a look at some details and possible problems

Explicit batch-size

If you're using Spring Boot 2.2.x and Spring Cloud, you have to set management.metrics.export.cloudwatch.batch-size=20 as described above. But why? Couldn't it be resolved on the library's level? Indeed, in micrometer-registry-cloudwatch, there is an interface CloudWatchConfig with the default method, where the value is checked and an exception thrown in case of being greater than 20. But if you will take a look at the org.springframework.cloud.aws.autoconfigure.metrics.CloudWatchExportAutoConfiguration, then the configuration is made through org.springframework.cloud.aws.autoconfigure.metrics.CloudWatchPropertiesConfigAdapter

@Bean
@ConditionalOnMissingBean
public CloudWatchConfig cloudWatchConfig(CloudWatchProperties cloudWatchProperties) {
  return new CloudWatchPropertiesConfigAdapter(cloudWatchProperties);
}

And the adapter overrides batchSize():

@Override
public int batchSize() {
  return get(CloudWatchProperties::getBatchSize, CloudWatchConfig.super::batchSize);
}

Meaning that first, the value will be taken from CloudWatchPropeties. And it does not contain any validation so the default value of 10000 will be set.

AWS requests

An application/service can not simply make requests to Amazon services. The requests must contain credentials and be authorized. There is a credentials provider chain for this in AWS SDK and an Instance Profile among them. It allows obtaining credentials from the EC2 metadata. For this to work, check EC2 was assigned with an appropriate role.

This role needs to grant access to make requests to CloudWatch and be available for EC2. A role can be assigned to the EC2 instance either at creation time or afterward. Role assignment happens on the fly.

Turn metrics off

For local development or test environment, it might be an overhead to export metrics to CloudWatch. To turn the export off, set management.metrics.export.cloudwatch.enabled=false. This will just turn off the export not gathering. If you have web module dependency, you will still see them through /metrics endpoint.

Micrometer gathers and exposes lots of different metrics. You can turn some of them off, individually or by category. For instance, by setting management.metrics.enable.some.metric=false property, all metrics, starting with some.metric will be turned off and NOT gathered at all.

Launch out of AWS

If you try to launch the service with minimin required setup out of AWS infrastructure, the next surprise awaits. For the application to start, an AWS region must be specified or obtained automatically. We have cloud.aws.region.auto property (similarly to cloud.aws.stack.auto), but setting it to false would not help. Need to explicitly set cloud.aws.region.static=us-east-1 (or any other AWS region).

Additionally, we need to be authenticated against AWS to send metrics or make any other requests to AWS. You can pass the credentials through application properties or environment variables

cloud.aws.credentials.access-key=YOUR_ACCESS_KEY
cloud.aws.credentials.secret-key=YOUR_SECRET_KEY

Conclusion

You may have noticed that to make it work and start exporting your metrics to CloudWatch, a couple of dependencies and properties should be added. That's where the details come into play...

Libraries and frameworks like Spring and AWS SDK are trying to make our lives easier and do hard work. On the other hand, any sidesteps could lead to stack traces, attempt to understand why the metrics are not being exported, why the application is not even starting and how to deal with it. The previous section and some EC2/CloudWatch details across the article are dedicated to all of that and will facilitate your understanding.

If you are using AWS infrastructure, it feels quite natural to utilize CloudWatch capabilities for your needs.

No doubt, the metics are the eyes and ears of our application. Nevertheless, one should understand how the metrics are gathers, accumulated, and aggregated. You will definitively face it during incidents. When talking about the micrometer library, you should refer to the official documentation for different Meters.

Links

Knowledge sharing is the key! Here are the links to the articles I used as well and find useful:

• Spring Boot: Metrics With Micrometer and AWS CloudWatch
• Spring Cloud. Using Amazon Web Services. Spring Boot auto-configuration
• Spring in Action 5, Craig Walls, Manning

You can find this project on GitHub

Share, comment, cheers!