Metrics
Another important aspect of observability and monitoring is the collection of metrics data, for example, to identify patterns and understand trends, and to make sure SLAs are met.
ADS collects metrics for requests according to the following key metrics:
Rate - the number of access requests ADS is serving
Errors - the number of failed access requests
Duration - distribution of the amount of time each access request takes
Rate of successful evaluations - the rate of successful access requests that evaluated to Permit, Deny, Indeterminate, or NotApplicable, respectively
Pull or push
There are two main models for how the communication of metrics data is orchestrated between the monitored application and the metrics backend, based on either a client pulling or a server pushing the metrics data.
ADS is compatible with both models and currently supports the following monitoring systems:
Prometheus (pull)
Azure Monitor (push)
InfluxDB (push)
Several monitoring systems can be used concurrently with ADS.
Pull
In this model, the metrics backend pulls data from the application every T time units (usually the time unit is "seconds"). This action is also referred to as polling or scraping. This is done by having the application expose an HTTP endpoint, which returns the current value of each metric without doing any calculation.
The polling period is configured in the metrics backend by the operator. It should be set to a value that yields enough data to satisfy monitoring needs and the ability to draw conclusions, while not negatively affecting the performance of the primary operations of the application. This may require some tuning and testing by the operator.
Push
In this model, the metrics backend waits for metrics data to be pushed (or sent) to it, at a time set by the application, that is, the monitored application is configured to send metrics data to the metrics backend every T time units (usually the time unit is "seconds").
This means that the metrics library used and the polling period is configured in the monitored application by the operator. As with the pull model, the polling period should be set to a value that yields enough data to satisfy monitoring needs and the ability to draw conclusions, while not negatively affecting the performance of the primary operations of the application. This may require some tuning and testing by the operator.
Metrics data
The metrics data provided by ADS is presented in the form of counters and timers collated into an output, ready to be accessed by the metrics backend. The following example shows the plain text format used for Prometheus.
# HELP decisions_total
# TYPE decisions_total counter
decisions_total{type="Permit",} 5.0
decisions_total{type="Indeterminate",} 0.0
decisions_total{type="Deny",} 1.0
decisions_total{type="NotApplicable",} 0.0
# HELP successful_requests_total
# TYPE successful_requests_total counter
successful_requests_total 8.0
# HELP error_requests_total
# TYPE error_requests_total counter
error_requests_total 1.0
# HELP duration_info_seconds_max
# TYPE duration_info_seconds_max gauge
duration_info_seconds_max 0.014
# HELP duration_info_seconds
# TYPE duration_info_seconds summary
duration_info_seconds{quantile="0.5",} 0.005210112
duration_info_seconds{quantile="0.75",} 0.014123008
duration_info_seconds{quantile="0.9",} 0.014123008
duration_info_seconds{quantile="0.99",} 0.014123008
duration_info_seconds{quantile="0.999",} 0.014123008
duration_info_seconds{quantile="1.0",} 0.014123008
duration_info_seconds_count 9.0
duration_info_seconds_sum 0.244
duration_info_seconds_bucket{le="0.001",} 182.0
duration_info_seconds_bucket{le="0.001048576",} 182.0
...
duration_info_seconds_bucket{le="30.0",} 1200.0
duration_info_seconds_bucket{le="+Inf",} 1200.0
Sample metrics output for Prometheus
Looking at the output per type, it can be broken down into the following sections:
# HELP successful_requests_total
# TYPE successful_requests_total counter
successful_requests_total 8.0
The number of successful access requests.
# HELP error_requests_total
# TYPE error_requests_total counter
error_requests_total 1.0
The number of errors.
# HELP duration_info_seconds_max
# TYPE duration_info_seconds_max gauge
duration_info_seconds_max 0.014
# HELP duration_info_seconds
# TYPE duration_info_seconds summary
duration_info_seconds{quantile="0.5",} 0.005210112
duration_info_seconds{quantile="0.75",} 0.014123008
duration_info_seconds{quantile="0.9",} 0.014123008
duration_info_seconds{quantile="0.99",} 0.014123008
duration_info_seconds{quantile="0.999",} 0.014123008
duration_info_seconds{quantile="1.0",} 0.014123008
duration_info_seconds_count 9.0
duration_info_seconds_sum 0.244
duration_info_seconds_bucket{le="0.001",} 182.0
duration_info_seconds_bucket{le="0.001048576",} 182.0
...
duration_info_seconds_bucket{le="30.0",} 1200.0
duration_info_seconds_bucket{le="+Inf",} 1200.0
The duration distribution for access requests. (Prometheus uses several buckets with histogram percentiles and for reasons of space the list is abbreviated.)
# HELP decisions_total
# TYPE decisions_total counter
decisions_total{type="Permit",} 5.0
decisions_total{type="Indeterminate",} 0.0
decisions_total{type="Deny",} 1.0
decisions_total{type="NotApplicable",} 0.0
The rate of successful access requests that evaluated to Permit, Deny, Indeterminate, or NotApplicable, respectively.
Note: Only requests that result in single decisions are included in the data for decision values.
To access the metrics data, you need to set up a configuration property in the deployment configuration file that includes the configuration of the metrics backend according to the chosen model to receive the data for visualization and/or further processing.
Note: In this implementation, metrics are collected for requests served from the REST endpoint of ADS. No extra features are implemented for the legacy SOAP endpoint. Any metrics data shown for the SOAP endpoint would be information produced by default by Dropwizard's Metrics library.
Shared configuration parameters for metrics
Statistics emanating from a timer like max, percentiles, and histogram counts decay over time to give greater weight to recent samples. To manage the decay behavior, ADS internally uses a ring buffer (an array with a pointer to a particular element) to keep track of the values for maximums and percentiles.
This functionality can be controlled by two optional parameters that are shared by all monitoring systems used for metrics processing. (As mentioned above, several monitoring systems can be used concurrently.)
- Open the deployment configuration file in a text editor.
- Add the property
metricsBackends:. - Add the subproperty
parameters:. - Add the parameters
decay:andbufferLength:with suitable values.
metricsBackends:
parameters:
decay: 2 minutes
bufferLength: 3
Shared configuration parameters for metrics
| Parameter | Default value | Possible values | Description |
|---|---|---|---|
| decay | 2 minutes | <a valid duration> minimum value 1 second | The decay parameter refers to how often the array pointer is moved to the next element of the ring buffer. The duration must be expressed as a positive integer and a time unit. NOTE: To avoid undersampling, Axiomatics strongly recommends using a value that is two times higher than the value used for the step parameter in the configuration of the metrics backend. For example, if the step is equal to 1 minute, the decay should be equal to 2 minutes. Axiomatics also recommends determining the value for the step parameter first and then adjusting the decay value accordingly. |
| bufferLength | 3 | <a positive integer> | The bufferLength parameter refers to the size of the ring buffer. NOTE: Axiomatics does not recommend using a value as low as 1. |
Shared parameters for metrics
Important: These parameters are optional. However, if added, the value for a parameter cannot be empty or null. That would be an invalid entry and the system would not be initialized. If the parameters are not included in the configuration, their default values will be used.
Configuration for Prometheus
To set up ADS for use with Prometheus, you need to add a property for Prometheus and at least one parameter in the deployment configuration file.
- Open the deployment configuration file in a text editor.
- Add the property
metricsBackends:with shared parameters as described in the previous section (if not already present). - For a minimal Prometheus configuration, where only the mandatory parameter is used, add the property
prometheus:and the parameterenabled:with the valuetrue.
metricsBackends:
parameters:
decay: 2 minute
bufferLength: 3
prometheus:
enabled: true
Minimal Prometheus configuration
Other configuration parameters are available for Prometheus. These are optional and can be added to suit your configuration needs. The following example shows an instance using all available parameters. The table provides a description of the parameters and their possible values.
metricsBackends:
parameters:
decay: 2 minutes
bufferLength: 3
prometheus:
enabled: true
descriptions: false
histogramFlavor: VictoriaMetrics
prefix: prom
step: 1 minute
Complete Prometheus configuration
| Parameter | Man./Opt. | Default value | Possible value | Description |
|---|---|---|---|---|
| enabled | mandatory | - | true, false | his parameter enables/disables the collection of data for the Prometheus metrics backend via an administration endpoint. The endpoint is GET /admin/metrics/prometheus under the administration endpoint. |
| descriptions | optional | true | true, false | Turn this parameter on if meter descriptions should be sent to Prometheus. Turn it off to minimize the amount of data sent on each scrape. |
| histogramFlavor | optional | Prometheus | Prometheus, VictoriaMetrics | The Histogram type to use for the meters DistributionSummary and Timer. |
| prefix | optional | prometheus | <string> | This is the prefix used by the metrics library employed internally by ADS. |
| step | optional | 1 minute | <a valid duration> minimum value 1 second | This parameter refers to how often data is sampled from gauges and percentiles. The duration must be expressed as a positive integer and a time unit. NOTE: Axiomatics strongly recommends that the step interval is the same as the pull (or scrape) interval set for Prometheus, and half that of the value of the decay parameter mentioned above. Axiomatics also recommends determining the value for the step parameter first and then adjusting the decay value accordingly. |
Prometheus configuration parameters
Important: If an optional parameter is added, its value cannot be empty or null. That would be an invalid entry and the system would not be initialized. If a parameter is not included in the configuration, its default value will be used.
These instructions are only referring to steps relevant to configure ADS for use with Prometheus. For other questions regarding Prometheus, please refer to Prometheus documentation.
Prometheus endpoint
This is the administration endpoint at which ADS exposes the current values of the metrics for Prometheus to pull (or scrape). The endpoint is only available when there is a valid Prometheus configuration enabled.
GET /admin/metrics/prometheus
Configuration for Azure Monitor
Important: The option of configuring Azure Monitor as a metrics backend in ADS as described below (using the OpenTelemetry Java agent) is no longer recommended, and its use is deprecated. This custom configuration option will be removed in a future release. When Azure Monitoring Application Insights is used, the Application Insights Java agent should be used for both tracing and metrics information. See Running ADS with the Application Insights Java agent.
Azure Monitor Application Insights is a feature of Azure Monitor that is used to monitor live applications. To set up ADS for use with Azure Monitor Application Insights, you need to provide a property for Azure Monitor and at least two parameters in the deployment configuration file.
Note: If ADS is launched with the Application Insights Java agent, Axiomatics recommends that the azureMonitor configuration in the deployment configuration file is disabled, and vice versa, do not use the Application Insights Java agent when the azureMonitor configuration is enabled.
- Open the deployment configuration file in a text editor.
- Add the property
metricsBackends:with shared parameters as described in the previous section (if not already present). - For a minimal Azure Monitor configuration, where only the mandatory parameters are used, add the subproperty
azureMonitor:and the parameterenabled:with the valuetrue, and the parameterinstrumentationKey:with the value of a string that represents the Instrumentation Key.
metricsBackends:
parameters:
decay: 2 minutes
bufferLength: 3
azureMonitor:
enabled: true
instrumentationKey: <String that represents the Instrumentation Key>
Minimal Azure Monitor Application Insights configuration
Other configuration parameters are available for Azure Monitor. These are optional and can be added to suit your configuration needs. The following example shows an instance using all available parameters. The table provides a description of the parameters and their possible values.
metricsBackends:
parameters:
decay: 2 minutes
bufferLength: 3
azuremonitor:
enabled: true
instrumentationKey: <String that represents the Instrumentation Key>
prefix: azuremonitor
step: 1 minute
Complete Azure Monitor Application Insights configuration
| Parameter | Man./Opt. | Default value | Possible values | Description |
|---|---|---|---|---|
| enabled | mandatory | - | true, false | This parameter enables/disables the collection of metrics data for the Azure Monitor metrics backend (Application Insights) via the Instrumentation Key. |
| instrumentationKey | mandatory | - | <a string that represents the Instrumentation Key> | The Instrumentation Key identifies the Application Insights resource that should be associated with the metrics data sent by ADS. It is the key integration point between ADS and the Application Insights monitoring service. |
| prefix | optional | azuremonitor | <string> | This is the prefix used by the metrics library employed internally by ADS. |
| step | optional | 1 minute | <a valid duration> minimum value 1 minute | This parameter refers to how often data is sampled from gauges and percentiles. It also governs the push interval, that is, the reporting frequency. The duration must be expressed as a positive integer and a time unit. NOTE: Axiomatics strongly recommends that the step interval is half that of the value of the decay parameter mentioned above. Axiomatics also recommends determining the value for the step parameter first and then adjusting the decay value accordingly. |
Azure Monitor Application Insights configuration parameters
Important: If an optional parameter is added, its value cannot be empty or null. That would be an invalid entry and the system would not be initialized. If a parameter is not included in the configuration, its default value will be used.
These instructions are only referring to steps relevant to configure ADS for use with Azure Monitor Application Insights. For other questions regarding Azure Monitor Application Insights, please refer to Azure Monitor Application Insights documentation.
Configuration for InfluxDB
When setting up ADS and InfluxDB, you need to create an InfluxDB account to include some of the parameters needed in the deployment configuration file.
- Open the deployment configuration file in a text editor.
- Add the property
metricsBackends:with shared parameters as described in the previous section (if not already present). - For a minimal InfluxDB configuration, where only the mandatory parameters are used, add the subproperty
influx:and the parameterenabled:with the valuetrue, the parameterorg:with the value of a string that represents the destination organization for writes, the parameterbucket:with the value of a string that represents destination bucket for writes, and the parametertoken:with the value of a string that represents the authentication token for the InfluxDB API.
Note: The parameters bucket, org, and token are created by the action of creating the InfluxDB account and the corresponding values can then be entered/copied into the configuration file. (The values for org and bucket are manually defined, while the token is automatically generated.)
metricsBackends:
parameters:
decay: 2 minute
bufferLength: 3
influx:
enabled: true
org: my-organization
bucket: my-bucket
token: <String that represents the authentication token>
Minimal InfluxDB configuration
Other configuration parameters are available for InfluxDB. These are optional and can be added to suit your configuration needs. The following example shows an instance using all available parameters. The table provides a description of the parameters and their possible values.
metricsBackends:
parameters:
decay: 2 minutes
bufferLength: 3
influx:
enabled: true
org: my-organization
prefix: influx
step: 1 minute
bucket: my-bucket
token: <String that represents the authentication token>
uri: http://localhost:8086
Complete InfluxDB configuration
| Parameter | Man./Opt. | Default value | Possible values | Description |
|---|---|---|---|---|
| enabled | mandatory | - | true, false | This enables/disables the InfluxDB time series platform that collects, stores, processes and visualizes metrics and events. |
| prefix | optional | influx | <string> | This is the prefix used by the metrics library employed internally by ADS. |
| org | mandatory | - | <string> | Specifies the destination organization for writes. Takes either the ID or Name interchangeably. This needs to be an org that exists in InfluxDB. |
| bucket | mandatory | - | <string> | Specifies the destination bucket for writes. Takes either the ID or Name interchangeably. This needs to be a bucket that exists in InfluxDB. |
| token | mandatory | <a string that represents the authentication token> | Authentication token for the InfluxDB API, to authorize API requests. This token is automatically generated in InfluxDB. | |
| step | optional | 1 minute | <a valid duration> minimum value 1 minute | This parameter refers to how often data is sampled from gauges and percentiles. It also governs the push interval, that is, the reporting frequency. The duration must be expressed as a positive integer and a time unit. NOTE: Axiomatics strongly recommends that the step interval is half that of the value of the decay parameter mentioned above. Axiomatics also recommends determining the value for the step parameter first and then adjusting the decay value |
| uri | optional | http://localhost:8086 | <string> | The URI for the Influx backend. This parameter can be set to allow writes only to a specific instance of InfluxDB. |
InfluxDB configuration parameters
ADS will push metrics data to influxDB according to the step setting. If step is set to 1 minute (and assuming that there is 1 request per second), then all the 60 requests will be written to influxDB every 1 minute.
Important: If an optional parameter is added, its value cannot be empty or null. That would be an invalid entry and the system would not be initialized. If a parameter is not included in the configuration, its default value will be used.
These instructions are only referring to steps relevant to configure ADS for use with InfluxDB. For other questions regarding InfluxDB, please refer to InfluxDB documentation.
Enabling TLS encryption
InfluxData strongly recommends enabling TLS, especially if you plan on sending requests to InfluxDB over a network.
Proper TLS certificates must be provided to InfluxDB, and to the JVM used to run ADS.
Refer to Enable TLS encryption for information on how to enable and configure TLS encryption.