openapi: 3.0.0
info:
contact:
name: Confluent
url: https://confluent.io
email: support@confluent.io
description: |
# Introduction
The Confluent Cloud Metrics API provides actionable operational metrics about your Confluent
Cloud deployment. This is a queryable HTTP API in which the user will `POST` a query written in
JSON and get back a time series of metrics specified by the query.
Comprehensive documentation is available on
[docs.confluent.io](https://docs.confluent.io/current/cloud/metrics-api.html).
# Available Metrics Reference
Please see the Metrics Reference for
a list of available metrics.
This information is also available programmatically via the
[descriptors endpoint](#tag/Version-2/paths/~1v2~1metrics~1{dataset}~1descriptors~1metrics/get).
# Authentication
Confluent uses API keys for integrating with Confluent Cloud. Applications must be
authorized and authenticated before they can access or manage resources in Confluent Cloud.
You can manage your API keys in the Confluent Cloud Dashboard or Confluent Cloud CLI.
An API key is owned by a User or Service Account and inherits the permissions granted
to the owner.
Today, you can divide API keys into two classes:
* **Cloud API Keys** - These grant access to the Confluent Cloud Control Plane APIs,
such as for Provisioning and Metrics integrations.
* **Cluster API Keys** - These grant access to a single Confluent cluster, such as a specific
Kafka or Schema Registry cluster.
**If you are going to use an API Key, you must use a Cloud API Key**. Cluster API Keys won't
work. Cloud API Keys can be created using the [Confluent Cloud CLI](https://docs.confluent.io/current/cloud/cli/).
```
ccloud api-key create --resource cloud
```
All API requests must be made over HTTPS. Calls made over plain HTTP will fail. API requests
without authentication will also fail.
The Confluent Cloud Metrics API also supports OAuth 2.0 by allowing Confluent Security Token Service (STS) tokens as
credentials to authenticate to metrics API. See steps to
[Authenticate access to Confluent Cloud APIs using Confluent STS tokens](https://docs.confluent.io/cloud/current/access-management/authenticate/oauth/access-rest-apis-sts.html#authenticate-access-to-ccloud-apis-using-confluent-security-token-service-sts-tokens)
# Versioning
Confluent APIs ensure stability for your integrations by avoiding the introduction
of breaking changes to customers unexpectedly. Confluent will make non-breaking
API changes without advance notice. Thus, API clients **must** follow the
[Compatibility Policy](#section/Versioning/Compatibility-Policy) below to ensure your
ingtegration remains stable. All APIs follow the API Lifecycle Policy described below,
which describes the guarantees API clients can rely on.
Breaking changes will be [widely communicated](#communication) in advance in accordance
with our [Deprecation Policy](#section/Versioning/Deprecation-Policy). Confluent will provide
timelines and a migration path for all API changes, where available. Be sure to subscribe
to one or more [communication channels](#communication) so you don't miss any updates!
One exception to these guidelines is for critical security issues. We will take any necessary
actions to mitigate any critical security issue as soon as possible, which may include disabling
the vulnerable functionality until a proper solution is available.
Do not consume any Confluent API unless it is documented in the API Reference. All undocumented
endpoints should be considered private, subject to change without notice, and not covered by any
agreements.
> Note: The "v1" in the URL is not a "major version" in the
[Semantic Versioning](https://semver.org/) sense. It is a "generational version" or
"meta version", as seen in other APIs like
Github API or the
Stripe API .
## Changelog
### 2022-12-01
#### The dataset `health-plus` is now available in preview.
See the [Datasets](#section/Object-Model/Datasets) section for more details.
### 2022-10-18
#### `cluster_active_link_count` metric is deprecated
The `io.confluent.kafka.server/cluster_active_link_count` is now deprecated. Please use the
`io.confluent.kafka.server/cluster_link_count` metric instead.
#### New metrics available in `/export`
The following metrics are now available in the `/export` endpoint:
* `io.confluent.kafka.server/request_bytes`
* `io.confluent.kafka.server/response_bytes`
* `io.confluent.kafka.server/cluster_link_destination_response_bytes`
* `io.confluent.kafka.server/cluster_link_source_response_bytes`
* `io.confluent.kafka.server/cluster_link_count`
* `io.confluent.kafka.server/cluster_link_mirror_topic_count`
* `io.confluent.kafka.server/cluster_link_mirror_topic_offset_lag`
* `io.confluent.kafka.server/cluster_link_mirror_topic_bytes`
### 2022-10-17
#### API Version 1 is marked sunset
All API Version 1 endpoints are no longer supported from 2022-10-17. API users
should migrate to API [Version 2](#tag/Version-2).
### 2021-09-23
#### API Version 1 is now deprecated
All API Version 1 endpoints are now deprecated and will be removed on 2022-04-04. API users
should migrate to API [Version 2](#tag/Version-2).
### 2021-08-24
#### Metric-specific aggregation functions
New metrics are being introduced that require alternative aggregation functions (e.g. `MAX`).
When querying those metrics, using `agg: "SUM"` will return an error.
It is recommended that clients **omit the `agg` field in the request** such that the required
aggregation function for the specific metric is automatically applied on the backend.
> Note: The initial version of Metrics API required clients to effectively hardcode `agg: "SUM"`
> in all queries. In early 2021, the `agg` field was made optional, but many clients have not
> been updated to omit the `agg` field.
#### Cursor-based pagination for `/query` endpoint
The `/query` endpoint now supports cursor-based pagination similar to the `/descriptors` and
`/attributes` endpoints.
### 2021-02-10
#### API Version 2 is now Generally Available (GA)
See the [Version 2](#tag/Version-2) section below for a detailed description of changes and
migration guide.
### 2020-12-04
#### API Version 2 *(Preview)*
Version 2 of the Metrics API is now available in Preview. See the [Version 2](#tag/Version-2)
section below for a detailed description of changes.
### 2020-07-08
#### Correction for `active_connection_count` metric
A bug in the `active_connection_count` metric that affected a subset of customers was fixed.
Customers exporting the metric to an external monitoring system may observe a discontinuity
between historical results and current results due to this one-time correction.
### 2020-04-01
This release includes the following changes from the preview release:
#### New `format` request attribute
The `/query` request now includes a `format` attribute which controls the result structuring in
the response body. See the `/query` endpoint definition for more details.
#### New `/available` endpoint
The new `/available` endpoint allows determining which metrics are available for a set of
resources (defined by labels). This endpoint can be used to determine which subset of metrics
are currently available for a specific resource (e.g. a Confluent Cloud Kafka cluster).
#### Metric type changes
The `CUMULATIVE_(INT|DOUBLE)` metric type enumeration was changed to `COUNTER_(INT|DOUBLE)`.
This was done to better align with OpenTelemetry conventions. In tandem with this change,
several metrics that were improperly classified as `GAUGE`s were re-classified as `COUNTER`s.
### Metric name changes
The `/delta` suffix has been removed from the following metrics:
* `io.confluent.kafka.server/received_bytes/delta`
* `io.confluent.kafka.server/sent_bytes/delta`
* `io.confluent.kafka.server/request_count/delta`
### 2020-09-15
#### Retire `/available` endpoint
The `/available` endpoint (which was in _Preview_ status) has been removed from the API.
The `/descriptors` endpoint can still be used to determine the universe of available
metrics for Metrics API.
**The legacy metric names are deprecated and will stop functioning on 2020-07-01.**
## API Lifecycle Policy
The following status labels are applicable to APIs, features, and SDK versions, based on
the current support status of each:
* **Early Access** – May change at any time. Not recommended for production usage. Not
officially supported by Confluent. Intended for user feedback only. Users must be granted
explicit access to the API by Confluent.
* **Preview** – Unlikely to change between Preview and General Availability. Not recommended
for production usage. Officially supported by Confluent for non-production usage.
For Closed Previews, users must be granted explicit access to the API by Confluent.
* **Generally Available (GA)** – Will not change at short notice. Recommended for production
usage. Officially supported by Confluent for non-production and production usage.
* **Deprecated** – No longer supported. Will be removed in the future at the announced date.
Use is discouraged and migration following the upgrade guide is recommended.
* **Sunset** – Removed, and no longer supported or available.
Resources, operations, and individual fields in the
OpenAPI spec will be annotated with
`x-lifecycle-stage`, `x-deprecated-at`, and `x-sunset-at`. These annotations will appear in the
corresponding API Reference Documentation. An API is "Generally Available" unless explicitly
marked otherwise.
## Compatibility Policy
Confluent APIs are governed by
Confluent Cloud Upgrade Policy in which we will make backward incompatible changes and
deprecations approximately once per year, and will provide 180 days notice via email to all
registered Confluent Cloud users.
### Backward Compatibility
> *An API version is backwards-compatible if a program written against the previous version of
> the API will continue to work the same way, without modification, against this version of the
> API.*
Confluent considers the following changes to be backwards-compatible:
* Adding new API resources.
* Adding new optional parameters to existing API requests (e.g., query string or body).
* Adding new properties to existing API responses.
* Changing the order of properties in existing API responses.
* Changing the length or format of object IDs or other opaque strings.
* Unless otherwise documented, you can safely assume object IDs we generate will never exceed
255 characters, but you should be able to handle IDs of up to that length.
If you're using MySQL, for example, you should store IDs in a
`VARCHAR(255) COLLATE utf8_bin` column.
* This includes adding or removing fixed prefixes (such as `lkc-` on kafka cluster IDs).
* This includes API keys, API tokens, and similar authentication mechanisms.
* This includes all strings described as "opaque" in the docs, such as pagination cursors.
* Omitting properties with null values from existing API responses.
### Client Responsibilities
* Resource and rate limits, and the default and maximum sizes of paginated data **are not**
considered part of the API contract and may change (possibly dynamically). It is the client's
responsibility to read the road signs and obey the speed limit.
* If a property has a primitive type and the API documentation does not explicitly limit its
possible values, clients **must not** assume the values are constrained to a particular set
of possible responses.
* If a property of an object is not explicitly declared as mandatory in the API, clients
**must not** assume it will be present.
* A resource **may** be modified to return a "redirection" response (e.g. `301`, `307`) instead
of directly returning the resource. Clients **must** handle HTTP-level redirects, and respect
HTTP headers (e.g. `Location`).
## Deprecation Policy
Confluent will announce deprecations at least 180 days in advance of a breaking change
and we will continue to maintain the deprecated APIs in their original form during this time.
Exceptions to this policy apply in case of critical security vulnerabilities or functional
defects.
### Communication
When a deprecation is announced, the details and any relevant migration
information will be available on the following channels:
* Publication in the [API Changelog](#section/Versioning/Changelog)
* Lifecycle, deprecation and "x-deprecated-at" annotations in the
OpenAPI spec
* Announcements on the
Developer Blog ,
Community Slack
(join! ),
Google Group ,
the @ConfluentInc twitter
account, and similar channels
* Enterprise customers may receive information by email to their specified Confluent contact,
if applicable.
# Object Model
The object model for the Metrics API is designed similarly to the
[OpenTelemetry](https://opentelemetry.io/) standard.
## Metrics
A _metric_ is a numeric attribute of a resource, measured at a specific point in time, labeled
with contextual metadata gathered at the point of instrumentation.
There are two types of metrics:
* `GAUGE`: An instantaneous measurement of a value.
Gauge metrics are implicitly averaged when aggregating over time.
> Example: `io.confluent.kafka.server/retained_bytes`
* `COUNTER`: The count of occurrences in a _single (one minute) sampling
interval_ (unless otherwise stated in the metric description).
Counter metrics are implicitly summed when aggregating over time.
> Example: `io.confluent.kafka.server/received_bytes`
The list of metrics and their labels is available at [/docs/descriptors](/docs/descriptors).
## Resources
A _resource_ represents an entity against which metrics are collected. For example, a Kafka
cluster, a Kafka Connector, a ksqlDB application, etc.
Each metric _descriptor_ is associated with one or more resource _descriptors_, representing
the resource types to which that metric can apply. A metric _data point_ is associated with a
single resource _instance_, identified by the resource labels on that metric data point.
For example, metrics emitted by Kafka Connect are associated to the `connector` resource type.
Data points for those metrics include resource labels identifying the specific `connector`
instance that emitted the metric.
The list of resource types and labels are discoverable via the `/descriptors/resources`
endpoint.
## Labels
A _label_ is a key-value attribute associated with a metric data point.
Labels can be used in queries to filter or group the results. Labels must be prefixed when
used in queries:
* `metric.` (for metric labels), for example `metric.topic`
* `resource..` (for resource labels), for example `resource.kafka.id`.
The set of valid label keys for a metric include:
* The label keys defined on that metric's descriptor itself
* The label keys defined on the resource descriptor for the metric's associated resource type
For example, the `io.confluent.kafka.server/received_bytes` metric has the following labels:
* `resource.kafka.id` - The Kafka cluster to which the metric pertains
* `metric.topic` - The Kafka topic to which the bytes were produced
* `metric.partition` - The partition to which the bytes were produced
## Datasets
A _dataset_ is a logical collection of metrics that can be queried together. The `dataset` is
a required URL template parameter for every endpoint in this API. The following datasets are
currently available:
Dataset
Description
cloud
Metrics originating from Confluent Cloud resources.
Requests to this dataset require a resource filter
(e.g. Kafka cluster ID, Connector ID, etc.) in the query for authorization purposes.
The client's API key must be authorized for the resource referenced in the filter.
cloud-custom
Metrics originating from custom Confluent Cloud resources (e.g. a Custom Connector).
Requests to this dataset require a resource filter
(e.g. Custom Connector ID, etc.) in the query for authorization purposes.
The client's API key must be authorized for the resource referenced in the filter.
health-plus
Metrics originating from Confluent Platform resources.
# Client Considerations and Best Practices
## Rate Limiting
To protect the stability of the API and keep it available to all users, Confluent employs
multiple safeguards. Users who send many requests in quick succession or perform too many
concurrent operations may be throttled or have their requested rejected with an error.
When a rate limit is breached, an HTTP `429 Too Many Requests` error is returned. The
following headers are sent back to provide assistance in dealing with rate limits.
| Header | Description |
| ----------------------- | -------------------------------------- |
| `rateLimit-limit` | The maximum number of requests you're permitted to make per time period. |
| `rateLimit-reset` | The relative time in seconds until the current rate limit window resets. **Important:** This differs from Github and Twitter\'s same-named header which uses UTC epoch seconds. We use relative time to avoid client/server time synchronization issues. |
| `rateLimit-remaining` | The number of requests remaining in the current rate-limit window. |
Rate limits are enforced at multiple scopes. You get two sets of the headers above, each
specifying the limit of one scope.
### Global Rate Limits
A global rate limit of **300 requests per IP address, per minute** is enforced.
### Per-endpoint Rate Limits
Additionally, some endpoint-specific rate limits are enforced.
| Endpoint | Rate limit |
| --------- | ---------- |
| `/v2/metrics/{dataset}/export` | 160 requests per resource, per hour, per principal. See the [export endpoint documentation](#tag/Version-2/paths/~1v2~1metrics~1{dataset}~1export/get) for details. |
## Retries
Implement retry logic in your client to gracefully handle transient API failures.
This should be done by watching for error responses and building in a retry mechanism.
This mechanism should follow a capped exponential backoff policy to prevent retry
amplification ("retry storms") and also introduce some randomness ("jitter") to avoid the
[thundering herd effect](https://en.wikipedia.org/wiki/Thundering_herd_problem).
## Metric Data Latency
Metric data points are typically available for query in the API within **5 minutes** of their
origination at the source. This latency can vary based on network conditions and processing
overhead. Clients that are polling (or "scraping") metrics into an external monitoring system
should account for this latency in their polling requests. API requests that fail to
incorporate the latency into the query `interval` may have incomplete data in the response.
## Pagination
Cursors, tokens, and corresponding pagination links may expire after a short amount of time.
In this case, the API will return a `400 Bad Request` error and the client will need to restart
from the beginning.
The client should have no trouble pausing between rate limiting windows, but persisting cursors
for hours or days is not recommended.
version: ''
title: Confluent Cloud Metrics API
x-api-id: 4be9bd3c-ea9b-4efe-89aa-946c36b50161
x-audience: external-public
x-logo:
url: https://assets.confluent.io/m/1661ef5e4ff82d3d/
servers:
- url: https://api.telemetry.confluent.cloud
security:
- api-key: []
- confluent-sts-access-token: []
tags:
- name: Version 2
description: |

Version 2 of the Metrics API adds the ability to query metrics for Kafka Connect, ksqlDB,
and Schema Registry.
This capability is enabled by the introduction of a [Resource](#section/Object-Model/Resources)
abstraction into the API object model. Resources represent the entity against which metrics
are collected.
### Migration Guide
The following endpoint URLs have changed in version 2:
| Endpoint | Version 1 (Sunset) | Version 2 |
| ----------------------- | -------------------------------- | ---------------------------------------- |
| Metrics discovery | `/metrics/{dataset}/descriptors` | `/metrics/{dataset}/descriptors/metrics` |
The label prefix syntax has changed in version 2:
| Label | Version 1 (Sunset) | Version 2 |
| ----------------------- | ---------------------------- | ----------------------- |
| Resource labels *(new)* | *N/A* | `resource.` |
| Kafka cluster ID | `metric.label.cluster_id` | `resource.kafka.id` |
| All other metric labels | `metric.label.` | `metric.` |
This example shows a request to `/v1/metrics/cloud/query` migrated into the new `v2` syntax
#### Version 1 Request
```json
{
"group_by": [
"metric.label.topic"
],
"aggregations": [{
"metric": "io.confluent.kafka.server/received_bytes",
"agg": "SUM"
}],
"filter": {
"field": "metric.label.cluster_id",
"op": "EQ",
"value": "lkc-00000"
},
"granularity": "ALL",
"intervals" : [
"2020-01-01T00:00:00Z/PT1H"
]
}
```
#### Version 2 Request
```json
{
"group_by": [
"metric.topic"
],
"aggregations": [{
"metric": "io.confluent.kafka.server/received_bytes",
"agg": "SUM"
}],
"filter": {
"field": "resource.kafka.id",
"op": "EQ",
"value": "lkc-00000"
},
"granularity": "ALL",
"intervals" : [
"2020-01-01T00:00:00Z/PT1H"
]
}
```
paths:
/v2/metrics/{dataset}/descriptors/metrics:
get:
summary: List metric descriptors
tags:
- Version 2
description: |
Lists all the metric descriptors for a dataset.
A metric descriptor represents metadata for a metric, including its data type and labels.
This metadata is provided programmatically to enable clients to dynamically adjust as new
metrics are added to the dataset, rather than hardcoding metric names in client code.
parameters:
- name: dataset
in: path
required: true
description: The [dataset](#section/Object-Model/Datasets) to list metric descriptors for.
schema:
$ref: '#/components/schemas/Dataset'
- name: page_size
description: The maximum number of results to return. The page size is an integer in the range from 1 through 1000.
in: query
required: false
schema:
type: integer
minimum: 1
maximum: 1000
default: 100
- name: page_token
description: The next page token. The token is returned by the previous request as part of `meta.pagination`.
in: query
required: false
schema:
$ref: '#/components/schemas/PageToken'
- name: resource_type
description: The type of the resource to list metric descriptors for.
in: query
required: false
schema:
$ref: '#/components/schemas/ResourceType'
responses:
'200':
description: Successful response
content:
application/json:
schema:
$ref: '#/components/schemas/ListMetricDescriptorsResponse'
examples:
listResponse:
value:
data:
- description: The delta count of bytes received from the network. Each sample is the number of bytes received since the previous data sample. The count is sampled every 60 seconds.
labels:
- description: The name of the Kafka topic.
key: topic
exportable: true
name: io.confluent.kafka.server/received_bytes
type: COUNTER_INT64
exportable: true
unit: By
lifecycle_stage: GENERAL_AVAILABILITY
resources:
- kafka
- description: The delta count of bytes sent over the network. Each sample is the number of bytes sent since the previous data point. The count is sampled every 60 seconds.
labels:
- description: The name of the Kafka topic.
key: topic
exportable: true
name: io.confluent.kafka.server/sent_bytes
type: COUNTER_INT64
exportable: true
unit: By
lifecycle_stage: GENERAL_AVAILABILITY
resources:
- kafka
links: null
meta:
pagination:
page_size: 3
total_size: 3
'429':
$ref: '#/components/responses/RateLimitError'
default:
description: Error
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
/v2/metrics/{dataset}/descriptors/resources:
get:
tags:
- Version 2
summary: List resource descriptors
description: |
Lists all the resource descriptors for a dataset.
parameters:
- name: dataset
in: path
required: true
description: The [dataset](#section/Object-Model/Datasets) to list resource descriptors for.
schema:
$ref: '#/components/schemas/Dataset'
- name: page_size
description: The maximum number of results to return. The page size is an integer in the range from 1 through 1000.
in: query
required: false
schema:
type: integer
minimum: 1
maximum: 1000
default: 100
- name: page_token
description: The next page token. The token is returned by the previous request as part of `meta.pagination`.
in: query
required: false
schema:
$ref: '#/components/schemas/PageToken'
responses:
'200':
description: Successful response
content:
application/json:
schema:
$ref: '#/components/schemas/ListResourceDescriptorsResponse'
'429':
$ref: '#/components/responses/RateLimitError'
default:
description: Error
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
/v2/metrics/{dataset}/query:
post:
summary: Query metric values
tags:
- Version 2
description: |
Query for metric values in a dataset.
parameters:
- name: dataset
in: path
required: true
description: The [dataset](#section/Object-Model/Datasets) to query.
schema:
$ref: '#/components/schemas/Dataset'
- name: page_token
description: The next page token. The token is returned by the previous request as part of `meta.pagination`. Pagination is only supported for requests containing a `group_by` element.
in: query
required: false
schema:
$ref: '#/components/schemas/PageToken'
requestBody:
content:
application/json:
schema:
$ref: '#/components/schemas/QueryRequest'
examples:
query:
value:
group_by:
- metric.topic
aggregations:
- metric: io.confluent.kafka.server/sent_bytes
agg: SUM
filter:
op: AND
filters:
- field: resource.kafka.id
op: EQ
value: lkc-1234
- op: NOT
filter:
field: metric.topic
op: EQ
value: topicA
order_by:
- metric: io.confluent.kafka.server/sent_bytes
agg: SUM
order: DESCENDING
granularity: PT1H
intervals:
- 2019-10-17T20:17:00.000Z/PT2H
limit: 5
responses:
'200':
description: Successful response
content:
application/json:
schema:
$ref: '#/components/schemas/QueryResponse'
examples:
flatResponse:
value:
data:
- timestamp: '2019-10-17T20:17:00.000Z'
metric.topic: foo
value: 9741
- timestamp: '2019-10-17T20:18:00.000Z'
metric.topic: foo
value: 9246
- timestamp: '2019-10-17T20:17:00.000Z'
metric.topic: bar
value: 844.1
- timestamp: '2019-10-17T20:18:00.000Z'
metric.topic: bar
value: 821.1
groupedResponse:
value:
data:
- metric.topic: foo
points:
- timestamp: '2019-10-17T20:17:00.000Z'
value: 9741
- timestamp: '2019-10-17T20:18:00.000Z'
value: 9246
- metric.topic: bar
points:
- timestamp: '2019-10-17T20:17:00.000Z'
value: 844.1
- timestamp: '2019-10-17T20:18:00.000Z'
value: 821.1
'429':
$ref: '#/components/responses/RateLimitError'
default:
description: Error
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
/v2/metrics/{dataset}/export:
get:
tags:
- Version 2
summary: Export metric values
description: |
Export current metric values in [OpenMetrics format](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md)
or [Prometheus format](https://prometheus.io/docs/instrumenting/exposition_formats/#text-based-format),
suitable for import into an external monitoring system. Returns the single most recent
data point for each metric, for each distinct combination of labels.
#### Supported datasets and metrics
By default only the `cloud` and `cloud-custom` datasets are supported for this endpoint.
Some metrics and labels within the datasets may not be exportable. To request
a particular metric or label be added, please contact [Confluent Support](https://support.confluent.io).
#### Metric translation
Metric and label names are translated to adhere to [Prometheus restrictions](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels).
The `resource.` and `metric.` prefixes from label names are also dropped to simplify consumption in downstream systems.
Counter metrics are classified as the Prometheus `gauge` type to conform to required semantics.
> The `counter` type in Prometheus must be monotonically increasing, whereas Confluent
Metrics API counters are represented as deltas.
#### Timestamp offset
To account for [metric data latency](#section/Client-Considerations-and-Best-Practices/Metric-Data-Latency),
this endpoint returns metrics from the current timestamp minus a fixed offset. The current
offset is 5 minutes rounded down to the start of the minute. For example, if a request is
received at `12:06:41`, the returned metrics will have the timestamp `12:01:00` and represent the
data for the interval `12:01:00` through `12:02:00` (exclusive).
> **NOTE:** Confluent may choose to lengthen or shorten this offset based on operational
considerations. _Doing so is considered a backwards-compatible change_.
To accommodate this offset, the timestamps in the response should be honored when importing
the metrics. For example, in prometheus this can be controlled using the `honor_timestamps`
flag.
#### Rate limits
Since metrics are available at minute granularity, it is expected that clients scrape this
endpoint at most once per minute. To allow for ad-hoc testing, the rate limit is enforced
at hourly granularity. To accommodate retries, the rate limit is 160 requests per hour
rather than 60 per hour.
The rate limit is evaluated on a per-resource basis. For example, the following requests would
each be allowed a 160-requests-per-hour rate:
* `GET /v2/metrics/cloud/export?resource.kafka.id=lkc-1&resource.kafka.id=lkc-2`
* `GET /v2/metrics/cloud/export?resource.kafka.id=lkc-3`
Rate limits for this endpoint are also scoped to the authentication principal. This allows multiple systems
to export metrics for the same resources by configuring each with a separate service account.
If the rate limit is exceeded, the response body will include a message indicating which
resource exceeded the limit.
```json
{
"errors": [
{
"status": "429",
"detail": "Too many requests have been made for the following resources: kafka.id:lkc-12345. Please see the documentation for current rate limits."
}
]
}
```
#### Resource parameter formats
Resource parameters support two formats for maximum compatibility:
**Dot format (traditional):** `resource.kafka.id`, `resource.connector.id`, etc.
**Underscore format (Prometheus):** `resource_kafka_id`, `resource_connector_id`, etc.
Both formats support two ways to specify multiple resources:
1. **Multiple parameters:** `?resource.kafka.id=lkc-1&resource.kafka.id=lkc-2`
2. **Comma-separated values:** `?resource.kafka.id=lkc-1,lkc-2,lkc-3`
You can also mix both formats in the same request:
* `?resource.kafka.id=lkc-1,lkc-2&resource_kafka_id=lkc-3`
Examples:
* `GET /v2/metrics/cloud/export?resource.kafka.id=lkc-1,lkc-2,lkc-3`
* `GET /v2/metrics/cloud/export?resource_kafka_id=lkc-1,lkc-2&resource.connector.id=lcc-1`
* `GET /v2/metrics/cloud/export?resource.kafka.id=lkc-1&resource_kafka_id=lkc-2&resource_connector_id=lcc-1,lcc-2`
#### Return limits
To ensure values are returned in a timely fashion, data points returned per `resource type` per `metric` is limited to `30000`.
If you have more than `30000` unique combinations of labels then the response will be truncated to return
the first `30000` data points.
For example, if a request is made for 4 Kafka clusters(resource type 1) and 3 Kafka connectors(resource type 2)
* Assuming each Kafka cluster has `8000` topics and there are `5` metrics with only topic label
It translates to `32000` unique label combinations per metric(capped to 30000) which in turn
translates to `150,000` data points for 5 metrics.
* Assuming each Kafka connector has `3` metrics
It translates to 9 data points for 3 Kafka connectors.
The data points returned for this request totals to 150,009.
In this case, we would recommend making separate scrape jobs up to 3 kafka clusters in order to stay within the return limits for a resource type.
#### Example Prometheus scrape configuration
Here is an example [prometheus configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/)
for scraping this endpoint:
```yaml
scrape_configs:
- job_name: Confluent Cloud
scrape_interval: 1m
scrape_timeout: 1m
honor_timestamps: true
static_configs:
- targets:
- api.telemetry.confluent.cloud
scheme: https
basic_auth:
username:
password:
metrics_path: /v2/metrics/cloud/export
params:
"resource.kafka.id":
- lkc-1
- lkc-2
```
#### Ignoring failed metrics
Using `ignore_failed_metrics` param, you can get a partial response consisting of successful metrics. Unsuccessful metrics would be ignored
if the failure count was below the configured failure threshold percentage, otherwise an error would be returned.
In case a partial response is returned, it would contain an additional StateSet metric `export_status` with one sample for each metric.
If the associated metric fetch was unsuccessful, the value of the sample for that metric will be 1, otherwise it will be 0.
Each sample will have a metric and a resource label which denotes the name of the metric for which the status is shown and name of the resource to which the metric belongs respectively.
A sample partial response is shown in the Responses section wherein two metrics were requested and one of them was unsuccessful.
parameters:
- name: dataset
in: path
required: true
description: The [dataset](#section/Object-Model/Datasets) to export metrics for.
schema:
$ref: '#/components/schemas/Dataset'
- name: resource.kafka.id
description: The ID of the Kafka cluster to export metrics for. This parameter can be specified multiple times (e.g. `?resource.kafka.id=lkc-1&resource.kafka.id=lkc-2`). You can also use comma-separated values in a single parameter (e.g. `?resource.kafka.id=lkc-1,lkc-2,lkc-3`).
in: query
required: false
style: form
schema:
type: array
items:
type: string
- name: resource_kafka_id
description: The ID of the Kafka cluster to export metrics for (Prometheus format). This parameter can be specified multiple times (e.g. `?resource_kafka_id=lkc-1&resource_kafka_id=lkc-2`). You can also use comma-separated values in a single parameter (e.g. `?resource_kafka_id=lkc-1,lkc-2,lkc-3`). This is an alternative to `resource.kafka.id` for compatibility with Prometheus scraping.
in: query
required: false
style: form
schema:
type: array
items:
type: string
- name: resource.connector.id
description: The ID of the Connector to export metrics for. This parameter can be specified multiple times (e.g. `?resource.connector.id=lcc-1&resource.connector.id=lcc-2`). You can also use comma-separated values in a single parameter (e.g. `?resource.connector.id=lcc-1,lcc-2,lcc-3`).
in: query
required: false
style: form
schema:
type: array
items:
type: string
- name: resource_connector_id
description: The ID of the Connector to export metrics for (Prometheus format). This parameter can be specified multiple times (e.g. `?resource_connector_id=lcc-1&resource_connector_id=lcc-2`). You can also use comma-separated values in a single parameter (e.g. `?resource_connector_id=lcc-1,lcc-2,lcc-3`). This is an alternative to `resource.connector.id` for compatibility with Prometheus scraping.
in: query
required: false
style: form
schema:
type: array
items:
type: string
- name: resource.custom_connector.id
description: The ID of the custom Connector to export metrics for. This parameter can be specified multiple times (e.g. `?resource.custom_connector.id=lcc-1&resource.custom_connector.id=lcc-2`). You can also use comma-separated values in a single parameter (e.g. `?resource.custom_connector.id=lcc-1,lcc-2,lcc-3`).
in: query
required: false
style: form
schema:
type: array
items:
type: string
- name: resource_custom_connector_id
description: The ID of the custom Connector to export metrics for (Prometheus format). This parameter can be specified multiple times (e.g. `?resource_custom_connector_id=lcc-1&resource_custom_connector_id=lcc-2`). You can also use comma-separated values in a single parameter (e.g. `?resource_custom_connector_id=lcc-1,lcc-2,lcc-3`). This is an alternative to `resource.custom_connector.id` for compatibility with Prometheus scraping.
in: query
required: false
style: form
schema:
type: array
items:
type: string
- name: resource.ksql.id
description: The ID of the ksqlDB application to export metrics for. This parameter can be specified multiple times (e.g. `?resource.ksql.id=lksql-1&resource.ksql.id=lksql-2`). You can also use comma-separated values in a single parameter (e.g. `?resource.ksql.id=lksql-1,lksql-2,lksql-3`).
in: query
required: false
style: form
schema:
type: array
items:
type: string
- name: resource_ksql_id
description: The ID of the ksqlDB application to export metrics for (Prometheus format). This parameter can be specified multiple times (e.g. `?resource_ksql_id=lksql-1&resource_ksql_id=lksql-2`). You can also use comma-separated values in a single parameter (e.g. `?resource_ksql_id=lksql-1,lksql-2,lksql-3`). This is an alternative to `resource.ksql.id` for compatibility with Prometheus scraping.
in: query
required: false
style: form
schema:
type: array
items:
type: string
- name: resource.schema_registry.id
description: The ID of the Schema Registry to export metrics for. This parameter can be specified multiple times (e.g. `?resource.schema_registry.id=lsr-1&resource.schema_registry.id=lsr-2`). You can also use comma-separated values in a single parameter (e.g. `?resource.schema_registry.id=lsr-1,lsr-2,lsr-3`).
in: query
required: false
style: form
schema:
type: array
items:
type: string
- name: resource_schema_registry_id
description: The ID of the Schema Registry to export metrics for (Prometheus format). This parameter can be specified multiple times (e.g. `?resource_schema_registry_id=lsr-1&resource_schema_registry_id=lsr-2`). You can also use comma-separated values in a single parameter (e.g. `?resource_schema_registry_id=lsr-1,lsr-2,lsr-3`). This is an alternative to `resource.schema_registry.id` for compatibility with Prometheus scraping.
in: query
required: false
style: form
schema:
type: array
items:
type: string
- name: resource.compute_pool.id
description: The ID of the Flink Compute Pool to export metrics for. This parameter can be specified multiple times (e.g. `?resource.compute_pool.id=lcp-1&resource.compute_pool.id=lcp-2`). You can also use comma-separated values in a single parameter (e.g. `?resource.compute_pool.id=lcp-1,lcp-2,lcp-3`).
in: query
required: false
style: form
schema:
type: array
items:
type: string
- name: resource_compute_pool_id
description: The ID of the Flink Compute Pool to export metrics for (Prometheus format). This parameter can be specified multiple times (e.g. `?resource_compute_pool_id=lcp-1&resource_compute_pool_id=lcp-2`). You can also use comma-separated values in a single parameter (e.g. `?resource_compute_pool_id=lcp-1,lcp-2,lcp-3`). This is an alternative to `resource.compute_pool.id` for compatibility with Prometheus scraping.
in: query
required: false
style: form
schema:
type: array
items:
type: string
- name: metric
description: The metric to export. If this parameter is not specified, all metrics for the resource will be exported. This parameter can be specified multiple times.
in: query
required: false
style: form
schema:
type: array
items:
type: string
- name: ignore_failed_metrics
description: Ignore failed metrics and export only successful metrics if allowed failure threshold is not breached. If this parameter is set to true, a StateSet metric (export_status) will be included in the response to report which metrics were successful and which failed.
in: query
required: false
style: form
schema:
type: boolean
responses:
'200':
description: Successful response
content:
text/plain;version=0.0.4:
schema:
description: Metric values formatted in [Prometheus text-based format](https://prometheus.io/docs/instrumenting/exposition_formats/#text-based-format).
type: string
examples:
ignore_failed_metrics_FALSE:
value: |
# HELP confluent_kafka_server_received_bytes The delta count of bytes of the customer's data received from the network. Each sample is the number of bytes received since the previous data sample. The count is sampled every 60 seconds.
# TYPE confluent_kafka_server_received_bytes gauge
confluent_kafka_server_received_bytes{kafka_id="lkc-1",topic="topicA"} 10.0 1609459200
confluent_kafka_server_received_bytes{kafka_id="lkc-1",topic="topicB"} 20.0 1609459200
confluent_kafka_server_received_bytes{kafka_id="lkc-2",topic="topicA"} 30.0 1609459200
# HELP confluent_kafka_server_sent_bytes The delta count of bytes of the customer's data sent to the network. Each sample is the number of bytes sent since the previous data sample. The count is sampled every 60 seconds.
# TYPE confluent_kafka_server_sent_bytes gauge
confluent_kafka_server_sent_bytes{kafka_id="lkc-1",topic="topicA"} 90.0 1609459200
confluent_kafka_server_sent_bytes{kafka_id="lkc-1",topic="topicB"} 80.0 1609459200
confluent_kafka_server_sent_bytes{kafka_id="lkc-2",topic="topicA"} 70.0 1609459200
ignore_failed_metrics_TRUE:
value: |
# HELP export_status Status of exported metrics. Value 1 denotes status was error. Value 0 denotes status was not error.
# TYPE export_status gauge
export_status{export_status="error",metric="io.confluent.kafka.server/received_bytes",resource="kafka",} 0.0
export_status{export_status="error",metric="io.confluent.kafka.server/sent_bytes",resource="kafka",} 1.0
# HELP confluent_kafka_server_received_bytes The delta count of bytes of the customer's data received from the network. Each sample is the number of bytes received since the previous data sample. The count is sampled every 60 seconds.
# TYPE confluent_kafka_server_received_bytes gauge
confluent_kafka_server_received_bytes{kafka_id="lkc-1",topic="topicA",} 30.0 1609459200000
confluent_kafka_server_received_bytes{kafka_id="lkc-2",topic="topicA",} 70.0 1609459200000
application/openmetrics-text;version=1.0.0;charset=utf-8:
schema:
description: Metric values formatted in [OpenMetrics text format](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md).
type: string
examples:
ignore_failed_metrics_FALSE:
value: |
# TYPE confluent_kafka_server_received_bytes gauge
# UNIT confluent_kafka_server_received_bytes bytes
# HELP confluent_kafka_server_received_bytes The delta count of bytes of the customer's data received from the network. Each sample is the number of bytes received since the previous data sample. The count is sampled every 60 seconds.
confluent_kafka_server_received_bytes{kafka_id="lkc-1",topic="topicA"} 30.0 1609459200.000
confluent_kafka_server_received_bytes{kafka_id="lkc-1",topic="topicB"} 70.0 1609459200.000
confluent_kafka_server_received_bytes{kafka_id="lkc-2",topic="topicA"} 230.0 1609459200.000
confluent_kafka_server_received_bytes{kafka_id="lkc-2",topic="topicB"} 270.0 1609459200.000
# TYPE confluent_kafka_server_sent_bytes gauge
# UNIT confluent_kafka_server_received_bytes bytes
# HELP confluent_kafka_server_sent_bytes The delta count of bytes of the customer's data sent over the network. Each sample is the number of bytes sent since the previous data point. The count is sampled every 60 seconds.
confluent_kafka_server_sent_bytes{kafka_id="lkc-1",topic="topicA"} 34.0 1609459200.000
confluent_kafka_server_sent_bytes{kafka_id="lkc-1",topic="topicB"} 74.0 1609459200.000
#EOF
ignore_failed_metrics_TRUE:
value: |
# TYPE export_status stateset
# HELP export_status Status of exported metrics. Value 1 denotes status was error. Value 0 denotes status was not error.
export_status{export_status="error",metric="io.confluent.kafka.server/received_bytes",resource="kafka"} 0.0
export_status{export_status="error",metric="io.confluent.kafka.server/sent_bytes",resource="kafka"} 1.0
# TYPE confluent_kafka_server_received_bytes gauge
# UNIT confluent_kafka_server_received_bytes bytes
# HELP confluent_kafka_server_received_bytes The delta count of bytes of the customer's data received from the network. Each sample is the number of bytes received since the previous data sample. The count is sampled every 60 seconds.
confluent_kafka_server_received_bytes{kafka_id="lkc-1",topic="topicA"} 30.0 1609459200000
confluent_kafka_server_received_bytes{kafka_id="lkc-2",topic="topicA"} 70.0 1609459200000
# EOF
'429':
description: Rate Limit Exceeded
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
default:
description: Error
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
/v2/metrics/{dataset}/attributes:
post:
summary: Query label values
tags:
- Version 2
description: |
Enumerates label values for a single metric.
parameters:
- name: dataset
in: path
required: true
description: The dataset to query.
schema:
$ref: '#/components/schemas/Dataset'
- name: page_token
description: The next page token. The token is returned by the previous request as part of `meta.pagination`.
in: query
required: false
schema:
type: string
requestBody:
content:
application/json:
schema:
$ref: '#/components/schemas/AttributesRequest'
examples:
request:
value:
metric: io.confluent.kafka.server/sent_bytes
group_by:
- metric.topic
filter:
field: resource.kafka.id
op: EQ
value: lkc-09d19
limit: 3
intervals:
- 2019-10-16T16:30:20Z/2019-10-24T18:57:00Z
responses:
'200':
description: Successful response
content:
application/json:
schema:
$ref: '#/components/schemas/AttributesResponse'
examples:
response:
value:
data:
- metric.topic: bar
- metric.topic: baz
- metric.topic: foo
meta:
pagination:
page_size: 3
next_page_token: dG9waWNC
'429':
$ref: '#/components/responses/RateLimitError'
default:
description: Error
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
components:
securitySchemes:
api-key:
type: http
scheme: basic
description: |
API keys must be sent as an `Authorization: Basic {key}` header with the Key ID as the
username and the Key Secret as the password. Remember that HTTP Basic authorization
requires you to colon-separate and base64 encode your key. For example, if your API Key ID
is `ABCDEFGH123456789` and the corresponding API Key Secret is
`XNCIW93I2L1SQPJSJ823K1LS902KLDFMCZPWEO`, then the authorization header will be
Authorization: Basic QUJDREVGR0gxMjM0NTY3ODk6WE5DSVc5M0kyTDFTUVBKU0o4MjNLMUxTOTAyS0xERk1DWlBXRU8=
This example header can be generated (using Mac OS X syntax) from the API key with
$ echo -n "ABCDEFGH123456789:XNCIW93I2L1SQPJSJ823K1LS902KLDFMCZPWEO" | base64
confluent-sts-access-token:
type: http
scheme: bearer
description: |
Confluent Security Token Service (STS) issues access tokens (`confluent-sts-access-token`) by exchanging an
external token (`external-access-token`) for a confluent-sts-access-token. It can be generated using the steps
mentioned [here](https://docs.confluent.io/cloud/current/access-management/authenticate/oauth/access-rest-apis-sts.html#)
confluent-sts-access-token must be sent as an `Authorization: Bearer {confluent-sts-access-token}` header.
schemas:
Dataset:
type: string
description: A logical collection of metrics that can be described and queried. Valid values are `cloud` and `health-plus`.
PageToken:
description: An opaque token that you use to get the next page of results.
type: string
ResourceType:
description: A named type for a resource (e.g. `kafka`, `connector`).
type: string
Pagination:
description: The pagination information.
properties:
page_size:
description: The page size requested by the user.
format: int32
type: integer
total_size:
description: The total number of items, if the total size can be determined in advance; otherwise, `total_size` is not included.
format: int32
type: integer
next_page_token:
$ref: '#/components/schemas/PageToken'
required:
- page_size
example:
page_size: 10
total_size: 25
Meta:
type: object
description: Meta object
properties:
pagination:
$ref: '#/components/schemas/Pagination'
required:
- pagination
Links:
description: link information
properties:
next:
description: The next page of resources in a collection. The link is present only if the current document is part of a collection.
type: string
ErrorResponse:
type: object
properties:
errors:
type: array
items:
type: object
properties:
id:
description: A unique identifier for this particular occurrence of the problem.
type: string
status:
description: The HTTP status code applicable to this problem, expressed as a string value.
type: string
code:
description: An application-specific error code, expressed as a string value.
type: string
detail:
description: A human-readable explanation specific to this occurrence of the problem.
type: string
meta:
$ref: '#/components/schemas/Meta'
links:
$ref: '#/components/schemas/Links'
required:
- errors
example:
errors:
- detail: fieldA must not be null
- detail: fieldB must not be null
LabelDescriptor:
description: The description of a label.
type: object
properties:
description:
description: The description of the metric.
type: string
key:
description: The key of the label.
type: string
exportable:
description: Is this label included in the `/export` endpoint response?
type: boolean
required:
- description
- key
example:
key: topic
description: The name of the Kafka topic.
exportable: true
MetricDescriptor:
description: Defines a metric type and its schema.
type: object
properties:
description:
description: The description of the metric.
type: string
labels:
description: |
Labels for filtering and aggregating this metric. For example, you can
filter the `io.confluent.kafka.server/received_bytes` metric by the
`topic` label.
type: array
items:
$ref: '#/components/schemas/LabelDescriptor'
name:
description: |
The unique name of the metric, for example,
`io.confluent.kafka.server/received_bytes`.
type: string
unit:
description: |
The unit that the metric value is reported in. Units follow the format described in
[the Unified Code for Units of Measure](http://unitsofmeasure.org/ucum.html).
For example, `By` for bytes and `By/s` for bytes per second.
type: string
type:
description: |
The type of the measurement. The metric type follows the
[OpenTelemetry](https://opentelemetry.io/) exposition format.
* `GAUGE_(INT64|DOUBLE)`: An instantaneous measurement of a value.
Gauge metrics are implicitly averaged when aggregating over time.
* `COUNTER_(INT64|DOUBLE)`: The count of occurrences in a _single (one minute) sampling
interval_ (unless otherwise stated in the metric description).
Counter metrics are implicitly summed when aggregating over time.
type: string
x-extensible-enum:
- GAUGE_INT64
- GAUGE_DOUBLE
- COUNTER_INT64
- COUNTER_DOUBLE
lifecycle_stage:
description: |
The support lifecycle stage for this metric:
* `PREVIEW`: May change at any time
* `GENERAL_AVAILABILITY`: Fully released and stable. Will not change without notice.
* `DEPRECATED`: No longer supported. Will be removed in the future at the annouced date.
type: string
x-extensible-enum:
- PREVIEW
- GENERAL_AVAILABILITY
- DEPRECATED
exportable:
description: Is this metric included in the `/export` endpoint response?
type: boolean
resources:
description: The resource types to which this metric pertains.
type: array
items:
type: string
required:
- description
- labels
- name
- unit
- type
- lifecycle_stage
- exportable
- resources
example:
description: The delta count of bytes received from the network. The count is sampled every 60 seconds.
name: io.confluent.kafka.server/received_bytes
unit: By
type: GAUGE_INT64
exportable: true
lifecycle_stage: GENERAL_AVAILABILITY
resources:
- kafka
labels:
- key: topic
description: The name of the Kafka topic.
exportable: true
ListMetricDescriptorsResponse:
description: The ListMetricDescriptors response.
type: object
properties:
data:
description: The metric descriptors for the specified dataset.
items:
$ref: '#/components/schemas/MetricDescriptor'
type: array
meta:
$ref: '#/components/schemas/Meta'
links:
$ref: '#/components/schemas/Links'
required:
- data
- meta
ResourceDescriptor:
description: |
A Resource represents the entity producing metrics.
For example: a Kafka cluster a Kafka Connector, etc.
type: object
properties:
type:
$ref: '#/components/schemas/ResourceType'
description:
description: The description of the resource.
type: string
labels:
description: |
Labels for the resource.
Resource labels can be used for filtering and aggregating metrics
associated with a resource.
type: array
items:
$ref: '#/components/schemas/LabelDescriptor'
required:
- type
- description
- labels
example:
type: kafka
description: A Kafka cluster.
labels:
- key: kafka.id
description: ID of the kafka cluster
exportable: true
ListResourceDescriptorsResponse:
description: The list of resource descriptors for a dataset
type: object
properties:
data:
description: The resource descriptors for the specified dataset.
type: array
items:
$ref: '#/components/schemas/ResourceDescriptor'
meta:
$ref: '#/components/schemas/Meta'
links:
$ref: '#/components/schemas/Links'
required:
- data
- meta
AggregationFunction:
type: string
enum:
- SUM
- MIN
- MAX
Aggregation:
type: object
x-java-class: io.confluent.observability.metricsquery.api.Aggregation
properties:
metric:
description: The metric to aggregate.
type: string
agg:
description: The aggregation function for the label buckets defined in `group_by`.
allOf:
- $ref: '#/components/schemas/AggregationFunction'
required:
- metric
example:
metric: io.confluent.kafka.server/bytes_in
agg: SUM
Granularity:
description: |
Defines the time buckets that the aggregation is performed for.
Buckets are specified in
[ISO-8601 duration syntax](https://en.wikipedia.org/wiki/ISO_8601#Durations), but
only the enumerated values are supported. Buckets are aligned to UTC boundaries.
The special `ALL` value defines a single bucket for all intervals.
The allowed granularity for a query is restricted by the length of that query's `interval`.
Do not confuse intervals with retention time. Confluent uses granularity and intervals to
validate requests. Retention time is the length of time Confluent stores data. For example,
requests with a granularity of `PT1M` can have a maximum interval of six hours. A request
with a granularity of `PT1M` and an interval of 12 hours would fail validation. For more
information about retention time, see the
[FAQ](https://docs.confluent.io/cloud/current/monitoring/metrics-api.html#what-is-the-retention-time-of-metrics-in-the-metrics-api).
Granularity equal to or greater than `PT1H` can use any interval.
| Granularity | Maximum Interval Length |
|----------------------|--------------------------|
| `PT1M` (1 minute) | 6 hours |
| `PT5M` (5 minutes) | 1 day |
| `PT15M` (15 minutes) | 4 days |
| `PT30M` (30 minutes) | 7 days |
| `PT1H` (1 hour) | Any |
| `PT4H` (4 hours) | Any |
| `PT6H` (6 hours) | Any |
| `PT12H` (12 hours) | Any |
| `P1D` (1 day) | Any |
| `ALL` | Any |
type: string
format: ISO-8601 duration (PnDTnHnMn.nS) or ALL
enum:
- PT1M
- PT5M
- PT15M
- PT30M
- PT1H
- PT4H
- PT6H
- PT12H
- P1D
- ALL
example: PT1H
FieldFilter:
title: Field Filter
type: object
x-java-class: io.confluent.observability.metricsquery.api.Filter
properties:
op:
type: string
description: |
The comparison operator for the filter.
Note that labels are compared _lexicographically_.
The `GT` or `GTE` operators can be used to page through grouped result sets that exceed
the query limit.
enum:
- EQ
- GT
- GTE
field:
description: |
The field to filter on; see [here](#section/Object-Model/Labels) on using labels as
filter fields.
type: string
example: metric.topic
value:
oneOf:
- type: string
- type: integer
required:
- op
- value
example:
op: EQ
field: resource.kafka.id
value: lkc-1234
Filter:
description: Metric filter.
oneOf:
- $ref: '#/components/schemas/FieldFilter'
- $ref: '#/components/schemas/CompoundFilter'
- $ref: '#/components/schemas/UnaryFilter'
CompoundFilter:
title: Compound Filter
type: object
x-java-class: io.confluent.observability.metricsquery.api.Filter
properties:
op:
type: string
enum:
- AND
- OR
filters:
type: array
items:
$ref: '#/components/schemas/Filter'
required:
- op
- filters
example:
op: OR
filters:
- field: resource.kafka.id
op: EQ
value: lkc-1234
- field: resource.kafka.id
op: EQ
value: lkc-5678
UnaryFilter:
title: Unary Filter
type: object
x-java-class: io.confluent.observability.metricsquery.api.Filter
properties:
op:
type: string
enum:
- NOT
filter:
$ref: '#/components/schemas/Filter'
required:
- op
- filter
example:
op: NOT
filter:
field: metric.topic
op: EQ
value: topicA
OrderBy:
type: object
properties:
metric:
type: string
agg:
$ref: '#/components/schemas/AggregationFunction'
order:
type: string
enum:
- ASCENDING
- DESCENDING
default: DESCENDING
required:
- metric
example:
metric: io.confluent.kafka.server/bytes_in
agg: SUM
order: DESCENDING
Interval:
type: string
format: ISO-8601 interval (/ | / | /)
ResponseFormat:
description: |
Desired response format for query results.
* `FLAT` (default): Each item in the response `data` array represents a data point in the
timeseries. Each data point contains the `timestamp`, metric aggregation `value` and
attributes for the `group_by` labels.
* `GROUPED`: Each item in the response `data` array represents a group. Each group contains
attributes for the `group_by` labels and an array of `points` for the metric aggregation
timeseries. **Only allowed when `group_by` is non-empty.**
Please see the response schema and accompanying examples for more details.
type: string
enum:
- FLAT
- GROUPED
default: FLAT
QueryRequest:
type: object
x-java-class: io.confluent.observability.metricsquery.api.MetricsQueryRequest
properties:
aggregations:
description: Specifies which metrics to query and the aggregation operator to apply across the `group_by` labels. **Currently, only one aggregation per request is supported.**
type: array
items:
$ref: '#/components/schemas/Aggregation'
minItems: 1
maxItems: 1
group_by:
description: |
Specifies how data gets bucketed by label(s); see [here](#section/Object-Model/Labels)
on using labels for grouping query results.
type: array
items:
type: string
granularity:
$ref: '#/components/schemas/Granularity'
filter:
$ref: '#/components/schemas/Filter'
order_by:
description: |
Sort ordering for result groups. **Only valid for `granularity: "ALL"`.**
If not specified, defaults to the first `aggregation` in descending order.
Note that this ordering applies to the groups.
Within a group (or for ungrouped results), data points are always ordered by `timestamp`
in descending order.
type: array
items:
$ref: '#/components/schemas/OrderBy'
maxItems: 1
intervals:
description: |
Defines the time range(s) that the query runs over.
A time range is an ISO-8601 interval.
The keyword `now` can be used in place of a timestamp to refer to the current time.
Offset and truncation modifiers can be also be applied to the `now` expression:
| Modifier | Syntax | Examples |
| --- | --- | --- |
| Offset | `(+\|-)(m\|h\|d)` | `-2m` (minus 2 minutes) `-1h` (minus 1 hour) |
| Truncation | `\|(m\|h\|d)` | `\|m` (round down to start of minute) `\|h` (round down to start of hour) |
All hour/day truncation is performed against the UTC timezone.
If `now` is `2020-01-01T02:13:27Z`, some examples are:
* `now-2m|m`: `now` minus 2 minutes, truncated to start of minute.
Resolves to `2020-01-01T02:11:00Z`
* `now|h`: `now` truncated to start of hour.
Resolves to `2020-01-01T02:00:00Z`
* `now-1d|d`: `now` minus 1 day, truncated to start of day.
Resolves to `2019-12-31T00:00:00Z`
When using `now`, it is recommended to apply a negative offset to avoid incomplete data
(see [metric availability delays](#section/Client-Considerations-and-Best-Practices/Metric-Data-Latency))
and align to minute boundaries (e.g. `now-2m|m`).
type: array
items:
$ref: '#/components/schemas/Interval'
minItems: 1
limit:
description: The maximum number of _groups_ to return. Only supported with a non-empty `group_by` field. The maximum number of data points in the response is equal to `limit * (interval / granularity)`. For example, with an interval of 1 day, granularity of `PT1H`, and limit of `2` there will be a maximum of 48 data points in the response (24 for each group). For granularity of `ALL`, the limit defaults to 1000.
type: integer
minimum: 1
maximum: 1000
default: 100
format:
$ref: '#/components/schemas/ResponseFormat'
required:
- intervals
- aggregations
- granularity
Point:
type: object
properties:
timestamp:
description: The timestamp for this time bucket, aligned to UTC boundaries.
type: string
format: date-time
value:
description: The value for the requested aggregation for this time bucket.
type: number
additionalProperties: true
required:
- timestamp
- value
example:
timestamp: '2019-10-17T20:17:00.000Z'
resource.kafka.id: lkc-12345entry
metric.topic: foo
value: 9741
FlatQueryResponse:
title: Flat Response
type: object
properties:
data:
description: An array of results for this query. Each item includes `timestamp`, `value`, and an attribute for each label specified in the request's `group_by`.
type: array
items:
$ref: '#/components/schemas/Point'
meta:
$ref: '#/components/schemas/Meta'
required:
- data
example:
data:
- timestamp: '2019-10-17T20:17:00.000Z'
metric.topic: foo
value: 9741
- timestamp: '2019-10-17T20:18:00.000Z'
metric.topic: foo
value: 9246
- timestamp: '2019-10-17T20:17:00.000Z'
metric.topic: bar
value: 844.1
- timestamp: '2019-10-17T20:18:00.000Z'
metric.topic: bar
value: 821.1
meta:
pagination:
page_size: 3
total_size: 3
GroupedQueryResponse:
title: Grouped Response
type: object
properties:
data:
description: An array of results for this query. Each item represents a group bucket having a distinct set of label values for the request's `group_by`. The groups are ordered lexicographically by the label values for that group.
type: array
items:
type: object
properties:
points:
type: array
items:
$ref: '#/components/schemas/Point'
additionalProperties: true
required:
- points
meta:
$ref: '#/components/schemas/Meta'
required:
- data
example:
data:
- metric.topic: foo
points:
- timestamp: '2019-10-17T20:17:00.000Z'
value: 9741
- timestamp: '2019-10-17T20:18:00.000Z'
value: 9246
- metric.topic: bar
points:
- timestamp: '2019-10-17T20:17:00.000Z'
value: 844.1
- timestamp: '2019-10-17T20:18:00.000Z'
value: 821.1
meta:
pagination:
page_size: 3
total_size: 3
QueryResponse:
oneOf:
- $ref: '#/components/schemas/FlatQueryResponse'
- $ref: '#/components/schemas/GroupedQueryResponse'
AttributesRequest:
type: object
x-java-class: io.confluent.observability.metricsquery.attributes.AttributesRequest
properties:
metric:
type: string
description: The metric that the label values are enumerated for.
group_by:
description: The label(s) that the values are enumerated for.
type: array
items:
type: string
minItems: 1
filter:
$ref: '#/components/schemas/Filter'
intervals:
description: Defines the time range(s) for which available metrics will be listed. A time range is an ISO-8601 interval. When unspecified, the value defaults to the last hour before the request was made
type: array
items:
$ref: '#/components/schemas/Interval'
minItems: 1
limit:
type: integer
minimum: 1
maximum: 1000
default: 100
required:
- group_by
AttributesResponse:
type: object
properties:
data:
description: The enumerated labels, as an array of key/value pairs.
type: array
items:
description: A key/value pair for the label.
type: object
additionalProperties:
type: string
example:
topic: foo
meta:
$ref: '#/components/schemas/Meta'
links:
$ref: '#/components/schemas/Links'
required:
- data
- meta
example:
data:
- metric.label.topic: foo
- metric.label.topic: bar
- metric.label.topic: baz
meta:
pagination:
page_size: 3
total_size: 3
responses:
RateLimitError:
description: Rate Limit Exceeded
x-tagGroups:
- name: API Endpoints
tags:
- Version 2