= Vert.x Micrometer Metrics

This project is an implementation of the Vert.x Metrics Service Provider Interface (SPI).
It uses link:http://micrometer.io/[Micrometer] for managing metrics and reporting to several backends.

== Features

* Vert.x core tools monitoring: TCP/HTTP client and servers, `link:../../apidocs/io/vertx/core/datagram/DatagramSocket.html[DatagramSocket]`
, `link:../../apidocs/io/vertx/core/eventbus/EventBus.html[EventBus]` and pools
* User defined metrics through Micrometer
* Reporting to any backend supported by Micrometer
* Built-in options for https://www.influxdata.com/[InfluxDB], https://prometheus.io/[Prometheus] and JMX reporting.

== InfluxDB

=== Prerequisites

Follow the https://docs.influxdata.com/influxdb/latest/introduction/getting_started/[instructions to get InfluxDb up and running].

=== Getting started

The modules _vertx-micrometer-metrics_ and _micrometer-registry-influx_ must be present in the classpath.

Maven users should add this to their project POM file:

[source,xml,subs="+attributes"]
----
<dependency>
 <groupId>io.vertx</groupId>
 <artifactId>vertx-micrometer-metrics</artifactId>
 <version>3.8.1</version>
</dependency>
<dependency>
 <groupId>io.micrometer</groupId>
 <artifactId>micrometer-registry-influx</artifactId>
 <version>${micrometer.version}</version>
</dependency>
----

And Gradle users, to their build file:

[source,groovy,subs="+attributes"]
----
compile 'io.vertx:vertx-micrometer-metrics:3.8.1'
compile 'io.micrometer:micrometer-registry-influx:${micrometer.version}'
----

=== Configuration examples

Vert.x does not enable SPI implementations by default. You must enable metric collection in the Vert.x options.

[source,groovy]
----
def vertx = Vertx.vertx([
  metricsOptions:[
    influxDbOptions:[
      enabled:true
    ],
    enabled:true
  ]
])

----

==== Using a specific URI and database name

[source,groovy]
----
def vertx = Vertx.vertx([
  metricsOptions:[
    influxDbOptions:[
      enabled:true,
      uri:"http://influxdb.example.com:8888",
      db:"sales-department"
    ],
    enabled:true
  ]
])

----

==== With authentication

[source,groovy]
----
def vertx = Vertx.vertx([
  metricsOptions:[
    influxDbOptions:[
      enabled:true,
      userName:"username",
      password:"password"
    ],
    enabled:true
  ]
])

----

== Prometheus

=== Prerequisites

Follow the https://prometheus.io/docs/prometheus/latest/getting_started/[instructions to get Prometheus up and running].

=== Getting started

The modules _vertx-micrometer-metrics_ and _micrometer-registry-prometheus_ must be present in the classpath.
You may also probably need _vertx-web_, to expose the metrics.

Maven users should add this to their project POM file:

[source,xml,subs="+attributes"]
----
<dependency>
 <groupId>io.vertx</groupId>
 <artifactId>vertx-micrometer-metrics</artifactId>
 <version>3.8.1</version>
</dependency>
<dependency>
 <groupId>io.micrometer</groupId>
 <artifactId>micrometer-registry-prometheus</artifactId>
 <version>${micrometer.version}</version>
</dependency>
----

And Gradle users, to their build file:

[source,groovy,subs="+attributes"]
----
compile 'io.vertx:vertx-micrometer-metrics:3.8.1'
compile 'io.micrometer:micrometer-registry-prometheus:${micrometer.version}'
----

=== Configuration examples

Vert.x does not enable SPI implementations by default. You must enable metric collection in the Vert.x options

[source,groovy]
----
def vertx = Vertx.vertx([
  metricsOptions:[
    prometheusOptions:[
      enabled:true
    ],
    enabled:true
  ]
])

----

==== Using an embedded HTTP server with custom endpoint

[source,groovy]
----
def vertx = Vertx.vertx([
  metricsOptions:[
    prometheusOptions:[
      enabled:true,
      startEmbeddedServer:true,
      embeddedServerOptions:[
        port:8080
      ],
      embeddedServerEndpoint:"/metrics/vertx"
    ],
    enabled:true
  ]
])

----

If the embedded server endpoint is not specified, it defaults to _/metrics_.

==== Binding metrics to an existing Vert.x Web router

[source,groovy]
----
def vertx = Vertx.vertx([
  metricsOptions:[
    prometheusOptions:[
      enabled:true
    ],
    enabled:true
  ]
])

// Later on, creating a router
def router = Router.router(vertx)
router.route("/metrics").handler(PrometheusScrapingHandler.create())
vertx.createHttpServer().requestHandler(router).listen(8080)

----

== JMX

=== Getting started

The modules _vertx-micrometer-metrics_ and _micrometer-registry-jmx_ must be present in the classpath.

Maven users should add this to their project POM file:

[source,xml,subs="+attributes"]
----
<dependency>
 <groupId>io.vertx</groupId>
 <artifactId>vertx-micrometer-metrics</artifactId>
 <version>3.8.1</version>
</dependency>
<dependency>
 <groupId>io.micrometer</groupId>
 <artifactId>micrometer-registry-jmx</artifactId>
 <version>${micrometer.version}</version>
</dependency>
----

And Gradle users, to their build file:

[source,groovy,subs="+attributes"]
----
compile 'io.vertx:vertx-micrometer-metrics:3.8.1'
compile 'io.micrometer:micrometer-registry-jmx:${micrometer.version}'
----

=== Configuration examples

Vert.x does not enable SPI implementations by default. You must enable metric collection in the Vert.x options

[source,groovy]
----
def vertx = Vertx.vertx([
  metricsOptions:[
    jmxMetricsOptions:[
      enabled:true
    ],
    enabled:true
  ]
])

----

==== With step and domain

In Micrometer, `step` refers to the reporting period, in seconds. `domain` is the JMX domain under which
MBeans are registered.

[source,groovy]
----
def vertx = Vertx.vertx([
  metricsOptions:[
    jmxMetricsOptions:[
      enabled:true,
      step:5,
      domain:"my.metrics.domain"
    ],
    enabled:true
  ]
])

----

== Other backends or combinations

Even if not all backends supported by Micrometer are implemented in Vert.x options, it is still possible
to create any Micrometer registry and pass it to Vert.x.

The list of available backends includes Graphite, Ganglia, Atlas, link:http://micrometer.io/docs[etc].
It also enables the link:http://micrometer.io/docs/concepts#_composite_registries[Micrometer Composite Registry]
in order to report the same metrics to multiple backends.

In this example, metrics are reported both for JMX and Graphite:

[source,groovy]
----
def myRegistry = new io.micrometer.core.instrument.composite.CompositeMeterRegistry()
myRegistry.add(new io.micrometer.jmx.JmxMeterRegistry({ s ->
  null
}, io.micrometer.core.instrument.Clock.SYSTEM))
myRegistry.add(new io.micrometer.graphite.GraphiteMeterRegistry({ s ->
  null
}, io.micrometer.core.instrument.Clock.SYSTEM))

def vertx = Vertx.vertx([
  metricsOptions:[
    micrometerRegistry:myRegistry,
    enabled:true
  ]
])

----

== Advanced usage

Please refer to `link:../../apidocs/io/vertx/micrometer/MicrometerMetricsOptions.html[MicrometerMetricsOptions]` for an exhaustive list of options.

=== Averages and quantiles in Prometheus

By default, when using the Prometheus registry, histogram-kind metrics will not contain averages or quantile stats.

Averages don't come out of the box but they are typically link:https://prometheus.io/docs/practices/histograms/#count-and-sum-of-observations[computed at query time],
with `promql`. Example, for HTTP client response time average during the last 5 minutes:

[source]
----
 rate(vertx_http_client_responseTime_seconds_sum[5m])
/
 rate(vertx_http_client_responseTime_seconds_count[5m])
----

To compute quantiles, there are two options available. The first is to activate quantile stats globally
and make them usable for Prometheus function `histogram_quantile`:

[source,groovy]
----
def vertx = Vertx.vertx([
  metricsOptions:[
    prometheusOptions:[
      enabled:true,
      publishQuantiles:true
    ],
    enabled:true
  ]
])

----

And then, for example the `promql` query for the HTTP client response time, 99th percentile over the last 5 minutes:
[source]
----
 histogram_quantile(0.99, sum(rate(vertx_http_client_responseTime_seconds_bucket[5m])) by (le))
----

The advantage of this option is that it can be leveraged in `promql`, aggregable across dimensions.
The downside is that it creates a lot of timeseries for stats under the hood.

The second option is to create limited stats, non-aggregable across dimensions.
It requires to access directly the Micrometer / Prometheus registry:

[source,groovy]
----
def registry = io.vertx.micrometer.backends.BackendRegistries.getDefaultNow()
registry.config().meterFilter(new io.micrometer.core.instrument.config.MeterFilter())

----

See also, more on histograms and percentiles:

* from link:https://micrometer.io/docs/concepts#_histograms_and_percentiles[Micrometer doc]
* from link:https://prometheus.io/docs/prometheus/latest/querying/functions/#histogram_quantile[Prometheus doc]

Furthermore, you can check some link:https://github.com/vert-x3/vertx-examples/tree/master/micrometer-metrics-examples[full working examples].
They come along with few instructions to setup with Prometheus and view dashboards in Grafana.

=== Disable some metric domains

Restricting the Vert.x modules being monitored can be done using
`link:../../apidocs/io/vertx/micrometer/MicrometerMetricsOptions.html#disabledMetricsCategories[disabledMetricsCategories]`.

For a full list of domains, see `link:../../apidocs/io/vertx/micrometer/MetricsDomain.html[MetricsDomain]`

=== User-defined metrics

The Micrometer registries are accessible, in order to create new metrics or fetch the existing ones.
By default, an unique registry is used and will be shared across the Vert.x instances of the JVM:

[source,groovy]
----
def registry = io.vertx.micrometer.backends.BackendRegistries.getDefaultNow()

----

It is also possible to have separate registries per Vertx instance, by giving a registry name in metrics options.
Then it can be retrieved specifically:

[source,groovy]
----
def vertx = Vertx.vertx([
  metricsOptions:[
    influxDbOptions:[
      enabled:true
    ],
    registryName:"my registry",
    enabled:true
  ]
])

// Later on:
def registry = io.vertx.micrometer.backends.BackendRegistries.getNow("my registry")

----

As an example, here is a custom timer that will track the execution time of a piece of code that is regularly called:

[source,groovy]
----
def registry = io.vertx.micrometer.backends.BackendRegistries.getDefaultNow()
def timer = io.micrometer.core.instrument.Timer.builder("my.timer").description("a description of what this timer does").register(registry)

vertx.setPeriodic(1000, { l ->
  timer.record({ ->
    // Running here some operation to monitor
  })
})

----

For more examples, documentation about the Micrometer registry and how to create metrics, check
link:http://micrometer.io/docs/concepts#_registry[Micrometer doc].

=== Reusing an existing registry

It is possible to reuse an existing Micrometer registry (or `CollectorRegistry` from the Prometheus Java client),
and inject it into the Vert.x metrics options:

[source,groovy]
----
// This registry might be used to collect metrics other than Vert.x ones
def registry = new io.micrometer.prometheus.PrometheusMeterRegistry(io.micrometer.prometheus.PrometheusConfig.DEFAULT)

// You could also reuse an existing registry from the Prometheus Java client:
def prometheusClientRegistry = new io.prometheus.client.CollectorRegistry()
registry = new io.micrometer.prometheus.PrometheusMeterRegistry(io.micrometer.prometheus.PrometheusConfig.DEFAULT, prometheusClientRegistry, io.micrometer.core.instrument.Clock.SYSTEM)

// It's reused in MicrometerMetricsOptions.
// Prometheus options configured here, such as "setPublishQuantiles(true)", will affect the whole registry.
def vertx = Vertx.vertx([
  metricsOptions:[
    prometheusOptions:[
      enabled:true,
      publishQuantiles:true
    ],
    micrometerRegistry:registry,
    enabled:true
  ]
])

----

=== JVM or other instrumentations

Since plain access to Micrometer registries is provided, it is possible to leverage the Micrometer API.
For instance, to instrument the JVM:

[source,groovy]
----
def registry = io.vertx.micrometer.backends.BackendRegistries.getDefaultNow()

new io.micrometer.core.instrument.binder.jvm.ClassLoaderMetrics().bindTo(registry)
new io.micrometer.core.instrument.binder.jvm.JvmMemoryMetrics().bindTo(registry)
new io.micrometer.core.instrument.binder.jvm.JvmGcMetrics().bindTo(registry)
new io.micrometer.core.instrument.binder.system.ProcessorMetrics().bindTo(registry)
new io.micrometer.core.instrument.binder.jvm.JvmThreadMetrics().bindTo(registry)

----

_From link:http://micrometer.io/docs/ref/jvm[Micrometer documentation]._

=== Labels and matchers

Vert.x Micrometer Metrics defines a set of labels (aka tags or fields) that are used to provide dimensionality
to a metric. For instance, metrics related to event bus messages have an _address_ label, which allows then to query
timeseries for a specific event bus address, or compare timeseries per address, or perform any kind of aggregation
that the query API allows.

While setting up metrics options, you can specify which labels you want to enable or not:

[source,groovy]
----
def vertx = Vertx.vertx([
  metricsOptions:[
    prometheusOptions:[
      enabled:true
    ],
    labels:java.util.EnumSet.of(Label.REMOTE, Label.LOCAL, Label.HTTP_CODE, Label.HTTP_PATH),
    enabled:true
  ]
])

----

The full list of labels is detailed here: `link:../../apidocs/io/vertx/micrometer/Label.html[Label]`.

WARNING: Enabling labels may result in a high cardinality in values, which can cause troubles on the metrics backend and affect performances.
So it must be used with care.
In general, it is fine to enable labels when the set of possible values is bounded.

For that reason, labels enabled by default are restricted to the ones with known bounded values.

It is possible to interact with labels further than just enabling/disabling. There are two ways for that:

==== Using Matchers

`link:../../apidocs/io/vertx/micrometer/Match.html[Match]`  objects can be used to filter or rename some label value
by matching it with either an exact string or a regular expression (the former being more efficient).

Here is an example to restrict HTTP server metrics to those with label _local=localhost:8080_ only:

[source,groovy]
----
def vertx = Vertx.vertx([
  metricsOptions:[
    prometheusOptions:[
      enabled:true
    ],
    labelMatchs:[
      [
        domain:"HTTP_SERVER",
        label:"local",
        value:"localhost:8080"
      ]
    ],
    enabled:true
  ]
])

----

When an _alias_ is specified in the Match, it will be used to rename value instead of filtering.

Matchers are especially useful to control labelling through configuration as they are set via
`link:../../apidocs/io/vertx/micrometer/MicrometerMetricsOptions.html[MicrometerMetricsOptions]`.

==== Using Micrometer's MeterFilter

Micrometer's link:http://micrometer.io/docs/concepts#_meter_filters[MeterFilter API] can be accessed directly in order to define rules on labels.
Compared to Matchers, it offers more features in manipulating the labels, but cannot be defined from configuration.
So both have their advantages.

Here is an example to replace the actual `path` label of HTTP requests with a generic form using regex:

[source,groovy]
----
def registry = io.vertx.micrometer.backends.BackendRegistries.getDefaultNow()
def pattern = java.util.regex.Pattern.compile("/foo/bar/.*")

registry.config().meterFilter(io.micrometer.core.instrument.config.MeterFilter.replaceTagValues(Label.HTTP_PATH.toString(), { actualPath ->
  def m = pattern.matcher(actualPath)
  if (m.matches()) {
    return "/foo/bar/:id"
  }
  return actualPath
}, ""))

----

NOTE: Matchers use MeterFilters under the hood.

=== Snapshots

A `link:../../apidocs/io/vertx/micrometer/MetricsService.html[MetricsService]` can be created out of a `link:../../apidocs/io/vertx/core/metrics/Measured.html[Measured]` object
in order to take a snapshot of its related metrics and measurements.
The snapshot is returned as a `link:../../apidocs/io/vertx/core/json/JsonObject.html[JsonObject]`.

A well known _Measured_ object is simply `link:../../apidocs/io/vertx/core/Vertx.html[Vertx]`:

[source,groovy]
----
def metricsService = MetricsService.create(vertx)
def metrics = metricsService.getMetricsSnapshot()
println(metrics)

----

Other components, such as an `link:../../apidocs/io/vertx/core/eventbus/EventBus.html[EventBus]` or a `link:../../apidocs/io/vertx/core/http/HttpServer.html[HttpServer]` are
measurable:

[source,groovy]
----
def server = vertx.createHttpServer()
def metricsService = MetricsService.create(server)
def metrics = metricsService.getMetricsSnapshot()
println(metrics)

----

Finally it is possible to filter the returned metrics from their base names:

[source,groovy]
----
def metricsService = MetricsService.create(vertx)
// Client + server
def metrics = metricsService.getMetricsSnapshot("vertx.http")
println(metrics)

----

== Vert.x core tools metrics

This section lists all the metrics generated by monitoring the Vert.x core tools.

NOTE: The metric backends may have different conventions or rules for naming metrics.
The names described below are given with underscore separators, but the actual names may vary
depending on the backend used.

=== Net Client

[cols="35,20,10,35", options="header"]
|===
|Metric name
|Labels
|Type
|Description

|`vertx_net_client_connections`
|`local`, `remote`
|Gauge
|Number of connections to the remote host currently opened.

|`vertx_net_client_bytesReceived`
|`local`, `remote`
|Summary
|Number of bytes received from the remote host.

|`vertx_net_client_bytesSent`
|`local`, `remote`
|Summary
|Number of bytes sent to the remote host.

|`vertx_net_client_errors`
|`local`, `remote`, `class`
|Counter
|Number of errors.

|===

=== HTTP Client

[cols="35,20,10,35", options="header"]
|===
|Metric name
|Labels
|Type
|Description

|`vertx_http_client_connections`
|`local`, `remote`
|Gauge
|Number of connections to the remote host currently opened.

|`vertx_http_client_bytesReceived`
|`local`, `remote`
|Summary
|Number of bytes received from the remote host.

|`vertx_http_client_bytesSent`
|`local`, `remote`
|Summary
|Number of bytes sent to the remote host.

|`vertx_http_client_errors`
|`local`, `remote`, `class`
|Counter
|Number of errors.

|`vertx_http_client_requests`
|`local`, `remote`, `path`, `method`
|Gauge
|Number of requests waiting for a response.

|`vertx_http_client_requestCount`
|`local`, `remote`, `path`, `method`
|Counter
|Number of requests sent.

|`vertx_http_client_responseTime`
|`local`, `remote`, `path`, `method`, `code`
|Timer
|Response time.

|`vertx_http_client_responseCount`
|`local`, `remote`, `path`, `method`, `code`
|Counter
|Number of received responses.

|`vertx_http_client_wsConnections`
|`local`, `remote`
|Gauge
|Number of websockets currently opened.

|===

=== Net Server

[cols="35,20,10,35", options="header"]
|===
|Metric name
|Labels
|Type
|Description

|`vertx_net_server_connections`
|`local`, `remote`
|Gauge
|Number of opened connections to the Net Server.

|`vertx_net_server_bytesReceived`
|`local`, `remote`
|Summary
|Number of bytes received by the Net Server.

|`vertx_net_server_bytesSent`
|`local`, `remote`
|Summary
|Number of bytes sent by the Net Server.

|`vertx_net_server_errors`
|`local`, `remote`, `class`
|Counter
|Number of errors.

|===

=== HTTP Server

[cols="35,20,10,35", options="header"]
|===
|Metric name
|Labels
|Type
|Description

|`vertx_http_server_connections`
|`local`, `remote`
|Gauge
|Number of opened connections to the HTTP Server.

|`vertx_http_server_bytesReceived`
|`local`, `remote`
|Summary
|Number of bytes received by the HTTP Server.

|`vertx_http_server_bytesSent`
|`local`, `remote`
|Summary
|Number of bytes sent by the HTTP Server.

|`vertx_http_server_errors`
|`local`, `remote`, `class`
|Counter
|Number of errors.

|`vertx_http_server_requests`
|`local`, `remote`, `path`, `method`
|Gauge
|Number of requests being processed.

|`vertx_http_server_requestCount`
|`local`, `remote`, `path`, `method`, `code`
|Counter
|Number of processed requests.

|`vertx_http_server_requestResetCount`
|`local`, `remote`, `path`, `method`
|Counter
|Number of requests reset.

|`vertx_http_server_processingTime`
|`local`, `remote`, `path`, `method`, `code`
|Timer
|Request processing time.

|`vertx_http_client_wsConnections`
|`local`, `remote`
|Gauge
|Number of websockets currently opened.

|===

=== Datagram socket

[cols="35,20,10,35", options="header"]
|===
|Metric name
|Labels
|Type
|Description

|`vertx_datagram_bytesReceived`
|`local`
|Summary
|Total number of bytes received on the `<host>:<port>` listening address.

|`vertx_datagram_bytesSent`
|(none)
|Summary
|Total number of bytes sent to the remote host.

|`vertx_datagram_errors`
|`class`
|Counter
|Total number of errors.

|===

=== Event Bus

[cols="35,20,10,35", options="header"]
|===
|Metric name
|Labels
|Type
|Description

|`vertx_eventbus_handlers`
|`address`
|Gauge
|Number of event bus handlers in use.

|`vertx_eventbus_errors`
|`address`,`class`
|Counter
|Number of errors.

|`vertx_eventbus_bytesWritten`
|`address`
|Summary
|Total number of bytes sent while sending messages to event bus cluster peers.

|`vertx_eventbus_bytesRead`
|`address`
|Summary
|Total number of bytes received while reading messages from event bus cluster peers.

|`vertx_eventbus_pending`
|`address`,`side` (local/remote)
|Gauge
|Number of messages not processed yet. One message published will count for `N` pending if `N` handlers
are registered to the corresponding address.

|`vertx_eventbus_published`
|`address`,`side` (local/remote)
|Counter
|Number of messages published (publish / subscribe).

|`vertx_eventbus_sent`
|`address`,`side` (local/remote)
|Counter
|Number of messages sent (point-to-point).

|`vertx_eventbus_received`
|`address`,`side` (local/remote)
|Counter
|Number of messages received.

|`vertx_eventbus_delivered`
|`address`,`side` (local/remote)
|Counter
|Number of messages delivered to handlers.

|`vertx_eventbus_replyFailures`
|`address`,`failure`
|Counter
|Number of message reply failures.

|`vertx_eventbus_processingTime`
|`address`
|Timer
|Processing time for handlers listening to the `address`.

|===

== Vert.x pool metrics

This section lists all the metrics generated by monitoring Vert.x pools.

There are two types currently supported:

* _worker_ (see `link:../../apidocs/io/vertx/core/WorkerExecutor.html[WorkerExecutor]`)
* _datasource_ (created with Vert.x JDBC client)

NOTE: Vert.x creates two worker pools upfront, _worker-thread_ and _internal-blocking_.

[cols="35,20,10,35", options="header"]
|===
|Metric name
|Labels
|Type
|Description

|`vertx_pool_queue_delay`
|`pool_type`,`pool_name`
|Timer
|Time waiting for a resource (queue time).

|`vertx_pool_queue_size`
|`pool_type`,`pool_name`
|Gauge
|Number of elements waiting for a resource.

|`vertx_pool_usage`
|`pool_type`,`pool_name`
|Timer
|Time using a resource (i.e. processing time for worker pools).

|`vertx_pool_inUse`
|`pool_type`,`pool_name`
|Gauge
|Number of resources used.

|`vertx_pool_completed`
|`pool_type`,`pool_name`
|Counter
|Number of elements done with the resource (i.e. total number of tasks executed for worker pools).

|`vertx_pool_ratio`
|`pool_type`,`pool_name`
|Gauge
|Pool usage ratio, only present if maximum pool size could be determined.

|===

== Verticle metrics

[cols="35,20,10,35", options="header"]
|===
|Metric name
|Labels
|Type
|Description

|`vertx_verticle_deployed`
|`name`
|Gauge
|Number of verticle instances deployed.

|===