= Hawkular Metrics

This project is an implementation of the Vert.x Metrics Service Provider Interface (SPI): metrics built from Vert.x
events will be sent to Hawkular, an http://www.hawkular.org[open source monitoring and management solution].

== Features

* Vert.x core tools monitoring: TCP/HTTP client and servers, `link:../../scaladocs/io/vertx/scala/core/datagram/DatagramSocket.html[DatagramSocket]`,
`link:../../scaladocs/io/vertx/scala/core/eventbus/EventBus.html[EventBus]` and handlers
* User defined metrics via an `link:../../scaladocs/io/vertx/scala/core/eventbus/EventBus.html[EventBus]` bridge.

== Prerequisites

Follow the http://www.hawkular.org/hawkular-services/docs/quickstart-guide/[instructions to get Hawkular up and running].

NOTE: You can use a standalone https://github.com/hawkular/hawkular-metrics[Hawkular Metrics] server as well.

== Getting started

The _vertx-hawkular-metrics-scala_2.12_ module must be present in the classpath.

Maven users should add this to their project POM file:

[source,xml,subs="+attributes"]
----
<dependency>
  <groupId>io.vertx</groupId>
  <artifactId>vertx-hawkular-metrics-scala_2.12</artifactId>
  <version>3.5.2.CR2</version>
</dependency>
----

And Gradle users, to their build file:

[source,groovy,subs="+attributes"]
----
compile 'io.vertx:vertx-hawkular-metrics-scala_2.12:3.5.2.CR2'
----

Vert.x does not enable SPI implementations by default. You must enable metric collection in the Vert.x options:

[source,scala]
----
var vertx = Vertx.vertx(VertxOptions()
  .setMetricsOptions(VertxHawkularOptions()
    .setEnabled(true)
  )
)

----

== Configuration

=== Remote Hawkular server

By default, _vertx-hawkular-metrics-scala_2.12_ sends metrics to a Hawkular server listening on `localhost` port `8080`.
But in production, the Hawkular server will likely run on a separate machine:

[source,scala]
----
var vertx = Vertx.vertx(VertxOptions()
  .setMetricsOptions(VertxHawkularOptions()
    .setEnabled(true)
    .setHost("hawkular.example.com")
    .setPort(8080)
  )
)

----

=== Tenant selection

Hawkular Metrics is a multi-tenant solution, and _vertx-hawkular-metrics-scala_2.12_ can send metrics for a tenant other than `default`:

[source,scala]
----
var vertx = Vertx.vertx(VertxOptions()
  .setMetricsOptions(VertxHawkularOptions()
    .setEnabled(true)
    .setTenant("sales-department")
  )
)

----

=== Connecting to a Hawkular server

Requests sent to a Hawkular server must be authenticated and tenant must be set to `hawkular`:

[source,scala]
----
var vertx = Vertx.vertx(VertxOptions()
  .setMetricsOptions(VertxHawkularOptions()
    .setEnabled(true)
    .setTenant("hawkular")
    .setAuthenticationOptions(AuthenticationOptions()
      .setEnabled(true)
      .setId("username")
      .setSecret("password")
    )
  )
)

----

=== Openshift Metrics token authentication

When working with Openshift's internal Metrics server, you can configure token authentication with a custom HTTP header:

[source,scala]
----
var vertx = Vertx.vertx(VertxOptions()
  .setMetricsOptions(VertxHawkularOptions()
    .setEnabled(true)
    .setTenant("my-namespace")
    .setHttpHeaders(new io.vertx.core.json.JsonObject().put("Authorization", "Bearer xkjdksf9890-shjkjhkjlkjlk"))
  )
)

----

=== HTTPS and other HTTP related options

_vertx-hawkular-metrics-scala_2.12_ communicates with the Hawkular server over HTTP. In order to communicate over HTTPS, set the
`ssl` flag to true:

[source,scala]
----
var vertx = Vertx.vertx(VertxOptions()
  .setMetricsOptions(VertxHawkularOptions()
    .setEnabled(true)
    .setHost("hawkular.example.com")
    .setPort(443)
    .setHttpOptions(HttpClientOptions()
      .setSsl(true)
    )
  )
)

----

NOTE: The usual `link:../../vertx-core/dataobjects.html#HttpClientOptions[HttpClientOptions]` properties can be used for SSL setup or client
tuning.

=== Metric tags

http://www.hawkular.org/hawkular-metrics/docs/user-guide/#_tagging[Tags] can be applied to metrics:

[source,scala]
----
var vertx = Vertx.vertx(VertxOptions()
  .setMetricsOptions(VertxHawkularOptions()
    .setEnabled(true)
    .setTags(new io.vertx.core.json.JsonObject().put("dc", "mars01").put("rack", "web-paca").put("host", "host13"))
  )
)

----

_vertx-hawkular-metrics-scala_2.12_ maintains a LRU cache of tagged metrics to avoid repeating tagging requests.
The cache size can be configured and defaults to `4096` metric names.

It is also possible to apply tags to a specific set of metrics defined via exact match or regex match:

[source,scala]
----
var vertx = Vertx.vertx(VertxOptions()
  .setMetricsOptions(VertxHawkularOptions()
    .setEnabled(true)
    .setMetricTagsMatches(List(MetricTagsMatch()
      .setValue("myapp.foo.my-metric")
      .setTags(new io.vertx.core.json.JsonObject().put("myapp", "foo"))
    , MetricTagsMatch()
      .setType(MatchType.REGEX)
      .setValue(".*\\.foo\\.*")
      .setTags(new io.vertx.core.json.JsonObject().put("myapp", "foo"))
    ))
  )
)

----

WARNING: If you use regex match, a wrong regex can potentially match a lot of metrics.

NOTE: When evaluating tags to apply, metric specific tags have higher priority than global tags.
In other words, a metric specific tag may overwrite a global tag.

Please refer to `link:../dataobjects.html#VertxHawkularOptions[VertxHawkularOptions]` for an exhaustive list of options.

== Vert.x core tools metrics

This section lists all the metrics generated by monitoring the Vert.x core tools.

=== Net Client

[cols="15,50,35", options="header"]
|===
|Metric type
|Metric name
|Description

|Gauge
|`vertx.net.client.<host>:<port>.connections`
|Number of connections to the remote host currently opened.

|Counter
|`vertx.net.client.<host>:<port>.bytesReceived`
|Total number of bytes received from the remote host.

|Counter
|`vertx.net.client.<host>:<port>.bytesSent`
|Total number of bytes sent to the remote host.

|Counter
|`vertx.net.client.<host>:<port>.errorCount`
|Total number of errors.

|===

=== HTTP Client

[cols="15,50,35", options="header"]
|===
|Metric type
|Metric name
|Description

|Gauge
|`vertx.http.client.<host>:<port>.connections`
|Number of connections to the remote host currently opened.

|Counter
|`vertx.http.client.<host>:<port>.bytesReceived`
|Total number of bytes received from the remote host.

|Counter
|`vertx.http.client.<host>:<port>.bytesSent`
|Total number of bytes sent to the remote host.

|Counter
|`vertx.http.client.<host>:<port>.errorCount`
|Total number of errors.

|Gauge
|`vertx.http.client.<host>:<port>.requests`
|Number of requests waiting for a response.

|Counter
|`vertx.http.client.<host>:<port>.requestCount`
|Total number of requests sent.

|Counter
|`vertx.http.client.<host>:<port>.responseTime`
|Cumulated response time.

|Gauge
|`vertx.http.client.<host>:<port>.wsConnections`
|Number of websockets currently opened.

|===

=== Datagram socket

[cols="15,50,35", options="header"]
|===
|Metric type
|Metric name
|Description

|Counter
|`vertx.datagram.<host>:<port>.bytesReceived`
|Total number of bytes received on the `<host>:<port>` listening address.

|Counter
|`vertx.datagram.<host>:<port>.bytesSent`
|Total number of bytes sent to the remote host.

|Counter
|`vertx.datagram.errorCount`
|Total number of errors.

|===

=== Net Server

[cols="15,50,35", options="header"]
|===
|Metric type
|Metric name
|Description

|Gauge
|`vertx.net.server.<host>:<port>.connections`
|Number of opened connections to the Net Server listening on the `<host>:<port>` address.

|Counter
|`vertx.net.server.<host>:<port>.bytesReceived`
|Total number of bytes received by the Net Server listening on the `<host>:<port>` address.

|Counter
|`vertx.net.server.<host>:<port>.bytesSent`
|Total number of bytes sent to the Net Server listening on the `<host>:<port>` address.

|Counter
|`vertx.net.server.<host>:<port>.errorCount`
|Total number of errors.

|===

=== HTTP Server

[cols="15,50,35", options="header"]
|===
|Metric type
|Metric name
|Description

|Gauge
|`vertx.http.server.<host>:<port>.connections`
|Number of opened connections to the HTTP Server listening on the `<host>:<port>` address.

|Counter
|`vertx.http.server.<host>:<port>.bytesReceived`
|Total number of bytes received by the HTTP Server listening on the `<host>:<port>` address.

|Counter
|`vertx.http.server.<host>:<port>.bytesSent`
|Total number of bytes sent to the HTTP Server listening on the `<host>:<port>` address.

|Counter
|`vertx.http.server.<host>:<port>.errorCount`
|Total number of errors.

|Gauge
|`vertx.http.client.<host>:<port>.requests`
|Number of requests being processed.

|Counter
|`vertx.http.client.<host>:<port>.requestCount`
|Total number of requests processed.

|Counter
|`vertx.http.client.<host>:<port>.processingTime`
|Cumulated request processing time.

|Gauge
|`vertx.http.client.<host>:<port>.wsConnections`
|Number of websockets currently opened.

|===

=== Event Bus

[cols="15,50,35", options="header"]
|===
|Metric type
|Metric name
|Description

|Gauge
|`vertx.eventbus.handlers`
|Number of event bus handlers.

|Counter
|`vertx.eventbus.errorCount`
|Total number of errors.

|Counter
|`vertx.eventbus.bytesWritten`
|Total number of bytes sent while sending messages to event bus cluster peers.

|Counter
|`vertx.eventbus.bytesRead`
|Total number of bytes received while reading messages from event bus cluster peers.

|Gauge
|`vertx.eventbus.pending`
|Number of messages not processed yet. One message published will count for `N` pending if `N` handlers
are registered to the corresponding address.

|Gauge
|`vertx.eventbus.pendingLocal`
|Like `vertx.eventbus.pending`, for local messages only.

|Gauge
|`vertx.eventbus.pendingRemote`
|Like `vertx.eventbus.pending`, for remote messages only.

|Counter
|`vertx.eventbus.publishedMessages`
|Total number of messages published (publish / subscribe).

|Counter
|`vertx.eventbus.publishedLocalMessages`
|Like `vertx.eventbus.publishedMessages`, for local messages only.

|Counter
|`vertx.eventbus.publishedRemoteMessages`
|Like `vertx.eventbus.publishedMessages`, for remote messages only.

|Counter
|`vertx.eventbus.sentMessages`
|Total number of messages sent (point-to-point).

|Counter
|`vertx.eventbus.sentLocalMessages`
|Like `vertx.eventbus.sentMessages`, for local messages only.

|Counter
|`vertx.eventbus.sentRemoteMessages`
|Like `vertx.eventbus.sentMessages`, for remote messages only.

|Counter
|`vertx.eventbus.receivedMessages`
|Total number of messages received.

|Counter
|`vertx.eventbus.receivedLocalMessages`
|Like `vertx.eventbus.receivedMessages`, for remote messages only.

|Counter
|`vertx.eventbus.receivedRemoteMessages`
|Like `vertx.eventbus.receivedMessages`, for remote messages only.

|Counter
|`vertx.eventbus.deliveredMessages`
|Total number of messages delivered to handlers.

|Counter
|`vertx.eventbus.deliveredLocalMessages`
|Like `vertx.eventbus.deliveredMessages`, for remote messages only.

|Counter
|`vertx.eventbus.deliveredRemoteMessages`
|Like `vertx.eventbus.deliveredMessages`, for remote messages only.

|Counter
|`vertx.eventbus.replyFailures`
|Total number of message reply failures.

|Counter
|`vertx.eventbus.<address>.processingTime`
|Cumulated processing time for handlers listening to the `address`.

|===

== Vert.x pool metrics

This section lists all the metrics generated by monitoring Vert.x pools.

There are two types currently supported:

* _worker_ (see `link:../../scaladocs/io/vertx/scala/core/WorkerExecutor.html[WorkerExecutor]`)
* _datasource_ (created with Vert.x JDBC client)

NOTE: Vert.x creates two worker pools upfront, _vert.x-worker-thread_ and _vert.x-internal-blocking_.

All metrics are prefixed with `<type>.<name>.`. For example, `worker.vert.x-internal-blocking.`.

[cols="15,50,35", options="header"]
|===
|Metric type
|Metric name
|Description

|Counter
|`vertx.pool.<type>.<name>.delay`
|Cumulated time waiting for a resource (queue time).

|Gauge
|`vertx.pool.<type>.<name>.queued`
|Current number of elements waiting for a resource.

|Counter
|`vertx.pool.<type>.<name>.queueCount`
|Total number of elements queued.

|Counter
|`vertx.pool.<type>.<name>.usage`
|Cumulated time using a resource (i.e. processing time for worker pools).

|Gauge
|`vertx.pool.<type>.<name>.inUse`
|Current number of resources used.

|Counter
|`vertx.pool.<type>.<name>.completed`
|Total number of elements done with the resource (i.e. total number of tasks executed for worker pools).

|Gauge
|`vertx.pool.<type>.<name>.maxPoolSize`
|Maximum pool size, only present if it could be determined.

|Gauge
|`vertx.pool.<type>.<name>.inUse`
|Pool usage ratio, only present if maximum pool size could be determined.

|===

== Verticle metrics

[cols="15,50,35", options="header"]
|===
|Metric type
|Metric name
|Description

|Gauge
|`vertx.verticle.<name>`
|Number of verticle instances deployed.

|===

== User defined metrics

Users can send their own metrics to the Hawkular server. In order to do so, the event bus metrics bridge must be
enabled:

[source,scala]
----
var vertx = Vertx.vertx(VertxOptions()
  .setMetricsOptions(VertxHawkularOptions()
    .setEnabled(true)
    .setMetricsBridgeEnabled(true)
  )
)

----

By default, the metrics bus handler is listening to the `hawkular.metrics` address. But the bridge address
can be configured:

[source,scala]
----
var vertx = Vertx.vertx(VertxOptions()
  .setMetricsOptions(VertxHawkularOptions()
    .setEnabled(true)
    .setMetricsBridgeEnabled(true)
    .setMetricsBridgeAddress("__hawkular_metrics")
  )
)

----

The metrics bridge handler expects messages in the JSON format. The JSON object must at least provide a metric
`id` and a numerical `value`:

[source,scala]
----
var message = new io.vertx.core.json.JsonObject().put("id", "myapp.files.opened").put("value", 7)
vertx.eventBus().publish("hawkular.metrics", message)

----

The handler will assume the metric is a gauge and will assign a timestamp corresponding to the time when the message was processed.
If the metric is a counter or availability, or if you prefer explicit configuration, set the `type` and/or `timestamp` attributes:

[source,scala]
----
var counterMetric = new io.vertx.core.json.JsonObject().put("id", "myapp.files.opened").put("type", "counter").put("timestamp", 189898098098908).put("value", 7)
vertx.eventBus().publish("hawkular.metrics", counterMetric)

var availabilityMetric = new io.vertx.core.json.JsonObject().put("id", "myapp.mysubsystem.status").put("type", "availability").put("value", "up")
vertx.eventBus().publish("hawkular.metrics", availabilityMetric)

----

NOTE: Hawkular understands all timestamps as milliseconds since January 1, 1970, 00:00:00 UTC.