# Hawkular Metrics

This project is an implementation of the Vert.x Metrics Service Provider Interface (SPI): metrics built from Vert.x events will be sent to Hawkular, an open source monitoring and management solution.

## Features

- Vert.x core tools monitoring: TCP/HTTP client and servers, `DatagramSocket`, `EventBus` and handlers

- User defined metrics via an `EventBus` bridge.

## Prerequisites

Follow the instructions to get Hawkular up and running.

Note | You can use a standalone Hawkular Metrics server as well.

## Getting started

The vertx-hawkular-metrics-scala_2.12 module must be present in the classpath.

Maven users should add this to their project POM file:

    <dependency>
      <groupId>io.vertx</groupId>
      <artifactId>vertx-hawkular-metrics-scala_2.12</artifactId>
      <version>3.5.3.CR1</version>
    </dependency>

And Gradle users, to their build file:

    compile 'io.vertx:vertx-hawkular-metrics-scala_2.12:3.5.3.CR1'

Vert.x does not enable SPI implementations by default. You must enable metric collection in the Vert.x options:

    var vertx = Vertx.vertx(VertxOptions()
      .setMetricsOptions(VertxHawkularOptions()
        .setEnabled(true)
      )
    )

## Configuration

### Remote Hawkular server

By default, vertx-hawkular-metrics-scala_2.12 sends metrics to a Hawkular server listening on `localhost` port `8080`. But in production, the Hawkular server will likely run on a separate machine:

    var vertx = Vertx.vertx(VertxOptions()
      .setMetricsOptions(VertxHawkularOptions()
        .setEnabled(true)
        .setHost("hawkular.example.com")
        .setPort(8080)
      )
    )

### Tenant selection

Hawkular Metrics is a multi-tenant solution, and vertx-hawkular-metrics-scala_2.12 can send metrics for a tenant other than `default`:

    var vertx = Vertx.vertx(VertxOptions()
      .setMetricsOptions(VertxHawkularOptions()
        .setEnabled(true)
        .setTenant("sales-department")
      )
    )

### Connecting to a Hawkular server

Requests sent to a Hawkular server must be authenticated and tenant must be set to `hawkular`:

    var vertx = Vertx.vertx(VertxOptions()
      .setMetricsOptions(VertxHawkularOptions()
        .setEnabled(true)
        .setTenant("hawkular")
        .setAuthenticationOptions(AuthenticationOptions()
          .setEnabled(true)
          .setId("username")
          .setSecret("password")
        )
      )
    )

### Openshift Metrics token authentication

When working with Openshift’s internal Metrics server, you can configure token authentication with a custom HTTP header:

    var vertx = Vertx.vertx(VertxOptions()
      .setMetricsOptions(VertxHawkularOptions()
        .setEnabled(true)
        .setTenant("my-namespace")
        .setHttpHeaders(new io.vertx.core.json.JsonObject().put("Authorization", "Bearer xkjdksf9890-shjkjhkjlkjlk"))
      )
    )

### HTTPS and other HTTP related options

vertx-hawkular-metrics-scala_2.12 communicates with the Hawkular server over HTTP. In order to communicate over HTTPS, set the `ssl` flag to true:

    var vertx = Vertx.vertx(VertxOptions()
      .setMetricsOptions(VertxHawkularOptions()
        .setEnabled(true)
        .setHost("hawkular.example.com")
        .setPort(443)
        .setHttpOptions(HttpClientOptions()
          .setSsl(true)
        )
      )
    )

Note | The usual `HttpClientOptions` properties can be used for SSL setup or client tuning.

### Metric tags

Tags can be applied to metrics:

    var vertx = Vertx.vertx(VertxOptions()
      .setMetricsOptions(VertxHawkularOptions()
        .setEnabled(true)
        .setTags(new io.vertx.core.json.JsonObject().put("dc", "mars01").put("rack", "web-paca").put("host", "host13"))
      )
    )

vertx-hawkular-metrics-scala_2.12 maintains a LRU cache of tagged metrics to avoid repeating tagging requests. The cache size can be configured and defaults to `4096` metric names.

It is also possible to apply tags to a specific set of metrics defined via exact match or regex match:

    var vertx = Vertx.vertx(VertxOptions()
      .setMetricsOptions(VertxHawkularOptions()
        .setEnabled(true)
        .setMetricTagsMatches(List(MetricTagsMatch()
          .setValue("myapp.foo.my-metric")
          .setTags(new io.vertx.core.json.JsonObject().put("myapp", "foo"))
        , MetricTagsMatch()
          .setType(MatchType.REGEX)
          .setValue(".*\\.foo\\.*")
          .setTags(new io.vertx.core.json.JsonObject().put("myapp", "foo"))
        ))
      )
    )

Warning | If you use regex match, a wrong regex can potentially match a lot of metrics.

Note | When evaluating tags to apply, metric specific tags have higher priority than global tags. In other words, a metric specific tag may overwrite a global tag.

Please refer to `VertxHawkularOptions` for an exhaustive list of options.

## Vert.x core tools metrics

This section lists all the metrics generated by monitoring the Vert.x core tools.

### Net Client

Metric type | Metric name | Description
--- | --- | ---
Gauge | `vertx.net.client.<host>:<port>.connections` | Number of connections to the remote host currently opened.
Counter | `vertx.net.client.<host>:<port>.bytesReceived` | Total number of bytes received from the remote host.
Counter | `vertx.net.client.<host>:<port>.bytesSent` | Total number of bytes sent to the remote host.
Counter | `vertx.net.client.<host>:<port>.errorCount` | Total number of errors.

### HTTP Client

Metric type | Metric name | Description
--- | --- | ---
Gauge | `vertx.http.client.<host>:<port>.connections` | Number of connections to the remote host currently opened.
Counter | `vertx.http.client.<host>:<port>.bytesReceived` | Total number of bytes received from the remote host.
Counter | `vertx.http.client.<host>:<port>.bytesSent` | Total number of bytes sent to the remote host.
Counter | `vertx.http.client.<host>:<port>.errorCount` | Total number of errors.
Gauge | `vertx.http.client.<host>:<port>.requests` | Number of requests waiting for a response.
Counter | `vertx.http.client.<host>:<port>.requestCount` | Total number of requests sent.
Counter | `vertx.http.client.<host>:<port>.responseTime` | Cumulated response time.
Gauge | `vertx.http.client.<host>:<port>.wsConnections` | Number of websockets currently opened.

### Datagram socket

Metric type | Metric name | Description
--- | --- | ---
Counter | `vertx.datagram.<host>:<port>.bytesReceived` | Total number of bytes received on the `<host>:<port>` listening address.
Counter | `vertx.datagram.<host>:<port>.bytesSent` | Total number of bytes sent to the remote host.
Counter | `vertx.datagram.errorCount` | Total number of errors.

### Net Server

Metric type | Metric name | Description
--- | --- | ---
Gauge | `vertx.net.server.<host>:<port>.connections` | Number of opened connections to the Net Server listening on the `<host>:<port>` address.
Counter | `vertx.net.server.<host>:<port>.bytesReceived` | Total number of bytes received by the Net Server listening on the `<host>:<port>` address.
Counter | `vertx.net.server.<host>:<port>.bytesSent` | Total number of bytes sent to the Net Server listening on the `<host>:<port>` address.
Counter | `vertx.net.server.<host>:<port>.errorCount` | Total number of errors.

### HTTP Server

Metric type | Metric name | Description
--- | --- | ---
Gauge | `vertx.http.server.<host>:<port>.connections` | Number of opened connections to the HTTP Server listening on the `<host>:<port>` address.
Counter | `vertx.http.server.<host>:<port>.bytesReceived` | Total number of bytes received by the HTTP Server listening on the `<host>:<port>` address.
Counter | `vertx.http.server.<host>:<port>.bytesSent` | Total number of bytes sent to the HTTP Server listening on the `<host>:<port>` address.
Counter | `vertx.http.server.<host>:<port>.errorCount` | Total number of errors.
Gauge | `vertx.http.client.<host>:<port>.requests` | Number of requests being processed.
Counter | `vertx.http.client.<host>:<port>.requestCount` | Total number of requests processed.
Counter | `vertx.http.client.<host>:<port>.processingTime` | Cumulated request processing time.
Gauge | `vertx.http.client.<host>:<port>.wsConnections` | Number of websockets currently opened.

### Event Bus

Metric type | Metric name | Description
--- | --- | ---
Gauge | `vertx.eventbus.handlers` | Number of event bus handlers.
Counter | `vertx.eventbus.errorCount` | Total number of errors.
Counter | `vertx.eventbus.bytesWritten` | Total number of bytes sent while sending messages to event bus cluster peers.
Counter | `vertx.eventbus.bytesRead` | Total number of bytes received while reading messages from event bus cluster peers.
Gauge | `vertx.eventbus.pending` | Number of messages not processed yet. One message published will count for `N` pending if `N` handlers are registered to the corresponding address.
Gauge | `vertx.eventbus.pendingLocal` | Like `vertx.eventbus.pending`, for local messages only.
Gauge | `vertx.eventbus.pendingRemote` | Like `vertx.eventbus.pending`, for remote messages only.
Counter | `vertx.eventbus.publishedMessages` | Total number of messages published (publish / subscribe).
Counter | `vertx.eventbus.publishedLocalMessages` | Like `vertx.eventbus.publishedMessages`, for local messages only.
Counter | `vertx.eventbus.publishedRemoteMessages` | Like `vertx.eventbus.publishedMessages`, for remote messages only.
Counter | `vertx.eventbus.sentMessages` | Total number of messages sent (point-to-point).
Counter | `vertx.eventbus.sentLocalMessages` | Like `vertx.eventbus.sentMessages`, for local messages only.
Counter | `vertx.eventbus.sentRemoteMessages` | Like `vertx.eventbus.sentMessages`, for remote messages only.
Counter | `vertx.eventbus.receivedMessages` | Total number of messages received.
Counter | `vertx.eventbus.receivedLocalMessages` | Like `vertx.eventbus.receivedMessages`, for remote messages only.
Counter | `vertx.eventbus.receivedRemoteMessages` | Like `vertx.eventbus.receivedMessages`, for remote messages only.
Counter | `vertx.eventbus.deliveredMessages` | Total number of messages delivered to handlers.
Counter | `vertx.eventbus.deliveredLocalMessages` | Like `vertx.eventbus.deliveredMessages`, for remote messages only.
Counter | `vertx.eventbus.deliveredRemoteMessages` | Like `vertx.eventbus.deliveredMessages`, for remote messages only.
Counter | `vertx.eventbus.replyFailures` | Total number of message reply failures.
Counter | `vertx.eventbus.<address>.processingTime` | Cumulated processing time for handlers listening to the `address`.

## Vert.x pool metrics

This section lists all the metrics generated by monitoring Vert.x pools.

There are two types currently supported:

- worker (see `WorkerExecutor`)

- datasource (created with Vert.x JDBC client)

Note | Vert.x creates two worker pools upfront, vert.x-worker-thread and vert.x-internal-blocking.

All metrics are prefixed with `<type>.<name>.`. For example, `worker.vert.x-internal-blocking.`.

Metric type | Metric name | Description
--- | --- | ---
Counter | `vertx.pool.<type>.<name>.delay` | Cumulated time waiting for a resource (queue time).
Gauge | `vertx.pool.<type>.<name>.queued` | Current number of elements waiting for a resource.
Counter | `vertx.pool.<type>.<name>.queueCount` | Total number of elements queued.
Counter | `vertx.pool.<type>.<name>.usage` | Cumulated time using a resource (i.e. processing time for worker pools).
Gauge | `vertx.pool.<type>.<name>.inUse` | Current number of resources used.
Counter | `vertx.pool.<type>.<name>.completed` | Total number of elements done with the resource (i.e. total number of tasks executed for worker pools).
Gauge | `vertx.pool.<type>.<name>.maxPoolSize` | Maximum pool size, only present if it could be determined.
Gauge | `vertx.pool.<type>.<name>.inUse` | Pool usage ratio, only present if maximum pool size could be determined.

## Verticle metrics

Metric type | Metric name | Description
--- | --- | ---
Gauge | `vertx.verticle.<name>` | Number of verticle instances deployed.

## User defined metrics

Users can send their own metrics to the Hawkular server. In order to do so, the event bus metrics bridge must be enabled:

    var vertx = Vertx.vertx(VertxOptions()
      .setMetricsOptions(VertxHawkularOptions()
        .setEnabled(true)
        .setMetricsBridgeEnabled(true)
      )
    )

By default, the metrics bus handler is listening to the `hawkular.metrics` address. But the bridge address can be configured:

    var vertx = Vertx.vertx(VertxOptions()
      .setMetricsOptions(VertxHawkularOptions()
        .setEnabled(true)
        .setMetricsBridgeEnabled(true)
        .setMetricsBridgeAddress("__hawkular_metrics")
      )
    )

The metrics bridge handler expects messages in the JSON format. The JSON object must at least provide a metric `id` and a numerical `value`:

    var message = new io.vertx.core.json.JsonObject().put("id", "myapp.files.opened").put("value", 7)
    vertx.eventBus().publish("hawkular.metrics", message)

The handler will assume the metric is a gauge and will assign a timestamp corresponding to the time when the message was processed. If the metric is a counter or availability, or if you prefer explicit configuration, set the `type` and/or `timestamp` attributes:

    var counterMetric = new io.vertx.core.json.JsonObject().put("id", "myapp.files.opened").put("type", "counter").put("timestamp", 189898098098908).put("value", 7)
    vertx.eventBus().publish("hawkular.metrics", counterMetric)

    var availabilityMetric = new io.vertx.core.json.JsonObject().put("id", "myapp.mysubsystem.status").put("type", "availability").put("value", "up")
    vertx.eventBus().publish("hawkular.metrics", availabilityMetric)

Note | Hawkular understands all timestamps as milliseconds since January 1, 1970, 00:00:00 UTC.