The metrics of XDC chain

The metrics has been improved in the PR #758 greatly.

1. Metrics flags

XDC includes a variety of optional metrics that can be reported to the user. However, metrics are disabled by default to save on the computational overhead for the average user.

The following flags exist in the PR #758:

--metrics                       (default=false):
    Enable metrics collection and reporting
--metrics-addr                  (default="127.0.0.1"):
    Enable stand-alone metrics HTTP server listening interface
--metrics-port                  (default=6060):
    Metrics HTTP server listening port
--metrics-influxdb              (default=false):
    Enable metrics export/push to an external InfluxDB database
--metrics-influxdb.endpoint     (default="<http://localhost:8086>"):
    InfluxDB API endpoint to report metrics to
--metrics-influxdb.database     (default="xdc"):
    InfluxDB database name to push reported metrics to
--metrics-influxdb.username     (default="test"):
    Username to authorize access to the database
--metrics-influxdb.password     (default="test"):
    Password to authorize access to the database
--metrics-influxdb.tags         (default="host=localhost"):
    Comma-separated InfluxDB tags (key/values) attached to all measurements
--metrics-influxdbv2            (default=false):
    Enable metrics export/push to an external InfluxDB v2 database
--metrics-influxdb.token        (default="test"):
    Token to authorize access to the database (v2 only)
--metrics-influxdb.bucket       (default=xdc"):
    InfluxDB bucket name to push reported metrics to (v2 only)
--metrics-influxdb.organization (default="xdc"):
    InfluxDB organization name (v2 only)

2. Metric types

XDC's metrics can be classified into four types: meters, timers, counters and gauges.

2.1 Meters

Analogous to physical meters (electricity, water, etc), XDC's meters are capable of measuring the amount of "things" that pass through and at the rate at which they do. A meter doesn't have a specific unit of measure (byte, block, malloc, etc), it just counts arbitrary events. At any point in time a meter can report:

• Total number of events that passed through the meter
• Mean throughput rate of the meter since startup (events / second)
• Weighted throughput rate in the last 1, 5 and 15 minutes (events / second) ("weighted" means that recent seconds count more that in older ones*)

2.2 Timers

Timers are extensions of meters, the duration of an event is collected alongside a log of its occurrence. Similarly to meters, a timer can also measure arbitrary events but each requires a duration to be assigned individually. In addition generating all of the meter report types, a timer also reports:

• Percentiles (5, 20, 50, 80, 95), reporting that some percentage of the events took less than the reported time to execute (e.g. Percentile 20 = 1.5s would mean that 20% of the measured events took less time than 1.5 seconds to execute; inherently 80%(=100%-20%) took more that 1.5s)
• Percentile 5: minimum durations (this is as fast as it gets)
• Percentile 50: well behaved samples (boring, just to give an idea)
• Percentile 80: general performance (these should be optimised)
• Percentile 95: worst case outliers (rare, just handle gracefully)

2.3 Counters

A counter is a single int64 value that can be incremented and decremented. The current value of the counter can be queried.

2.4 Gauges

A gauge is a single int64 value. Its value can increment and decrement - as with a counter - but can also be set arbitrarily.

3. Querying metrics

XDC collects metrics if the --metrics flag is provided at startup. Those metrics are available via an HTTP server if the --metrics-addr flag is also provided. By default the metrics are served at 127.0.0.1:6060/debug/metrics but a custom IP address can be provided. A custom port can also be provided to the --metrics-port flag. For example, to serve all metrics at the default address and port:

XDC <other commands> --metrics --metrics-addr 127.0.0.1

Navigating the browser to the given metrics address displays all the available metrics in the form of JSON data that looks similar to:

{
    "chain/account/commits.50-percentile": 1763,
    "chain/account/commits.75-percentile": 1763,
    "chain/account/commits.95-percentile": 1763,
    "chain/account/commits.99-percentile": 1763,
    "chain/account/commits.count": 1,
    "chain/account/commits.mean": 1763,
    "chain/account/hashes.50-percentile": 962,
    "chain/account/hashes.75-percentile": 962,
    "chain/account/hashes.95-percentile": 962,
    "chain/account/hashes.99-percentile": 962,
    "chain/account/hashes.count": 1,
    "chain/account/hashes.mean": 962,
    ...
}

XDC also supports dumping metrics directly into an influx database. In order to activate this, the --metrics-influxdb flag must be provided at startup. The API endpoint, username, password and other influxdb tags can also be provided.

We also provide Prometheus-formatted metrics data, which can be obtained through the http://127.0.0.1:6060/debug/metrics/prometheus URL, eg:

# TYPE chain_account_commits_count counter
chain_account_commits_count 1

# TYPE chain_account_commits summary
chain_account_commits {quantile="0.5"} 2305
chain_account_commits {quantile="0.75"} 2305
chain_account_commits {quantile="0.95"} 2305
chain_account_commits {quantile="0.99"} 2305
chain_account_commits {quantile="0.999"} 2305
chain_account_commits {quantile="0.9999"} 2305

# TYPE chain_account_hashes_count counter
chain_account_hashes_count 1

# TYPE chain_account_hashes summary
chain_account_hashes {quantile="0.5"} 1283
chain_account_hashes {quantile="0.75"} 1283
chain_account_hashes {quantile="0.95"} 1283
chain_account_hashes {quantile="0.99"} 1283
chain_account_hashes {quantile="0.999"} 1283
chain_account_hashes {quantile="0.9999"} 1283

...

4. Summary

XDC can be configured to report metrics to an HTTP server or database. These functions are disabled by default but can be configured by passing the appropriate commands on startup.