Prometheus Integration

Provisioning Observability Metrics

To set up and configure the Observability Metrics service, please reach out to our support team at [email protected]. Provide the following details in your request:

• Your organization name.

• The list of vaults you want to receive metrics for.

Our support team will provide the Prometheus endpoints and credentials needed to access your metrics.

Example

If you want to receive metrics for the vault tntasd123 in the organization my-test-organization, the endpoint and credentials might look like this:

Prometheus URL: https://metrics.apps.verygood.systems/vaults/tntasd123/api/metrics
Username: prometheus
Password: hunter2

Scrape Data

The Observability Metrics Prometheus endpoint exposes a /metrics endpoint that can be scraped from any Prometheus-compatible service.

Prometheus Scrape Compatible APMs

Many APMs will support entering the /metrics endpoint directly. If you use an APM like Grafana Cloud, you can pass the credentials in as basic auth credentials when configuring the endpoint.

APMs without Prometheus Scrape

If you have a Prometheus-compatible system that does not support scrape (e.g., you may need to leverage remote_write or remote_read you can still make this work by hosting your own Prometheus instance, which will scrape and then push or expose the metrics.

Pushing VGS Prometheus metrics to Datadog

To connect to your Datadog account and begin collecting metrics, follow these steps:

Prerequisites

• Datadog API Key: Obtain this from your Datadog account.

• Datadog Site Parameter: Identify the appropriate site for your account.

Step 1. Configure the Datadog Agent

Create a configuration file named dd-agent-conf.yml. This file specifies:

• The Prometheus endpoints to be scraped

• Basic authentication credentials

• Specific metrics to collect

Here's an example configuration:

init_config:
instances:
  - prometheus_url: https://metrics.apps.verygood.systems/vaults/tntasd123/api/metrics # note, your Vault ID must be added to the path
    username: prometheus
    password: hunter2
    namespace: vgs
    send_distribution_buckets: true
    max_returned_metrics: 10000
    send_histograms_buckets: true
    label_to_hostname: vgs_tier
    metrics: # List of metrics to collect (required)
    - vault_record_usage: record_usage
    - vault_record_usage_failure: record_usage_failure
    - http_proxy_request
    - http_proxy_response
    - http_proxy_processing_duration_ms
    - http_upstream_response
    - http_upstream_processing_duration_ms
    - sftp_proxy_request
    - sftp_proxy_processing_duration_ms
    - iso_8583_proxy_request
    - iso_8583_proxy_processing_duration_ms

Step 2. Start the Datadog Agent in Docker

Run the following command to start the Datadog Agent in a Docker container. Replace path/to/your/dd-agent-conf.yml with the actual path to your configuration file.

docker run -d --name dd-agent \
    -v /var/run/docker.sock:/var/run/docker.sock:ro \
    -v /proc/:/host/proc/:ro \
    -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
    -e DD_API_KEY="my-dd-api-key" \
    -e DD_SITE="datadoghq.com" \
    -v path/to/your/dd-agent-conf.yml:/etc/datadog-agent/conf.d/openmetrics.d/conf.yaml \
    gcr.io/datadoghq/agent:latest

Step 3. Verify the Agent

Once the Docker container has successfully started, the agent will begin periodic checks and push metrics to your Datadog account. Verify the metrics in your Datadog dashboard.

Further Reading

For more information on integrating Prometheus metrics with Datadog, check out these resources:

• Getting Started with Datadog

• Prometheus and OpenMetrics metrics collection from a host

Pushing VGS Prometheus metrics to Self-Hosted Grafana Instances

To push metrics to a self-hosted Grafana instance, you'll need to configure a Prometheus server to scrape metrics and set up a Grafana data source to connect to the server. This example demonstrates how to create both the Prometheus server and Grafana instance on a single Docker container.

Step 1. Configure Prometheus and Grafana Services

a. Create the Prometheus Configuration. Create a file named prometheus.yml to define the endpoints to scrape and the basic authentication details.

Example configuration:

scrape_configs:
  - job_name: 'vgs'
    static_configs:
        - targets: ['metrics.apps.verygood.systems']
    metrics_path: /vaults/tntasd123/api/metrics
    scheme: https
    basic_auth:
      username: prometheus
      password: hunter2

b. Create the Docker Compose Configuration. Create a file named docker-compose.yml to configure the Prometheus server and Grafana instance.

Example configuration:

services:
  prometheus:
    image: prom/prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"
  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"
    depends_on:
      - prometheus

Step 2. Start the Services

Run the following command in the directory containing the configuration files to start both services:

docker-compose up -d

Step 3. Set Up a Prometheus Data Source in Grafana

• Log in to the Grafana instance (Default credentials: Username: admin / Password: admin).

• From the homepage, navigate to Data Sources in the side menu.

• Click Add data source and search for Prometheus.

• Under the Connection section, set the URL to the Prometheus instance: http://prometheus:9090 in this example

• Click Save & Test to verify the connection.

Step 4. Explore Your Metrics

With the Prometheus server and Grafana instance now connected, you can begin exploring metrics directly in Grafana. Use the Explore section or create custom dashboards to visualize your data.

Further Reading

For more information on integrating Prometheus metrics with Grafana, check out these resources:

• Configure the Prometheus data source

Pushing VGS Prometheus metrics to Grafana Cloud

To push metrics to Grafana Cloud via an agent, you must set up a Prometheus connection and configure the Grafana Agent. Follow the steps below:

Step 1: Create a Prometheus Connection

• Log in to your Grafana Cloud account.

• From the homepage, click Connections in the left-side menu.

• Under Connections, click Add new connection.

• Select Hosted Prometheus metrics.

• Under Choose a method for forwarding metrics, select From my local Prometheus server.

• Under Choose your use case for forwarding metrics, select Send metrics from a single Prometheus instance.

• Under Select how to send the metrics from Prometheus to Grafana Cloud, select Directly.

• Under Update Prometheus configuration, create a new token. Save the generated prometheus.yml snippet to your clipboard for later use (this will be needed to configure the agent).

Within the remote_write section, you can specify any Prometheus-compatible server endpoint to send data there.

global:
  scrape_interval: 60s
remote_write:
  - url: https://my-prometheus-connection.grafana.net/api/prom/push
    basic_auth:
      username: my-grafana-username
      password: my-grafana-password
scrape_configs:
  - job_name: node
    static_configs:
      - targets: ["localhost:9090"]

Step 2. Configure the Grafana Agent

Create a file named agent-config.yml. This file will define the Prometheus endpoints to scrape and the basic authentication details.

1. Example configuration:

server:
  log_level: info

metrics:
  configs:
    - name: example-metrics-config
      scrape_configs:
        - job_name: 'tntasd123' # This can be any value but your vgs Vault ID is a clear choice
          static_configs:
            - targets: ['metrics.apps.verygood.systems']
          metrics_path: /vaults/tntasd123/api/metrics # note, your Vault ID must be added to the path
          scheme: https
          basic_auth:
            username: prometheus
            password: hunter2
      remote_write: # Paste the Prometheus connection configuration from the previous step
        - url: 'https://my-prometheus-connection.grafana.net/api/prom/push'
          basic_auth:
            username: my-grafana-username
            password: my-grafana-password

Step 3. Start the Grafana Agent in Docker

Run the following command to start the Grafana Agent in a Docker container. Replace path/to/your/agent-config.yml with the correct path to your configuration file:

docker run -d --name grafana-agent \
  -v path/to/your/agent-config.yml:/etc/agent/agent.yaml \
  grafana/agent:latest

Step 4. Verify Metrics in Grafana Cloud

Once the Docker container is running, the Grafana Agent will begin scraping metrics and pushing them to your Grafana Cloud account. You can verify this by checking the metrics in your Grafana dashboard.

Further Reading

For more information on integrating Prometheus metrics with Grafana, check out these resources:

• Configure the Prometheus data source

• metrics_config | Grafana Agent documentation

Dashboard Templates

VGS supplies a set of pre-built dashboards to get you up and running quickly.