You can use the OpenShift Container Platform monitoring stack to record and view health checks and metrics for your Knative services. This section describes the following topics:

  • What metrics Knative services expose by default

  • How to configure exposing custom metrics

  • How to configure the monitoring stack to scrape the exposed metrics

  • How to view the metrics of a service

Scraping the metrics does not affect autoscaling of a Knative service, because scraping requests do not go through the activator. Consequently, no scraping takes place if no pods are running.

Additional resources

Knative service metrics exposed by default

Table 1. Metrics exposed by default for each Knative service on port 9090
Metric name, unit, and type Description Metric tags

queue_requests_per_second

Metric unit: dimensionless

Metric type: gauge

Number of requests per second that hit the queue proxy.

Formula: stats.RequestCount / r.reportingPeriodSeconds

stats.RequestCount is calculated directly from the networking pkg stats for the given reporting duration.

destination_configuration="event-display", destination_namespace="pingsource1", destination_pod="event-display-00001-deployment-6b455479cb-75p6w", destination_revision="event-display-00001"

queue_proxied_operations_per_second

Metric unit: dimensionless

Metric type: gauge

Number of proxied requests per second.

Formula: stats.ProxiedRequestCount / r.reportingPeriodSeconds

stats.ProxiedRequestCount is calculated directly from the networking pkg stats for the given reporting duration.

queue_average_concurrent_requests

Metric unit: dimensionless

Metric type: gauge

Number of requests currently being handled by this pod.

Average concurrency is calculated at the networking pkg side as follows:

  • When a req change happens, the time delta between changes is calculated. Based on the result, the current concurrency number over delta is computed and added to the current computed concurrency. Additionally, a sum of the deltas is kept.

    Current concurrency over delta is computed as follows:

    global_concurrency × delta

  • Each time a reporting is done, the sum and current computed concurrency are reset.

  • When reporting the average concurrency the current computed concurrency is divided by the sum of deltas.

  • When a new request comes in, the global concurrency counter is increased. When a request is completed, the counter is decreased.

destination_configuration="event-display", destination_namespace="pingsource1", destination_pod="event-display-00001-deployment-6b455479cb-75p6w", destination_revision="event-display-00001"

queue_average_proxied_concurrent_requests

Metric unit: dimensionless

Metric type: gauge

Number of proxied requests currently being handled by this pod:

stats.AverageProxiedConcurrency

destination_configuration="event-display", destination_namespace="pingsource1", destination_pod="event-display-00001-deployment-6b455479cb-75p6w", destination_revision="event-display-00001"

process_uptime

Metric unit: seconds

Metric type: gauge

The number of seconds that the process has been up.

destination_configuration="event-display", destination_namespace="pingsource1", destination_pod="event-display-00001-deployment-6b455479cb-75p6w", destination_revision="event-display-00001"

Table 2. Metrics exposed by default for each Knative service on port 9091
Metric name, unit, and type Description Metric tags

request_count

Metric unit: dimensionless

Metric type: counter

The number of requests that are routed to queue-proxy.

configuration_name="event-display", container_name="queue-proxy", namespace_name="apiserversource1", pod_name="event-display-00001-deployment-658fd4f9cf-qcnr5", response_code="200", response_code_class="2xx", revision_name="event-display-00001", service_name="event-display"

request_latencies

Metric unit: milliseconds

Metric type: histogram

The response time in milliseconds.

configuration_name="event-display", container_name="queue-proxy", namespace_name="apiserversource1", pod_name="event-display-00001-deployment-658fd4f9cf-qcnr5", response_code="200", response_code_class="2xx", revision_name="event-display-00001", service_name="event-display"

app_request_count

Metric unit: dimensionless

Metric type: counter

The number of requests that are routed to user-container.

configuration_name="event-display", container_name="queue-proxy", namespace_name="apiserversource1", pod_name="event-display-00001-deployment-658fd4f9cf-qcnr5", response_code="200", response_code_class="2xx", revision_name="event-display-00001", service_name="event-display"

app_request_latencies

Metric unit: milliseconds

Metric type: histogram

The response time in milliseconds.

configuration_name="event-display", container_name="queue-proxy", namespace_name="apiserversource1", pod_name="event-display-00001-deployment-658fd4f9cf-qcnr5", response_code="200", response_code_class="2xx", revision_name="event-display-00001", service_name="event-display"

queue_depth

Metric unit: dimensionless

Metric type: gauge

The current number of items in the serving and waiting queue, or not reported if unlimited concurrency. breaker.inFlight is used.

configuration_name="event-display", container_name="queue-proxy", namespace_name="apiserversource1", pod_name="event-display-00001-deployment-658fd4f9cf-qcnr5", response_code="200", response_code_class="2xx", revision_name="event-display-00001", service_name="event-display"

Knative service with custom application metrics

You can extend the set of metrics exported by a Knative service. The exact implementation depends on your application and the language used.

The following listing implements a sample Go application that exports the count of processed events custom metric.

package main

import (
  "fmt"
  "log"
  "net/http"
  "os"

  "github.com/prometheus/client_golang/prometheus" (1)
  "github.com/prometheus/client_golang/prometheus/promauto"
  "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
  opsProcessed = promauto.NewCounter(prometheus.CounterOpts{ (2)
     Name: "myapp_processed_ops_total",
     Help: "The total number of processed events",
  })
)


func handler(w http.ResponseWriter, r *http.Request) {
  log.Print("helloworld: received a request")
  target := os.Getenv("TARGET")
  if target == "" {
     target = "World"
  }
  fmt.Fprintf(w, "Hello %s!\n", target)
  opsProcessed.Inc() (3)
}

func main() {
  log.Print("helloworld: starting server...")

  port := os.Getenv("PORT")
  if port == "" {
     port = "8080"
  }

  http.HandleFunc("/", handler)

  // Separate server for metrics requests
  go func() { (4)
     mux := http.NewServeMux()
     server := &http.Server{
        Addr: fmt.Sprintf(":%s", "9095"),
        Handler: mux,
     }
     mux.Handle("/metrics", promhttp.Handler())
     log.Printf("prometheus: listening on port %s", 9095)
     log.Fatal(server.ListenAndServe())
  }()

   // Use same port as normal requests for metrics
  //http.Handle("/metrics", promhttp.Handler()) (5)
  log.Printf("helloworld: listening on port %s", port)
  log.Fatal(http.ListenAndServe(fmt.Sprintf(":%s", port), nil))
}
1 Including the Prometheus packages.
2 Defining the opsProcessed metric.
3 Incrementing the opsProcessed metric.
4 Configuring to use a separate server for metrics requests.
5 Configuring to use the same port as normal requests for metrics and the metrics subpath.

Configuration for scraping custom metrics

Custom metrics scraping is performed by an instance of Prometheus purposed for user workload monitoring. After you enable user workload monitoring and create the application, you need a configuration that defines how the monitoring stack will scrape the metrics.

The following sample configuration defines the ksvc for your application and configures the service monitor. The exact configuration depends on your application and how it exports the metrics.

apiVersion: serving.knative.dev/v1 (1)
kind: Service
metadata:
  name: helloworld-go
spec:
  template:
    metadata:
      labels:
        app: helloworld-go
      annotations:
    spec:
      containers:
      - image: docker.io/skonto/helloworld-go:metrics
        resources:
          requests:
            cpu: "200m"
        env:
        - name: TARGET
          value: "Go Sample v1"
---
apiVersion: monitoring.coreos.com/v1 (2)
kind: ServiceMonitor
metadata:
  labels:
  name: helloworld-go-sm
spec:
  endpoints:
  - port: queue-proxy-metrics
    scheme: http
  - port: app-metrics
    scheme: http
  namespaceSelector: {}
  selector:
    matchLabels:
       name:  helloworld-go-sm
---
apiVersion: v1 (3)
kind: Service
metadata:
  labels:
    name:  helloworld-go-sm
  name:  helloworld-go-sm
spec:
  ports:
  - name: queue-proxy-metrics
    port: 9091
    protocol: TCP
    targetPort: 9091
  - name: app-metrics
    port: 9095
    protocol: TCP
    targetPort: 9095
  selector:
    serving.knative.dev/service: helloworld-go
  type: ClusterIP
1 Application specification.
2 Configuration of which application’s metrics are scraped.
3 Configuration of the way metrics are scraped.

Examining metrics of a service

After you have configured the application to export the metrics and the monitoring stack to scrape them, you can examine the metrics in the web console.

Procedure
  1. Optional: Run requests against your application that you will be able to see in the metrics:

    $ hello_route=$(oc get ksvc helloworld-go -n ns1 -o jsonpath='{.status.url}') && \
        curl $hello_route
    Example output
    Hello Go Sample v1!
  2. In the web console, navigate to the MonitoringMetrics interface.

  3. In the input field, enter the query for the metric you want to observe, for example:

    revision_app_request_count{namespace="ns1", job="helloworld-go-sm"}

    Another example:

    myapp_processed_ops_total{namespace="ns1", job="helloworld-go-sm"}
  4. Observe the visualized metrics: