Observability Guide

Adding/Removing observability stack

make observability-setup

This will setup Loki, Grafana Alloy, and Grafana Dashboard in your cluster.

make observability-teardown

Removes the observability stack from your cluster.

Adding Example Grafana Dashboard

Follow the steps after installing the stack to setup your secret and port forward the Grafana dashboard.

1. Get your 'admin' user password by running:

# Taken from the output of grafana installation
kubectl get secret --namespace monitoring my-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

1. Port-forward to expose Grafana at http://localhost:3000/login

kubectl --namespace monitoring port-forward deployment/my-grafana 3000

MCP Tool Annotation Hints Visualization

Please note that you should try running some MCP traffic through your cluster to see data in this visualization. You can do so using the mcp inspector tool as shown in the main README.

1. Go to Dashboards, create -> add visualization.
2. Select your Loki datasource.
3. Set some type of graph (bar graph or stat visualizations are easiest to read).
4. Add four queries using the code editor

# Query A
sum by (destructive) (
  count_over_time({namespace="mcp-system"} |= `x-mcp-annotation-hints` 
  | regexp `x-mcp-annotation-hints\\"[^\"]*raw_value:\\\"(?P<annotation_hints>[^\\\"]+)\\\"` 
  | regexp `"readOnly=(?P<readOnly>[^,]+),destructive=(?P<destructive>[^,]+),idempotent=(?P<idempotent>[^,]+),openWorld=(?P<openWorld>[^\"]+)"` 
  | destructive = "true" [$__auto])
)

# Query B
sum by (readOnly) (
  count_over_time({namespace="mcp-system"} |= `x-mcp-annotation-hints` 
  | regexp `x-mcp-annotation-hints\\"[^\"]*raw_value:\\\"(?P<annotation_hints>[^\\\"]+)\\\"` 
  | regexp `"readOnly=(?P<readOnly>[^,]+),destructive=(?P<destructive>[^,]+),idempotent=(?P<idempotent>[^,]+),openWorld=(?P<openWorld>[^\"]+)"` 
  | readOnly = "false" [$__auto])
)

# Query C
sum by (idempotent) (
  count_over_time({namespace="mcp-system"} |= `x-mcp-annotation-hints` 
  | regexp `x-mcp-annotation-hints\\"[^\"]*raw_value:\\\"(?P<annotation_hints>[^\\\"]+)\\\"` 
  | regexp `"readOnly=(?P<readOnly>[^,]+),destructive=(?P<destructive>[^,]+),idempotent=(?P<idempotent>[^,]+),openWorld=(?P<openWorld>[^\"]+)"` 
  | idempotent = "false" [$__auto])
)

# Query D
sum by (openWorld) (

  count_over_time({namespace="mcp-system"} |= `x-mcp-annotation-hints` 
  | regexp `x-mcp-annotation-hints\\"[^\"]*raw_value:\\\"(?P<annotation_hints>[^\\\"]+)\\\"` 
  | regexp `"readOnly=(?P<readOnly>[^,]+),destructive=(?P<destructive>[^,]+),idempotent=(?P<idempotent>[^,]+),openWorld=(?P<openWorld>[^\"]+)"` 
  | openWorld = "true\\" [$__auto])
)

1. Press "Run queries" to see the results.

This visualization shows you the number of true or false hints that are set coming out of the mcp-system namespace. This mostly involves logs coming from the mcp-router and relies on the INFO log that emits the headers when the response is being set in the EXT_PROC module for Envoy.

Example log line this visualization reads from:

time=2025-11-08T21:41:34.147Z level=INFO msg="Sending MCP body routing instructions to Envoy: request_body:{response:{header_mutation:{set_headers:{header:{key:\"x-mcp-method\"  raw_value:\"tools/call\"}}  set_headers:{header:{key:\"x-mcp-annotation-hints\"  raw_value:\"readOnly=false,destructive=true,idempotent=false,openWorld=true\"}}  set_headers:{header:{key:\"x-mcp-toolname\"  raw_value:\"headers\"}}  set_headers:{header:{key:\"x-mcp-servername\"  raw_value:\"mcp-test/mcp-server2-route\"}}  set_headers:{header:{key:\"mcp-session-id\"  raw_value:\"mcp-session-f4c2a956-b3cc-4a80-b583-ae08a760e63b\"}}  set_headers:{header:{key:\":authority\"  raw_value:\"mcp-server-2\"}}  set_headers:{header:{key:\"content-length\"  raw_value:\"119\"}}}  body_mutation:{body:\"{\\\"id\\\":11,\\\"jsonrpc\\\":\\\"2.0\\\",\\\"method\\\":\\\"tools/call\\\",\\\"params\\\":{\\\"_meta\\\":{\\\"progressToken\\\":11},\\\"arguments\\\":{},\\\"name\\\":\\\"headers\\\"}}\"}  clear_route_cache:true}}"