Building Useful Grafana and Loki Monitoring with a Few Cursor Conversations
· 17 min read · Views --

Building Useful Grafana and Loki Monitoring with a Few Cursor Conversations

Author: Alex Xiang

Engineering

This article records a very practical observability task: taking an existing Kubernetes service whose stdout logs already flow into Loki, and building useful Grafana dashboards and alerts without first becoming a full-time observability engineer.

The interesting part is not that AI wrote a few queries. The interesting part is how to ask the right questions, verify every LogQL expression, and gradually turn scattered logs into charts that can help during incidents.

Why Grafana, Loki, and Prometheus

Prometheus is good for numeric metrics: CPU, memory, request counters, latency histograms, queue length, and so on. Loki is good for logs. It indexes labels and stores log lines more cheaply than a full-text log search system. Grafana sits on top and provides dashboards, exploration, and alerting.

In this case, the service already printed request logs such as:

[REQ] --> POST /api/v1/query
[REQ] <-- POST /api/v1/query status=200 elapsed_ms=126

That means we could get a lot of value without changing the service first. We could parse existing logs, build request-rate panels, group by path and status, estimate latency, and add alerts.

This is not the perfect long-term architecture. For strict SLOs, proper metrics and histograms are better. But when the immediate goal is “see what is happening and alert someone when it breaks,” Loki plus Grafana is a pragmatic start.

Start from Real Labels

The first common mistake is copying a LogQL query from someone else’s environment:

{namespace="prod", app="tool-gateway"}

That only works if your Loki labels actually contain namespace and app with those values. The first step should be Grafana Explore, then Label browser.

Useful checks:

  • Which label identifies the Kubernetes namespace?
  • Which label identifies the application?
  • Are there pod labels, container labels, or job labels?
  • Are multiple replicas included?

Only after confirming labels should you write query expressions.

Counting Requests

If each completed request has one exit log line, request rate can be derived from the exit line:

sum(
  rate({namespace="prod", app="tool-gateway"} |= "[REQ] <--" [5m])
)

For QPS by path, extract the path and group by it:

sum by (path) (
  rate(
    {namespace="prod", app="tool-gateway"}
      |= "[REQ] <--"
      | regexp `(?P<method>GET|POST|PUT|DELETE|PATCH) (?P<path>/[^ ]+)`
    [5m]
  )
)

The key is to count only the completion log, not both entry and exit logs. Otherwise every request may be counted twice.

Status-code Panels

Status-code grouping is usually the first panel that helps during incidents:

sum by (status) (
  rate(
    {namespace="prod", app="tool-gateway"}
      |= "[REQ] <--"
      | regexp `status=(?P<status>[0-9]{3})`
    [5m]
  )
)

From there, error rate can be built with two expressions:

  • numerator: 5xx completion logs,
  • denominator: all completion logs,
  • expression: numerator divided by denominator.

Grafana’s expression editor can be awkward at first, but keeping each intermediate query visible makes debugging much easier.

Latency from Logs

If the exit log includes elapsed_ms=126, a simple average latency panel can be built:

avg_over_time(
  {namespace="prod", app="tool-gateway"}
    |= "[REQ] <--"
    | regexp `elapsed_ms=(?P<elapsed_ms>[0-9.]+)`
    | unwrap elapsed_ms
  [5m]
)

For multiple replicas and multiple paths, be careful with aggregation. A simple average of averages can be misleading if one pod handles far more traffic than another.

For rough operational visibility, log-derived average latency may be enough. For accurate p95 and p99, application metrics with histograms are better.

Alerts and Webhooks

Grafana Unified Alerting can evaluate LogQL queries and send notifications to a webhook. The alert design should stay simple:

AlertQuery Idea
High 5xx rate5xx request rate divided by total request rate
No traffictotal completion log rate falls to zero
Slow responseaverage or estimated percentile latency above threshold
Error keywordrate of logs containing known exception markers

When webhook messages contain [no value], the problem is usually label or template mismatch. Test with a simple alert first, then add labels and fields gradually.

How Cursor Helped

Cursor was useful because the task involved many small questions:

  • What does this Loki query mean?
  • Why does unwrap fail here?
  • How do I group by path?
  • Why does the chart show no data?
  • How do Grafana expressions combine query results?
  • How should an alert template reference labels?

The important workflow was not “let AI generate the final dashboard.” It was:

  1. Ask for one small LogQL expression.
  2. Run it in Explore.
  3. Compare it with real logs.
  4. Fix labels and parsing.
  5. Move it into a panel only after it works.
  6. Add alerts after the dashboard is trustworthy.

AI shortened the search path, but the verification still had to happen in Grafana.

Lessons Learned

Log-based observability is a good first step when a service already prints structured request logs. It is fast, cheap, and does not require immediate code changes.

But it has limits. Logs are text, not metrics. Parsing can break when log formats change. Percentiles from logs are less reliable than histogram metrics. Alert queries can become expensive if they scan too much text.

The practical strategy is:

  • use Loki quickly to see traffic and failures,
  • add Prometheus metrics for long-term SLOs,
  • keep dashboards small and focused,
  • validate every query against real log lines,
  • do not trust an AI-generated LogQL query until it returns the expected data.

That is the real value of AI-assisted ops work: not skipping fundamentals, but getting through unfamiliar syntax and tool UI faster while still checking the result like an engineer.