Alerting on Redis Eviction Rate and Keyspace Misses

Your cache is quietly evicting hot keys and every eviction turns into a database read, but your dashboards still show a green instance because used_memory never crossed a static threshold — the damage is in the rate of change, not the absolute counters. This page wires up an alert path that catches that class of incident early: scrape INFO stats and INFO memory from a live Redis 7.x instance, expose evicted_keys, keyspace_misses, and keyspace_hits as Prometheus counters, convert the monotonic totals into per-second rates with rate(), and fire PromQL alerts with for: windows split into page-now and open-a-ticket severities. Every step runs against a real instance so you can verify the signal before you trust it in an incident. The underlying eviction mechanics assumed here are decided in LRU vs LFU Eviction Policies.

# Prerequisites

• Redis 6.2+ or 7.x, reachable from your exporter host and readable with INFO.
• redis-py 5.x on Python 3.10+ (pip install "redis>=5,<6" prometheus-client). Examples use the redis.asyncio client.
• Prometheus 2.40+ scraping the exporter, plus Alertmanager for routing severities.
• A maxmemory and maxmemory-policy already set explicitly — an unbounded noeviction cache never evicts and hides the very signal you are alerting on.
• Baseline knowledge of your normal keyspace_hits/keyspace_misses split so a threshold means something.

# Step-by-Step Implementation

# 1. Scrape the raw counters from INFO

Pull the cumulative counters from INFO stats and the memory picture from INFO memory in one async round trip so a single scrape carries every field the alerts depend on.

import redis.asyncio as redis

client = redis.Redis(host="redis-primary", port=6379, decode_responses=True)

async def scrape() -> dict[str, int]:
    # INFO returns parsed key/value pairs; request only the sections we use.
    stats = await client.info("stats")
    memory = await client.info("memory")
    return {
        "evicted_keys": int(stats["evicted_keys"]),
        "keyspace_hits": int(stats["keyspace_hits"]),
        "keyspace_misses": int(stats["keyspace_misses"]),
        "used_memory": int(memory["used_memory"]),
        "maxmemory": int(memory["maxmemory"]),
    }

# 2. Expose the counters as Prometheus metrics

evicted_keys, keyspace_hits, and keyspace_misses are monotonic totals that reset only on restart, so expose them through a custom collector as CounterMetricFamily — that tells Prometheus to treat a reset correctly instead of reading it as a negative spike.

from prometheus_client.core import CounterMetricFamily, GaugeMetricFamily
from prometheus_client.registry import Collector

class RedisStatsCollector(Collector):
    def collect(self):
        s = self._latest  # dict from scrape(), refreshed on each scrape interval
        yield CounterMetricFamily(
            "redis_evicted_keys_total", "Keys evicted under maxmemory pressure",
            value=s["evicted_keys"])
        yield CounterMetricFamily(
            "redis_keyspace_hits_total", "Lookups that found a key", value=s["keyspace_hits"])
        yield CounterMetricFamily(
            "redis_keyspace_misses_total", "Lookups that missed", value=s["keyspace_misses"])
        yield GaugeMetricFamily(
            "redis_memory_used_bytes", "Current used memory", value=s["used_memory"])

# 3. Convert the totals into per-second rates

Alert on movement, not level: rate() divides the increase in a counter by the window length and transparently handles counter resets, giving you evictions-per-second and misses-per-second regardless of how long the process has been up.

# Evictions per second, averaged over the last 5 minutes.
rate(redis_evicted_keys_total[5m])

# Miss rate per second over the same window.
rate(redis_keyspace_misses_total[5m])

# 4. Derive a live hit ratio alongside the rates

A rising miss rate is only alarming when the ratio also degrades, so compute the hit ratio from the two rate vectors and alert when it falls through a floor rather than on raw miss volume.

# Hit ratio over a 5m window: hits / (hits + misses). Falls toward 0 when the
# cache stops absorbing reads. Guard against divide-by-zero on an idle instance.
sum(rate(redis_keyspace_hits_total[5m]))
  /
clamp_min(
  sum(rate(redis_keyspace_hits_total[5m])) + sum(rate(redis_keyspace_misses_total[5m])),
  1)

# 5. Write PromQL alert rules with for: windows

Wrap each expression in an alerting rule with a for: clause so a single noisy scrape cannot page anyone — the condition must hold continuously for the whole window before it fires.

groups:
  - name: redis-eviction-misses
    rules:
      - alert: RedisEvictionRateHigh
        expr: rate(redis_evicted_keys_total[5m]) > 100
        for: 10m
        labels: { severity: page }
        annotations:
          summary: "Redis evicting more than 100 keys/s for 10m — instance in labels"
      - alert: RedisHitRatioLow
        expr: |
          sum(rate(redis_keyspace_hits_total[5m]))
          / clamp_min(sum(rate(redis_keyspace_hits_total[5m]))
              + sum(rate(redis_keyspace_misses_total[5m])), 1) < 0.85
        for: 15m
        labels: { severity: ticket }
        annotations:
          summary: "Redis hit ratio below 0.85 for 15m — instance in labels"

# 6. Split page-now from open-a-ticket thresholds

Give each symptom two rules at different severity and for: windows so a slow drift files a ticket during business hours while a sharp, sustained spike pages the on-call immediately, and route them by the severity label in Alertmanager.

      - alert: RedisEvictionRateElevated
        expr: rate(redis_evicted_keys_total[5m]) > 20
        for: 30m
        labels: { severity: ticket }     # slow burn -> ticket, no page
        annotations:
          summary: "Sustained low-level eviction — see instance label"
          runbook_url: "https://runbooks.internal/redis/eviction"

# 7. Attach the runbook to the alert

Point every alert at a runbook so the responder does not start from a blank page at 3 a.m.; the annotation renders as a link in the notification and the text below is what it should contain.

        annotations:
          runbook_url: "https://runbooks.internal/redis/eviction"
          description: >
            1. Confirm eviction is real: redis-cli INFO stats | grep evicted_keys.
            2. Check headroom: compare used_memory to maxmemory.
            3. If the working set outgrew the box, scale memory or shard.
            4. If a key family exploded, cap its TTL or fix an unbounded write.

# Failure Modes

Alert flapping around the threshold. The eviction rate hovers right at 100/s, so the alert fires, resolves, and re-fires every scrape, spamming the on-call. Diagnose by graphing the raw expression against its threshold and counting state transitions:

# Count ALERTING<->OK transitions for the rule over the last hour.
curl -s 'http://prometheus:9090/api/v1/query?query=changes(ALERTS{alertname="RedisEvictionRateHigh"}[1h])'

Fix by widening the for: window (a spike must persist to matter) and, if it still oscillates, adding hysteresis — alert above 100/s but only resolve below 60/s using two rules — so the resolve threshold sits below the fire threshold.

Rate reads negative or zero after a restart. A Redis restart resets evicted_keys to 0; if the counter is mistyped as a gauge, rate() sees a drop and returns a bogus value. Diagnose by inspecting the metric type Prometheus recorded:

redis-cli INFO stats | grep -E "evicted_keys|keyspace_"   # confirm counters exist
curl -s http://exporter:9121/metrics | grep "# TYPE redis_evicted_keys_total"

The # TYPE ... counter line must say counter, not gauge — Step 2 uses CounterMetricFamily precisely so rate() compensates for the reset instead of emitting a false negative.

Misses spike but eviction is flat — you are alerting on the wrong symptom. A cache stampede from synchronized TTL expiry drives keyspace_misses up while evicted_keys stays still, so an eviction-only alert misses a real incident. Diagnose by comparing the two rates side by side:

redis-cli INFO stats | grep -E "expired_keys|evicted_keys|keyspace_misses"

A miss surge tracking expired_keys rather than evicted_keys points at expiry, not memory — mitigate with jittered TTLs from TTL vs Explicit Invalidation rather than adding memory.

# Verification

Confirm the exporter is publishing the three counters and that Prometheus scraped them:

curl -s http://exporter:9121/metrics | grep -E "redis_(evicted_keys|keyspace_hits|keyspace_misses)_total"

Force a controlled eviction on a throwaway instance and confirm the rate expression responds within one scrape interval:

redis-cli CONFIG SET maxmemory 8mb
redis-cli CONFIG SET maxmemory-policy allkeys-lru
redis-cli DEBUG POPULATE 1000000        # overflow memory to trigger eviction
redis-cli INFO stats | grep evicted_keys # should climb; rate() turns non-zero

Validate the alert rule itself with promtool so a syntax error never ships silently:

promtool check rules redis-eviction-misses.yml
promtool test rules redis-eviction-misses.test.yml   # unit-test the thresholds

# FAQ

# Should I alert on eviction or on expiry?

They are different signals with different fixes. evicted_keys climbs when memory hits maxmemory and the policy sacrifices keys to admit writes — the fix is more memory, sharding, or a tighter working set. expired_keys climbs when TTLs fire on schedule, which is normal; a miss surge tracking expiry usually means synchronized expiry rather than a memory problem. Alert on eviction rate for capacity, and watch expiry alongside misses to distinguish the two.

# Why alert on the rate instead of the absolute counter?

evicted_keys and keyspace_misses are cumulative totals that only ever grow, so any static threshold on the raw value fires once and then stays fired forever after enough uptime. rate() measures how fast the counter is moving right now, which is the quantity that actually correlates with user-visible pain, and it resets cleanly when the process restarts. Alerting on the derivative is what turns a lagging counter into a leading indicator.

# What hit-ratio floor should I set?

There is no universal number — set the floor relative to your own steady-state baseline, not an internet rule of thumb. A read-heavy catalog cache might sit at 0.98 and a floor of 0.90 catches real regressions; a write-heavy session store might legitimately run at 0.80. Measure the ratio over a normal week, then set the alert a few points below the fifth-percentile of that distribution so ordinary variance does not page anyone.

# How do I stop the alert from flapping?

Two levers. First, lengthen the for: window so the condition must hold continuously before firing — most transient spikes resolve inside a couple of scrape intervals. Second, add hysteresis by separating the fire and resolve thresholds: fire above a high value but only clear below a distinctly lower one, so a metric oscillating around a single line cannot toggle the alert. Together they convert a jittery raw signal into a stable, actionable one.

---

Up: Monitoring and Alerting for Redis Caches

# Related

• Building a Cache Hit-Ratio SLO Dashboard — turn the same hit/miss counters into an error budget.
• LRU vs LFU Eviction Policies — what decides which keys the eviction alert is counting.
• Monitoring and Alerting for Redis Caches — the parent guide to Redis cache observability.
• Redis Caching Architecture & Invalidation Fundamentals — where these counters fit in the wider stack.