Alerting

By email, Slack, SMS, push notification or a call in the middle of the night, depending on how critical the incident is.

The core is a reversal of the way of working: it is not the person who looks for the error; the error finds the person. For an online shop this is vital. A checkout that has been throwing errors for two hours costs hard cash, and depending on traffic you lose four-figure revenue during this time without noticing. Alerting shortens the span between "something is broken" and "someone knows about it" from hours to seconds. A compact overview of the concept is also provided by the Wikipedia article on network monitoring, the field in which alerting historically emerged.

Monitoring, alerting, observability: what is the difference?

The three terms are often lumped together but mean different things. Monitoring is the continuous collection and display of measurements: server load, response times, error rates. It shows you the state. Alerting is the layer on top that derives an action from these measurements: if value X exceeds threshold Y, notify Z. Observability goes further and asks whether you can derive from the collected data at all why something happened.

In short: monitoring sees, alerting calls, observability explains. Without alerting, monitoring is just a dashboard that nobody looks at when it matters. Alerting is therefore not the most expensive of the three layers, but often the one with the greatest consequences, because it decides whether a measurement leads to a reaction at all.

What triggers an alert

Alerts arise from conditions on metrics or logs. The typical triggers in an e-commerce environment:

• Threshold alerts: a metric rises above or falls below a fixed value. Example: CPU utilisation above 90 per cent or the conversion rate below half the daily average.
• Error rate alerts: the proportion of failed requests (HTTP 5xx) rises above a defined limit.
• Availability alerts: a health check or a synthetic probe can no longer reach the shop, the classic for "site is down".
• Anomaly alerts: instead of fixed thresholds, the system detects a deviation from learned normal behaviour. Helpful for metrics with a strong daily rhythm, where a fixed value does not fit.
• Heartbeat or dead man's switch alerts: these raise the alarm when an expected signal fails to arrive, for example a nightly cron job that no longer reports that it has run.

How a good alert is structured

An alert is more than "something is red". It only becomes useful through context. A well-built alert answers three questions immediately: what has happened, how bad is it, and what should the recipient do now? A notification with the text "DiskUsageWarning on prod-db-01" without a severity and without guidance on what to do only creates stress, not a reaction.

That is why severity levels have become established. A pragmatic grading:

The most common sin: declaring everything critical. Then the phone rings at night because a log file directory is 80 per cent full. The result is alert fatigue, and that is more dangerous than no alerting at all.

Alert fatigue: when too many alarms become the real risk

When a team receives dozens of alerts every day, most of which are harmless or disappear by themselves, it becomes desensitised. At some point notifications are clicked away reflexively, and then the one alarm that really mattered gets lost. In practice, this phenomenon is the main reason why alerting fails, not a lack of technology.

A few levers that work together help against alert fatigue:

• Deduplication: a thousand identical errors create one alert, not a thousand. Similar signals are combined.
• Grouping: related alerts from one incident end up in one notification instead of triggering fifty individual messages.
• Escalation: if nobody responds, the alarm moves on to the next person after X minutes, instead of being repeated more loudly.
• Honest thresholds: limits are calibrated against normal behaviour, not set low out of fear.
• Regular clear-outs: every alert that has never led to an action in recent months should be reviewed.

In its widely cited site reliability engineering book, Google formulates the guideline that every alert that wakes a person up must be actionable; otherwise it should be abolished. You can read this freely in the Google SRE Book: Monitoring Distributed Systems.

A concrete practical example

A Shopware shop sells around 500 orders a day on average, with a clear daily pattern: few in the morning, a peak in the evening. The operator sets up the following alerting rule: "If the number of successfully completed orders over a period of 30 minutes is more than 70 per cent below the expected value for this time of day, trigger a high alert in the operations Slack channel."

On a Tuesday evening at 8.15 pm, actually peak time, the order rate suddenly drops to almost zero. The anomaly alert fires after eight minutes. The message lands in the channel with context: expected orders 45, actual 3, a link to the dashboard, last deploy 22 minutes ago. The team looks immediately and finds the cause: a deployment has broken a payment method configuration, and the checkout aborts at the payment step.

Without alerting, someone might only have noticed the next morning, and more than half of the evening's revenue would have been lost. With alerting, the error was fixed after 25 minutes. The calculation is simple: the few hours of effort for setting up the rule paid for themselves on exactly this one evening.

Tools at a glance: from Prometheus to n8n

The tool landscape is broad, and different tools cover different levels:

• Prometheus with Alertmanager: the de facto standard for metric-based alerting in the open-source camp. Prometheus collects time series and defines alert rules, and the Alertmanager takes care of deduplication, grouping, silencing and routing to the right channels.
• Grafana: provides the visualisation and can raise alerts itself, fed from the same data sources.
• PagerDuty and Opsgenie: specialised in the notification and escalation side: on-call schedules, phone chains, escalation across several levels.
• Cloud-native engines: AWS CloudWatch, Google Cloud Monitoring and Azure Monitor bring their own alerting if the stack is there anyway.
• No-code workflows with n8n: for business-related alerting that sits closer to day-to-day business than to infrastructure. An n8n error workflow, for example, catches every failed run of an automation workflow and sends a formatted Slack message to the team. In the same way, an order or stock level threshold can be monitored without a single line of code.

The last point in particular is interesting for mid-sized companies: not every sensible alert needs an observability platform. A lean automation that observes a business signal and notifies the right team in an emergency can be set up with tools such as n8n in hours rather than weeks. We show such a set-up step by step in our article alerting with n8n, and how no-code alerting fits into larger processes is part of process automation & AI workflows.

Setting thresholds sensibly

The most difficult question in alerting is not "whether" but "from when". If you set the threshold too low, the alert fires constantly and becomes noise. If you set it too high, you only notice the problem when it already hurts. There is no universally correct value; it results from the normal behaviour of your system and from the question of when a person really has to intervene.

A proven approach: observe first over several weeks without alerting. Learn how a metric behaves in normal operation, with its daily, weekly and seasonal fluctuations. Only then do you set a threshold that does not catch the normal ups and downs but does catch genuine outliers. Static thresholds work well for metrics with a clear upper limit (storage space, error rate). For metrics with a strong rhythm, such as the order rate, dynamic or anomaly-based thresholds do better.

A second lever against false alarms is duration. Instead of alerting immediately at the first breach, you wait to see whether the state persists over a period of time. This is what a typical Prometheus alert rule that does exactly that looks like:

groups:
  - name: shop-alerts
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "More than 5% 5xx errors for 5 minutes"
          runbook: "https://wiki.example.com/runbooks/5xx"

The for: 5m condition ensures that a single two-second spike does not trigger an alarm; only a state that lasts five minutes is serious. This condition filters out a large proportion of nervous false alarms.

Alert on business metrics, not just technology

Most alerting set-ups concentrate on infrastructure: servers, databases, storage space, network. That is necessary but not sufficient. A shop can run flawlessly from a technical point of view (all servers green, all databases responding) and still not make any money, because there is a bug in the ordering process that no server metric shows.

That is why business metrics belong in alerting. The most effective signals for an online shop are often the most obvious ones:

• Orders per period: a sudden drop is the most reliable early warning signal for a broken checkout.
• Conversion rate: if it slumps without a drop in traffic, something is wrong in the funnel.
• Payment failure rate: if the proportion of failed payments rises, there is often a payment provider or a faulty configuration behind it.
• Basket abandonment at the last step: a sudden increase points to an error exactly where it is most expensive.

The appeal of these metrics: they measure the effect, not the cause. You do not need to know which of the hundred technical things can break; you notice it because customers are no longer buying. That is robust alerting, because it also catches errors nobody thought of.

Runbooks and escalation: what happens after the alarm

An alert without a plan is only half the job. Mature teams attach a runbook to every important alert: a short guide to what should be checked when it fires and which first steps help. Ideally, the alert links directly to this runbook. Then the person who is woken at three in the morning does not first have to think about where to start: check this dashboard first, then restart that service, and if in doubt escalate to this person.

This includes a clean escalation chain: if the first on-call recipient does not respond within the agreed time, the next one automatically takes over. Tools such as PagerDuty or Opsgenie map exactly these chains, including on-call rotas, so that it is not always the same person lying awake at night. This way, no critical alarm slips through just because someone happens to be at the cinema.

Common mistakes

We see three patterns again and again. First, alerting is set up and then never touched again, until half the staff have muted the notifications. Second, alerts are only raised on technical metrics (server, database), never on business metrics (orders, revenue). Yet you notice a broken payment button far more reliably from a drop in orders than from CPU load. Third, there are no quiet periods (maintenance windows) for planned work, so every deployment triggers an alert storm and the team is conditioned to ignore alarms.

Good alerting is not a product you buy and tick off. It is a discipline: few, sharp, action-guiding signals instead of many nervous ones. When an alert fires, the normal reaction should be "oh, I need to look at that", not "that thing again, I'll ignore it".

Frequently asked questions

What is the difference between alerting and monitoring?
Monitoring collects and displays measurements; it is the constant view of the system's state. Alerting is the layer above it that derives an active notification from these values as soon as a defined condition is met. Without alerting, monitoring remains a dashboard that only works as long as someone is looking at it.

What is alert fatigue and how do I avoid it?
Alert fatigue is the desensitisation of a team caused by too many, mostly harmless alarms. The consequence: even important alerts are ignored reflexively. Remedies are honestly calibrated thresholds, deduplication, grouping of related alerts, escalation chains and regularly clearing out rules that never lead to an action.

Which channels are suitable for alerts?
That depends on the severity. Non-critical matters belong in a Slack or Teams channel that is checked during day-to-day business. Really critical incidents (shop down, checkout broken) need a channel that gets through even at night: SMS, push or a call via an on-call service such as PagerDuty.

Do I need expensive specialist tools for alerting?
Not necessarily. For metric-based infrastructure alerting, Prometheus with Alertmanager is open source. For business-related alerting, for example a drop in orders or a failed workflow, a no-code automation with n8n that monitors a threshold and reports to the team is often enough. What matters is not the tool, but that the alerts are actionable and reach a person who is awake.

Should I alert on symptoms or on causes?
On symptoms. "Customers cannot pay" is a useful alert because it describes a real impact. "CPU at 85 per cent" is often just a number; as long as the shop is running normally, high utilisation is no reason to wake anyone up. Symptom-based alerting also catches errors that nobody thought of when setting it up.