AWS Always Looked Vast and Expensive. So I Built a Dead-Man's-Switch to See for Myself.

For years I had no real reason to touch AWS. And the little I’d seen of it looked so vast that I just assumed it would be expensive and complicated on top of that.

Opening the console invites a wall of services onto your screen, each with its own acronym and subsections. That alone was enough sensory overload to make me file the whole platform under “costly and complicated” and move on. So I stayed in the world I knew and told myself I’d climb that curve eventually.

That assumption turned out to be almost entirely wrong.

I only found out by using the Edge Methodology and diving deep into the unknown. At the time, I was concurrently working on Patcher’s API and needed a dead-man’s-switch. It’s small enough to be safe and real enough to be useful, seemingly the perfect candidate for this exercise.

The verdict, before the build

A dead-man’s-switch is straightforward conceptually, basically just send an alert if a scheduled check-in fails. You can build the smallest honest version with a launchd service, curl, and a Slack webhook in an afternoon.

So why does mine run on a small pile of AWS services? Genuinely, I wanted to kill two birds with one stone and dive into learning AWS head-first. The switch has to run somewhere else, otherwise we’d just get silence if the box goes down (the exact event I most need to hear about).

The architecture below is engineering for the off-site requirement and a deliberate learning detour for the AWS specifically part.

flowchart LR
    EB["EventBridge<br/>every 15 min"] --> L["Lambda<br/>monitor"]
    L -->|"GET /health, /mcp, /stats"| API["Patcher API"]
    L <-->|"last known status"| DDB[("DynamoDB")]
    L -->|"read webhook"| SSM["SSM Parameter Store"]
    L -->|"only on a status change"| SLACK["Slack"]

What “down” actually means

Before writing any code, we need to define what broken means in our context.

The first obvious answer is a crashed server. Fairly straightforward, we have a /health endpoint to poll and can alert on any response that isn’t a healthy 200. But what else?

Well, for a catalog API that serves application metadata, I’d want to be alerted if that data ever stopped refreshing. Serving stale data is better than serving no data, sure, but not by much. If the daily refresh of the catalog fails, the API keeps returning 200 OK for weeks while the data rots. A plain uptime check would never notice.

Different kinds of “alive”

• Liveness: does the endpoint answer, with the right status code and (optionally) an expected string in the body?
• Freshness: is the data recent? The API exposes a /stats endpoint with a last_refresh timestamp. If that timestamp is older than the refresh interval plus some headroom, the catalog is stale, even though the server is responsive.

Getting into AWS (and proving the bill won’t bite)

Signing up is the ordinary part: an email, a credit card for identity verification, a phone confirmation. The part worth saying out loud, because it’s what put the cost assumption to bed:

Almost everything in this project lives inside the AWS Free Tier by a wide margin. The monitor runs 96 times a day. The Free Tier covers a million Lambda invocations a month. The database is billed per request and stores a handful of tiny rows. There’s one stored secret. Add it up and the expected monthly cost is, genuinely, $0.

If the cost question is what’s kept you away too, do two things on day one:

1. Set a billing alarm. AWS Budgets will email you the instant projected spend crosses a threshold you set (mine is a couple of dollars). It’s free, and it converts “what if I get a surprise bill” into “I’ll get a warning long before any bill is a surprise.”
2. Don’t use your root account for daily work. Which brings us to the part I want to make sure to do properly.

IAM, the part worth doing right

When you sign up, AWS hands you a “root” identity that can do anything, including close the account and run up the bill. You don’t use it for everyday work, instead you create scoped identities.

Two IAM ideas worth internalizing:

• An IAM role is a set of permissions that a thing (a person, or in our case a Lambda function) can temporarily assume. The role isn’t a login, it’s a costume with exactly the powers needed for one job.
• Least privilege means that costume grants the minimum set of actions and nothing more.

Here’s the actual policy my Lambda runs under. The label on each block denotes what it is allowed to do: write its own logs, read and write one database table, read one secret, decrypt that one secret. That’s the entire list.

data "aws_iam_policy_document" "lambda" {
  statement {
    sid       = "Logs"
    actions   = ["logs:CreateLogStream", "logs:PutLogEvents"]
    resources = ["${aws_cloudwatch_log_group.lambda.arn}:*"]
  }
  statement {
    sid       = "StatusTable"
    actions   = ["dynamodb:GetItem", "dynamodb:PutItem"]
    resources = [aws_dynamodb_table.status.arn]
  }
  statement {
    sid       = "ReadWebhook"
    actions   = ["ssm:GetParameter"]
    resources = [local.slack_webhook_arn]
  }
  statement {
    sid       = "DecryptWebhook"
    actions   = ["kms:Decrypt"]
    resources = [data.aws_kms_alias.ssm.target_key_arn]
  }
}

If this function were ever compromised, the fallout is scoped only to reading the uptime table and single Slack URL. It cannot touch anything else in the account, because it was never granted the power to. Least privilege access, executed to a T.

What a Lambda actually is

A Lambda function is a chunk of code that AWS runs for you, on demand, with no server for you to provision, patch, or keep alive. You hand AWS a zip of code and say “run the handler function when X happens.” X can be an HTTP request, a file landing in storage, a message on a queue, or, in our case, a timer.

You pay only for the milliseconds it runs. When it’s idle, which is almost always, it costs nothing. That billing model is exactly why a once-every-15-minutes monitor is effectively free.

Other times to reach for Lambda, beyond this could include:

• Scheduled chores: nightly cleanups, report generation, this monitor.
• Reacting to events: resize an image the moment it’s uploaded, process a webhook, fan out a queue message.
• Glue between services: a small function that translates one system’s output into another system’s input.

It’s a poor fit when you need something running continuously, holding long-lived connections, or doing heavy sustained compute. For short, bursty, event-shaped work, it’s ideal.

Here’s the entire monitor. Outside of boto3, it’s all stdlib Python. The script probes each target, checks freshness, compares against the last-known status in DynamoDB, and messages Slack only when something changed.

"""
Probe Patcher's public endpoints on a schedule and alert Slack on status
transitions. Last-known status per target lives in DynamoDB, so a sustained
outage pages once when it goes down and once when it recovers, not every run.
"""

import json
import os
import urllib.error
import urllib.request
from datetime import datetime, timezone

import boto3

PROBE_TIMEOUT = 10
# Cloudflare 403s the default Python-urllib UA as a bot; identify ourselves instead.
USER_AGENT = "brainframe-monitor/1.0 (+https://github.com/liquidz00/Patcher)"
TARGETS = json.loads(os.environ["TARGETS"])
TABLE_NAME = os.environ["DDB_TABLE"]
WEBHOOK_PARAM = os.environ["SLACK_WEBHOOK_PARAM"]
STATS_URL = os.environ.get("STATS_URL", "")  # empty disables the freshness check
FRESH_MAX_AGE_HOURS = float(os.environ.get("FRESH_MAX_AGE_HOURS", "26"))

_table = boto3.resource("dynamodb").Table(TABLE_NAME)
_ssm = boto3.client("ssm")
_webhook_url = None  # cached across warm invocations


# A 3xx means the server answered, so treat it as alive rather than chasing
# the redirect (mcp.patcherctl.dev/mcp returns 307 by design).
class _NoRedirect(urllib.request.HTTPRedirectHandler):
    def redirect_request(self, req, fp, code, msg, headers, newurl):
        return None


_opener = urllib.request.build_opener(_NoRedirect)


def _probe(target):
    """Return ``("up" | "down", reason)`` for a single endpoint."""
    req = urllib.request.Request(target["url"], headers={"User-Agent": USER_AGENT})
    try:
        with _opener.open(req, timeout=PROBE_TIMEOUT) as resp:
            code = resp.status
            body = resp.read(8192).decode("utf-8", "replace")
    except urllib.error.HTTPError as exc:
        code, body = exc.code, ""
    except Exception as exc:
        return "down", f"request failed: {exc}"

    if not 200 <= code < 400:
        return "down", f"HTTP {code}"
    match = target.get("body_match")
    if match and match not in body:
        return "down", f"body missing {match!r}"
    return "up", f"HTTP {code}"


def _probe_freshness():
    """Return ``("up" | "down", reason)`` from the catalog's last-refresh age."""
    req = urllib.request.Request(STATS_URL, headers={"User-Agent": USER_AGENT})
    try:
        with _opener.open(req, timeout=PROBE_TIMEOUT) as resp:
            data = json.loads(resp.read())
    except Exception as exc:
        return "down", f"stats fetch failed: {exc}"

    stamp = data.get("last_refresh")
    if not stamp:
        return "down", "catalog reports no last_refresh"
    age_hours = (datetime.now(timezone.utc) - datetime.fromisoformat(stamp)).total_seconds() / 3600
    if age_hours > FRESH_MAX_AGE_HOURS:
        return "down", f"catalog stale: refreshed {age_hours:.0f}h ago"
    return "up", f"fresh ({age_hours:.0f}h ago)"


def _last_status(name):
    item = _table.get_item(Key={"target": name}).get("Item")
    return item["status"] if item else "up"  # assume healthy until proven otherwise


def _save_status(name, status, reason):
    _table.put_item(
        Item={
            "target": name,
            "status": status,
            "reason": reason,
            "updated": datetime.now(timezone.utc).isoformat(),
        }
    )


def _post_slack(text):
    global _webhook_url
    if _webhook_url is None:
        _webhook_url = _ssm.get_parameter(Name=WEBHOOK_PARAM, WithDecryption=True)["Parameter"]["Value"]

    payload = json.dumps({"text": text}).encode("utf-8")
    req = urllib.request.Request(
        _webhook_url,
        data=payload,
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    urllib.request.urlopen(req, timeout=PROBE_TIMEOUT)


def handler(event, context):
    results = [(target["name"], *_probe(target)) for target in TARGETS]
    if STATS_URL:
        results.append(("catalog freshness", *_probe_freshness()))

    transitions = []
    for name, status, reason in results:
        if status != _last_status(name):
            transitions.append((name, status, reason))
            _save_status(name, status, reason)

    if transitions:
        lines = [
            f"🔴 DOWN  `{name}`: {reason}" if status == "down" else f"✅ RECOVERED  `{name}`"
            for name, status, reason in transitions
        ]
        _post_slack("\n".join(lines))

    return {"checked": len(results), "transitions": len(transitions)}

The detail I’m proudest of is the transition logic at the bottom. Storing last-known status per target in DynamoDB is what turns a noisy “still down, still down, still down” alarm into a civilized one: it pages me once when something goes down, and once when it recovers.

Standing it up with Terraform

I could have clicked all of this together in the AWS web console. I didn’t, because I wanted the infrastructure to be reproducible text, not a series of remembered clicks I’d have to redo (and probably get subtly wrong) if I ever rebuilt it.

Terraform is the tool for that. You describe the infrastructure you want in a configuration language (HCL), and Terraform figures out the API calls to make reality match. Change the text, run terraform apply, and it computes the diff.

Lambda

resource "aws_lambda_function" "monitor" {
  function_name    = local.function_name
  role             = aws_iam_role.lambda.arn
  runtime          = "python3.13"
  handler          = "handler.handler"
  filename         = data.archive_file.lambda.output_path
  source_code_hash = filebase64sha256("${path.module}/lambda/handler.py")
  timeout          = 60

  environment {
    variables = {
      TARGETS             = jsonencode(var.targets)
      DDB_TABLE           = aws_dynamodb_table.status.name
      SLACK_WEBHOOK_PARAM = local.slack_webhook_name
      STATS_URL           = var.stats_url
      FRESH_MAX_AGE_HOURS = tostring(var.freshness_max_age_hours)
    }
  }

  depends_on = [aws_cloudwatch_log_group.lambda]
}

Two lines there do quiet, important work:

1. source_code_hash is a fingerprint of the handler file, so when I change the Python, Terraform notices and redeploys. Leave it out and your code edits silently never ship.
2. The environment block is how configuration (which URLs to probe, the staleness threshold) reaches the function without being hardcoded into it.

Timer

The timer that drives the whole thing is an EventBridge schedule. This is the “check in on a schedule” half of the dead-man’s-switch:

resource "aws_cloudwatch_event_rule" "schedule" {
  name                = "${local.function_name}-schedule"
  schedule_expression = var.schedule_expression
}

What it’s watching

Configuration of what to watch lives in variables, so the next person (or future me) can change targets without ever touching the logic:

variable "aws_region" {
  description = "AWS region for all resources."
  type        = string
  default     = "us-east-1"
}

variable "schedule_expression" {
  description = "EventBridge rate/cron for the probe."
  type        = string
  default     = "rate(15 minutes)"
}

variable "stats_url" {
  description = "Catalog /stats endpoint for the freshness check. Empty string disables it."
  type        = string
  default     = "https://api.patcherctl.dev/stats"
}

variable "freshness_max_age_hours" {
  description = "Alert if the catalog's last_refresh is older than this many hours (daily refresh + headroom)."
  type        = number
  default     = 26
}

variable "targets" {
  description = "Endpoints to probe. body_match, if set, is a substring the response body must contain."
  type = list(object({
    name       = string
    url        = string
    body_match = optional(string)
  }))
  default = [
    {
      name       = "api /health"
      url        = "https://api.patcherctl.dev/health"
      body_match = "\"status\":\"ok\""
    },
    {
      name = "mcp"
      url  = "https://mcp.patcherctl.dev/mcp"
    },
  ]
}

The optional bits

Two components of this specific configuration are completely optional. I’ve kept this section shorter intentionally to keep the focus of the post on AWS.

Slack

You’ll need a Slack workspace and a Slack app. Both you can create completely for free.

Webhook

The webhook used throughout this article comes from a Slack app. Create one at api.slack.com/apps, turn on Incoming Webhooks, add a webhook to the channel you want alerts in, and copy the URL. That URL is a password (anyone with it can post to your channel), which is why it goes into Parameter Store as an encrypted SecureString rather than a plain variable.

Keeping the Slack secret out of state

A subtle gotcha with Terraform worth calling out: anything Terraform creates gets written into its state file in plaintext, including secrets. I didn’t want my Slack webhook URL sitting in state. So I created the webhook manually and stored the encrypted value with a single AWS CLI command (below). Terraform only ever references it by its unique ID (its ARN) to grant read access, it never sees the value itself.

aws ssm put-parameter \
  --name /brainframe/monitor/slack_webhook \
  --type SecureString \
  --value 'https://hooks.slack.com/services/XXX/YYY/ZZZ'

locals {
  function_name = "brainframe-monitor"

  # Created out-of-band so the webhook value never lands in Terraform state.
  # Terraform only references its ARN to grant the Lambda read access.
  slack_webhook_name = "/brainframe/monitor/slack_webhook"
  slack_webhook_arn  = "arn:aws:ssm:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:parameter${local.slack_webhook_name}"
}

HashiCorp Cloud Platform (HCP)

For the truly curious, my Terraform runs remotely on HCP Terraform and authenticates to AWS via OIDC (a keyless trust handshake). This means there are no static AWS keys stored anywhere. That’s a bootstrapping rabbit hole of its own and deserves its own post. For a first project, running Terraform locally with an AWS SSO profile is completely fine.

So, was the overkill worth it?

Time for the honest ledger I promised.

What AWS genuinely bought me

• Off-site independence. This is the non-negotiable one. The monitor survives a Linode outage and a home-internet outage, because it doesn’t depend on either. A cron job on the monitored box could never make that claim.
• Effectively zero operational burden. Nothing to patch, nothing to keep running, no server of my own to babysit. It has run untouched and free.
• Reproducibility. The entire thing is ~120 lines of HCL and ~120 of Python. I could rebuild it in a fresh account in minutes.

What it cost

• Concepts. To deploy a 40-line script I had to meet IAM roles, policies, EventBridge, DynamoDB, and Parameter Store. That’s a real learning tax, even if each piece is small.
• More surface area. Five services instead of one box means five things to understand, even if none of them ever needs attention.

The line where it tips

If you have one hobby service and a spare always-on machine somewhere other than the thing you’re monitoring, a launchd or cron timer running curl and a Slack post is the right answer. Genuinely. Don’t let anyone (or any AI) talk you into a cloud account for that.

It tips toward managed infrastructure when the monitor’s independence actually matters: when “where do I even run this so it survives the outage?” has no good answer among the machines you own, or when you’re monitoring something whose failure could take your monitor down with it. At that point, “somewhere else, that costs nothing when idle, defined as code” is not overkill. It’s just correct.

For me, the off-site requirement was real, so some off-site monitor was always justified. Choosing AWS Lambda specifically, over an equally-valid GitHub Actions cron, was the learning detour. I’d make the same call again, because the thing I was actually buying was experience in a platform I’d written off as too costly and complicated to bother with. That experience turned out to be the cheapest thing on the whole invoice.

The full stack lives in the brainframe repository under workspaces/brainframe/monitor/. If you’ve been avoiding AWS on the same assumption I was, build something tiny. Set the billing alarm, stay in the free tier, and let a thing that actually works prove the assumption wrong.