Docs / Use Cases / Retail Streaming Analytics
Diagram: produce.py writes order events to Firehose (DirectPut), which delivers newline-delimited JSON to S3, cataloged by Glue and queried with Athena SQL.

Retail Streaming Analytics

A real streaming ingestion-to-analytics pipeline: order and clickstream events in, SQL aggregates out — the first example in this repo to exercise Firehose, Glue, and Athena together as one real pipeline, not three separate service demos.

What the demo does. Provisions a raw S3 bucket, an Athena-results S3 bucket, a Firehose delivery stream (DirectPut, destination extended_s3), and a Glue database + table over the delivered files. scripts/produce.py writes JSON order events straight to Firehose; Firehose buffers and flushes them to S3 as newline-delimited JSON; Glue catalogs the S3 location; Athena runs SQL directly against it. Source: 24-retail-streaming-analytics/ in the examples repo.
📝

The newline gotcha

Firehose concatenates each record's payload back-to-back with no delimiter of its own — real AWS behavior, not a LocalEmu quirk. Skip the trailing \n and Athena sees one malformed blob instead of N records.

🔌

A real, documented gap

Firehose's KinesisStreamAsSource mode accepts the config and polls the stream, but no record ever reaches S3. This example uses DirectPut instead, a legitimate real-AWS pattern on its own.

📊

Three services, one pipeline

Firehose, Glue, and Athena aren't demoed in isolation here — the test suite proves the full ingestion-to-query path with a fixed, known batch of events.

Step-by-Step Walkthrough

Step 1: Start LocalEmu

Terminal
$ localemu start   # no special flags

Step 2: Deploy the pipeline with Terraform

./scripts/deploy.sh local
$ ./scripts/deploy.sh local
$ cd terraform && terraform apply -auto-approve -var "target=local"

→ deployed to local. outputs:
athena_results_bucket = "le-analytics-athena-results-us-east-1"
delivery_stream_name  = "le-analytics-events"
glue_database         = "retail_analytics_db"
glue_table            = "events"
raw_bucket             = "le-analytics-raw-us-east-1"

7 resources: 2 S3 buckets, the Firehose IAM role and policy, the Firehose delivery stream itself, and the Glue database + table pointed at the delivery stream's S3 prefix.

Step 3: Produce events

scripts/produce.py
# scripts/produce.py: write straight to Firehose (DirectPut), one
# JSON record per PutRecord call, each terminated with its own \n.
$ python3 scripts/produce.py 50
put 50 records to le-analytics-events

# buffering_size=1 / buffering_interval=10 (terraform/main.tf) ->
# wait ~15s for the flush to land in S3 before querying.

Each record is json.dumps(record) + "\n", written with a direct firehose.put_record call — no Kinesis stream in between.

Step 4: Query with Athena

Terminal
$ aws athena start-query-execution \
    --query-string "SELECT event, COUNT(*), ROUND(SUM(amount),2) FROM retail_analytics_db.events GROUP BY event" \
    --result-configuration OutputLocation=s3://le-analytics-athena-results-us-east-1/

$ aws athena get-query-results --query-execution-id <id>
event             | _col1 | _col2
order_placed      | ..    | ..
order_shipped      | ..    | ..
order_cancelled    | ..    | ..

Plain SQL against the Glue-cataloged table, no different from querying real S3-backed data on real AWS.

What the run proves

Six integration tests against a real, fixed, known batch of 6 events (3 order_placed, 2 order_shipped, 1 order_cancelled), so a wrong aggregation is a genuine bug, not sampling noise.

Test Proves
test_total_event_count_includes_this_runs_batchCOUNT(*) for this run's batch is exactly 6
test_group_by_event_typeGROUP BY event gives placed=(3, $180.00), shipped=(2, $150.00), cancelled=(1, $30.00)
test_filter_by_customerWHERE customer='alice' AND event='order_placed' sums to $150.00
test_cancellation_ratecancelled / placed = 1/3, computed in SQL with a CASE WHEN
test_delivered_files_are_newline_delimited_jsonthe delivered S3 object really ends in \n — the exact gotcha this lab is built around
test_unknown_order_prefix_returns_zero_not_an_errora query matching nothing returns COUNT(*) = 0, not an error

Design note: the Kinesis-as-source gap, and why DirectPut instead

The architecture you'd expect for "real" streaming analytics is producer → Kinesis → Firehose → S3, so that's what was tried first. CreateDeliveryStream with a KinesisStreamSourceConfiguration succeeds, and the consumer genuinely polls the stream (confirmed via continuous kinesis.GetRecords => 200 in the server log) — but no record ever reaches S3, with the exact same destination config that delivers in seconds under plain DirectPut.

This example uses DirectPut instead — a legitimate, common real-AWS pattern on its own (e.g. a Lambda-based ingestion layer writing straight to Firehose), not a downgrade — but the Kinesis-as-source gap is real and worth filing upstream if you want the "producer writes to Kinesis, Firehose fans out to S3" shape to actually work.

Full source: scripts/produce.py

The event producer. Writes straight to Firehose with DirectPut, appending the newline each record needs.

#!/usr/bin/env python3
"""Simulate a retail order clickstream, writing straight to Firehose.

Firehose concatenates record payloads back-to-back in the delivered file
with no delimiter of its own (real AWS behavior, not a LocalEmu quirk) -
each record MUST end with its own newline, or Athena's JSON reader sees
one malformed blob instead of N newline-delimited records.
"""
from __future__ import annotations

import json
import os
import random
import subprocess
import sys
from pathlib import Path

import boto3

HERE = Path(__file__).resolve().parent

EVENTS = ["order_placed", "order_shipped", "order_cancelled"]
CUSTOMERS = ["alice", "bob", "carol", "dave"]


def _tf_output(name: str) -> str:
    out = subprocess.check_output(["terraform", "output", "-raw", name], cwd=str(HERE.parent / "terraform"))
    return out.decode().strip()


def main() -> None:
    count = int(sys.argv[1]) if len(sys.argv) > 1 else 20

    firehose = boto3.client(
        "firehose",
        endpoint_url=os.environ.get("AWS_ENDPOINT_URL", "http://localhost:4566"),
        region_name="us-east-1",
        aws_access_key_id="AKIAIOSFODNN7EXAMPLE",
        aws_secret_access_key="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    )
    stream_name = _tf_output("delivery_stream_name")

    for i in range(count):
        record = {
            "order_id": f"O-{3000 + i}",
            "event": random.choice(EVENTS),
            "amount": round(random.uniform(10, 200), 2),
            "customer": random.choice(CUSTOMERS),
        }
        firehose.put_record(
            DeliveryStreamName=stream_name,
            Record={"Data": (json.dumps(record) + "\n").encode()},
        )

    print(f"put {count} records to {stream_name}")


if __name__ == "__main__":
    main()

Files

Repository layout.

24-retail-streaming-analytics/
├── README.md
├── terraform/                (7 resources, one provider)
├── tests/
│   └── test_analytics.py     (6 tests)
└── scripts/
    ├── deploy.sh
    ├── test.sh
    ├── teardown.sh
    └── produce.py