Retail Streaming Analytics
A real streaming ingestion-to-analytics pipeline: order and clickstream events in, SQL aggregates out — the first example in this repo to exercise Firehose, Glue, and Athena together as one real pipeline, not three separate service demos.
DirectPut, destination extended_s3),
and a Glue database + table over the delivered files.
scripts/produce.py writes JSON order events
straight to Firehose; Firehose buffers and flushes them to S3 as newline-delimited JSON; Glue
catalogs the S3 location; Athena runs SQL directly against it.
Source: 24-retail-streaming-analytics/ in the examples repo.
The newline gotcha
Firehose concatenates each record's payload back-to-back with no delimiter of its own — real AWS behavior, not a LocalEmu quirk. Skip the trailing \n and Athena sees one malformed blob instead of N records.
A real, documented gap
Firehose's KinesisStreamAsSource mode accepts the config and polls the stream, but no record ever reaches S3. This example uses DirectPut instead, a legitimate real-AWS pattern on its own.
Three services, one pipeline
Firehose, Glue, and Athena aren't demoed in isolation here — the test suite proves the full ingestion-to-query path with a fixed, known batch of events.
Step-by-Step Walkthrough
Step 1: Start LocalEmu
$ localemu start # no special flags Step 2: Deploy the pipeline with Terraform
$ ./scripts/deploy.sh local
$ cd terraform && terraform apply -auto-approve -var "target=local"
→ deployed to local. outputs:
athena_results_bucket = "le-analytics-athena-results-us-east-1"
delivery_stream_name = "le-analytics-events"
glue_database = "retail_analytics_db"
glue_table = "events"
raw_bucket = "le-analytics-raw-us-east-1" 7 resources: 2 S3 buckets, the Firehose IAM role and policy, the Firehose delivery stream itself, and the Glue database + table pointed at the delivery stream's S3 prefix.
Step 3: Produce events
# scripts/produce.py: write straight to Firehose (DirectPut), one
# JSON record per PutRecord call, each terminated with its own \n.
$ python3 scripts/produce.py 50
put 50 records to le-analytics-events
# buffering_size=1 / buffering_interval=10 (terraform/main.tf) ->
# wait ~15s for the flush to land in S3 before querying. Each record is json.dumps(record) + "\n", written with a direct firehose.put_record call — no Kinesis stream in between.
Step 4: Query with Athena
$ aws athena start-query-execution \
--query-string "SELECT event, COUNT(*), ROUND(SUM(amount),2) FROM retail_analytics_db.events GROUP BY event" \
--result-configuration OutputLocation=s3://le-analytics-athena-results-us-east-1/
$ aws athena get-query-results --query-execution-id <id>
event | _col1 | _col2
order_placed | .. | ..
order_shipped | .. | ..
order_cancelled | .. | .. Plain SQL against the Glue-cataloged table, no different from querying real S3-backed data on real AWS.
What the run proves
Six integration tests against a real, fixed, known batch of 6 events (3 order_placed, 2 order_shipped, 1 order_cancelled), so a wrong aggregation is a genuine bug, not sampling noise.
| Test | Proves |
|---|---|
| test_total_event_count_includes_this_runs_batch | COUNT(*) for this run's batch is exactly 6 |
| test_group_by_event_type | GROUP BY event gives placed=(3, $180.00), shipped=(2, $150.00), cancelled=(1, $30.00) |
| test_filter_by_customer | WHERE customer='alice' AND event='order_placed' sums to $150.00 |
| test_cancellation_rate | cancelled / placed = 1/3, computed in SQL with a CASE WHEN |
| test_delivered_files_are_newline_delimited_json | the delivered S3 object really ends in \n — the exact gotcha this lab is built around |
| test_unknown_order_prefix_returns_zero_not_an_error | a query matching nothing returns COUNT(*) = 0, not an error |
Design note: the Kinesis-as-source gap, and why DirectPut instead
The architecture you'd expect for "real" streaming analytics is producer → Kinesis → Firehose → S3, so that's what was tried first. CreateDeliveryStream with a KinesisStreamSourceConfiguration succeeds, and the consumer genuinely polls the stream (confirmed via continuous kinesis.GetRecords => 200 in the server log) — but no record ever reaches S3, with the exact same destination config that delivers in seconds under plain DirectPut.
This example uses DirectPut instead — a legitimate, common real-AWS pattern on its own (e.g. a Lambda-based ingestion layer writing straight to Firehose), not a downgrade — but the Kinesis-as-source gap is real and worth filing upstream if you want the "producer writes to Kinesis, Firehose fans out to S3" shape to actually work.
Full source: scripts/produce.py
The event producer. Writes straight to Firehose with DirectPut, appending the newline each record needs.
#!/usr/bin/env python3
"""Simulate a retail order clickstream, writing straight to Firehose.
Firehose concatenates record payloads back-to-back in the delivered file
with no delimiter of its own (real AWS behavior, not a LocalEmu quirk) -
each record MUST end with its own newline, or Athena's JSON reader sees
one malformed blob instead of N newline-delimited records.
"""
from __future__ import annotations
import json
import os
import random
import subprocess
import sys
from pathlib import Path
import boto3
HERE = Path(__file__).resolve().parent
EVENTS = ["order_placed", "order_shipped", "order_cancelled"]
CUSTOMERS = ["alice", "bob", "carol", "dave"]
def _tf_output(name: str) -> str:
out = subprocess.check_output(["terraform", "output", "-raw", name], cwd=str(HERE.parent / "terraform"))
return out.decode().strip()
def main() -> None:
count = int(sys.argv[1]) if len(sys.argv) > 1 else 20
firehose = boto3.client(
"firehose",
endpoint_url=os.environ.get("AWS_ENDPOINT_URL", "http://localhost:4566"),
region_name="us-east-1",
aws_access_key_id="AKIAIOSFODNN7EXAMPLE",
aws_secret_access_key="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
)
stream_name = _tf_output("delivery_stream_name")
for i in range(count):
record = {
"order_id": f"O-{3000 + i}",
"event": random.choice(EVENTS),
"amount": round(random.uniform(10, 200), 2),
"customer": random.choice(CUSTOMERS),
}
firehose.put_record(
DeliveryStreamName=stream_name,
Record={"Data": (json.dumps(record) + "\n").encode()},
)
print(f"put {count} records to {stream_name}")
if __name__ == "__main__":
main() Files
Repository layout.
24-retail-streaming-analytics/
├── README.md
├── terraform/ (7 resources, one provider)
├── tests/
│ └── test_analytics.py (6 tests)
└── scripts/
├── deploy.sh
├── test.sh
├── teardown.sh
└── produce.py