Amazon Kinesis and Managed Flink

Overview

Amazon Kinesis and Amazon Managed Service for Apache Flink are AWS's real-time data streaming services — Kinesis for ingesting and delivering high-throughput event streams, and Managed Flink for stateful, complex computations on those streams.

Before diving in: understand the two-stage model that all streaming architectures follow:

Data Producers                  Ingest Layer              Process / Deliver Layer
(apps, devices, logs)  ──────▶  [ Kinesis ]  ──────────▶  [ Flink / Lambda / Redshift ]

1. Ingest — collect raw events from many sources at high speed (Kinesis's job)
2. Process / Deliver — transform, aggregate, filter, or route events to a destination (Flink, Lambda, or Firehose's job)

The Kinesis Family

Kinesis is not one service — it is a family of four related services:

Kinesis Data Streams (KDS)

KDS is the core streaming buffer — a durable, ordered, replayable log of records. Producers push events in; multiple consumers read from it independently.

Key Concepts

How Capacity Works

Total stream throughput = Number of Shards × (1 MB/s in / 2 MB/s out)

Example: 10 shards = 10 MB/s ingestion, 20 MB/s consumption

• On-Demand mode: AWS automatically scales shards based on traffic (simpler, higher cost)
• Provisioned mode: shard count is set manually (more control, lower cost)

KDS vs. SQS — When to Use Which

Kinesis Data Firehose

Firehose is a fully managed, zero-code delivery pipeline — it reads from a stream (or accepts records directly) and delivers them to a destination, with optional transformation along the way.

Destinations

Key Properties

• No consumer code needed — fully managed; configure a destination, and Firehose handles the rest
• Buffer — batches records before delivery (configurable: 1–900 seconds or 1–128 MB, whichever comes first)
• Lambda transformation — optionally invoke a Lambda function to transform each batch before delivery
• Not real-time — minimum ~60 second delivery latency due to buffering (this is by design)

Amazon Managed Service for Apache Flink

What is Apache Flink?

Apache Flink is an open-source, distributed stream processing framework. It excels at stateful, complex, real-time computations on event streams — aggregations over time windows, anomaly detection, joining two streams together, and more.

AWS offers it as a fully managed service called Amazon Managed Service for Apache Flink (previously named Kinesis Data Analytics for Apache Flink).

What "Stateful" Means

Most stream operations are stateless (filter a record, transform a field). Stateful means the computation needs to remember past events:

• "How many orders in the last 5 minutes?" → requires a time window (state = running count)
• "Alert if the same card is used in 3 different countries within 1 hour" → requires keyed state per card
• "Join a click event stream with a user profile stream" → requires stream-to-stream join

Flink manages this state durably — even if the application restarts, it picks up where it left off.

Key Concepts

Example: Tumbling Window Aggregation

-- Count orders per product every 1 minute (Flink SQL API)
SELECT
  product_id,
  COUNT(*) AS order_count,
  TUMBLE_END(event_time, INTERVAL '1' MINUTE) AS window_end
FROM orders
GROUP BY product_id, TUMBLE(event_time, INTERVAL '1' MINUTE);

When to Use Flink vs. Lambda

The Full Analytics Pipeline — How These Services Fit Together

[Producers]          [Ingest]              [Process]           [Store / Serve]
IoT sensors    ──▶  Kinesis Data    ──▶   Managed Flink   ──▶  DynamoDB (live state)
Mobile apps          Streams               (stateful               S3 (data lake)
Web clickstream                            real-time)          Redshift (warehouse)
Log events     ──▶  Kinesis Data    ──▶   ─────────────── ──▶  S3 / Redshift /
                    Firehose                (no-code pipe)       OpenSearch

SAA/SAP Exam Tips

Cross-Cloud Equivalents

Kinesis Data Streams

Kinesis Data Firehose

Apache Flink / Managed Flink

Pricing Model

Kinesis Data Streams:

• On-Demand: per GB ingested + per GB retrieved
• Provisioned: per shard-hour + per PUT payload unit

Kinesis Data Firehose:

• Per GB of data ingested; additional charge for format conversion (Parquet/ORC)

Managed Flink:

• Per Kinesis Processing Unit (KPU) per hour (1 KPU = 1 vCPU + 4 GB RAM)

Related Services / See Also

• Amazon Redshift — common Firehose and Flink destination for analytical warehousing
• Amazon Athena — query the S3 data that Firehose lands
• Amazon SQS and SNS — message queuing / fan-out; different pattern from streaming
• Amazon MSK — managed Apache Kafka; alternative ingestion layer to KDS