Apache Flink delivers low latency stream processing with exactly-once guarantees across distributed workloads. This architecture page explains how Flink handles state, execution, and fault tolerance so you can build reliable data applications.
| Component | Role | Key Feature | Benefit |
|---|---|---|---|
| JobManager | Coordinates execution | Master node | Centralized control |
| TaskManager | Executes tasks | Worker node | Parallel processing |
| DataStream API | Programming model | Event-at-a-time ops | Low latency pipelines |
| State Backend | State storage | RocksDB & heap | Scalable state |
| Checkpointing | Consistency | Barrier aligned | Exactly-once |
DataStream Execution Engine
The DataStream execution engine processes continuous event streams with pipelined operators. Each operator runs in a task slot, and chaining reduces network shuffle to keep latency predictable. This design supports stateful transformations while maintaining high throughput.
Task slots isolate workload classes and memory usage within the same TaskManager. Backpressure is handled by async barriers, so slow downstream tasks do not block upstream sources. The engine schedules operator chains based on resource availability and preferred parallelism.
Network buffers and credit-based flow control prevent out-of-memory issues during bursts. Watermarks track event time progress and enable window computations across distributed partitions. Together, these mechanisms deliver low latency and consistent throughput at scale.
Distributed State Management
Stateful applications rely on managed keyed state and operator state to store intermediate results. Flink’s state backends keep snapshots in memory or RocksDB, with asynchronous checkpoints ensuring consistent recovery without blocking processing.
RocksDB state backend scales beyond heap limits by storing state on disk while keeping hot data in memory. Incremental checkpoints reduce network traffic by sending only changed files, which lowers recovery time and bandwidth usage.
State TTL cleans up expired keys automatically, preventing leaks in long-running jobs. By combining local fast access with remote durable storage, Flink balances performance, size, and correctness for demanding workloads.
Fault Tolerance Mechanism
Checkpointing captures consistent snapshots of state and stream positions across all tasks. Barriers injected into the data flow align with state snapshots, ensuring exactly-once semantics even when failures occur.
On failure, the JobManager selects a recent completed checkpoint and restarts tasks to replay from that barrier. This replay model avoids data loss and duplication, and recovery time depends on snapshot size and cluster resources.
Savepoints provide manual checkpoints for planned upgrades and maintenance. They enable pausing, updating code, or resuming on another cluster without losing progress in stateful computations.
Resource and Deployment Model
Flink runs on standalone clusters, Kubernetes, YARN, and cloud-managed services. The resource model abstracts CPU, memory, and managed memory, letting you tune task slots and operator chains per workload.
Application mode deployments package jobs as clusters per job, improving isolation and simplifying lifecycle management. High-availability setups use ZooKeeper or etco to elect a new leader if the active JobManager fails.
Dynamic scaling reschedules operator states across new parallel instances while preserving correctness. Configurable buffer timeout, backpressure monitoring, and memory controls help you balance latency and throughput in production.
Operational Best Practices
- Align checkpoint interval with your recovery time objectives and workload patterns.
- Use RocksDB for large state and configure incremental checkpoints to reduce network and storage overhead.
- Monitor backpressure, latency, and checkpoint duration to detect bottlenecks early.
- Leverage savepoints for planned upgrades, version migrations, and state inspection.
- Tune task slots, managed memory, and buffer timeouts to balance throughput and latency for your use case.
FAQ
Reader questions
How does checkpointing achieve exactly-once processing in Apache Flink?
Checkpointing uses distributed snapshots and stream position markers so that state and inputs are rewound precisely, preventing duplicates or data loss during recovery.
What is the difference between keyed state and operator state in Flink?
Keyed state is partitioned by key and accessible only from the keyed context, while operator state is tied to a specific operator task and can be redistributed during rescaling.
Can RocksDB state backend handle large state with limited heap memory?
Yes, RocksDB spills state to disk and retains only hot data in heap, enabling state sizes larger than memory while maintaining acceptable access latency.
How do savepoints support zero-downtime upgrades and cluster migration?
Savepoints are manually triggered, complete snapshots that let you pause a job, update code or cluster, and resume from the same state without data inconsistencies.