Native s3 Filesystem Blog#860
Conversation
|
cc @davidradl if you have some time, since you're native 🙂 |
|
|
||
| Apache Flink relies on the underlying filesystem for much of its work: reading and writing application data, materializing streaming sinks, and storing checkpoints and savepoints for recovery. For years, S3 support in Flink meant choosing between two Hadoop-based plugins, each with its own trade-offs and configuration quirks. With Flink 2.3, there is a better option. | ||
|
|
||
| Today we're introducing `flink-s3-fs-native`, A ground-up, Hadoop-free S3 filesystem built specifically for Flink. It ships as an experimental opt-in plugin in Flink 2.3, is already running in production at scale at major technology companies, and delivers measurable, reproducible performance gains. |
There was a problem hiding this comment.
I wonder if the op-in plugin could be turned into a hyperlink to a section that describes it.
| | **~2x faster checkpoints** | 48.8 s average vs 90.1 s with the Presto plugin; up to 4.5x at small state sizes | | ||
| | **Drop-in replacement** | Swap the JAR, keep your existing `flink-conf.yaml`, restart your cluster | | ||
| | **No Hadoop dependency** | ~13 MB JAR vs ~30–93 MB; no CVE triage on Hadoop transitive dependencies | | ||
| | **AWS SDK v2** | Async-first I/O; AWS SDK v1 entered maintenance mode December 2025 | |
There was a problem hiding this comment.
hyper links for the AWS things and end date would be helpful.
|
|
||
| ### Test environment | ||
|
|
||
| The benchmark ran on Amazon EKS (ap-south-1) with a Flink 2.1.1 cluster composed of 1 JobManager (2 GB memory, 1 core) and 2 TaskManagers (6 GB memory, 1.5 cores, 4 task slots each) for a total parallelism of 8. The workload targeted 20 GB of RocksDB state with full, non-incremental checkpoints every 60 seconds in EXACTLY_ONCE mode. The test ran for approximately 77 minutes. Configurations for both plugins were identical except for the plugin JAR itself. |
There was a problem hiding this comment.
Add a caveat that users performance might differ.
Can we point to the payloads - so users can run exactly this benchmark?
There was a problem hiding this comment.
updated with new wording PTAL
| - **Enhanced observability** : S3 operation metrics (latency, retry counts, throughput) exposed through Flink's metric system, giving platform teams visibility into S3 I/O behavior. | ||
| - **Stream-based S3 read/write** : Improving memory efficiency for large object operations. | ||
|
|
||
| **Phase 2: Recommended default.** Once stability is proven across a broad set of community deployments, the native plugin will be promoted to the recommended default for new Flink installations. Documentation, quickstarts, and tutorials will be updated accordingly. |
There was a problem hiding this comment.
how will be know that "proven across a broad set of community deployments" has happened.
There was a problem hiding this comment.
Clarified in Phase 2, promotion is a community decision on the dev@ list, and the concrete signals are sustained adoption feedback from production users plus no open Blocker/Critical JIRAs against the native plugin across at least one full release cycle.
|
|
||
| **Phase 2: Recommended default.** Once stability is proven across a broad set of community deployments, the native plugin will be promoted to the recommended default for new Flink installations. Documentation, quickstarts, and tutorials will be updated accordingly. | ||
|
|
||
| **Phase 3: Legacy deprecation.** The Hadoop and Presto plugins will be formally deprecated with a defined support window before removal. |
There was a problem hiding this comment.
we could deprecate in phase 1. As we do not intend to enhance these connectors.
|
|
||
| `flink-s3-fs-native` is part of Apache Flink and is developed in the open. The module lives at `flink-filesystems/flink-s3-fs-native` in the [Flink repository](https://github.com/apache/flink). | ||
|
|
||
| The migration is safe and requires minimal deployment changes. If your team is already evaluating or running this in production, we want to hear from you. Your feedback directly shapes the path from experimental to default. |
There was a problem hiding this comment.
Maybe be explicit as to how to contact us about this. Is there a tag that should be used in a dev list post?
|
|
||
| ```bash | ||
| # 1. Remove your existing plugin | ||
| rm plugins/flink-s3-fs-hadoop-*.jar # or flink-s3-fs-presto-*.jar |
There was a problem hiding this comment.
If we stick to the pattern what the official exiting doc page suggests the this is more like
rm -rf plugins/flink-s3-fs-hadoop/
There was a problem hiding this comment.
|
fyi @Samrat002 when you have completed an action you should resolve the conversation so it is obvious what is outstanding. |
|
|
||
| ```bash | ||
| # 1. Remove your existing plugin | ||
| rm -rf plugins/flink-s3-fs-hadoop/ # or plugins/flink-s3-fs-presto/ |
There was a problem hiding this comment.
what is the comment here ?
Izeren
left a comment
Izeren
left a comment
There was a problem hiding this comment.
Thank you @Samrat002, overall, looks good to me, I have left a couple of comments, PTAL.
| |---|---| | ||
| | **[~2x faster checkpoints](#performance)** | 48.8 s average vs 90.1 s with the Presto plugin; up to 4.5x at small state sizes | | ||
| | **Drop-in replacement** | Swap the JAR, keep your existing `flink-conf.yaml`, restart your cluster | | ||
| | **No Hadoop dependency** | ~13 MB JAR vs ~30–93 MB; no CVE triage on Hadoop transitive dependencies | |
There was a problem hiding this comment.
Is it worth calling out specific Hadoop dependencies that are painful and not required in the new FS?
| **At a glance** | ||
|
|
||
| | | | | ||
| |---|---| |
There was a problem hiding this comment.
Do we want to call out streaming implementation support here? Or are we keeping it for next releases post?
|
|
||
| The decision to replace the S3 plugin is not just a performance choice. It has direct operational and financial consequences. | ||
|
|
||
| **Security and compliance teams** have long carried the burden of triaging CVEs in `hadoop-common`'s transitive dependency tree. That tree is large, changes frequently, and generates a steady stream of vulnerability disclosures unrelated to S3 or Flink. Removing it permanently eliminates that toil. Fewer dependencies mean fewer CVEs, fewer emergency patch cycles, and fewer security review gates for new deployments. |
There was a problem hiding this comment.
"eliminates" is a strong word. I am sure there will be still remaining surface related to SDK v2 bumps and network clients (like Netty).
|
|
||
| **Platform and infrastructure teams** running multi-tenant Flink clusters benefit from a clean, unified `s3.*` configuration namespace. The native plugin's configuration model is designed for Flink. No Hadoop-style key mirroring, no adapter translation layer, no debugging sessions caused by settings silently not propagating. | ||
|
|
||
| **Risk and compliance teams** should note that the AWS SDK for Java 1.x has been in [maintenance mode since July 31, 2024](https://aws.amazon.com/blogs/developer/the-aws-sdk-for-java-1-x-is-in-maintenance-mode-effective-july-31-2024/) and reaches **end-of-support on December 31, 2025**, after which it receives no further updates or releases. The foundation that both existing plugins depend on is therefore reaching end-of-life, which means no new features and a winding-down stream of bug and security fixes. Continuing to operate on SDK v1 is an accumulating technical and compliance liability. The native plugin is built entirely on [AWS SDK v2](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/home.html). |
There was a problem hiding this comment.
"reaches"
This refers to the past, has it already reached?
|
|
||
| **Risk and compliance teams** should note that the AWS SDK for Java 1.x has been in [maintenance mode since July 31, 2024](https://aws.amazon.com/blogs/developer/the-aws-sdk-for-java-1-x-is-in-maintenance-mode-effective-july-31-2024/) and reaches **end-of-support on December 31, 2025**, after which it receives no further updates or releases. The foundation that both existing plugins depend on is therefore reaching end-of-life, which means no new features and a winding-down stream of bug and security fixes. Continuing to operate on SDK v1 is an accumulating technical and compliance liability. The native plugin is built entirely on [AWS SDK v2](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/home.html). | ||
|
|
||
| **Operations teams** benefit from faster checkpoints in two concrete ways: |
There was a problem hiding this comment.
That is not only operation teams' benefit. All customers using exactly once would get lower e2e latency when checkpoints are faster
|
|
||
| **Exactly-once recoverable writes.** `NativeS3RecoverableWriter` uses S3 multipart uploads to provide exactly-once semantics for Flink's sink connectors and checkpoint metadata. Uploads are resumable on failure. The writer can recover an in-progress multipart upload and continue from the last committed part. | ||
|
|
||
| **Per-bucket configuration.** A single Flink cluster will be able to access multiple S3 buckets with distinct credentials, regions, endpoints, and encryption policies, configured via `s3.bucket.<bucket-name>.<property>`. This is planned for Flink 2.4. |
There was a problem hiding this comment.
Should we also include plans for CTR support?
| # 4. Restart your cluster | ||
| ``` | ||
|
|
||
| Existing checkpoints and savepoints on S3 remain fully readable. The native filesystem is read/write compatible with data written by either the Hadoop or Presto plugins. |
There was a problem hiding this comment.
Worth calling out that users can also package both jars and use Flink config override to choose which factory to use for the same schema. As well as the fact that state is compatible both ways, so rollback is easy and safe
|
|
||
| **A note on s5cmd.** Users of `s5cmd` for bulk state downloads should be aware that the native plugin does not use `s5cmd`. Instead, it relies on `S3TransferManager`'s async concurrent transfer engine, which demonstrated superior throughput in our benchmarks. No external binary dependency is required. | ||
|
|
||
| **Safety net.** If both a legacy plugin JAR and the native JAR are accidentally present in `plugins/`, Flink will not crash. The Hadoop plugin takes precedence. There is no data loss risk from a misconfigured migration. |
There was a problem hiding this comment.
It is not just a safety net it is deliberate opportunity to "choose" which plugin to use via priority configuration that by default selects hadoop, but can be overridden
|
|
||
| ## Two Plugins, One Filesystem, and No Good Answer | ||
|
|
||
| If you've configured S3 for Flink before, you likely know that Flink ships two S3 filesystem plugins, and both register on the same `s3://` scheme. Only one can be active at a time. Choosing between them has been a source of confusion for years. |
There was a problem hiding this comment.
| If you've configured S3 for Flink before, you likely know that Flink ships two S3 filesystem plugins, and both register on the same `s3://` scheme. Only one can be active at a time. Choosing between them has been a source of confusion for years. | |
| If you've configured S3 for Flink before, you likely know that Flink ships two S3 filesystem plugins, and both register on the same `s3://` scheme. Only one can be active at a time. Choosing between them has been a source of confusion for years. Even once one has been chosen, its use still [perplexes](https://rmoff.net/2024/08/06/troubleshooting-flink-sql-s3-problems/) many end-users because of the similarly-named but different configurations required. |
|
2.3 is just out. @Samrat002 can you plz have a look at the comments, resolve them to push it out? |
5 participants
Flink Native s3 first release blog
Todo : Fix the file name and the correct release date in
docs/content/posts/2026-06-14-announcing-native-s3-fs.md