Add Flink 2.0 and Spark 4.1 upgrades to PR #4#6
Draft
adrianodyn wants to merge 3 commits intodynatrace-research:upgrade-java-kstreams-hzcastfrom
Draft
Add Flink 2.0 and Spark 4.1 upgrades to PR #4#6adrianodyn wants to merge 3 commits intodynatrace-research:upgrade-java-kstreams-hzcastfrom
adrianodyn wants to merge 3 commits intodynatrace-research:upgrade-java-kstreams-hzcastfrom
Conversation
- Bump Flink to 2.0.0 (flink-connector-kafka 4.0.1-2.0), Shadow 8.1.1, Jib 3.4.5.
- Switch toolchain to Java 21.
- Migrate sources to the Flink 2.x DataStream/Sink V2 APIs and the new
KafkaRecordSerializationSchema (replaces the deprecated KafkaSerializationSchema).
- Update Kryo serializers and SerDes for the Flink 2.x type-info changes.
- Kubernetes manifests:
* Rename ConfigMap key flink-conf.yaml -> config.yaml (Flink 2.x).
* Update volume items in JobManager/TaskManager deployments accordingly.
* Add the standard Flink 2.x JVM --add-opens / --add-exports list to
env.java.opts.all (the official image sets these in its bundled
config.yaml, but our ConfigMap mount overrides /opt/flink/conf).
- Bump Spark to 4.1.1 / Hadoop 3.4.2; switch toolchain and runtime to Java 21.
- Rewrite Dockerfile on ubuntu:22.04 + openjdk-21-jre-headless; ship the
shadow fat jar (shuffle-spark-1.0-SNAPSHOT-all.jar) and add an
IMAGE_BUILD_ID build marker for deploy verification.
- Application: refactor SparkStructuredStreamingShuffle to support both
MicroProfile Config (USE_MICROPROFILE_CONFIG=true) and a pure env-var
fallback. Default trigger mode is the standard Spark micro-batch
(spark.trigger.mode=default); 'realtime', 'processing' and 'continuous'
modes are configurable via spark.trigger.mode + spark.trigger.interval.
- Update microprofile-config.properties defaults and add a short README
documenting the new trigger-mode configuration.
- Tweak Kryo registrator and spark-master/spark-defaults.conf for the new
driver/executor JVM module-access requirements.
- Kubernetes manifest:
* Use the -all (shadow) jar path.
* Pass the standard Spark-on-Java-21 --add-opens / --add-exports list
via spark.driver.extraJavaOptions and spark.executor.extraJavaOptions.
* Set the env vars now required by the env-var fallback
(KAFKA_TOPIC_INPUT/OUTPUT, CONSUMER_OUTPUT_RATE,
CONSUMER_STATE_SIZE_BYTES, CONSUMER_INIT_COUNT_SEED).
- aws/kafka-storage-class.yaml: switch from the in-tree kubernetes.io/aws-ebs provisioner (removed in K8s 1.23+) to the AWS EBS CSI driver (ebs.csi.aws.com), with csi.storage.k8s.io/fstype: xfs and allowVolumeExpansion: true. - values.yaml: drop the now-invalid 'jvmOptions: -Xms/-Xmx: null' override for Strimzi Kafka (newer Strimzi CRDs reject null for these string fields). Also override STRIMZI_KUBERNETES_VERSION on the cluster operator so Strimzi 0.38 stops trying to parse the Kubernetes 1.34+ version reply (which it doesn't understand).
Comment on lines
+7
to
+8
| # In-tree "kubernetes.io/aws-ebs" provisioner was removed in K8s 1.23+ and | ||
| # is fully gone by 1.35. Use the AWS EBS CSI driver instead. |
Member
There was a problem hiding this comment.
Suggested change
| # In-tree "kubernetes.io/aws-ebs" provisioner was removed in K8s 1.23+ and | |
| # is fully gone by 1.35. Use the AWS EBS CSI driver instead. |
Comment on lines
+20
to
+26
| # Note: Theodolite chart used to set "-Xms"/"-Xmx" to null here to let | ||
| # Strimzi auto-size the JVM heap, but newer Strimzi CRDs reject null | ||
| # for these fields (they must be strings). Leave jvmOptions unset so | ||
| # the chart's default string values apply. To override, use e.g.: | ||
| # jvmOptions: | ||
| # "-Xms": "4G" | ||
| # "-Xmx": "4G" |
Member
There was a problem hiding this comment.
Suggested change
| # Note: Theodolite chart used to set "-Xms"/"-Xmx" to null here to let | |
| # Strimzi auto-size the JVM heap, but newer Strimzi CRDs reject null | |
| # for these fields (they must be strings). Leave jvmOptions unset so | |
| # the chart's default string values apply. To override, use e.g.: | |
| # jvmOptions: | |
| # "-Xms": "4G" | |
| # "-Xmx": "4G" |
Could be removed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This builds on top of #4 and adds:
(EBS CSI provisioner; Strimzi CRD/JVM-options compat)