Tuesday, 18 April 2023

AWS Kinesis Flink Application not supported in terraform

UPDATE (18/04/23): There is now a terraform resource for an Apache Flink Kinesis Application, please see here for the terraform documentation.

So until the below PR is addressed:

https://github.com/terraform-providers/terraform-provider-aws/pull/11652

I've been using null_resources for Apache Flink Kinesis Application. It's not great, but it's a workaround. See below example:

resource "null_resource" "kinesis_application" {
  provisioner "local-exec" {
    when    = create
    command = "aws kinesisanalyticsv2 create-application --cli-input-json '{\"ApplicationName\":\"${local.name}\",\"RuntimeEnvironment\":\"FLINK-1_8\",\"ServiceExecutionRole\":\"${aws_iam_role.main.arn}\",\"ApplicationConfiguration\":{\"FlinkApplicationConfiguration\":{\"MonitoringConfiguration\":{\"ConfigurationType\":\"CUSTOM\",\"MetricsLevel\":\"APPLICATION\",\"LogLevel\":\"WARN\"},\"ParallelismConfiguration\":{\"ConfigurationType\":\"CUSTOM\",\"Parallelism\":1,\"ParallelismPerKPU\":1,\"AutoScalingEnabled\":true}},\"EnvironmentProperties\":{\"PropertyGroups\":[{\"PropertyGroupId\":\"KdaConfigProperties\",\"PropertyMap\":{\"aws.kinesis.input.stream\":\"${local.project}-event_aggregator-kda-input-${local.environment}\",\"aws.kinesis.output.stream\":\"${local.project}-event_aggregator-kda-output-${local.environment}\",\"aws.region\":\"us-east-1\",\"AggregationEnabled\":\"false\",\"flink.stream.initpos\":\"TRIM_HORIZON\"}}]},\"ApplicationCodeConfiguration\":{\"CodeContent\":{\"S3ContentLocation\":{\"BucketARN\":\"${aws_s3_bucket.main.arn}\",\"FileKey\":\"event-aggregator.jar\"}},\"CodeContentType\":\"ZIPFILE\"},\"ApplicationSnapshotConfiguration\":{\"SnapshotsEnabled\":true}},\"CloudWatchLoggingOptions\":[{\"LogStreamARN\":\"${aws_cloudwatch_log_stream.main.arn}\"}]}' --profile ${var.aws_profile}"
  }

  provisioner "local-exec" {
    when    = destroy
    command = <<EOT
export TIMESTAMP=$(aws kinesisanalyticsv2 describe-application --application-name ${local.name} --profile ${var.aws_profile} --query 'ApplicationDetail.CreateTimestamp');
aws kinesisanalyticsv2 delete-application --create-timestamp $TIMESTAMP --application-name ${local.name} --profile ${var.aws_profile}
EOT
  }
}


No comments:

Post a Comment

Platform Engineering: Developer Experience

Intro The role of Platform Engineer as evolved through the ages. I’ve gone through all the names: Sys Admin, Infrastructure Engineer, DevOps...