From df945974e614b2df76970c2318825e0ac12583ad Mon Sep 17 00:00:00 2001 From: Viren Nadkarni Date: Tue, 17 Dec 2024 15:04:14 +0530 Subject: [PATCH 1/6] Managed Flink: Add tagging support --- content/en/user-guide/aws/kinesisanalyticsv2/index.md | 1 - 1 file changed, 1 deletion(-) diff --git a/content/en/user-guide/aws/kinesisanalyticsv2/index.md b/content/en/user-guide/aws/kinesisanalyticsv2/index.md index f366209fc8..0715f7d727 100644 --- a/content/en/user-guide/aws/kinesisanalyticsv2/index.md +++ b/content/en/user-guide/aws/kinesisanalyticsv2/index.md @@ -169,7 +169,6 @@ $ awslocal s3api list-objects --bucket sink-bucket - Only S3 zipfile code is supported - Values of 20,000 ms for `execution.checkpointing.interval` and 5,000 ms for `execution.checkpointing.min-pause` are used for checkpointing. They can not be overridden. -- [Tagging](https://docs.aws.amazon.com/managed-flink/latest/java/how-tagging.html) is not supported - In-place [version upgrades](https://docs.aws.amazon.com/managed-flink/latest/java/how-in-place-version-upgrades.html) and [roll-backs](https://docs.aws.amazon.com/managed-flink/latest/java/how-system-rollbacks.html) are not supported - [Snapshot/savepoint management](https://docs.aws.amazon.com/managed-flink/latest/java/how-snapshots.html) is not implemented - CloudWatch and CloudTrail integration is not implemented From 09e7ef20002fd4107b0a985eae98b5efb5fa3c76 Mon Sep 17 00:00:00 2001 From: Viren Nadkarni Date: Fri, 27 Dec 2024 12:08:50 +0530 Subject: [PATCH 2/6] Add section on CloudWatch Logging --- .../aws/kinesisanalyticsv2/index.md | 52 ++++++++++++++++++- 1 file changed, 51 insertions(+), 1 deletion(-) diff --git a/content/en/user-guide/aws/kinesisanalyticsv2/index.md b/content/en/user-guide/aws/kinesisanalyticsv2/index.md index 0715f7d727..0be9fd7f52 100644 --- a/content/en/user-guide/aws/kinesisanalyticsv2/index.md +++ b/content/en/user-guide/aws/kinesisanalyticsv2/index.md @@ -153,6 +153,55 @@ You can verify this with: $ awslocal s3api list-objects --bucket sink-bucket {{< /command >}} +## CloudWatch Logging + +LocalStack supports CloudWatch Logs integration to help monitor the Flink cluster for application events or configuration problems. +The logging option can be added at the time of creating the Flink application using the [CreateApplication](https://docs.aws.amazon.com/managed-flink/latest/apiv2/API_CreateApplication.html) operation. +Logging options can also be managed at a later point using the [AddApplicationCloudWatchLoggingOption](https://docs.aws.amazon.com/managed-flink/latest/apiv2/API_AddApplicationCloudWatchLoggingOption.html) and [DeleteApplicationCloudWatchLoggingOption](https://docs.aws.amazon.com/managed-flink/latest/apiv2/API_DeleteApplicationCloudWatchLoggingOption.html) operations. + +There are following prerequisites for CloudWatch Logs integration: +- You must create the application's log group and log stream. Flink will not create it for you. +- You must add the permissions your application needs to write to the log stream to the service execution role. + Generally the following IAM actions are sufficient: `logs:DescribeLogGroups`, `logs:DescribeLogStreams` and `logs:PutLogEvents` + +To add a logging option: + +{{< command >}} +$ awslocal kinesisanalyticsv2 add-application-cloud-watch-logging-option \ + --application-name msaf-app \ + --cloud-watch-logging-option '{"LogStreamARN": "arn:aws:logs:us-east-1:000000000000:log-group:msaf-log-group:log-stream:msaf-log-stream"}' +{ + "ApplicationARN": "arn:aws:kinesisanalytics:us-east-1:000000000000:application/msaf-app", + "ApplicationVersionId": 2, + "CloudWatchLoggingOptionDescriptions": [ + { + "CloudWatchLoggingOptionId": "1.1", + "LogStreamARN": "arn:aws:logs:us-east-1:000000000000:log-group:msaf-log-group:log-stream:msaf-log-stream" + } + ] +} +{{< /command >}} + +Configured logging options can be retrieved using [DescribeApplication](https://docs.aws.amazon.com/managed-flink/latest/apiv2/API_DescribeApplication.html): + +{{< command >}} +$ awslocal kinesisanalyticsv2 describe-application --application-name msaf-app | jq .ApplicationDetail.CloudWatchLoggingOptionDescriptions +[ + { + "CloudWatchLoggingOptionId": "1.1", + "LogStreamARN": "arn:aws:logs:us-east-1:000000000000:log-group:msaf-log-group:log-stream:msaf-log-stream" + } +] +{{< /command >}} + +Log events can be retrieved from CloudWatch Logs using the appropriate operation. +To retrieve all events: + +{{< command >}} +$ awslocal logs get-log-events --log-group-name msaf-log-group --log-stream-name msaf-log-stream +{{< /command >}} + + ## Supported Flink Versions | Flink version | Supported by LocalStack | Supported by Apache | @@ -171,4 +220,5 @@ $ awslocal s3api list-objects --bucket sink-bucket They can not be overridden. - In-place [version upgrades](https://docs.aws.amazon.com/managed-flink/latest/java/how-in-place-version-upgrades.html) and [roll-backs](https://docs.aws.amazon.com/managed-flink/latest/java/how-system-rollbacks.html) are not supported - [Snapshot/savepoint management](https://docs.aws.amazon.com/managed-flink/latest/java/how-snapshots.html) is not implemented -- CloudWatch and CloudTrail integration is not implemented +- CloudTrail integration and CloudWatch metrics is not implemented. + The application logging level defaults to `INFO` and can not be overridden. From 25fa8d057e1aacd9db50e999392dd07998027614 Mon Sep 17 00:00:00 2001 From: Viren Nadkarni Date: Fri, 27 Dec 2024 12:23:12 +0530 Subject: [PATCH 3/6] Add section of resource tagging --- .../aws/kinesisanalyticsv2/index.md | 29 +++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/content/en/user-guide/aws/kinesisanalyticsv2/index.md b/content/en/user-guide/aws/kinesisanalyticsv2/index.md index 0be9fd7f52..09cae673c4 100644 --- a/content/en/user-guide/aws/kinesisanalyticsv2/index.md +++ b/content/en/user-guide/aws/kinesisanalyticsv2/index.md @@ -155,7 +155,7 @@ $ awslocal s3api list-objects --bucket sink-bucket ## CloudWatch Logging -LocalStack supports CloudWatch Logs integration to help monitor the Flink cluster for application events or configuration problems. +LocalStack MSAF supports CloudWatch Logs integration to help monitor the Flink cluster for application events or configuration problems. The logging option can be added at the time of creating the Flink application using the [CreateApplication](https://docs.aws.amazon.com/managed-flink/latest/apiv2/API_CreateApplication.html) operation. Logging options can also be managed at a later point using the [AddApplicationCloudWatchLoggingOption](https://docs.aws.amazon.com/managed-flink/latest/apiv2/API_AddApplicationCloudWatchLoggingOption.html) and [DeleteApplicationCloudWatchLoggingOption](https://docs.aws.amazon.com/managed-flink/latest/apiv2/API_DeleteApplicationCloudWatchLoggingOption.html) operations. @@ -201,6 +201,31 @@ To retrieve all events: $ awslocal logs get-log-events --log-group-name msaf-log-group --log-stream-name msaf-log-stream {{< /command >}} +## Resource Tagging + +You can manage resource tags using [TagResource](https://docs.aws.amazon.com/managed-flink/latest/apiv2/API_TagResource.html), [UntagResource](https://docs.aws.amazon.com/managed-flink/latest/apiv2/API_UntagResource.html) and [ListTagsForResource](https://docs.aws.amazon.com/managed-flink/latest/apiv2/API_ListTagsForResource.html). +Tags can also be specified when creating the Flink application using the [CreateApplication](https://docs.aws.amazon.com/managed-flink/latest/apiv2/API_CreateApplication.html) operation. + +{{< command >}} +$ awslocal kinesisanalyticsv2 tag-resource \ + --resource-arn arn:aws:kinesisanalytics:us-east-1:000000000000:application/msaf-app \ + --tags Key=country,Value=SE + +$ awslocal kinesisanalyticsv2 list-tags-for-resource \ + --resource-arn arn:aws:kinesisanalytics:us-east-1:000000000000:application/msaf-app +{ + "Tags": [ + { + "Key": "country", + "Value": "SE" + } + ] +} + +$ awslocal kinesisanalyticsv2 untag-resource \ + --resource-arn arn:aws:kinesisanalytics:us-east-1:000000000000:application/msaf-app \ + --tag-keys country +{{< /command >}} ## Supported Flink Versions @@ -209,7 +234,7 @@ $ awslocal logs get-log-events --log-group-name msaf-log-group --log-stream-name | 1.20.0 | yes | yes | | 1.19.1 | yes | yes | | 1.18.1 | yes | yes | -| 1.15.2 | yes | yes | +| 1.15.2 | yes | no | | 1.13.1 | yes | no | ## Limitations From 216ac95b861dac7f3ed91735a1c1e14dd769eb70 Mon Sep 17 00:00:00 2001 From: Viren Nadkarni Date: Fri, 27 Dec 2024 12:24:17 +0530 Subject: [PATCH 4/6] Fix lint --- content/en/user-guide/aws/kinesisanalyticsv2/index.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/content/en/user-guide/aws/kinesisanalyticsv2/index.md b/content/en/user-guide/aws/kinesisanalyticsv2/index.md index 09cae673c4..1c1bef3c8e 100644 --- a/content/en/user-guide/aws/kinesisanalyticsv2/index.md +++ b/content/en/user-guide/aws/kinesisanalyticsv2/index.md @@ -160,7 +160,8 @@ The logging option can be added at the time of creating the Flink application us Logging options can also be managed at a later point using the [AddApplicationCloudWatchLoggingOption](https://docs.aws.amazon.com/managed-flink/latest/apiv2/API_AddApplicationCloudWatchLoggingOption.html) and [DeleteApplicationCloudWatchLoggingOption](https://docs.aws.amazon.com/managed-flink/latest/apiv2/API_DeleteApplicationCloudWatchLoggingOption.html) operations. There are following prerequisites for CloudWatch Logs integration: -- You must create the application's log group and log stream. Flink will not create it for you. +- You must create the application's log group and log stream. + Flink will not create it for you. - You must add the permissions your application needs to write to the log stream to the service execution role. Generally the following IAM actions are sufficient: `logs:DescribeLogGroups`, `logs:DescribeLogStreams` and `logs:PutLogEvents` From 2dec7e61ab909692f8a45539de829fbf421f2cfd Mon Sep 17 00:00:00 2001 From: Viren Nadkarni Date: Fri, 27 Dec 2024 12:25:27 +0530 Subject: [PATCH 5/6] Add links to AWS docs --- content/en/user-guide/aws/kinesisanalyticsv2/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/user-guide/aws/kinesisanalyticsv2/index.md b/content/en/user-guide/aws/kinesisanalyticsv2/index.md index 1c1bef3c8e..00a862086e 100644 --- a/content/en/user-guide/aws/kinesisanalyticsv2/index.md +++ b/content/en/user-guide/aws/kinesisanalyticsv2/index.md @@ -155,7 +155,7 @@ $ awslocal s3api list-objects --bucket sink-bucket ## CloudWatch Logging -LocalStack MSAF supports CloudWatch Logs integration to help monitor the Flink cluster for application events or configuration problems. +LocalStack MSAF supports [CloudWatch Logs integration](https://docs.aws.amazon.com/managed-flink/latest/java/cloudwatch-logs.html) to help monitor the Flink cluster for application events or configuration problems. The logging option can be added at the time of creating the Flink application using the [CreateApplication](https://docs.aws.amazon.com/managed-flink/latest/apiv2/API_CreateApplication.html) operation. Logging options can also be managed at a later point using the [AddApplicationCloudWatchLoggingOption](https://docs.aws.amazon.com/managed-flink/latest/apiv2/API_AddApplicationCloudWatchLoggingOption.html) and [DeleteApplicationCloudWatchLoggingOption](https://docs.aws.amazon.com/managed-flink/latest/apiv2/API_DeleteApplicationCloudWatchLoggingOption.html) operations. @@ -204,7 +204,7 @@ $ awslocal logs get-log-events --log-group-name msaf-log-group --log-stream-name ## Resource Tagging -You can manage resource tags using [TagResource](https://docs.aws.amazon.com/managed-flink/latest/apiv2/API_TagResource.html), [UntagResource](https://docs.aws.amazon.com/managed-flink/latest/apiv2/API_UntagResource.html) and [ListTagsForResource](https://docs.aws.amazon.com/managed-flink/latest/apiv2/API_ListTagsForResource.html). +You can manage [resource tags](https://docs.aws.amazon.com/managed-flink/latest/java/how-tagging.html) using [TagResource](https://docs.aws.amazon.com/managed-flink/latest/apiv2/API_TagResource.html), [UntagResource](https://docs.aws.amazon.com/managed-flink/latest/apiv2/API_UntagResource.html) and [ListTagsForResource](https://docs.aws.amazon.com/managed-flink/latest/apiv2/API_ListTagsForResource.html). Tags can also be specified when creating the Flink application using the [CreateApplication](https://docs.aws.amazon.com/managed-flink/latest/apiv2/API_CreateApplication.html) operation. {{< command >}} From 3c68e83753e6312cb5867a042e0ef03535d5aec1 Mon Sep 17 00:00:00 2001 From: Viren Nadkarni Date: Fri, 27 Dec 2024 13:58:00 +0530 Subject: [PATCH 6/6] Add callout note about frequency of event reporting --- content/en/user-guide/aws/kinesisanalyticsv2/index.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/content/en/user-guide/aws/kinesisanalyticsv2/index.md b/content/en/user-guide/aws/kinesisanalyticsv2/index.md index 00a862086e..f29835f0ca 100644 --- a/content/en/user-guide/aws/kinesisanalyticsv2/index.md +++ b/content/en/user-guide/aws/kinesisanalyticsv2/index.md @@ -202,6 +202,10 @@ To retrieve all events: $ awslocal logs get-log-events --log-group-name msaf-log-group --log-stream-name msaf-log-stream {{< /command >}} +{{< callout >}} +Logs events are reported to CloudWatch every 10 seconds. +{{< /callout >}} + ## Resource Tagging You can manage [resource tags](https://docs.aws.amazon.com/managed-flink/latest/java/how-tagging.html) using [TagResource](https://docs.aws.amazon.com/managed-flink/latest/apiv2/API_TagResource.html), [UntagResource](https://docs.aws.amazon.com/managed-flink/latest/apiv2/API_UntagResource.html) and [ListTagsForResource](https://docs.aws.amazon.com/managed-flink/latest/apiv2/API_ListTagsForResource.html).