initial commit

Author: ksmin23

README.md

@@ -1,11 +1,20 @@
-## My Project
+# Amazon OpenSearch Ingestion CDK Python project!
 
-TODO: Fill this README out!
+This repository contains a set of example projects for [Amazon OpenSearch Ingestion](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ingestion.html)
 
-Be sure to:
+| Example | Architecture | Description |
+|---------|-------------|------|
+| [ingestion to opensearch domain](./opensearch) | ![osis-domain-pipeline](./opensearch/osis-domain-pipeline.svg) | data ingestion to an opensearch domain using OpenSearch Ingestion Pipelines |
+| [opensearch-serverless colleciton](./opensearch-serverless) | ![osis-collection-pipeline](./opensearch-serverless/osis-collection-pipeline.svg) | data ingestion to an opensearch serverless collection using OpenSearch Ingestion Pipelines |
 
-* Change the title in this README
-* Edit your repository description on GitHub
+Enjoy!
+
+## References
+
+ * [Amazon OpenSearch Ingestion Developer Guide](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ingestion.html)
+   * [Tutorial: Ingesting data into a domain using Amazon OpenSearch Ingestion](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/osis-get-started.html)
+   * [Tutorial: Ingesting data into a collection using Amazon OpenSearch Ingestion](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/osis-serverless-get-started.html)
+ * [Data Prepper](https://opensearch.org/docs/latest/data-prepper/index/) - a server-side data collector capable of filtering, enriching, transforming, normalizing, and aggregating data for downstream analytics and visualization.
 
 ## Security
 
@@ -14,4 +23,3 @@ See [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more inform
 ## License
 
 This library is licensed under the MIT-0 License. See the LICENSE file.
-

opensearch-serverless/.gitignore

@@ -0,0 +1,10 @@
+*.swp
+package-lock.json
+__pycache__
+.pytest_cache
+.venv
+*.egg-info
+
+# CDK asset staging directory
+.cdk.staging
+cdk.out

opensearch-serverless/README.md

@@ -0,0 +1,174 @@
+
+# Ingesting data into a collection using Amazon OpenSearch Ingestion
+
+![osis-collection-pipeline](./osis-collection-pipeline.svg)
+
+This is an Amazon OpenSearch ingestion project for CDK development with Python.
+
+This project builds on the following tutorial: [Ingesting data into a collection using Amazon OpenSearch Ingestion](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/osis-serverless-get-started.html).
+
+This project shows you how to use Amazon OpenSearch Ingestion to configure a simple pipeline and ingest data into an Amazon OpenSearch Serverless collection.
+
+The `cdk.json` file tells the CDK Toolkit how to execute your app.
+
+This project is set up like a standard Python project.  The initialization
+process also creates a virtualenv within this project, stored under the `.venv`
+directory.  To create the virtualenv it assumes that there is a `python3`
+(or `python` for Windows) executable in your path with access to the `venv`
+package. If for any reason the automatic creation of the virtualenv fails,
+you can create the virtualenv manually.
+
+To manually create a virtualenv on MacOS and Linux:
+
+```
+$ python3 -m venv .venv
+```
+
+After the init process completes and the virtualenv is created, you can use the following
+step to activate your virtualenv.
+
+```
+$ source .venv/bin/activate
+```
+
+If you are a Windows platform, you would activate the virtualenv like this:
+
+```
+% .venv\Scripts\activate.bat
+```
+
+Once the virtualenv is activated, you can install the required dependencies.
+
+```
+(.venv) $ pip install -r requirements.txt
+```
+
+At this point you can now synthesize the CloudFormation template for this code.
+
+<pre>
+(.venv) $ export CDK_DEFAULT_ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
+(.venv) $ export CDK_DEFAULT_REGION=$(curl -s 169.254.169.254/latest/dynamic/instance-identity/document | jq -r .region)
+(.venv) $ cdk synth -c iam_user_name=<i>your-iam-user-name</i> --all
+</pre>
+
+:warning: Amazon OpenSearch Serverless requires mandatory IAM permission for access to resources.
+You are required to add these two IAM permissions for your OpenSearch Serverless **"aoss:APIAccessAll"** for Data Plane API access, and **"aoss:DashboardsAccessAll"** for Dashboards access. Failure to add the two new IAM permissions will result in 403 errors starting on May 10th, 2023
+
+For a sample data-plane policy [here](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/security-iam-serverless.html#security_iam_id-based-policy-examples-data-plane.html):
+
+  - [Using OpenSearch Serverless in the console
+](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/security-iam-serverless.html#security_iam_serverless_id-based-policy-examples-console)
+  - [Administering OpenSearch Serverless collections](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/security-iam-serverless.html#security_iam_id-based-policy-examples-collection-admin)
+  - [Viewing OpenSearch Serverless collections](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/security-iam-serverless.html#security_iam_id-based-policy-examples-view-collections)
+  - [Using data-plane policies](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/security-iam-serverless.html#security_iam_id-based-policy-examples-data-plane)
+
+Use `cdk deploy` command to create the stack shown above.
+
+<pre>
+(.venv) $ cdk deploy -c iam_user_name=<i>your-iam-user-name</i> --all
+</pre>
+
+To add additional dependencies, for example other CDK libraries, just add
+them to your `setup.py` file and rerun the `pip install -r requirements.txt`
+command.
+
+## Clean Up
+
+Delete the CloudFormation stack by running the below command.
+
+<pre>
+(.venv) $ cdk destroy -c iam_user_name=<i>your-iam-user-name</i> --force --all
+</pre>
+
+## Useful commands
+
+ * `cdk ls`          list all stacks in the app
+ * `cdk synth`       emits the synthesized CloudFormation template
+ * `cdk deploy`      deploy this stack to your default AWS account/region
+ * `cdk diff`        compare deployed stack with current state
+ * `cdk docs`        open CDK documentation
+
+Enjoy!
+
+## Run Tests
+
+#### Step 1: Ingest some sample data
+
+First, get the ingestion URL from the **Pipeline settings** page:
+
+![osis-pipeline-settings](./assets/osis-pipeline-settings.png)
+
+Then, ingest some sample data. The following sample request uses [awscurl](https://github.com/okigan/awscurl) to send a single log file to the `my_logs` index:
+
+<pre>
+$ awscurl --service osis --region <i>us-east-1</i> \
+  -X POST \
+  -H "Content-Type: application/json" \
+  -d '[{"time":"2014-08-11T11:40:13+00:00","remote_addr":"122.226.223.69","status":"404","req
+uest":"GET http://www.k2proxy.com//hello.html HTTP/1.1","http_user_agent":"Mozilla/4.0 (compatible; WOW64; SLCC2;)"}]' \
+https://<i>{pipeline-endpoint}.us-east-1</i>.osis.amazonaws.com/log-pipeline/test_ingestion_path
+</pre>
+
+You should see a `200 OK` response.
+
+#### Step 2: Query the sample data
+
+Now, query the `my_logs` index to ensure that the log entry was successfully ingested:
+
+<pre>
+$ awscurl --service aoss --region <i>us-east-1</i> \
+     -X GET \
+     https://<i>{collection-id}.us-east-1</i>.aoss.amazonaws.com/my_logs/_search | jq -r '.'
+</pre>
+
+**Sample response:**
+
+<pre>
+{
+  "took": 367,
+  "timed_out": false,
+  "_shards": {
+    "total": 0,
+    "successful": 0,
+    "skipped": 0,
+    "failed": 0
+  },
+  "hits": {
+    "total": {
+      "value": 1,
+      "relation": "eq"
+    },
+    "max_score": 1,
+    "hits": [
+      {
+        "_index": "my_logs",
+        "_id": "1%3A0%3ALkidTIgBbiu_ytx_zXnH",
+        "_score": 1,
+        "_source": {
+          "time": "2014-08-11T11:40:13+00:00",
+          "remote_addr": "122.226.223.69",
+          "status": "404",
+          "request": "GET http://www.k2proxy.com//hello.html HTTP/1.1",
+          "http_user_agent": "Mozilla/4.0 (compatible; WOW64; SLCC2;)",
+          "@timestamp": "2023-05-24T07:16:29.708Z"
+        }
+      }
+    ]
+  }
+}
+</pre>
+
+## References
+
+ * [Tutorial: Ingesting data into a collection using Amazon OpenSearch Ingestion](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/osis-serverless-get-started.html)
+ * [Amazon OpenSearch Ingestion Developer Guide](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ingestion.html)
+ * [Data Prepper](https://opensearch.org/docs/latest/data-prepper/index/) - a server-side data collector capable of filtering, enriching, transforming, normalizing, and aggregating data for downstream analytics and visualization.
+ * [Top strategies for high volume tracing with Amazon OpenSearch Ingestion (2023-04-27)](https://aws.amazon.com/blogs/big-data/top-strategies-for-high-volume-tracing-with-amazon-opensearch-ingestion/)
+ * [Use cases for Amazon OpenSearch Ingestion
+](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/use-cases-overview.html) - some common use cases for Amazon OpenSearch Ingestion.
+ * [Best practices for Amazon OpenSearch Ingestion](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/osis-best-practices.html)
+ * [Identity and Access Management for Amazon OpenSearch Serverless](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/security-iam-serverless.html#security_iam_id-based-policy-examples-data-plane.html)
+ * [Setting up roles and users in Amazon OpenSearch Ingestion](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/pipeline-security-overview.html)
+ * [AWS Signature Version 4 Signing Examples](https://github.com/aws-samples/sigv4a-signing-examples)
+ * [awscurl](https://github.com/okigan/awscurl) - curl-like tool with AWS Signature Version 4 request signing.
+

opensearch-serverless/app.py

@@ -0,0 +1,31 @@
+#!/usr/bin/env python3
+import os
+
+from cdk_stacks import (
+  OpsCollectionPipelineRoleStack,
+  OpsServerlessTimeSeriesStack,
+  OpsServerlessIngestionStack
+)
+
+import aws_cdk as cdk
+
+
+AWS_ENV = cdk.Environment(account=os.getenv('CDK_DEFAULT_ACCOUNT'),
+  region=os.getenv('CDK_DEFAULT_REGION'))
+
+app = cdk.App()
+
+collection_pipeline_role = OpsCollectionPipelineRoleStack(app, 'OpsCollectionPipelineRoleStack')
+
+ops_serverless_ts_stack = OpsServerlessTimeSeriesStack(app, "OpsServerlessTSStack",
+  collection_pipeline_role.iam_role.role_arn,
+  env=AWS_ENV)
+ops_serverless_ts_stack.add_dependency(collection_pipeline_role)
+
+ops_serverless_ingestion_stack = OpsServerlessIngestionStack(app, "OpsServerlessIngestionStack",
+  collection_pipeline_role.iam_role.role_arn,
+  ops_serverless_ts_stack.collection_endpoint,
+  env=AWS_ENV)
+ops_serverless_ingestion_stack.add_dependency(ops_serverless_ts_stack)
+
+app.synth()

opensearch-serverless/assets/osis-pipeline-settings.png

@@ -0,0 +1,51 @@
+{
+  "app": "python3 app.py",
+  "watch": {
+    "include": [
+      "**"
+    ],
+    "exclude": [
+      "README.md",
+      "cdk*.json",
+      "requirements*.txt",
+      "source.bat",
+      "**/__init__.py",
+      "python/__pycache__",
+      "tests"
+    ]
+  },
+  "context": {
+    "@aws-cdk/aws-lambda:recognizeLayerVersion": true,
+    "@aws-cdk/core:checkSecretUsage": true,
+    "@aws-cdk/core:target-partitions": [
+      "aws",
+      "aws-cn"
+    ],
+    "@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver": true,
+    "@aws-cdk/aws-ec2:uniqueImdsv2TemplateName": true,
+    "@aws-cdk/aws-ecs:arnFormatIncludesClusterName": true,
+    "@aws-cdk/aws-iam:minimizePolicies": true,
+    "@aws-cdk/core:validateSnapshotRemovalPolicy": true,
+    "@aws-cdk/aws-codepipeline:crossAccountKeyAliasStackSafeResourceName": true,
+    "@aws-cdk/aws-s3:createDefaultLoggingPolicy": true,
+    "@aws-cdk/aws-sns-subscriptions:restrictSqsDescryption": true,
+    "@aws-cdk/aws-apigateway:disableCloudWatchRole": true,
+    "@aws-cdk/core:enablePartitionLiterals": true,
+    "@aws-cdk/aws-events:eventsTargetQueueSameAccount": true,
+    "@aws-cdk/aws-iam:standardizedServicePrincipals": true,
+    "@aws-cdk/aws-ecs:disableExplicitDeploymentControllerForCircuitBreaker": true,
+    "@aws-cdk/aws-iam:importedRoleStackSafeDefaultPolicyName": true,
+    "@aws-cdk/aws-s3:serverAccessLogsUseBucketPolicy": true,
+    "@aws-cdk/aws-route53-patters:useCertificate": true,
+    "@aws-cdk/customresources:installLatestAwsSdkDefault": false,
+    "@aws-cdk/aws-rds:databaseProxyUniqueResourceName": true,
+    "@aws-cdk/aws-codedeploy:removeAlarmsFromDeploymentGroup": true,
+    "@aws-cdk/aws-apigateway:authorizerChangeDeploymentLogicalId": true,
+    "@aws-cdk/aws-ec2:launchTemplateDefaultUserData": true,
+    "@aws-cdk/aws-secretsmanager:useAttachedSecretResourcePolicyForSecretTargetAttachments": true,
+    "@aws-cdk/aws-redshift:columnId": true,
+    "@aws-cdk/aws-stepfunctions-tasks:enableEmrServicePolicyV2": true,
+    "@aws-cdk/aws-ec2:restrictDefaultSecurityGroup": true,
+    "@aws-cdk/aws-apigateway:requestValidatorUniqueId": true
+  }
+}

opensearch-serverless/cdk.json

@@ -0,0 +1,3 @@
+from .collection_pipeline_role import OpsCollectionPipelineRoleStack
+from .opensearch_serverless_ts import OpsServerlessTimeSeriesStack
+from .opensearch_serverless_ingestion import OpsServerlessIngestionStack

opensearch-serverless/cdk_stacks/__init__.py

@@ -0,0 +1,35 @@
+import aws_cdk as cdk
+
+from aws_cdk import (
+  Stack,
+  aws_iam
+)
+from constructs import Construct
+
+
+class OpsCollectionPipelineRoleStack(Stack):
+
+  def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
+    super().__init__(scope, construct_id, **kwargs)
+
+    collection_pipeline_policy_doc = aws_iam.PolicyDocument()
+
+    collection_pipeline_policy_doc.add_statements(aws_iam.PolicyStatement(**{
+      "effect": aws_iam.Effect.ALLOW,
+      "resources": ["*"],
+      "actions": [
+        "aoss:BatchGetCollection"
+      ]
+    }))
+
+    pipeline_role = aws_iam.Role(self, 'OpenSearchIngestionPipelineRole',
+      role_name='OpenSearchCollectionPipelineRole',
+      assumed_by=aws_iam.ServicePrincipal('osis-pipelines.amazonaws.com'),
+      inline_policies={
+        'collection-pipeline-policy': collection_pipeline_policy_doc
+      }
+    )
+    self.iam_role = pipeline_role
+
+    cdk.CfnOutput(self, f'{self.stack_name}_Role', value=self.iam_role.role_name)
+    cdk.CfnOutput(self, f'{self.stack_name}_RoleArn', value=self.iam_role.role_arn)