Skip to content

Commit 03f1ba4

Browse files
asuncepedaasuncepeda
authored
add readme and base structure (#363)
* add readme and base structure * update with PR ids
1 parent c7eb993 commit 03f1ba4

File tree

8 files changed

+149
-0
lines changed

8 files changed

+149
-0
lines changed

‎.github/workflows/iterate_dynamodb_cd.yml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
##################################################
2+
### Visit https://github.com/tinybirdco/ci ###
3+
### for more details or custom CI/CD ###
4+
##################################################
5+
6+
name: Tinybird - CD Workflow
7+
8+
on:
9+
workflow_dispatch:
10+
push:
11+
paths:
12+
- 'iterate_dynamodb/**'
13+
branches:
14+
- main
15+
jobs:
16+
cd:
17+
uses: tinybirdco/ci/.github/workflows/cd.yml@v4.1.0
18+
with:
19+
data_project_dir: ./iterate_dynamodb
20+
secrets:
21+
tb_admin_token: ${{ secrets.TB_ADMIN_TOKEN_ITERATIONS }}
22+
tb_host: https://api.us-east.aws.tinybird.co

‎.github/workflows/iterate_dynamodb_ci.yml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
##################################################
2+
### Visit https://github.com/tinybirdco/ci ###
3+
### for more details or custom CI/CD ###
4+
##################################################
5+
6+
name: Tinybird - CI Workflow
7+
8+
on:
9+
workflow_dispatch:
10+
pull_request:
11+
paths:
12+
- 'iterate_dynamodb/**'
13+
branches:
14+
- main
15+
- '*-dynamo-*'
16+
types: [opened, reopened, labeled, unlabeled, synchronize, closed]
17+
18+
concurrency: ${{ github.workflow }}-${{ github.event.pull_request.number }}
19+
20+
jobs:
21+
ci:
22+
uses: tinybirdco/ci/.github/workflows/ci.yml@v4.1.0
23+
with:
24+
data_project_dir: ./iterate_dynamodb
25+
secrets:
26+
tb_admin_token: ${{ secrets.TB_ADMIN_TOKEN_ITERATIONS }}
27+
tb_host: https://api.us-east.aws.tinybird.co

‎iterate_dynamodb/.tinyenv‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# VERSION format is major.minor.patch-post where major, minor, patch and post are integer numbers
2+
# bump post to deploy to the current live Release, rollback to previous post version is not available
3+
# bump patch or minor to deploy a new Release and auto-promote it to live. Add TB_AUTO_PROMOTE=0 to create the Release in preview status
4+
# bump major to deploy a new Release in preview status
5+
VERSION=0.0.0
6+
7+
8+
9+
##########
10+
# OPTIONAL env vars
11+
# Deploy a new Release in preview status (default is 1)
12+
# TB_AUTO_PROMOTE=0
13+
14+
# Check if deploy requires backfilling on preview (default is 1)
15+
# TB_CHECK_BACKFILL_REQUIRED=0
16+
17+
# Force old Releases deletion on promote (default is 0)
18+
# Setting it to 1 will remove oldest rollback Releases even when some resource is still in use
19+
# TB_FORCE_REMOVE_OLDEST_ROLLBACK=0
20+
21+
# Don't print CLI version warning message if there's a new available version
22+
# TB_VERSION_WARNING=0
23+
24+
# Skip regression tests
25+
# TB_SKIP_REGRESSION=0
26+
27+
# Use `OBFUSCATE_REGEX_PATTERN` and `OBFUSCATE_PATTERN_SEPARATOR` environment variables to define a regex pattern and a separator (in case of a single string with multiple regex) to obfuscate secrets in the CLI output.
28+
# OBFUSCATE_REGEX_PATTERN="https://(www\.)?[^/]+||^Follow these instructions =>"
29+
# OBFUSCATE_PATTERN_SEPARATOR=||
30+
##########

‎iterate_dynamodb/README.md‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# Use cases for iteration of DynamoDB Data Sources
2+
3+
First of all, you must connect your Tinybird Workspace with your Git repository using the [documentation about Versions](https://www.tinybird.co/docs/production/working-with-version-control#connect-your-workspace-to-git-from-the-cli)
4+
5+
### Create a DynamoDB Data Source
6+
7+
- Create the DynamoDB Connection in your main Tinybird Workspace using Tinybird CLI, as explain in the [iteration documentation](https://www.tinybird.co/docs/ingest/dynamodb). Connections reside always in the main Branch.
8+
- Create a new git branch
9+
- Create a new DynamoDB .datasource data file in the branch, as in the example PR. You shouldn't need fixtures as the Data Source will be populated from the Connection, but you can add fixtures and/or tests if needed.
10+
- For information about DynamoDB options read the [connector documentation](https://www.tinybird.co/docs/ingest/dynamodb)
11+
- Commit your code and create a new PR/MR from the branch to `main`
12+
- Wait for CI to finish.
13+
- Check that everything is working as expected in the temporary Tinybird Branch, CI will automatically create it and name it like `tmp_ci_{PRid}`).
14+
- You can travel to this Branch through UI and check that the Data Source is working
15+
- You'll need wait for the initial synchronization job to end, check the job status in the dashboard and the logs in the Data Source `log` tab
16+
- You can add/update/delete items in your DynamoDB table and check that the changes reproduce in the Data Source.
17+
- If everything is fine, merge and wait for CD. Now you should have your Data Source in the main Workspace, using the Connection, and the temporary CI Branch will be deleted.
18+
19+
[Example Pull Request](https://github.com/tinybirdco/use-case-examples/pull/364)
20+
21+
22+
### Update DynamoDB Data Source with a new column, with same Connection
23+
24+
- Create another git branch
25+
- Add the new column to the schema and query
26+
- Commit your code and create a new PR from the branch to `main`
27+
- Wait for CI to finish.
28+
- Check that everything is working as expected in the temporary branch, CI will automatically create it and name it like `tmp_ci_{PRid}`).
29+
- You can travel to the Branch through UI and check that the Data Source has the changes applied
30+
- You can add/update/delete items in your DynamoDB table and check that the changes reproduce in the Data Source.
31+
- Merge and wait for CD. Now you should have your Data Source updated in the main Workspace, using the Connection, and the temporary CI Branch will be deleted.
32+
33+
[Example Pull Request](https://github.com/tinybirdco/use-case-examples/pull/365)
34+
35+
### Delete a DynamoDB Data Source
36+
37+
- Create another git branch
38+
- Remove the .datasource data file
39+
- If the Data Source has dependencies they must be deleted too, to the workflow to work
40+
- Commit your code and create a new PR from the branch to `main`
41+
- Wait for CI to finish.
42+
- Check that everything is working as expected in the temporary Branch, CI will automatically create it and name it like `tmp_ci_{PRid}`).
43+
- Merge and wait for CD. Now you should have your Data Source deleted in the main Workspace, using the Connection, and the temporary CI Branch will be deleted.
44+
- If the Connection isn't being used by any other Data Sources, you can remove it directly using CLI.
45+

‎iterate_dynamodb/datasources/fixtures/.gitkeep‎

Whitespace-only changes.

‎iterate_dynamodb/requirements.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
tinybird-cli==5.7.0

‎iterate_dynamodb/scripts/exec_test.sh‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
2+
#!/usr/bin/env bash
3+
4+
fail=0;
5+
6+
# We set this environment variable to avoid the CLI to show the warning about a new version
7+
export TB_VERSION_WARNING=0
8+
9+
for t in `find ./tests -name "*.test"`; do
10+
echo "** Running $t **"
11+
echo "** $(cat $t)"
12+
if res=$(bash $t $1 | diff -B ${t}.result -); then
13+
echo 'OK';
14+
else
15+
echo "failed, diff:";
16+
echo "$res";
17+
fail=1
18+
fi
19+
echo ""
20+
done;
21+
22+
if [ $fail == 1 ]; then
23+
exit -1;
24+
fi

‎iterate_dynamodb/tests/.gitkeep‎

Whitespace-only changes.

0 commit comments

Comments
 (0)