Skip to content

Commit f2ebfbc

Browse files
asuncepedaasuncepeda
authored
Merge pull request #358 from tinybirdco/12700-snowflake-cases
snowflake cases
2 parents 11c36bd + 24c6b00 commit f2ebfbc

File tree

8 files changed

+141
-2
lines changed

8 files changed

+141
-2
lines changed

‎.github/workflows/iterate_snowflake_cd.yml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
##################################################
2+
### Visit https://github.com/tinybirdco/ci ###
3+
### for more details or custom CI/CD ###
4+
##################################################
5+
6+
name: Tinybird - CD Workflow
7+
8+
on:
9+
workflow_dispatch:
10+
push:
11+
paths:
12+
- 'iterate_snowflake/**'
13+
branches:
14+
- main
15+
jobs:
16+
cd:
17+
uses: tinybirdco/ci/.github/workflows/cd.yml@v4.1.0
18+
with:
19+
data_project_dir: ./iterate_snowflake
20+
secrets:
21+
tb_admin_token: ${{ secrets.TB_ADMIN_TOKEN_ADD_COLUMN_SNOWFLAKE_DS }}
22+
tb_host: https://api.tinybird.co

‎.github/workflows/iterate_snowflake_ci.yml‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
##################################################
2+
### Visit https://github.com/tinybirdco/ci ###
3+
### for more details or custom CI/CD ###
4+
##################################################
5+
6+
name: Tinybird - CI Workflow
7+
8+
on:
9+
workflow_dispatch:
10+
pull_request:
11+
paths:
12+
- 'iterate_snowflake/**'
13+
branches:
14+
- main
15+
types: [opened, reopened, labeled, unlabeled, synchronize, closed]
16+
17+
concurrency: ${{ github.workflow }}-${{ github.event.pull_request.number }}
18+
19+
jobs:
20+
ci:
21+
uses: tinybirdco/ci/.github/workflows/ci.yml@v4.1.0
22+
with:
23+
data_project_dir: ./iterate_snowflake
24+
secrets:
25+
tb_admin_token: ${{ secrets.TB_ADMIN_TOKEN_ADD_COLUMN_SNOWFLAKE_DS }}
26+
tb_host: https://api.tinybird.co

‎DEVELOPMENT/README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
1. Create a new workspace in Tinybird using the UI and choose to use the `Web Analytics` Template. This will generate a new Workspace and push all the resources from the Web Analytics template
44
2. Create a repository in Github (private or public)
5-
3. Connect the workspace with the repository using the UI. Follow the steps and push all the changes to the repo
5+
3. Connect the workspace with the repository using the [documentation about versions](https://www.tinybird.co/docs/production/working-with-version-control#connect-your-workspace-to-git-from-the-cli)
66
4. At this point, you should have a workspace connected to Git and using the release 0.0.0
77
5. (Optional) You can use `mockingbird` to start fake data to our workspace. Just go to https://mockingbird.tinybird.co/ and follow the steps to start sending data to your workspace
8-
6. (Optional) You can run the script `utils/query_api.sh` to generate fake requests to the endpoints of the workspace
8+
6. (Optional) You can run the script `utils/query_api.sh` to generate fake requests to the endpoints of the workspace

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ This repository contains all the use cases you can iterate with Versions:
2323
- [Create a Materialized View with batch ingest](create_a_materialized_view_batch_ingest)
2424
- [Delete simple resource](delete_simple_resource)
2525
- [Iterating an API Endpoint](iterating_api_endpoint)
26+
- [Iterating a Snowflake Data Source](iterate_snowflake)
2627
- [Recover data from quarantine](recover_data_from_quarantine)
2728
- [Remove column from landing Data Source](remove_column_landing_data_source)
2829
- [Use new columns from a Shared Data Source](use_new_columns_from_shared_datasource)

‎iterate_snowflake/.tinyenv‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# VERSION format is major.minor.patch-post where major, minor, patch and post are integer numbers
2+
# bump post to deploy to the current live Release, rollback to previous post version is not available
3+
# bump patch or minor to deploy a new Release and auto-promote it to live. Add TB_AUTO_PROMOTE=0 to create the Release in preview status
4+
# bump major to deploy a new Release in preview status
5+
VERSION=0.0.0
6+
7+
8+
9+
##########
10+
# OPTIONAL env vars
11+
# Deploy a new Release in preview status (default is 1)
12+
# TB_AUTO_PROMOTE=0
13+
14+
# Check if deploy requires backfilling on preview (default is 1)
15+
# TB_CHECK_BACKFILL_REQUIRED=0
16+
17+
# Force old Releases deletion on promote (default is 0)
18+
# Setting it to 1 will remove oldest rollback Releases even when some resource is still in use
19+
# TB_FORCE_REMOVE_OLDEST_ROLLBACK=0
20+
21+
# Don't print CLI version warning message if there's a new available version
22+
# TB_VERSION_WARNING=0
23+
24+
# Skip regression tests
25+
# TB_SKIP_REGRESSION=0
26+
27+
# Use `OBFUSCATE_REGEX_PATTERN` and `OBFUSCATE_PATTERN_SEPARATOR` environment variables to define a regex pattern and a separator (in case of a single string with multiple regex) to obfuscate secrets in the CLI output.
28+
# OBFUSCATE_REGEX_PATTERN="https://(www\.)?[^/]+||^Follow these instructions =>"
29+
# OBFUSCATE_PATTERN_SEPARATOR=||
30+
##########

‎iterate_snowflake/README.md‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
# Use cases for iteration of Snowflake Data Sources
2+
3+
First of all, you must connect your Tinybird workspace with your Git repository using the [documentation about versions](https://www.tinybird.co/docs/production/working-with-version-control#connect-your-workspace-to-git-from-the-cli)
4+
5+
### Create a Snowflake Data Source
6+
7+
- Create the Snowflake Connection in your main tinybird workspace using Tinybird CLI, as explain in the [iteration documentation](https://www.tinybird.co/docs/ingest/snowflake). Snowflake connections reside always in the main branch.
8+
- Create a new git branch
9+
- Create a new Snowflake .datasource data file in the branch, as in the example PR. You shouldn't need fixtures as the Data Source will be populated from the connection, but you can add fixtures and/or tests if needed.
10+
- For information about Snowflake options read the [connector documentation](https://www.tinybird.co/docs/ingest/snowflake)
11+
- Commit your code and create a new PR/MR from the branch to `main`
12+
- Wait for CI. You can check in the temporary workspace branch (automatically created and named like `tmp_ci_*`) that everything works as expected.
13+
- Merge and wait for CD. Now you should have your Data Source in the main workspace, using the connection, and the temporary CI branch will be deleted.
14+
15+
[Example Pull Request](https://github.com/tinybirdco/use-case-examples/pull/tbd)
16+
17+
18+
### Update Snowflake Data Source with a new column, with same connection
19+
20+
- Create another git branch
21+
- Add the new column to the schema and query
22+
- Commit your code and create an ew PR from the branch to `main`
23+
- Wait for CI to finish. You can check in the temporary workspace branch (automatically created and named like `tmp_ci_*`) that everything works as expected.
24+
- Merge and wait for CD. Now you should have your Data Source updated in the main workspace, using the connection, and the temporary CI branch will be deleted.
25+
26+
[Example Pull Request](https://github.com/tinybirdco/use-case-examples/pull/tbd)
27+
28+
### Delete a Snowflake Data Source
29+
30+
- Create another git branch
31+
- Remove the .datasource data file
32+
- Commit your code and create an ew PR from the branch to `main`
33+
- Wait for CI to finish. You can check in the temporary workspace branch (automatically created and named like `tmp_ci_*`) that everything works as expected.
34+
- Merge and wait for CD. Now you should have your Data Source updated in the main workspace, using the connection, and the temporary CI branch will be deleted.
35+

‎iterate_snowflake/requirements.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
tinybird-cli==5.7.0

‎iterate_snowflake/scripts/exec_test.sh‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
2+
#!/usr/bin/env bash
3+
4+
fail=0;
5+
6+
# We set this environment variable to avoid the CLI to show the warning about a new version
7+
export TB_VERSION_WARNING=0
8+
9+
for t in `find ./tests -name "*.test"`; do
10+
echo "** Running $t **"
11+
echo "** $(cat $t)"
12+
if res=$(bash $t $1 | diff -B ${t}.result -); then
13+
echo 'OK';
14+
else
15+
echo "failed, diff:";
16+
echo "$res";
17+
fail=1
18+
fi
19+
echo ""
20+
done;
21+
22+
if [ $fail == 1 ]; then
23+
exit -1;
24+
fi

0 commit comments

Comments
 (0)