Skip to content

Commit 717e1fa

Browse files
jrdijrdi
authored
Tinybird Forward Deployment (#379)
* Initial commit * Fix port * Try to deploy the diff only when needed * Fix grep return code * Deploy to the cloud * Change grep pattern * Print CLI version * Add debug to tb deploy * Add copy for the initial populate * Make populate pipe a copy * Add MV
1 parent f6c3453 commit 717e1fa

File tree

7 files changed

+129
-0
lines changed

7 files changed

+129
-0
lines changed

‎.github/workflows/forward_deployment.yml‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
name: Tinybird - Forward Deployment demo
2+
3+
on:
4+
workflow_dispatch:
5+
schedule:
6+
- cron: "0 * * * *"
7+
pull_request:
8+
branches:
9+
- main
10+
- master
11+
types: [opened, reopened, labeled, unlabeled, synchronize]
12+
13+
concurrency: ${{ github.workflow }}-${{ github.event.pull_request.number }}
14+
15+
env:
16+
TINYBIRD_FWD_HOST: ${{ secrets.TINYBIRD_FWD_HOST }}
17+
TINYBIRD_FWD_TOKEN: ${{ secrets.TINYBIRD_FWD_TOKEN }}
18+
19+
jobs:
20+
forward_deployment:
21+
runs-on: ubuntu-latest
22+
defaults:
23+
run:
24+
working-directory: './forward_deployment'
25+
services:
26+
tinybird:
27+
image: tinybirdco/tinybird-local:beta
28+
ports:
29+
- 7181:7181
30+
steps:
31+
- uses: actions/checkout@v3
32+
- name: Install Tinybird CLI
33+
run: curl -LsSf https://tbrd.co/fwd | sh
34+
- name: Tinybird CLI version
35+
run: tb --version
36+
- name: Get datasource definition
37+
run: |
38+
DATASOURCE_SORTING_KEY=$(curl \
39+
-H "Authorization: Bearer $TINYBIRD_FWD_TOKEN" \
40+
-X GET "$TINYBIRD_FWD_HOST/v0/datasources/users.datasource" | grep 'ENGINE_SORTING_KEY "tuple()"' || true)
41+
42+
echo "DATASOURCE_SORTING_KEY=$DATASOURCE_SORTING_KEY" >> $GITHUB_ENV
43+
- name: Apply patch
44+
run: |
45+
if [ -n "$DATASOURCE_SORTING_KEY" ]; then
46+
cp -r .diff/* . && git diff
47+
else
48+
echo "No changes to apply"
49+
fi
50+
- name: Build project
51+
run: tb build
52+
- name: Deploy project
53+
run: tb --host $TINYBIRD_FWD_HOST --token $TINYBIRD_FWD_TOKEN --cloud deployment create --auto --wait
54+
- name: Populate datasource if empty
55+
run: |
56+
COUNT=$(tb --host $TINYBIRD_FWD_HOST --token $TINYBIRD_FWD_TOKEN --cloud sql "SELECT count() AS c FROM users" --format json | grep '"c":' | awk '{print $2}')
57+
if [ "$COUNT" -eq 0 ]; then
58+
tb --host $TINYBIRD_FWD_HOST --token $TINYBIRD_FWD_TOKEN --cloud copy run populate --wait
59+
else
60+
echo "Datasource is not empty, skipping population"
61+
fi
62+
- name: Check users datasource
63+
run: |
64+
COUNT=$(tb --host $TINYBIRD_FWD_HOST --token $TINYBIRD_FWD_TOKEN --cloud sql "SELECT count() AS c FROM users" --format json | grep '"c":' | awk '{print $2}')
65+
if [ "$COUNT" -eq 1000000000 ]; then
66+
echo "Datasource data is correct"
67+
else
68+
echo "Datasource data is incorrect"
69+
exit 1
70+
fi
71+
- name: Check user_stats datasource
72+
run: |
73+
COUNT=$(tb --host $TINYBIRD_FWD_HOST --token $TINYBIRD_FWD_TOKEN --cloud sql "SELECT count() AS c FROM user_stats" --format json | grep '"c":' | awk '{print $2}')
74+
if [ "$COUNT" -gt 0 ]; then
75+
echo "Datasource data is correct"
76+
else
77+
echo "Datasource data is incorrect"
78+
exit 1
79+
fi
80+

‎forward_deployment/.diff/datasources/users.datasource‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
SCHEMA >
2+
3+
id UInt64 `json:$.user_id`,
4+
name String `json:$.name`,
5+
category LowCardinality(String) `json:$.category`,
6+
created_at DateTime `json:$.created_at` DEFAULT now()
7+
8+
ENGINE MergeTree
9+
ENGINE_SORTING_KEY "id"

‎forward_deployment/.gitkeep‎

Whitespace-only changes.

‎forward_deployment/copies/populate.pipe‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
NODE populate_users
2+
SQL >
3+
SELECT id % 100000 AS id, name, arrayElement(['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j'], (category_id % 10) + 1) AS category, created_at
4+
FROM generateRandom('id UInt64, name String, category_id UInt8, created_at DateTime')
5+
LIMIT 1000000000
6+
7+
TYPE copy
8+
9+
TARGET_DATASOURCE users
10+
COPY_SCHEDULE @on-demand
11+
COPY_MODE replace

‎forward_deployment/datasources/user_stats.datasource‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
SCHEMA >
2+
3+
id UInt64,
4+
categories AggregateFunction(uniq, String),
5+
count AggregateFunction(count, UInt64)
6+
7+
ENGINE AggregatingMergeTree
8+
ENGINE_SORTING_KEY id

‎forward_deployment/datasources/users.datasource‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
SCHEMA >
2+
3+
id UInt64 `json:$.user_id`,
4+
name String `json:$.name`,
5+
category LowCardinality(String) `json:$.category`,
6+
created_at DateTime `json:$.created_at` DEFAULT now()
7+
8+
ENGINE MergeTree
9+
ENGINE_SORTING_KEY ""

‎forward_deployment/materializations/mv_1.pipe‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
NODE mv_user_stats
2+
SQL >
3+
4+
SELECT
5+
id,
6+
uniqState(category) AS categories,
7+
countState() AS count
8+
FROM users
9+
GROUP BY id
10+
11+
TYPE materialized
12+
DATASOURCE user_stats

0 commit comments

Comments
 (0)