Skip to content

Commit 3b0771b

Browse files
wierdvanderhaaramotl
wierdvanderhaar
authored and
amotl
committed
Scale: Add tutorial "Autoscale a CrateDB Cloud Cluster"
1 parent a2d7842 commit 3b0771b

File tree

1 file changed

+220
-1
lines changed

1 file changed

+220
-1
lines changed

‎docs/admin/clustering/scale/auto.md‎

Lines changed: 220 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,226 @@
22

33
# Autoscale a CrateDB Cloud Cluster
44

5-
CrateDB’s fully managed solution CrateDB Cloud comes with a REST API,
5+
_How to scale your CrateDB Cloud cluster using the REST API._
6+
7+
CrateDB’s fully managed solution [CrateDB Cloud] comes with a REST API,
68
which enables you to automate a lot of tasks. The tutorial demonstrates
79
how to use the REST API to scale your CrateDB Cloud Cluster based on a
810
threshold on the number of shards in a cluster.
11+
12+
> Goal: The goal of the demo is to show how easy it is to scale out a
13+
> CrateDB Cloud Cluster, using a simple Python script.
14+
15+
## Introduction
16+
One of the unique features of CrateDB is that it is a multi-model distributed SQL database with a shared-nothing architecture. This makes it the perfect database for those environments where scalability is needed.
17+
18+
CrateDB's fully managed solution comes with a REST API which enables you to automate a lot of tasks. As CrateDB is world famous for its, highly scalable architecture, one of those tasks could be to scale a cluster out (horizontally) when needed, during a peak, for example, and back when possible to reduce costs.
19+
20+
![CrateDB-Scaling-Process|690x389](https://global.discourse-cdn.com/flex020/uploads/crate/original/2X/a/a10f95850f05a064ce022b4ccca45131c1c481cd.png){w=690px align=center}
21+
22+
This small blog shows how you can use the API to scale out and scale back a cluster when needed.
23+
24+
25+
## Prerequisites
26+
* A running CrateDB Cloud Cluster. If you don't have one yet, please follow this link to get [started][]. Note: To be able to scale a cluster you need to have a dedicated-tier cluster.
27+
* Admin UI access to the cluster.
28+
* An active API Key/Secret combination. See [API-KEY][] for more info.
29+
Note: These keys are cloud-dependent. So, if you have multiple clusters across different cloud vendors, you need to generate different keys.
30+
* You will need both the organization ID and the cluster ID to be able to execute the correct API calls.
31+
* A Python runtime environment.
32+
33+
[started]: https://cratedb.com/lp-crfree
34+
[API-KEY]: https://cratedb.com/blog/introducing-api-tokens-for-cratedb-cloud
35+
36+
37+
## Manual Scaling
38+
Let's do things step by step first to give you a "feel" of what is possible.
39+
40+
Open a terminal and execute the following:
41+
42+
```bash
43+
export APIKEY='USE YOUR API_KEY'
44+
export APISECRET='USE YOUR API_SECRET'
45+
export auth_data=${APIKEY}:${APISECRET}
46+
```
47+
48+
Now that the authorization is defined, we can start calling APIs. For example:
49+
50+
```bash
51+
curl -s -u $auth_data \
52+
https://console.cratedb.cloud/api/v2/organizations/ | jq
53+
```
54+
55+
Output:
56+
```json
57+
[
58+
{
59+
"dc": {
60+
"created": "2023-08-24T13:53:34.691000+00:00",
61+
"modified": "2023-10-05T09:30:54.109000+00:00"
62+
},
63+
"email": "wierd@nonofyourbusiness.org",
64+
"id": "xxxxx-8055-4c48-b332-d06d583c1999",
65+
"name": "My Organization",
66+
"notifications_enabled": true,
67+
"plan_type": 3,
68+
"project_count": 1,
69+
"role_fqn": "org_admin"
70+
}
71+
]
72+
```
73+
74+
Or if you want to know the current amount of nodes in your cluster. Here, you need to use the organization ID that was projected in the previous API call. For the next steps to work fluently, we can populate some variables.
75+
76+
```bash
77+
orgid=$(curl -s -u $auth_data https://console.cratedb.cloud/api/v2/organizations/ | jq -r '.[0].id')
78+
echo ${orgid}
79+
```
80+
81+
If you have multiple clusters running you need to filter using that cluster_name to get the clusterid.
82+
83+
```bash
84+
# Replace these values with your authentication data and the name of the cluster you want to select
85+
cluster_name="autoscaling-demo"
86+
87+
# Use jq to filter the output based on the cluster name and extract its ID
88+
clusterid=$(curl -s -u $auth_data https://console.cratedb.cloud/api/v2/clusters/ | jq -r '.[] | select(.name == '\"$cluster_name\"') | .id')
89+
echo ${clusterid}
90+
```
91+
92+
Get the number of nodes
93+
```bash
94+
numnodes=$(curl -s -u $auth_data https://console.cratedb.cloud/api/v2/clusters/${clusterid}/ | jq -r '.num_nodes')
95+
echo ${numnodes}
96+
```
97+
Output
98+
```json
99+
3
100+
```
101+
So, if you want to scale out, you can call the scale API. Note that the `clusterid` is needed.
102+
103+
```bash
104+
curl -u $auth_data -X PUT "https://console.cratedb.cloud/api/v2/clusters/${clusterid}/scale/" -H "accept: application/json" -H "Content-Type: application/json" -d "{\"product_unit\":2}"
105+
```
106+
107+
Extra info:
108+
```product_unit``` is the number of nodes minus 1, as 1 is considered the minimum number. In case you want to increase a 3 node cluster you need to use 3 as the value for product_unit.
109+
110+
```bash
111+
jobid=$(curl -u $auth_data -X PUT "https://console.cratedb.cloud/api/v2/clusters/${clusterid}/scale/" -H "accept: application/json" -H "Content-Type: application/json" -d "{\"product_unit\":$numnodes}" | jq -r '.last_async_operation.id')
112+
```
113+
The job can be used to query the operations API to get the info on the status.
114+
115+
```bash
116+
# check status:
117+
curl -s -u $auth_data https://console.cratedb.cloud/api/v2/clusters/${clusterid}/operations/ | jq --arg jobid "$jobid" '.operations[] | select(.id == $jobid)'
118+
```
119+
The `status` should change from `IN_PROGRESS` to `SUCCEEDED`:
120+
```json
121+
{
122+
"dc": {
123+
"created": "2024-02-23T13:04:22.960000+00:00",
124+
"modified": "2024-02-23T13:06:52.157000+00:00"
125+
},
126+
"entity": "CLUSTER",
127+
"entity_id": "xxxxx-b44e-446f-adca-a092fd1df0aa",
128+
"feedback_data": {
129+
"message": "Successfully scaled cluster to 4 nodes."
130+
},
131+
"id": "xxxxx-3d33-4c4a-8c9c-93202931ed1c",
132+
"non_sensitive_data": {
133+
"current_num_nodes": 3,
134+
"current_product_unit": 2,
135+
"target_num_nodes": 4,
136+
"target_product_unit": 3
137+
},
138+
"status": "SUCCEEDED",
139+
"type": "SCALE"
140+
}
141+
```
142+
143+
144+
## Automatic Scaling
145+
146+
We created a Python script that can be used as a starting point to enable autoscaling (out and back). In this script, we check the number of shards in a cluster, and when this crosses a threshold, the cluster will be scaled out. When the threshold goes a certain number, the cluster will be scaled back.
147+
148+
Download the [autoscaling script from GitHub] and run it as indicated in the README.
149+
150+
You can control the autoscale behavior by defining the ``MAX_NUM_SHARDS`` and by tweaking the number of shards in the cluster. For example, by creating an extra table with x number of shards to see the autoscale in action. In the script, the ``MAX_NUM_SHARDS`` is set to 30, but this can be changed to a number of your liking. It is also good to understand that the script will trigger a scale-out when the number of shards is higher than 80% of the ``MAX_NUM_SHARDS``. Assuming the default number of 30, this will mean that when the number of shards exceeds 24, the autoscale will kick in.
151+
152+
In this example, I created a 3-node CrateDB Cloud cluster and created this table:
153+
154+
``` sql
155+
CREATE TABLE ta (
156+
"keyword" TEXT INDEX using fulltext
157+
,"ts" TIMESTAMP
158+
,"day" TIMESTAMP GENERATED ALWAYS AS date_trunc('day', ts)
159+
)
160+
CLUSTERED INTO 24 SHARDS;
161+
```
162+
163+
This will create 8 primary shards per node plus 8 replicas. This can be checked by looking at the number of shards. This can be done for example using the console by running this:
164+
165+
```sql
166+
select node ['name'], count(*)
167+
from sys.shards
168+
group by node ['name']
169+
order by 1
170+
limit 100;
171+
```
172+
173+
In this example, the amount of shards is 16 per node.
174+
175+
```
176+
node['name'] count(*)
177+
data-hot-0 16
178+
data-hot-1 16
179+
data-hot-2 16
180+
```
181+
182+
Run the Python script, assuming you filled in the ```API_KEY``` ```API_SECRET``` ```organization_id``` and ```cluster_id```.
183+
184+
To trigger the scale-out you can add a table. For example:
185+
186+
```sql
187+
CREATE TABLE tb (
188+
"keyword" TEXT INDEX using fulltext
189+
,"ts" TIMESTAMP
190+
,"day" TIMESTAMP GENERATED ALWAYS AS date_trunc('day', ts)
191+
)
192+
CLUSTERED INTO 18 SHARDS;
193+
```
194+
195+
As you can see in the output of the running Python script this change is triggering a scale-out:
196+
```text
197+
Current avg number of shards: 16.8
198+
Nothing to do!
199+
Current avg number of shards: 28.0
200+
Start scaling out from 3 to 4
201+
Scaling in progress...................
202+
Scaled up successfully!
203+
Current avg number of shards: 21.0
204+
Nothing to do!
205+
```
206+
207+
The scaling of a cluster takes time depending on the amount of data that needs to be rebalanced. In this example, we don't really have any data, so that will be pretty quick (+/- 3mins).
208+
209+
When you delete the tb table, you will see that the autoscale script will scale the cluster back to 3 nodes.
210+
```text
211+
Current avg number of shards: 21.0
212+
Nothing to do!
213+
Current avg number of shards: 12.0
214+
Start scaling down from 4 to 3
215+
Scaling in progress...................
216+
Scaled down successfully!
217+
Current avg number of shards: 16.0
218+
```
219+
220+
221+
## Conclusion
222+
223+
With this simple example, I showed how easy it is to scale out a CrateDB cluster either through manual or automated API calls.
224+
225+
226+
[autoscaling script from GitHub]: https://github.com/crate/cratedb-examples/tree/main/topic/autoscaling
227+
[CrateDB Cloud]: https://cratedb.com/product/cloud

0 commit comments

Comments
 (0)