Skip to content

Commit 4ffcecb

Browse files
skeet70skeet70
authored
Add a deterministic recovery example (#104)
* Add a deterministic recovery example It uses the standard recovery example to recover its deterministic secret, building on that with the steps required to recover the deterministic data. * Spelling/grammar pass * Add blurb about running examples
1 parent 6e742f8 commit 4ffcecb

File tree

6 files changed

+250
-56
lines changed

6 files changed

+250
-56
lines changed

‎examples/disaster-recovery-example/README.md‎

Lines changed: 92 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,12 +17,14 @@ situations. 'Cause archeologists need love too!
1717

1818
This example uses previously encrypted data from the other examples. When run, it deconstructs that data and reconstitutes the original decrypted data.
1919

20-
The credentials to make running this yourself aren't provided, but you can see the code and example output for more information about what is happening.
20+
The credentials to run this yourself aren't provided, but you can see the code and example output for more information about what is happening.
21+
22+
Information on IronCore document header shapes and protobuf formats can be found in the open-source [ironcore-documents](https://github.com/IronCoreLabs/ironcore-documents) repository.
2123

2224
## Recovery Process
2325

2426
### Retrieving the Encrypted Bytes, GCM tag, and IV
25-
In the `EncryptedDocumentMap` (`Map<String, byte[]>`) returned by the TSC, the `byte[]`’s are of the structure:
27+
In the `EncryptedDocumentMap` (`Map<String, byte[]>`) returned by the TSC, the `byte[]`s in v3 documents are of the structure:
2628

2729
```
2830
VERSION_NUMBER (1 byte, fixed at 3)
@@ -36,6 +38,8 @@ GCM_TAG (16 bytes)
3638

3739
`ENCRYPTED_DATA` are the bytes you’ll be decrypting, and the `DATA_IV` is needed to make the AES decryption call. You’ll also need the encryption key, obtained in the next step.
3840

41+
See [ironcore-documents](https://github.com/IronCoreLabs/ironcore-documents) combined with the open-source code of the specific version of the TSC or IronCore Alloy that you're using to know what the document header structure of your data will be.
42+
3943
### Retrieving the Document Encryption Key
4044
The DEK (Document Encryption Key) you need to decrypt the `ENCRYPTED_DATA` is the decrypted value of the EDEK (Encrypted DEK) that was also returned by the TSC.
4145

@@ -56,6 +60,9 @@ message EncryptedDeks {repeated EncryptedDek encryptedDeks = 1;}
5660

5761
You need to decode the EDEK base64 as an `EncryptedDeks` message, then for disaster recovery you're safe to just use the first `EncryptedDek` in the resulting list of them.
5862

63+
See [ironcore-documents](https://github.com/IronCoreLabs/ironcore-documents) combined with the open-source code of the specific version of the TSC or IronCore Alloy that you're using for the most up to date information on what protobuf formats to expect.
64+
65+
5966
#### Un-leased
6067
If the `EncryptedDek.leasedKeyId` is `0` (zero) you can decrypt the DEK by calling unwrap on your (or the tenant’s) KMS, passing the `EncryptedDek.encryptedDekData` bytes, using the correct credentials and key path. The result will be the DEK which can then be used in the “Decrypting the Document” step.
6168

@@ -78,6 +85,9 @@ all known keypaths / creds and map out the associations.
7885

7986

8087
## Example Run Output
88+
89+
Code for this process is in `src/index.ts`. It can be run with `yarn && yarn start` if you have IronCore GCP credentials. This same code could be modified or expanded on to work for your real data or recovery process, but it is not itself production quality code.
90+
8191
```console
8292
Unleased Encrypted Document: {
8393
"ssn": "\u0003IRON\u0000,\n\u001c������\"_���k,�\bj@���S\u001f�b3g�M\u001a\f\n\ntenant-gcp\u001a \u0002 Zک�\u0001��q*�\n\u000eq�/\u0004\u00122(��؋@ʹ�F\u0007K\u0005���",
@@ -100,3 +110,83 @@ Leased Document: {
100110
"name": "Jim Bridger"
101111
}
102112
```
113+
114+
## Deterministic
115+
116+
As mentioned in the [documentation on our website](https://ironcorelabs.com/docs/saas-shield/deterministic-encryption/#recovering-data) the recovery process for deterministically encrypted data requires a few extra steps. Deterministic data is encrypted with a key derived from a rotating tenant secret, which itself is wrapped by their KMS. To recover deterministic data, we need to:
117+
118+
1. Determine the ID of the secret used for our data.
119+
2. Retrieve that previously backed-up encrypted secret.
120+
3. Decrypt the secret (using the method described in [Recovery Process](#recovery-process)).
121+
4. Use the secret to hash a string incorporating the derivation path and tenant ID.
122+
5. Use that string as a key to decrypt the data.
123+
124+
Let's go over those steps in more detail. A concrete Typescript example of this process can be found in `examples/disaster-recovery-example/src/deterministic.ts`.
125+
126+
### Determine Secret ID
127+
128+
We need to retrieve the ID of the secret that was used to encrypt our piece of data so the encrypted secret can be retrieved from backups.
129+
130+
For deterministic data encrypted with the Tenant Security Clients (`tsc-java 6+`, `tsc-nodejs 3+`, not supported in `tsc-php` or `tsc-go`), the first 4 binary bytes contain the tenant secret ID, and the next two bytes are padding.
131+
132+
For deterministic data encrypted with IronCore Alloy, the first 4 binary bytes contain the tenant secret ID. See the [ironcore-alloy](https://github.com/IronCoreLabs/ironcore-alloy/blob/main/src/deterministic.rs) and [ironcore-documents](https://github.com/IronCoreLabs/ironcore-documents) source code for the most up-to-date information.
133+
134+
### Retrieve Secret
135+
136+
Encrypted tenant secrets are available from the Configuration Broker and should be regularly backed up.
137+
138+
In the UI a `.zip` of encrypted tenant secrets can be downloaded from the [Tenant Secrets page](https://config.staging.ironcorelabs.com/app/kms/secrets) of the Configuration Broker. Look for a clickable download icon near the page title.
139+
140+
In the Vendor API Bridge, a request to the [List Tenant Secrets](https://ironcore-labs.stoplight.io/docs/vendor-bridge/8ee4d2ba0dd9c-list-tenant-secrets) endpoint will return a `JSON` list of encrypted tenant secrets.
141+
142+
These encrypted secrets need to be backed up in a way that they can be reliably retrieved by secret ID in a disaster scenario.
143+
144+
### Decrypt Secret
145+
146+
The secret is encrypted by the tenant's KMS, and the process described in [Recovery Process](#recovery-process) is used to decrypt it. Notably, the secret will include a KMS config ID in its header (see [ironcore-documents](https://github.com/IronCoreLabs/ironcore-documents) for header formats), but without access to the Configuration Broker it won't be clear which of the tenants KMS key paths and credentials were referenced by that KMS config. In a true disaster scenario, you'll need to work with the tenant to either provide them with these secrets and have them try decryption with all their KMS keys, or have them provide you or your recovery tool with credentials that can access all their possible KMS keys to try.
147+
148+
### Create a Deterministic Key
149+
150+
Once you have the tenant's decrypted secret for this piece of data, you can create a deterministic key. Use the secret as a key to `HMAC-SHA512` sign over `"tenant_provided_id-derivation_path"`. The output of the hash is the deterministic key used to decrypt the actual data.
151+
152+
### Decrypt the Data
153+
154+
Use the deterministic key to `AES-SIV` decrypt the deterministic data. There is no `AES-SIV` associated data on IronCore deterministic values.
155+
156+
### Example Run Output
157+
158+
Code for this process is in `src/deterministic.ts`. It can be run with `yarn && yarn deterministic` if you have IronCore GCP credentials. This same code could be modified or expanded on to work for your real data or recovery process, but it is not itself production quality code.
159+
160+
```console
161+
Deterministic Encrypted Value:
162+
AAAT9wAAeFKwSikcVVSXpGj8z2/7bYkDpEue9kUlUxOj
163+
Recovered Secret ID:
164+
5111
165+
Retrieved Backup Secret: {
166+
"tenantProvidedId": "tenant-gcp",
167+
"numericId": 5111,
168+
"id": "4a7f76dd-a0fb-4642-b15b-8423a983cc88",
169+
"tenantSecretId": 5111,
170+
"secretFingerprint": "tMoo5p0wpn3J2AS3iJptVWAS19B9zvj1nDGltf40wos=",
171+
"secretPath": "secretPath",
172+
"kmsConfigId": 510,
173+
"migrationStatus": 1,
174+
"encryptedSecret": "CnYKcQokAAWBd/OMpxQKZ7Fzm7WnD1vprk6t5a2Cx1OS5ZaTd3Qez6Q9EkkA8xjtHtVE4ZD8CMqa1bWVwiTggI/McQvx7+bvuytA9ztnivj08xOIIJy2zV9aU2w9RykpEXPk7pXZCLT3vMtt+5dho9m8igg6EP4D",
175+
"rotationStatus": 1,
176+
"secretType": 2,
177+
"created": 1689877493455,
178+
"updated": 1689877493455
179+
}
180+
Decrypting Backup Secret with Tenant's GCP KMS using provided credentials...
181+
Decrypted Backup Secret:
182+
ScbKs2h0XSaBQ/vXp04RmChAffLjh2nxBgRT95j21FM=
183+
Hashing Over Tenant String:
184+
tenant-gcp-derivPath
185+
Recovered Deterministic Key:
186+
+5RA+h4hWWs28cDBaJ0xf6VekSt+NyqJLnj9/GLugMBgwPhV/Qcefs21asNN+iN3h1H8MSjRlgA+V7CyPfmFYw==
187+
AES-SIV Decrypting Deterministic Data...
188+
Recovered Data:
189+
SmltIEJyaWRnZXI=
190+
Original Data (String):
191+
Jim Bridger
192+
```

‎examples/disaster-recovery-example/package.json‎

Lines changed: 18 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,20 @@
11
{
2-
"name": "tsc-node-example",
3-
"version": "1.0.0",
4-
"main": "src/index.js",
5-
"license": "MIT",
6-
"devDependencies": {
7-
"typescript": "^4.3.5"
8-
},
9-
"dependencies": {
10-
"@google-cloud/kms": "^2.6.0",
11-
"@types/node": "^16.6.0",
12-
"ts-proto": "^1.82.5"
13-
},
14-
"scripts": {
15-
"start": "yarn proto && yarn tsc --target ES6 --sourceMap false --module CommonJS --outDir ./dist/src src/index.ts && node dist/src/index.js",
16-
"proto": "protoc --plugin=./node_modules/.bin/protoc-gen-ts_proto --ts_proto_out=./src ./transform.proto"
17-
}
2+
"name": "tsc-node-example",
3+
"version": "1.0.0",
4+
"main": "src/index.js",
5+
"license": "MIT",
6+
"devDependencies": {
7+
"typescript": "^4.3.5"
8+
},
9+
"dependencies": {
10+
"@google-cloud/kms": "^2.6.0",
11+
"@types/node": "^16.6.0",
12+
"miscreant": "^0.3.2",
13+
"ts-proto": "^1.82.5"
14+
},
15+
"scripts": {
16+
"start": "yarn proto && yarn tsc --target ES6 --sourceMap false --module CommonJS --outDir ./dist/src src/index.ts && node dist/src/index.js",
17+
"deterministic": "yarn proto && yarn tsc --target ES6 --sourceMap false --module CommonJS --outDir ./dist/src src/deterministic.ts && node dist/src/deterministic.js",
18+
"proto": "protoc --plugin=./node_modules/.bin/protoc-gen-ts_proto --ts_proto_out=./src ./transform.proto"
19+
}
1820
}

‎examples/disaster-recovery-example/src/deterministic.ts‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
import * as crypto from "crypto";
2+
import * as miscreant from "miscreant";
3+
import {retrieveDek} from "./index";
4+
5+
// Only run through the example if this module was executed (vs imported).
6+
if (require.main == module) {
7+
(async () => {
8+
/*
9+
These values were taken from the `deterministic-roundtrip` example 1, when run with `tenant-gcp` from the demo deployment.
10+
In a real situation these would be the values that you have stored in your database or other persistence layer.
11+
*/
12+
const deterministicEncryptedValue = Buffer.from([
13+
0, 0, 19, 247, 0, 0, 120, 82, 176, 74, 41, 28, 85, 84, 151, 164, 104, 252, 207, 111, 251, 109, 137, 3, 164, 75, 158, 246, 69, 37, 83, 19, 163,
14+
]);
15+
console.log(`Deterministic Encrypted Value:
16+
${deterministicEncryptedValue.toString("base64")}`);
17+
/*
18+
The first 6 bytes of the deterministic value are the secret ID. In this case it's 5111.
19+
*/
20+
const secretId = deterministicEncryptedValue.readUInt32BE(0);
21+
console.log(`Recovered Secret ID:
22+
${secretId}`);
23+
24+
/*
25+
We took that secretId and pulled out its secret from our manual backup with
26+
`cat secret-recovery.json | jq '.result | map(select(.tenantSecretId == secretId))'`.
27+
The same thing could be programatically done if you have the secret backup up in a DB or file,
28+
or still have access to the Config Broker by way of the Vendor API or UI.
29+
30+
WARNING: You must back up encrypted secrets periodically to avoid data loss! See current [IronCore documentation](https://docs.ironcorelabs.com)
31+
for information on obtaining backups of encrypted secrets.
32+
*/
33+
const backedUpSecret = {
34+
tenantProvidedId: "tenant-gcp",
35+
numericId: 5111,
36+
id: "4a7f76dd-a0fb-4642-b15b-8423a983cc88",
37+
tenantSecretId: 5111,
38+
secretFingerprint: "tMoo5p0wpn3J2AS3iJptVWAS19B9zvj1nDGltf40wos=",
39+
secretPath: "secretPath",
40+
kmsConfigId: 510,
41+
migrationStatus: 1,
42+
encryptedSecret:
43+
"CnYKcQokAAWBd/OMpxQKZ7Fzm7WnD1vprk6t5a2Cx1OS5ZaTd3Qez6Q9EkkA8xjtHtVE4ZD8CMqa1bWVwiTggI/McQvx7+bvuytA9ztnivj08xOIIJy2zV9aU2w9RykpEXPk7pXZCLT3vMtt+5dho9m8igg6EP4D",
44+
rotationStatus: 1,
45+
secretType: 2,
46+
created: 1689877493455,
47+
updated: 1689877493455,
48+
};
49+
console.log(`Retrieved Backup Secret: ${JSON.stringify(backedUpSecret, null, "\t")}`);
50+
51+
/*
52+
To decrypt the secret itself we need to follow the same process we do for other standard encrypted IronCore documents. See
53+
`disaster-recovery-example/src/index.ts` for that process. We'll call out to that file from here instead of repeating ourselves.
54+
*/
55+
console.log("Decrypting Backup Secret with Tenant's GCP KMS using provided credentials...");
56+
const decryptedSecret = await retrieveDek(backedUpSecret.encryptedSecret);
57+
console.log(`Decrypted Backup Secret:
58+
${Buffer.from(decryptedSecret).toString("base64")}`);
59+
60+
/*
61+
Use the decrypted secret as a key in a HMAC-SHA512 hash to get the deterministic key. You can see that you need
62+
to know what derivation path you called the TSC with for any given data in your system (which you already do or it's
63+
not decryptable).
64+
*/
65+
const derivationString = `${backedUpSecret.tenantProvidedId}-derivPath`;
66+
console.log(`Hashing Over Tenant String:
67+
${derivationString}`);
68+
const hmac = crypto.createHmac("sha512", decryptedSecret);
69+
hmac.update(derivationString);
70+
const deterministicKey = hmac.digest();
71+
console.log(`Recovered Deterministic Key:
72+
${deterministicKey.toString("base64")}`);
73+
74+
/*
75+
We can now use the recovered `deterministicKey` to AES-SIV decrypt the deterministic data. This part of the
76+
process will vary the most language to language as AES-SIV libraries all have slight differences in interface.
77+
There is no AES-SIV associated data attached to IronCore deterministic values.
78+
*/
79+
console.log("AES-SIV Decrypting Deterministic Data...");
80+
const cryptoProvider = new miscreant.PolyfillCryptoProvider();
81+
const siv = await miscreant.SIV.importKey(deterministicKey, "AES-SIV", cryptoProvider);
82+
// slice past the secretId and padding to the ciphertext
83+
const recoveredData = Buffer.from(await siv.open(deterministicEncryptedValue.slice(6), []));
84+
console.log(`Recovered Data:
85+
${recoveredData.toString("base64")}`);
86+
const recoveredString = recoveredData.toString("utf8");
87+
console.log(`Original Data (String):
88+
${recoveredString}`);
89+
})().catch((e) => {
90+
console.log(e);
91+
});
92+
}

‎examples/disaster-recovery-example/src/index.ts‎

Lines changed: 40 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,7 @@ const leasedEncryptedDocumentMap = {
5656
};
5757

5858
/**
59-
* An IronCore document has a specific shape:
59+
* An IronCore v3 document (which these are) has a specific shape:
6060
* VERSION_NUMBER (1 byte, fixed at 3)
6161
* IRONCORE_MAGIC (4 bytes, IRON in ASCII)
6262
* HEADER_LENGTH (2 bytes Uint16)
@@ -78,15 +78,15 @@ const HEADER_LENGTH_LENGTH = 2;
7878
* https://github.com/IronCoreLabs/tenant-security-client-nodejs/blob/main/src/kms/Crypto.ts as a starting point for more
7979
* complex logic related to this, as well as current versions to check for and what they may contain.
8080
*/
81-
const removeHeader = (documentBuffer: Buffer): Buffer => {
81+
export const removeHeader = (documentBuffer: Buffer): Buffer => {
8282
const protobufHeaderLength = documentBuffer.readUInt16BE(VERSION_LENGTH + IRONCORE_MAGIC_LENGTH);
8383
return documentBuffer.slice(VERSION_LENGTH + IRONCORE_MAGIC_LENGTH + HEADER_LENGTH_LENGTH + protobufHeaderLength);
8484
};
8585

8686
/**
8787
* We can split apart the encrypted document bytes into the pieces we actually need to decrypt the encrypted bytes.
8888
*/
89-
const decomposeEncryptedDocument = (documentBuffer: Buffer): {iv: Buffer; encryptedBytes: Buffer; gcmTag: Buffer} => ({
89+
export const decomposeEncryptedDocument = (documentBuffer: Buffer): {iv: Buffer; encryptedBytes: Buffer; gcmTag: Buffer} => ({
9090
iv: documentBuffer.slice(0, IV_LENGTH),
9191
encryptedBytes: documentBuffer.slice(IV_LENGTH, documentBuffer.length - GCM_TAG_LENGTH),
9292
gcmTag: documentBuffer.slice(documentBuffer.length - GCM_TAG_LENGTH),
@@ -109,7 +109,7 @@ const decomposeEncryptedDocument = (documentBuffer: Buffer): {iv: Buffer; encryp
109109
* * one unleased, `tenant-gcp`
110110
* * one leased, `tenant-gcp-l`
111111
*/
112-
const retrieveDek = async (edekBase64: string): Promise<Uint8Array> => {
112+
export const retrieveDek = async (edekBase64: string): Promise<Uint8Array> => {
113113
const encryptedDeks = EncryptedDeks.decode(Uint8Array.from(Buffer.from(edekBase64, "base64"))).encryptedDeks;
114114
const encryptedDek = encryptedDeks[0];
115115
const client = new KeyManagementServiceClient();
@@ -169,36 +169,39 @@ const decryptDocument = async (documentMap: {[fieldName: string]: Buffer}, edek:
169169
return Object.fromEntries(decryptedDocumentEntries);
170170
};
171171

172-
(async () => {
173-
console.log(
174-
`Unleased Encrypted Document: ${JSON.stringify(
175-
Object.fromEntries(
176-
Object.entries(unleasedEncryptedDocumentMap).map(([fieldName, encryptedFieldBytes]) => [fieldName, encryptedFieldBytes.toString("utf8")])
177-
),
178-
null,
179-
2
180-
)}`
181-
);
182-
const decryptedUnleasedDocumentBytes = await decryptDocument(unleasedEncryptedDocumentMap, unleasedEdek);
183-
const decryptedUnleasedDocumentText = Object.fromEntries(
184-
Object.entries(decryptedUnleasedDocumentBytes).map(([fieldName, decryptedFieldBytes]) => [fieldName, decryptedFieldBytes.toString("utf8")])
185-
);
186-
console.log(`Unleased Document: ${JSON.stringify(decryptedUnleasedDocumentText, null, 2)}`);
187-
188-
console.log(
189-
`Leased Encrypted Document: ${JSON.stringify(
190-
Object.fromEntries(
191-
Object.entries(leasedEncryptedDocumentMap).map(([fieldName, encryptedFieldBytes]) => [fieldName, encryptedFieldBytes.toString("utf8")])
192-
),
193-
null,
194-
2
195-
)}`
196-
);
197-
const decryptedLeasedDocumentBytes = await decryptDocument(leasedEncryptedDocumentMap, leasedEdek);
198-
const decryptedLeasedDocumentText = Object.fromEntries(
199-
Object.entries(decryptedLeasedDocumentBytes).map(([fieldName, decryptedFieldBytes]) => [fieldName, decryptedFieldBytes.toString("utf8")])
200-
);
201-
console.log(`Leased Document: ${JSON.stringify(decryptedLeasedDocumentText, null, 2)}`);
202-
})().catch((e) => {
203-
console.log(e);
204-
});
172+
// Only run through the example if this module was executed (vs imported).
173+
if (require.main == module) {
174+
(async () => {
175+
console.log(
176+
`Unleased Encrypted Document: ${JSON.stringify(
177+
Object.fromEntries(
178+
Object.entries(unleasedEncryptedDocumentMap).map(([fieldName, encryptedFieldBytes]) => [fieldName, encryptedFieldBytes.toString("utf8")])
179+
),
180+
null,
181+
2
182+
)}`
183+
);
184+
const decryptedUnleasedDocumentBytes = await decryptDocument(unleasedEncryptedDocumentMap, unleasedEdek);
185+
const decryptedUnleasedDocumentText = Object.fromEntries(
186+
Object.entries(decryptedUnleasedDocumentBytes).map(([fieldName, decryptedFieldBytes]) => [fieldName, decryptedFieldBytes.toString("utf8")])
187+
);
188+
console.log(`Unleased Document: ${JSON.stringify(decryptedUnleasedDocumentText, null, 2)}`);
189+
190+
console.log(
191+
`Leased Encrypted Document: ${JSON.stringify(
192+
Object.fromEntries(
193+
Object.entries(leasedEncryptedDocumentMap).map(([fieldName, encryptedFieldBytes]) => [fieldName, encryptedFieldBytes.toString("utf8")])
194+
),
195+
null,
196+
2
197+
)}`
198+
);
199+
const decryptedLeasedDocumentBytes = await decryptDocument(leasedEncryptedDocumentMap, leasedEdek);
200+
const decryptedLeasedDocumentText = Object.fromEntries(
201+
Object.entries(decryptedLeasedDocumentBytes).map(([fieldName, decryptedFieldBytes]) => [fieldName, decryptedFieldBytes.toString("utf8")])
202+
);
203+
console.log(`Leased Document: ${JSON.stringify(decryptedLeasedDocumentText, null, 2)}`);
204+
})().catch((e) => {
205+
console.log(e);
206+
});
207+
}

0 commit comments

Comments
 (0)