Skip to content

Commit 6511cff

Browse files
xinlian12annie-macFabianMeiswinkelannie-mac
authored
addAadSupportInSpark (Azure#32393)
* add aad support in spark --------- Co-authored-by: annie-mac <annie-mac@DESKTOP-M86HBMH.redmond.corp.microsoft.com> Co-authored-by: Fabian Meiswinkel <fabian@meiswinkel.com> Co-authored-by: annie-mac <annie-mac@annie-macs-MacBook-Pro.local>
1 parent d748dcf commit 6511cff

File tree

49 files changed

+1997
-663
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

49 files changed

+1997
-663
lines changed

‎sdk/cosmos/azure-cosmos-spark_3-1_2-12/CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
### 4.17.0-beta.1 (Unreleased)
44

55
#### Features Added
6+
* Added Service Principle based AAD Auth - See [PR 32393](https://github.com/Azure/azure-sdk-for-java/pull/32393)
67

78
#### Breaking Changes
89

‎sdk/cosmos/azure-cosmos-spark_3-2_2-12/CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
### 4.17.0-beta.1 (Unreleased)
44

55
#### Features Added
6+
* Added Service Principle based AAD Auth - See [PR 32393](https://github.com/Azure/azure-sdk-for-java/pull/32393)
67

78
#### Breaking Changes
89

‎sdk/cosmos/azure-cosmos-spark_3-3_2-12/CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
### 4.17.0-beta.1 (Unreleased)
44

55
#### Features Added
6+
* Added Service Principle based AAD Auth - See [PR 32393](https://github.com/Azure/azure-sdk-for-java/pull/32393)
67

78
#### Breaking Changes
89

‎sdk/cosmos/azure-cosmos-spark_3_2-12/docs/configuration-reference.md‎

Lines changed: 17 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,13 +2,23 @@
22

33

44
## Generic Configuration
5-
| Config Property Name | Default | Description |
6-
| :--- | :---- | :--- |
7-
| `spark.cosmos.accountEndpoint` | None | Cosmos DB Account Endpoint Uri |
8-
| `spark.cosmos.accountKey` | None | Cosmos DB Account Key |
9-
| `spark.cosmos.database` | None | Cosmos DB database name |
10-
| `spark.cosmos.container` | None | Cosmos DB container name |
11-
5+
| Config Property Name | Default | Description |
6+
|:---------------------------------| :---- |:-----------------------------------------------------------------------------------------------------------|
7+
| `spark.cosmos.accountEndpoint` | None | Cosmos DB Account Endpoint Uri |
8+
| `spark.cosmos.accountKey` | None | Cosmos DB Account Key |
9+
| `spark.cosmos.database` | None | Cosmos DB database name |
10+
| `spark.cosmos.container` | None | Cosmos DB container name |
11+
| `spark.cosmos.subscriptionId` | None | The subscriptionId of the CosmosDB account. Required for `ServicePrinciple` authentication. |
12+
| `spark.cosmos.tenantId` | None | The tenantId of the CosmosDB account. Required for `ServicePrinciple` authentication. |
13+
| `spark.cosmos.resourceGroupName` | None | The resource group of the CosmosDB account. Required for `ServicePrinciple` authentication. |
14+
| `spark.cosmos.azureEnvironment` | `Azure` | The azure environment of the CosmosDB account: `Azure`, `AzureChina`, `AzureUsGovernment`, `AzureGermany`. |
15+
16+
### AAD Auth Config
17+
| Config Property Name | Default | Description |
18+
|:-------------------------------------| :---- |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
19+
| `spark.cosmos.auth.type` | `MasterKey` | There are two auth types are supported currently: `MasterKey`(PrimaryReadWriteKeys, SecondReadWriteKeys, PrimaryReadOnlyKeys, SecondReadWriteKeys), `ServicePrinciple`. |
20+
| `spark.cosmos.auth.aad.clientId` | None | The clientId/ApplicationId of the service principle. Required for `ServicePrinciple` authentication. |
21+
| `spark.cosmos.auth.aad.clientSecret` | None | The client secret/password of the service principle. Required for `ServicePrinciple` authentication. |
1222

1323
### Additional Tuning
1424
| Config Property Name | Default | Description |

‎sdk/cosmos/azure-cosmos-spark_3_2-12/pom.xml‎

Lines changed: 31 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,33 @@
8181
</exclusion>
8282
</exclusions>
8383
</dependency>
84+
<dependency>
85+
<groupId>com.azure.resourcemanager</groupId>
86+
<artifactId>azure-resourcemanager-cosmos</artifactId>
87+
<version>2.23.0</version> <!-- {x-version-update;com.azure.resourcemanager:azure-resourcemanager-cosmos;dependency} -->
88+
</dependency>
89+
<dependency>
90+
<groupId>com.azure</groupId>
91+
<artifactId>azure-identity</artifactId>
92+
<version>1.7.3</version> <!-- {x-version-update;com.azure:azure-identity;dependency} -->
93+
<exclusions>
94+
<exclusion>
95+
<groupId>com.azure</groupId>
96+
<artifactId>azure-core</artifactId>
97+
</exclusion>
98+
</exclusions>
99+
</dependency>
100+
<dependency>
101+
<groupId>com.azure</groupId>
102+
<artifactId>azure-core-http-netty</artifactId>
103+
<version>1.12.8</version> <!-- {x-version-update;com.azure:azure-core-http-netty;dependency} -->
104+
<exclusions>
105+
<exclusion>
106+
<groupId>com.azure</groupId>
107+
<artifactId>azure-core</artifactId>
108+
</exclusion>
109+
</exclusions>
110+
</dependency>
84111

85112
<!-- Test -->
86113
<dependency>
@@ -396,10 +423,6 @@
396423
<pattern>com.fasterxml</pattern>
397424
<shadedPattern>${shadingPrefix}.com.fasterxml</shadedPattern>
398425
</relocation>
399-
<relocation>
400-
<pattern>io.netty</pattern>
401-
<shadedPattern>${shadingPrefix}.io.netty</shadedPattern>
402-
</relocation>
403426
<relocation>
404427
<pattern>com.ctc.wstx</pattern>
405428
<shadedPattern>${shadingPrefix}.com.ctc.wstx</shadedPattern>
@@ -448,6 +471,10 @@
448471
<pattern>com.google</pattern>
449472
<shadedPattern>${shadingPrefix}.com.google</shadedPattern>
450473
</relocation>
474+
<relocation>
475+
<pattern>com.azure.resourcemanager</pattern>
476+
<shadedPattern>${shadingPrefix}.com.azure.resourcemanager</shadedPattern>
477+
</relocation>
451478
</relocations>
452479
<filters>
453480
<filter>

‎sdk/cosmos/azure-cosmos-spark_3_2-12/src/main/scala/com/azure/cosmos/spark/ChangeFeedPartitionReader.scala‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,8 +10,8 @@ import com.azure.cosmos.spark.ChangeFeedPartitionReader.LsnPropertyName
1010
import com.azure.cosmos.spark.CosmosPredicates.requireNotNull
1111
import com.azure.cosmos.spark.CosmosTableSchemaInferrer.LsnAttributeName
1212
import com.azure.cosmos.spark.diagnostics.{DiagnosticsContext, DiagnosticsLoader, LoggerHelper, SparkTaskContext}
13-
import org.apache.spark.TaskContext
1413
import com.fasterxml.jackson.databind.JsonNode
14+
import org.apache.spark.TaskContext
1515
import org.apache.spark.broadcast.Broadcast
1616
import org.apache.spark.sql.Row
1717
import org.apache.spark.sql.catalyst.InternalRow

‎sdk/cosmos/azure-cosmos-spark_3_2-12/src/main/scala/com/azure/cosmos/spark/ChangeFeedTable.scala‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ package com.azure.cosmos.spark
55
import com.azure.cosmos.CosmosException
66
import com.azure.cosmos.implementation.CosmosClientMetadataCachesSnapshot
77
import com.azure.cosmos.models.PartitionKey
8+
import com.azure.cosmos.spark.catalog.CosmosCatalogCosmosSDKClient
89
import com.azure.cosmos.spark.diagnostics.LoggerHelper
910
import com.fasterxml.jackson.databind.node.ObjectNode
1011
import org.apache.spark.broadcast.Broadcast
@@ -166,12 +167,12 @@ private class ChangeFeedTable(val session: SparkSession,
166167
}
167168

168169
val state = new CosmosClientMetadataCachesSnapshot()
169-
state.serialize(clientCacheItems(0).get.client)
170+
state.serialize(clientCacheItems(0).get.cosmosClient)
170171

171172
var throughputControlState: Option[CosmosClientMetadataCachesSnapshot] = None
172173
if (clientCacheItems(1).isDefined) {
173174
throughputControlState = Some(new CosmosClientMetadataCachesSnapshot())
174-
throughputControlState.get.serialize(clientCacheItems(1).get.client)
175+
throughputControlState.get.serialize(clientCacheItems(1).get.cosmosClient)
175176
}
176177

177178
val metadataSnapshots = CosmosClientMetadataCachesSnapshots(state, throughputControlState)

0 commit comments

Comments
 (0)