Cosmos spark3.2 support (Azure#26527)

* Version bump after cosmos spark release 4.5.3

* Skeleton for Spark 3.1 and 3.2 support

* Mergin main

* Update external_dependencies.txt

* Updating external dependencies for Spark 3.1 and 3.2

* Update pom.xml

* Update pom.xml

* Updating md files

* Update external_dependencies.txt

* Fixing versioning

* Update version_client.txt

* Update .docsettings.yml

* Update .docsettings.yml

* fixing links

* fixing links

* adding versions to child poms

* adding groupid to child poms

* Removing parent section from dependency-reduced-pom

* Update pom.xml

* Fixing Build_Analyze issues

* Update CHANGELOG.md

* Update ci.yml

Author: FabianMeiswinkel

.gitignore

@@ -96,3 +96,7 @@ scalastyle-output.xml
 # Emacs #
 
 #changebundle.txt#
+
+# Flatten-pom
+.flattened-pom.xml
+flatter.pom

eng/.docsettings.yml

@@ -44,6 +44,7 @@ known_presence_issues:
   - ['sdk/cognitiveservices/ms-azure-cs-videosearch', '#2847']
   - ['sdk/cognitiveservices/ms-azure-cs-visualsearch', '#2847']
   - ['sdk/cognitiveservices/ms-azure-cs-websearch', '#2847']
+  - ['sdk/cosmos/azure-cosmos-spark_3_2-12', '#2847']
   - ['sdk/eventhubs/microsoft-azure-eventhubs-extensions', '#2847']
   - ['sdk/keyvault/microsoft-azure-keyvault', '#2847']
   - ['sdk/keyvault/microsoft-azure-keyvault-core', '#2847']
@@ -111,11 +112,12 @@ known_content_issues:
   - ['sdk/cosmos/azure-cosmos/README.md', '#3113']
   - ['sdk/cosmos/azure-cosmos-encryption/README.md', '#3113']
   - ['sdk/cosmos/azure-cosmos-spark_3-1_2-12/README.md', '#3113']
-  - ['sdk/cosmos/azure-cosmos-spark_3-1_2-12/docs/catalog-api.md', '#3113']
-  - ['sdk/cosmos/azure-cosmos-spark_3-1_2-12/docs/configuration-reference.md', '#3113']
-  - ['sdk/cosmos/azure-cosmos-spark_3-1_2-12/docs/local-emulator.md', '#3113']
-  - ['sdk/cosmos/azure-cosmos-spark_3-1_2-12/docs/quick-start.md', '#3113']
-  - ['sdk/cosmos/azure-cosmos-spark_3-1_2-12/dev/README.md', '#3113']
+  - ['sdk/cosmos/azure-cosmos-spark_3-2_2-12/README.md', '#3113']
+  - ['sdk/cosmos/azure-cosmos-spark_3_2-12/dev/README.md', '#3113']
+  - ['sdk/cosmos/azure-cosmos-spark_3_2-12/docs/catalog-api.md', '#3113']
+  - ['sdk/cosmos/azure-cosmos-spark_3_2-12/docs/configuration-reference.md', '#3113']
+  - ['sdk/cosmos/azure-cosmos-spark_3_2-12/docs/local-emulator.md', '#3113']
+  - ['sdk/cosmos/azure-cosmos-spark_3_2-12/docs/quick-start.md', '#3113']
   - ['sdk/cosmos/README.md', '#3113']
   - ['sdk/e2e/README.md', '#3113']
   - ['sdk/eventgrid/microsoft-azure-eventgrid/README.md', '#3113']

eng/versioning/external_dependencies.txt

@@ -312,8 +312,10 @@ cosmos_org.hdrhistogram:HdrHistogram;2.1.12
 
 ## Cosmos Spark connector under sdk\cosmos\azure-cosmos-spark_3-1_2-12\pom.xml
 # Cosmos Spark connector runtime dependencies - provided by Spark runtime/host
-cosmos_org.apache.spark:spark-sql_2.12;3.1.1
-cosmos_org.apache.spark:spark-hive_2.12;3.1.1
+cosmos-spark_3-1_org.apache.spark:spark-sql_2.12;3.1.1
+cosmos-spark_3-2_org.apache.spark:spark-sql_2.12;3.2.0
+cosmos-spark_3-1_org.apache.spark:spark-hive_2.12;3.1.1
+cosmos-spark_3-2_org.apache.spark:spark-hive_2.12;3.2.0
 cosmos_org.scala-lang:scala-library;2.12.10
 cosmos_org.scala-lang.modules:scala-java8-compat_2.12;0.8.0
 cosmos_io.projectreactor:reactor-scala-extensions_2.12;0.8.0
@@ -329,6 +331,9 @@ cosmos_org.scalamock:scalamock_2.12;5.0.0
 cosmos_org.scalatest:scalatest-maven-plugin;2.0.2
 cosmos_net.alchim31.maven:scala-maven-plugin;4.5.4
 cosmos_org.scalastyle:scalastyle-maven-plugin;1.0.0
+cosmos_org.apache.maven.plugins:maven-shade-plugin;1.2.7
+cosmos_org.codehaus.mojo:flatten-maven-plugin;1.2.7
+cosmos_com.coderplus.maven.plugins:copy-rename-maven-plugin;1.0.1
 
 # sdk\core\azure-core-serializer-avro-jackson\pom.xml
 # This dependency is needed since Jackson Avro uses an older dependency on Apache Avro which is another library.

eng/versioning/version_client.txt

@@ -81,7 +81,7 @@ com.azure:azure-core-tracing-opentelemetry;1.0.0-beta.19;1.0.0-beta.20
 com.azure:azure-cosmos;4.25.0;4.26.0-beta.1
 com.azure:azure-cosmos-benchmark;4.0.1-beta.1;4.0.1-beta.1
 com.azure:azure-cosmos-dotnet-benchmark;4.0.1-beta.1;4.0.1-beta.1
-com.azure.cosmos.spark:azure-cosmos-spark_3-1_2-12;4.6.0-beta.1;4.6.0-beta.1
+com.azure.cosmos.spark:azure-cosmos-spark_3_2-12;4.6.0-beta.1;4.6.0-beta.1
 com.azure:azure-cosmos-encryption;1.0.0-beta.9;1.0.0-beta.10
 com.azure:azure-data-appconfiguration;1.2.4;1.2.5
 com.azure:azure-data-appconfiguration-perf;1.0.0-beta.1;1.0.0-beta.1

sdk/cosmos/azure-cosmos-spark_3-1_2-12/CHANGELOG.md

@@ -95,7 +95,7 @@
 * Cosmos DB Spark 3.1.1 Connector Preview `4.0.0-beta.3` Release.
 #### Configuration Renames
 * Renamed data source name `cosmos.changeFeed` to `cosmos.oltp.changeFeed`, see [PR](https://github.com/Azure/azure-sdk-for-java/pull/21121).
-* Configuration renamed. See [PR](https://github.com/Azure/azure-sdk-for-java/pull/21004) for list of changes. See [Configuration-Reference](https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/cosmos/azure-cosmos-spark_3-1_2-12/docs/configuration-reference.md) for more details.
+* Configuration renamed. See [PR](https://github.com/Azure/azure-sdk-for-java/pull/21004) for list of changes. See [Configuration-Reference](https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/cosmos/azure-cosmos-spark_3_2-12/docs/configuration-reference.md) for more details.
 
 #### Key Bug Fixes
 * Added validation for all config-settings with a name starting with "spark.cosmos."

sdk/cosmos/azure-cosmos-spark_3-1_2-12/CONTRIBUTING.md

@@ -10,8 +10,8 @@ To build the project, run maven commands.
 
 ```bash
 git clone https://github.com/Azure/azure-sdk-for-java.git 
-cd sdk/cosmos/azure-cosmos-spark_3_2.12
-mvnw clean install
+cd sdk/cosmos/azure-cosmos-spark_3-1_2.12
+mvn clean install
 ```
 
 ## Test

sdk/cosmos/azure-cosmos-spark_3-1_2-12/README.md

@@ -10,9 +10,9 @@ https://github.com/Azure/azure-sdk-for-java/issues/new
 
 ### Documentation
 
-- [Getting started](https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/cosmos/azure-cosmos-spark_3-1_2-12/docs/quick-start.md)
-- [Catalog API](https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/cosmos/azure-cosmos-spark_3-1_2-12/docs/catalog-api.md)
-- [Configuration Parameter Reference](https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/cosmos/azure-cosmos-spark_3-1_2-12/docs/configuration-reference.md)
+- [Getting started](https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/cosmos/azure-cosmos-spark_3_2-12/docs/quick-start.md)
+- [Catalog API](https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/cosmos/azure-cosmos-spark_3_2-12/docs/catalog-api.md)
+- [Configuration Parameter Reference](https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/cosmos/azure-cosmos-spark_3_2-12/docs/configuration-reference.md)
 
 [//]: # (//TODO: moderakh add more sections)
 [//]: # (//TODO: moderakh Enable Client Logging)
@@ -24,8 +24,10 @@ https://github.com/Azure/azure-sdk-for-java/issues/new
 
 ### Version Compatibility
 
+#### azure-cosmos-spark_3-1_2-12
 | Connector     | Supported Spark Versions | Minimum Java Version | Supported Scala Versions | Supported Databricks Runtimes |
 | ------------- | ------------------------ | -------------------- | -----------------------  | ----------------------------- |
+| 4.6.0         | 3.1.1 - 3.1.2            | 8                    | 2.12                     | 8.\*, 9.\*                    |
 | 4.5.3         | 3.1.1 - 3.1.2            | 8                    | 2.12                     | 8.\*, 9.\*                    |
 | 4.5.2         | 3.1.1 - 3.1.2            | 8                    | 2.12                     | 8.\*, 9.\*                    |
 | 4.5.1         | 3.1.1 - 3.1.2            | 8                    | 2.12                     | 8.\*, 9.\*                    |
@@ -42,6 +44,11 @@ https://github.com/Azure/azure-sdk-for-java/issues/new
 | 4.0.0-beta.2  | 3.1.1                    | 8                    | 2.12                     | 8.\*                          |
 | 4.0.0-beta.1  | 3.1.1                    | 8                    | 2.12                     | 8.\*                          |
 
+#### azure-cosmos-spark_3-2_2-12
+| Connector     | Supported Spark Versions | Minimum Java Version | Supported Scala Versions | Supported Databricks Runtimes |
+| ------------- | ------------------------ | -------------------- | -----------------------  | ----------------------------- |
+| 4.6.0         | 3.2.0 - 3.2.1            | 8                    | 2.12                     | 10.\*                         |
+
 ### Download
 
 You can use the maven coordinate of the jar to auto install the Spark Connector to your Databricks Runtime 8 from Maven: