Feature/oauth copyfromurl (#22086)

* Set up STG78 branch (#21634)

* Generated off latest version of blob swagger (#21644)

* swagger generation and public API overloads

* blob clients obey new generation

* minor changes to bearer token logic

* TEST WORKING

* extra testing

* ported oauthcopy source to blobs

* options API changes and docstrings

* fixed constructor issues

* fix missed source auth misses

* script tags on pom

* removes arg checks in option model constructors

rerecorded tests

* pom fix

* fix misnamed page blob tests

* Test fixes

adds missing recording.
extra playback logic for oauth tokens that should fix a CI-specific
issue.

* fixed pom script comment

* downgrade some depenency verions

* rerecord sas tests

* snippets and checkstyle

* rerecorded tests

fixed recording infra for file share tests

* test fixes

* rerecorded versioning tests

* HttpAuthorization added to core-experimental

azure-storage-blob and azure-storage-file-share now depend on
core-experimental for new class and updated their APIs to use the new
class.

* refactored options type names

* Reenabled quick query parquet support (#21785)

* reidentified samples for options class name change

* Implemented Immutable Storage with Versioning (#21718)

* File listing v2 (#21720)

* swagger generation and hookup

regenerated files swagger for listing
hooked up generated and handwritten models
expanded client API to support new models

* testing and fixes

expanded test parameters for file listing.
swagger transform added.

* rerecorded file listing test

* more testing

also fixed a method name casing

* docstrings and samples

* checkstyle

* reverted primative boolean extended info arg

REST api accepts true or an absence of value. Alligning with .NET for
now on Boolean vs boolean.

* checkpoint

* pr feedback

* misused variable fix

* fixed test

* send nothing, not false: include-extended-info

other minor fixes
rerecorded tests

* import cleanup

* swagger regeneration and test fixes

* API comments

extendedInfo boolean => Boolean
renamed options class

* reidentified samples for options class name change

Co-authored-by: jschrepp-MSFT <41338290+jschrepp-MSFT@users.noreply.github.com>

* Fix test.

* remove HttpAuthorization in prep for merge

added in separate PR. removing to be ready for that offical addition.

* fix build from merge

* fix pom versioning CI issues

* snippet line length

* fix merge error with immutable versioning

* PR feedback

* Minor cleanup

* generator cleanup and regeneration

* Keep files-shares tests dependent on latest blob

* test refactor/rerecord

* deleted old tests that were copied instead of moved

* min service version reformat

* import cleanup

* post-merge.

* getOauthToken refactor

* reformatting

Co-authored-by: Gauri Prasad <51212198+gapra-msft@users.noreply.github.com>
Co-authored-by: jschrepp-MSFT <41338290+jschrepp-MSFT@users.noreply.github.com>
Co-authored-by: gapra <gapra@microsoft.com>
Co-authored-by: Kamil Sobol <kasobol@microsoft.com>
Co-authored-by: Kamil Sobol <61715331+kasobol-msft@users.noreply.github.com>

Author: 3 people

sdk/storage/azure-storage-blob/pom.xml

@@ -58,6 +58,11 @@
       <artifactId>azure-core</artifactId>
       <version>1.17.0</version> <!-- {x-version-update;com.azure:azure-core;dependency} -->
     </dependency>
+    <dependency>
+      <groupId>com.azure</groupId>
+      <artifactId>azure-core-experimental</artifactId>
+      <version>1.0.0-beta.15</version> <!-- {x-version-update;com.azure:azure-core-experimental;current} -->
+    </dependency>
     <dependency>
       <groupId>com.azure</groupId>
       <artifactId>azure-core-http-netty</artifactId>

sdk/storage/azure-storage-blob/src/main/java/com/azure/storage/blob/options/AppendBlobAppendBlockFromUrlOptions.java

@@ -0,0 +1,121 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.
+// Licensed under the MIT License.
+
+package com.azure.storage.blob.options;
+
+import com.azure.core.annotation.Fluent;
+import com.azure.core.experimental.http.HttpAuthorization;
+import com.azure.core.util.CoreUtils;
+import com.azure.storage.blob.models.AppendBlobRequestConditions;
+import com.azure.storage.blob.models.BlobRange;
+import com.azure.storage.blob.models.BlobRequestConditions;
+
+/**
+ * Extended options that may be passed when appending a block from a source URL.
+ */
+@Fluent
+public class AppendBlobAppendBlockFromUrlOptions {
+    private final String sourceUrl;
+    private BlobRange sourceRange;
+    private byte[] sourceContentMd5;
+    private AppendBlobRequestConditions destinationRequestConditions;
+    private BlobRequestConditions sourceRequestConditions;
+    private HttpAuthorization sourceAuthorization;
+
+    /**
+     * @param sourceUrl The source URL to copy from. URLs outside of Azure may only be copied to block blobs.
+     */
+    public AppendBlobAppendBlockFromUrlOptions(String sourceUrl) {
+        this.sourceUrl = sourceUrl;
+    }
+
+    /**
+     * @return Source URL to copy from.
+     */
+    public String getSourceUrl() {
+        return sourceUrl;
+    }
+
+    /**
+     * @return Range of bytes to read from the source.
+     */
+    public BlobRange getSourceRange() {
+        return sourceRange;
+    }
+
+    /**
+     * @param sourceRange Range of bytes to read from the source.
+     * @return The updated options.
+     */
+    public AppendBlobAppendBlockFromUrlOptions setSourceRange(BlobRange sourceRange) {
+        this.sourceRange = sourceRange;
+        return this;
+    }
+
+    /**
+     * @return MD5 of the source content to be appended.
+     */
+    public byte[] getSourceContentMd5() {
+        return CoreUtils.clone(sourceContentMd5);
+    }
+
+    /**
+     * @param sourceContentMd5 MD5 of the source content to be appended.
+     * @return The updated options.
+     */
+    public AppendBlobAppendBlockFromUrlOptions setSourceContentMd5(byte[] sourceContentMd5) {
+        this.sourceContentMd5 = CoreUtils.clone(sourceContentMd5);
+        return this;
+    }
+
+    /**
+     * @return {@link AppendBlobRequestConditions} for writing to destination.
+     */
+    public AppendBlobRequestConditions getDestinationRequestConditions() {
+        return destinationRequestConditions;
+    }
+
+    /**
+     * @param destinationRequestConditions {@link AppendBlobRequestConditions} for writing to destination.
+     * @return The updated options.
+     */
+    public AppendBlobAppendBlockFromUrlOptions setDestinationRequestConditions(AppendBlobRequestConditions destinationRequestConditions) {
+        this.destinationRequestConditions = destinationRequestConditions;
+        return this;
+    }
+
+    /**
+     * @return {@link BlobRequestConditions} for accessing source.
+     */
+    public BlobRequestConditions getSourceRequestConditions() {
+        return sourceRequestConditions;
+    }
+
+    /**
+     * @param sourceRequestConditions {@link BlobRequestConditions} for accessing source.
+     * @return The updated options.
+     */
+    public AppendBlobAppendBlockFromUrlOptions setSourceRequestConditions(BlobRequestConditions sourceRequestConditions) {
+        this.sourceRequestConditions = sourceRequestConditions;
+        return this;
+    }
+
+    /**
+     * @return auth header for accessing source.
+     */
+    public HttpAuthorization getSourceAuthorization() {
+        return sourceAuthorization;
+    }
+
+    /**
+     * Sets "Authorization" header for accessing source URL. Currently only "Bearer" authentication is accepted by
+     * Storage.
+     *
+     * @param sourceAuthorization auth header for accessing source.
+     * @return The updated options.
+     */
+    public AppendBlobAppendBlockFromUrlOptions setSourceAuthorization(HttpAuthorization sourceAuthorization) {
+        this.sourceAuthorization = sourceAuthorization;
+        return this;
+    }
+}

sdk/storage/azure-storage-blob/src/main/java/com/azure/storage/blob/options/BlobCopyFromUrlOptions.java

@@ -4,6 +4,7 @@
 package com.azure.storage.blob.options;
 
 import com.azure.core.annotation.Fluent;
+import com.azure.core.experimental.http.HttpAuthorization;
 import com.azure.core.http.RequestConditions;
 import com.azure.storage.blob.models.AccessTier;
 import com.azure.storage.blob.models.BlobImmutabilityPolicy;
@@ -23,6 +24,7 @@ public class BlobCopyFromUrlOptions {
     private AccessTier tier;
     private RequestConditions sourceRequestConditions;
     private BlobRequestConditions destinationRequestConditions;
+    private HttpAuthorization sourceAuthorization;
     private BlobImmutabilityPolicy immutabilityPolicy;
     private Boolean legalHold;
 
@@ -122,6 +124,25 @@ public BlobCopyFromUrlOptions setDestinationRequestConditions(BlobRequestConditi
         return this;
     }
 
+    /**
+     * @return auth header for access to source.
+     */
+    public HttpAuthorization getSourceAuthorization() {
+        return sourceAuthorization;
+    }
+
+    /**
+     * Sets "Authorization" header for accessing source URL. Currently only "Bearer" authentication is accepted by
+     * Storage.
+     *
+     * @param sourceAuthorization auth header for access to source.
+     * @return The updated options.
+     */
+    public BlobCopyFromUrlOptions setSourceAuthorization(HttpAuthorization sourceAuthorization) {
+        this.sourceAuthorization = sourceAuthorization;
+        return this;
+    }
+
     /**
      * @return {@link BlobImmutabilityPolicy}
      */

sdk/storage/azure-storage-blob/src/main/java/com/azure/storage/blob/options/BlobUploadFromUrlOptions.java

@@ -4,6 +4,7 @@
 package com.azure.storage.blob.options;
 
 import com.azure.core.util.CoreUtils;
+import com.azure.core.experimental.http.HttpAuthorization;
 import com.azure.storage.blob.models.AccessTier;
 import com.azure.storage.blob.models.BlobHttpHeaders;
 import com.azure.storage.blob.models.BlobRequestConditions;
@@ -23,6 +24,7 @@ public class BlobUploadFromUrlOptions {
     private BlobRequestConditions destinationRequestConditions;
     private BlobRequestConditions sourceRequestConditions;
     private Boolean copySourceBlobProperties;
+    private HttpAuthorization sourceAuthorization;
 
     /**
      * @param sourceUrl The source URL to upload from.
@@ -159,4 +161,23 @@ public BlobUploadFromUrlOptions setCopySourceBlobProperties(Boolean copySourceBl
         this.copySourceBlobProperties = copySourceBlobProperties;
         return this;
     }
+
+    /**
+     * @return auth header for access to source.
+     */
+    public HttpAuthorization getSourceAuthorization() {
+        return sourceAuthorization;
+    }
+
+    /**
+     * Sets "Authorization" header for accessing source URL. Currently only "Bearer" authentication is accepted by
+     * Storage.
+     *
+     * @param sourceAuthorization auth header for access to source.
+     * @return The updated options.
+     */
+    public BlobUploadFromUrlOptions setSourceAuthorization(HttpAuthorization sourceAuthorization) {
+        this.sourceAuthorization = sourceAuthorization;
+        return this;
+    }
 }

sdk/storage/azure-storage-blob/src/main/java/com/azure/storage/blob/options/BlockBlobStageBlockFromUrlOptions.java

@@ -0,0 +1,130 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.
+// Licensed under the MIT License.
+
+package com.azure.storage.blob.options;
+
+import com.azure.core.annotation.Fluent;
+import com.azure.core.experimental.http.HttpAuthorization;
+import com.azure.core.util.CoreUtils;
+import com.azure.storage.blob.models.BlobRange;
+import com.azure.storage.blob.models.BlobRequestConditions;
+
+/**
+ * Extended options that may be passed when staging a block from a source URL.
+ */
+@Fluent
+public class BlockBlobStageBlockFromUrlOptions {
+    private final String base64BlockId;
+    private final String sourceUrl;
+    private BlobRange sourceRange;
+    private byte[] sourceContentMd5;
+    private String leaseId;
+    private BlobRequestConditions sourceRequestConditions;
+    private HttpAuthorization sourceAuthorization;
+
+    /**
+     * @param base64BlockId The block ID to assign the new block.
+     * @param sourceUrl The source URL to copy from. URLs outside of Azure may only be copied to block blobs.
+     */
+    public BlockBlobStageBlockFromUrlOptions(String base64BlockId, String sourceUrl) {
+        this.base64BlockId = base64BlockId;
+        this.sourceUrl = sourceUrl;
+    }
+
+    /**
+     * @return The block ID to assign the new block.
+     */
+    public String getBase64BlockId() {
+        return base64BlockId;
+    }
+
+    /**
+     * @return The source URL to upload from.
+     */
+    public String getSourceUrl() {
+        return sourceUrl;
+    }
+
+    /**
+     * @return Range of bytes to read from the source.
+     */
+    public BlobRange getSourceRange() {
+        return sourceRange;
+    }
+
+    /**
+     * @param sourceRange Range of bytes to read from the source.
+     * @return The updated options.
+     */
+    public BlockBlobStageBlockFromUrlOptions setSourceRange(BlobRange sourceRange) {
+        this.sourceRange = sourceRange;
+        return this;
+    }
+
+    /**
+     * @return MD5 of the source content.
+     */
+    public byte[] getSourceContentMd5() {
+        return CoreUtils.clone(sourceContentMd5);
+    }
+
+    /**
+     * @param sourceContentMd5 MD5 of the source content.
+     * @return The updated options.
+     */
+    public BlockBlobStageBlockFromUrlOptions setSourceContentMd5(byte[] sourceContentMd5) {
+        this.sourceContentMd5 = CoreUtils.clone(sourceContentMd5);
+        return this;
+    }
+
+    /**
+     * @return Lease ID for accessing source content.
+     */
+    public String getLeaseId() {
+        return leaseId;
+    }
+
+    /**
+     * @param leaseId Lease ID for accessing source content.
+     * @return The updated options.
+     */
+    public BlockBlobStageBlockFromUrlOptions setLeaseId(String leaseId) {
+        this.leaseId = leaseId;
+        return this;
+    }
+
+    /**
+     * @return {@link BlobRequestConditions} for accessing source content.
+     */
+    public BlobRequestConditions getSourceRequestConditions() {
+        return sourceRequestConditions;
+    }
+
+    /**
+     * @param sourceRequestConditions {@link BlobRequestConditions} for accessing source content.
+     * @return The updated options.
+     */
+    public BlockBlobStageBlockFromUrlOptions setSourceRequestConditions(BlobRequestConditions sourceRequestConditions) {
+        this.sourceRequestConditions = sourceRequestConditions;
+        return this;
+    }
+
+    /**
+     * @return auth header to access source.
+     */
+    public HttpAuthorization getSourceAuthorization() {
+        return sourceAuthorization;
+    }
+
+    /**
+     * Sets "Authorization" header for accessing source URL. Currently only "Bearer" authentication is accepted by
+     * Storage.
+     *
+     * @param sourceAuthorization auth header to access source.
+     * @return The updated options.
+     */
+    public BlockBlobStageBlockFromUrlOptions setSourceAuthorization(HttpAuthorization sourceAuthorization) {
+        this.sourceAuthorization = sourceAuthorization;
+        return this;
+    }
+}