Feature/storage/transactional crc (Azure#23056)

* Transactional hashing models

* StageBlock now has transactional hashing support

* ContentHasher supports precalculated hash

GetHash will now read all options and pull a possible precalculated hash
into its response instead of calculating the result from scratch

* New overloads

appendblobclient and blockblobclient now have options bag overloads for
their baseline upload methods.

* Append and page blob updates

Those clients now have new APIs for base uploads, deprecated old
overloads, and updated writestreams to use the new internal signature.

* Alligned XxxOpenWriteOptions for hashing options

* Blob Write Streams Updated

* uploadTransactionalHashing in datalake and shares

* fixed mock issues

Fixed how some mocks were constructed due to new signatures for some
internal methods

* blob download range transactional hashing

* crc46 implementation and download tests

* several read methods updated to transactional hash

* testing

* datalake/share download

* build fix

* renames | Blob/datalake downloadTo hashing

* tests and fixes

* Range required for download hashing

testing, recording.

* more tests. property renames for consistency.

* Azure.Storage.Models => Azure.Storage

Moved transactional hashing options classes down a namespace

* fixes and rerecording transactional hashing tests

* recording fix

* recording

* test fixes

* all hashing tests scan messages for headers

* minor fixes and test organization

* test fine-tuning and rerecording

* open write tests refactored for reusability

* test refactors

* More test refactoring

* refactored parallel upload tests

* finished blob test refactoring

* exportapi

* fix export

* fixed a scope issue

* fixed datalake.append scope and common mock

* fixed mock setup

* More mocking issues

* irerecord

* full rerecording transactional hash tests

* revert force record mode

* testing CI failure

removing an inserted policy from hashing partitioned download tests to
determine  if it is interfering in recording interpretation.

* Revert "testing CI failure"

This reverts commit 4bbb8d3.

* made encryption/hashing test liveonly

* datalake transactional hashing tests

* port hashing tests to file shares

* transactional hash arg validation

* parallel upload doesn't accept precalculated hash

* rerecord transactional hashing tests

* restored accidental recording delete

* rerecorded OpenRead tests

* rerecord misc file share tests

* New storagecrc64 API shape

storage crc calculator prepared to inherit NonCryptographicHashAlgorithm
when released.
Static crc calculator converted to use ReadOnlySpan<byte> instead of
byte[].
Static crc calculator converted to use BinaryPrimitives instead of
BitConverter. NOTE this forces the calculation into a specific
endianness which is acceptable based on CRC properties.

* exportapi

* misc pr feedback

* Fixed a docstring

* fixed diagnostic scope issue

* Documented TransactionalHashingTestSkeletons.cs

* test comments

* Some PR responses

Some options classes renamed
GetHashResult memory optimizations.
Disabled clientside crypto/hashing in combination

* new downloadto options have consistent API shape

* datalake hashing tests now use hns tenant

* More PR comments

* export api

* new options bag APIs memory optimization

* fixed failed tests

* Initial hashing test conversion

Converted TransactionalHashingTestSkeletons (collection of static helper
methods) to TransactionalHashingTestBase : StorageTestBase. Tests are
fully defined on this class. Subclasses per-client will implement
abstract methods as instructions for uploads/downloads/etc., where
before they were lambdas passed to the static helper methods every
single time.

* moved transactional hashing tests under new inheritance tree

* deleted old hashing test session records

* fixed playback bug and rerecorded to accomodate

* Recorded inconclusive hashing tests

* more recording inconclusive tests

* quick fix

* nitpicks/cleanup

* ported v11 crc test

* file header

* reverted some auto IDE csproj changes

* corrected a formating reversion

* pr feedback

* arg validation

* pr feedback

* Changelog

Co-authored-by: jschrepp-MSFT <41338290+jschrepp-MSFT@users.noreply.github.com>

Author: jaschrep-msft

sdk/storage/Azure.Storage.Blobs/CHANGELOG.md

@@ -5,6 +5,7 @@
 - Added support for Encryption Scope SAS.
 - Added support for Encryption Scopes with BlobBaseClient.SyncCopyFromUriAsync().
 - Added support for generating SAS URLs with the Permanent Delete ('y') SAS permission.
+- Added support for SDK-calculated transactional hash checksums on data transfer.
 - Fixed bug where BlobContainerClient.GetBlobs() and .GetBlobsByHierarchyAsync() was not parsing the Object Replication Metadata correctly
 
 ## 12.10.0 (2021-09-08)

sdk/storage/Azure.Storage.Blobs/api/Azure.Storage.Blobs.netstandard2.0.cs

@@ -20,10 +20,12 @@ public AppendBlobWriteStream(
             long bufferSize,
             long position,
             AppendBlobRequestConditions conditions,
-            IProgress<long> progressHandler) : base(
+            IProgress<long> progressHandler,
+            UploadTransactionalHashingOptions hashingOptions) : base(
                 position,
                 bufferSize,
-                progressHandler)
+                progressHandler,
+                hashingOptions)
         {
             ValidateBufferSize(bufferSize);
             _appendBlobClient = appendBlobClient;
@@ -40,9 +42,12 @@ protected override async Task AppendInternal(
 
                 Response<BlobAppendInfo> response = await _appendBlobClient.AppendBlockInternal(
                     content: _buffer,
-                    transactionalContentHash: default,
-                    conditions: _conditions,
-                    progressHandler: _progressHandler,
+                    new AppendBlobAppendBlockOptions()
+                    {
+                        TransactionalHashingOptions = _hashingOptions,
+                        Conditions = _conditions,
+                        ProgressHandler = _progressHandler
+                    },
                     async: async,
                     cancellationToken: cancellationToken)
                     .ConfigureAwait(false);

sdk/storage/Azure.Storage.Blobs/src/AppendBlobClient.cs

@@ -1,4 +1,4 @@
-<Project Sdk="Microsoft.NET.Sdk">
+<Project Sdk="Microsoft.NET.Sdk">
   <PropertyGroup>
     <TargetFrameworks>$(RequiredTargetFrameworks)</TargetFrameworks>
   </PropertyGroup>
@@ -48,11 +48,15 @@
     <Compile Include="$(AzureStorageSharedSources)ClientsideEncryption\Models\*.cs" LinkBase="Shared" />
     <Compile Include="$(AzureStorageSharedSources)AggregatingProgressIncrementer.cs" LinkBase="Shared" />
     <Compile Include="$(AzureStorageSharedSources)Constants.cs" LinkBase="Shared" />
+    <Compile Include="$(AzureStorageSharedSources)ContentHasher.cs" LinkBase="Shared" />
     <Compile Include="$(AzureStorageSharedSources)Errors.cs" LinkBase="Shared" />
     <Compile Include="$(AzureStorageSharedSources)Errors.Clients.cs" LinkBase="Shared" />
     <Compile Include="$(AzureStorageSharedSources)GeoRedundantReadPolicy.cs" LinkBase="Shared" />
+    <Compile Include="$(AzureStorageSharedSources)HashAlgorithmHasher.cs" LinkBase="Shared" />
     <Compile Include="$(AzureStorageSharedSources)IDownloadedContent.cs" LinkBase="Shared" />
+    <Compile Include="$(AzureStorageSharedSources)IHasher.cs" LinkBase="Shared" />
     <Compile Include="$(AzureStorageSharedSources)LoggingExtensions.cs" LinkBase="Shared" />
+    <Compile Include="$(AzureStorageSharedSources)NonCryptographicHashAlgorithmHasher.cs" LinkBase="Shared" />
     <Compile Include="$(AzureStorageSharedSources)NonDisposingStream.cs" LinkBase="Shared" />
     <Compile Include="$(AzureStorageSharedSources)PartitionedUploader.cs" LinkBase="Shared" />
     <Compile Include="$(AzureStorageSharedSources)PooledMemoryStream.cs" LinkBase="Shared" />

sdk/storage/Azure.Storage.Blobs/src/AppendBlobWriteStream.cs

@@ -1624,13 +1624,19 @@ internal async Task<Response<BlobContentInfo>> StagedUploadInternal(
         {
             if (UsingClientSideEncryption)
             {
+                if (UsingClientSideEncryption && options.TransactionalHashingOptions != default)
+                {
+                    throw Errors.TransactionalHashingNotSupportedWithClientSideEncryption();
+                }
+
                 // content is now unseekable, so PartitionedUploader will be forced to do a buffered multipart upload
                 (content, options.Metadata) = await new BlobClientSideEncryptor(new ClientSideEncryptor(ClientSideEncryption))
                     .ClientSideEncryptInternal(content, options.Metadata, async, cancellationToken).ConfigureAwait(false);
             }
 
             var uploader = GetPartitionedUploader(
                 transferOptions: options?.TransferOptions ?? default,
+                options?.TransactionalHashingOptions,
                 operationName: $"{nameof(BlobClient)}.{nameof(Upload)}");
 
             return await uploader.UploadInternal(
@@ -1724,8 +1730,9 @@ private BlockBlobClient BlockBlobClient
 
         internal PartitionedUploader<BlobUploadOptions, BlobContentInfo> GetPartitionedUploader(
             StorageTransferOptions transferOptions,
+            UploadTransactionalHashingOptions hashingOptions,
             ArrayPool<byte> arrayPool = null,
             string operationName = null)
-            => BlockBlobClient.GetPartitionedUploader(transferOptions, arrayPool, operationName);
+            => BlockBlobClient.GetPartitionedUploader(transferOptions, hashingOptions, arrayPool, operationName);
     }
 }