Skip to content

Commit 0abe583

Browse files
amnguyernpmsft
andauthored
DataMovement Blobs Initial Merge (Azure#32483)
* Create packages for DM Common and Blobs * Making Test packages for DM Common and Blobs; Added Readme, Changelog, BreakingChanges stubs * WIP - Added BlobDirectoryUploadOptions and StorageTransferStatus * Initial creation of filesystem scanner for DMLib (Azure#21492) * Filesystem scanner refactored to non-static implementation (Azure#21715) * Created filesystem scanner for DM Common * Modifed scanner to properly handle missing permissions; added test cases for filesystem scanner * Tests remade using native temp files; Scanner now throws errors that precede first successful yield * Changed Posix compatibility dependency * Edited versioning and READMEs to adhere to pipelines * Refactored scanner to non-static implementation; provided user configurable options for error handling * Removed test dependencies from central package list * Refactored scanner to non-static implementation; provided user configurable options for error handling * Removed test dependencies from central package list * Scanner will only work on one path for now * Capitalization on FileSystemScanner * Changed scanner to internal * Refactored FS scanner to use factory model and work better with mocking (Azure#21894) * Refactored FS scanner to use factory model and work better with mocking * Rename class/simplify factory implementation * Return folders as well; preview of ctor logic changes (only throw if path nonexistent/malformed) * Changed parameter name for scan (continueOnError), re-exported API * More exported API changes * DMLib Skeleton start (Azure#22336) * WIP - Removed DataMovement Blobs package, consildate to one package * WIP - Storage Transfer Jobs * WIP - remove dm blobs * WIP - Added TraansferItemScheduler * Ran exportapis * WIP - Resolve package conflicts * Addressed most PR comments * Ran export-api script * Made job for each specific operation for blobs * Added specific copy directory jobs, added option bags for copy scenarios * Ran ExportApi script * Update comments in StorageTransferManager * Rename BlobUploadDirectoryOptions -> BlobDirectoryUploadOptions * Run ExportAPI * PR Comments * Merge fix * WIP * Directory Upload and Download basic tests work * Test recordings test * Rerecord tests * WIP - not all ListBlobs/GetBlobs tests for DirectoryClient pass * WIP - blobtransfermanager * WIP - Moving configuations for DM Blobs * WIP - blobtransferjobs * Updated storage solution file * WIP - pathScanner tests * WIP - champion scenarios * WIP - champ scenarios * WIP - small changes * WIP' * WIP * WIP * Create packages for DM Common and Blobs * Making Test packages for DM Common and Blobs; Added Readme, Changelog, BreakingChanges stubs * WIP - Added BlobDirectoryUploadOptions and StorageTransferStatus * Initial creation of filesystem scanner for DMLib (Azure#21492) * Filesystem scanner refactored to non-static implementation (Azure#21715) * Created filesystem scanner for DM Common * Modifed scanner to properly handle missing permissions; added test cases for filesystem scanner * Tests remade using native temp files; Scanner now throws errors that precede first successful yield * Changed Posix compatibility dependency * Edited versioning and READMEs to adhere to pipelines * Refactored scanner to non-static implementation; provided user configurable options for error handling * Removed test dependencies from central package list * Refactored scanner to non-static implementation; provided user configurable options for error handling * Removed test dependencies from central package list * Scanner will only work on one path for now * Capitalization on FileSystemScanner * Changed scanner to internal * Refactored FS scanner to use factory model and work better with mocking (Azure#21894) * Refactored FS scanner to use factory model and work better with mocking * Rename class/simplify factory implementation * Return folders as well; preview of ctor logic changes (only throw if path nonexistent/malformed) * Changed parameter name for scan (continueOnError), re-exported API * More exported API changes * DMLib Skeleton start (Azure#22336) * WIP - Removed DataMovement Blobs package, consildate to one package * WIP - Storage Transfer Jobs * WIP - remove dm blobs * WIP - Added TraansferItemScheduler * Ran exportapis * WIP - Resolve package conflicts * Addressed most PR comments * Ran export-api script * Made job for each specific operation for blobs * Added specific copy directory jobs, added option bags for copy scenarios * Ran ExportApi script * Update comments in StorageTransferManager * Rename BlobUploadDirectoryOptions -> BlobDirectoryUploadOptions * Run ExportAPI * PR Comments * Merge fix * Merge main update * WIP * Builds here without Azure.Storage.DataMovement.Blobs * Builds - DMLib common, DMlib blobs, DMlib samples * Added back in blobs tests * BlobTransferScheduler updated, logger updated, plan file updated * API generates * Rerun some tests, attempting to fix some parallel start problems * Resolve bad merge conflicts * DMLib builds but Blobs.Tests does not build * Conversion from internal job to job details * Run exports api update * Update logger information * Changed threadpool method to use inherit TransferScheduler * Remove previous implementation of getting job details, and combine into one * Removing mistake of committed files * Update to Job Plan header * Updating manager detail of API * Add abstract resumeJob to base storagetransfermanager * Update event arguments to have individual ones for each case, update progress handler to basic handler, update copy method * Removed base DM models, made base event args, made protected ctor StorageTransferManager * Changed Directory Download to DownloadTo, added overwrite options, updated internal pipeline transfer for directoryclient * change string to uri for local paths, remove unncessary things from blob job properties * WIP - changing job details out, added more champ scenarios regarding progress tracking * Updating Resume API, correcting event arg names, correctly linked internal deep copy for directory client * Readded upload directory tests with working json files, changed uploadDirectory API return type, Mild changes to some APIs, renamed part files * WIP * Cannot catch exception properly, tear downs upload call * Addressing Arch board comment * Some fixes from merging from main Remove test dependency on AesGcm for datamovement * WIP * Renamed Experimental to DataMovement * Fixed channel blocklist issue * WIP - changing event handler in uploader to trigger block status * Working commit block handler * WIP * Changes to Download and APIs regarding download * Copy Service Job cleanup * WIP - API changes to StorageResource and Controller * WIP * WIP - Aligning blobs API usage * WIP - Added dependenices to Azure.Storage.DataMovement.Test * WIP - Updated APIs to include checkpointing * WIP - ConsumeableStream -> GetConsumerableStream * WIP - make old API structure internal; todo: remove all old APIs * WIP - Remade API for blobs DM, removed CopyMethod * WIP -Update to StorageTransfer event args name * WIP - Removed factory calls, made dervived storage resource types public * Merged BlobDataController to main controller, renamed DataController to Transfermanager, removed ListType from StorageResource * WIP - Added Checkpointer API, removed unnecessary -1 enum values, updated job plan header bytes * WIP - removed options from respective derived storage resource calls, added options bag to blob storage resources * WIP - renamed CommitListTYpe to clearer type * WIP - Update to Copy Options api in blockblob storage, and samples * WIP - Updated APIs * WIP - Updated APIs to include offset streams * WIP - Rename writetooffsetoptions with storageresource prefixed * WIP - copy to and from and update to mmp job plan file * Added over the concurrency tuner * Remove ConfigureAwait from samples * WIP - changes to MMF, service to service copy and adding method to pass the token credential or bearer token to storage resource * WIP - fixes to event handler, removable of complete transfer check api * WIP - fix to closing stream when reading from local, setting blocklist order before commiting * WIP - tests * WIP - Remove unnecessary APIs and old code * Removing more unnecessary changes and test recordings for old tests * More removal of old test recordings * Removing BlobFolderClient / BlobVirtualDirectoryClient * Ran Export APIs, moved DataTransferExtensions to DataTransfer * ApiView Comments addressed * Renamed from Blobs.DataMovement to DataMovement.Blobs * Ran ExportApis * Updating assemblyinfo datamovement blobs namespace * Move over Storage Resource tests; Made some API corrections * Remove suppression on editorconfig * Added API for creation of blobs storage resource, max chunk size, more tests, fixes * Changed GetStorageResources to return a base class of storage resource; fixed bugs with append / sequential operations; Updated copy status handler for async copy * PR Comments - reverted necessary config files, moved constants to a separate file, rremvoed globalsupression files * Export APIs * PR Comments - removed merge mistakes, updated some xml comments, change some option bags, removed blobstorageresourcefactory, removed more globalsupression files * PR Comments - Move unnecessary return xml removed and removed localfilefactory * PR Comments - Removing leftover folder models from BlobVirtualFolderClient * Updating GetProperties comment XML, removing first value from cpu monitor reading, adding try block to delete file when failed download chunks occur * Fix to directory, and some test changes to use DataTransfer awaitcompletion * Update to tests and adding discovered length optimization * Ignore some tests for now, to push recording in a separate PR * Update readmes * Ignore more tests * Ignore more local directory tests * Temporarily remove nuget package link; readd when link works when package is released * Update snippets to include length Co-authored-by: Rushi Patel <84736613+rnpmsft@users.noreply.github.com>
1 parent bb2e13f commit 0abe583

File tree

176 files changed

+21810
-22
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

176 files changed

+21810
-22
lines changed

sdk/storage/Azure.Storage.Blobs/src/BlobBaseClient.cs

Lines changed: 108 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
using System.IO;
88
using System.Linq;
99
using System.Net.Http.Headers;
10+
using System.Runtime.CompilerServices;
1011
using System.Threading;
1112
using System.Threading.Tasks;
1213
using Azure.Core;
@@ -21,6 +22,13 @@
2122

2223
#pragma warning disable SA1402 // File may only contain a single type
2324

25+
[assembly: InternalsVisibleTo("Azure.Storage.DataMovement.Blobs, PublicKey=" +
26+
"0024000004800000940000000602000000240000525341310004000001000100d15ddcb2968829" +
27+
"5338af4b7686603fe614abd555e09efba8fb88ee09e1f7b1ccaeed2e8f823fa9eef3fdd60217fc" +
28+
"012ea67d2479751a0b8c087a4185541b851bd8b16f8d91b840e51b1cb0ba6fe647997e57429265" +
29+
"e85ef62d565db50a69ae1647d54d7bd855e4db3d8a91510e5bcbd0edfbbecaa20a7bd9ae74593d" +
30+
"aa7b11b4")]
31+
2432
namespace Azure.Storage.Blobs.Specialized
2533
{
2634
/// <summary>
@@ -240,7 +248,7 @@ public BlobBaseClient(string connectionString, string blobContainerName, string
240248
/// every request.
241249
/// </param>
242250
public BlobBaseClient(Uri blobUri, BlobClientOptions options = default)
243-
: this(blobUri, (HttpPipelinePolicy)null, options, null)
251+
: this(blobUri, (HttpPipelinePolicy)null, options, storageSharedKeyCredential: null)
244252
{
245253
}
246254

@@ -263,7 +271,7 @@ public BlobBaseClient(Uri blobUri, BlobClientOptions options = default)
263271
/// every request.
264272
/// </param>
265273
public BlobBaseClient(Uri blobUri, StorageSharedKeyCredential credential, BlobClientOptions options = default)
266-
: this(blobUri, credential.AsPolicy(), options, credential)
274+
: this(blobUri, credential.AsPolicy(), options, storageSharedKeyCredential: credential)
267275
{
268276
}
269277

@@ -290,7 +298,7 @@ public BlobBaseClient(Uri blobUri, StorageSharedKeyCredential credential, BlobCl
290298
/// This constructor should only be used when shared access signature needs to be updated during lifespan of this client.
291299
/// </remarks>
292300
public BlobBaseClient(Uri blobUri, AzureSasCredential credential, BlobClientOptions options = default)
293-
: this(blobUri, credential.AsPolicy<BlobUriBuilder>(blobUri), options, null)
301+
: this(blobUri, credential.AsPolicy<BlobUriBuilder>(blobUri), options, storageSharedKeyCredential: null)
294302
{
295303
}
296304

@@ -313,11 +321,71 @@ public BlobBaseClient(Uri blobUri, AzureSasCredential credential, BlobClientOpti
313321
/// every request.
314322
/// </param>
315323
public BlobBaseClient(Uri blobUri, TokenCredential credential, BlobClientOptions options = default)
316-
: this(blobUri, credential.AsPolicy(options), options, null)
324+
: this(blobUri, credential.AsPolicy(options), options, credential)
317325
{
318326
Errors.VerifyHttpsTokenAuth(blobUri);
319327
}
320328

329+
/// <summary>
330+
/// Initializes a new instance of the <see cref="BlobBaseClient"/>
331+
/// class.
332+
/// </summary>
333+
/// <param name="blobUri">
334+
/// A <see cref="Uri"/> referencing the blob that includes the
335+
/// name of the account, the name of the container, and the name of
336+
/// the blob.
337+
/// This is likely to be similar to "https://{account_name}.blob.core.windows.net/{container_name}/{blob_name}".
338+
/// </param>
339+
/// <param name="authentication">
340+
/// An optional authentication policy used to sign requests.
341+
/// </param>
342+
/// <param name="options">
343+
/// Optional client options that define the transport pipeline
344+
/// policies for authentication, retries, etc., that are applied to
345+
/// every request.
346+
/// </param>
347+
/// <param name="tokenCredential">
348+
/// The token credential used to sign requests.
349+
/// </param>
350+
internal BlobBaseClient(
351+
Uri blobUri,
352+
HttpPipelinePolicy authentication,
353+
BlobClientOptions options,
354+
TokenCredential tokenCredential)
355+
{
356+
Argument.AssertNotNull(blobUri, nameof(blobUri));
357+
options ??= new BlobClientOptions();
358+
_uri = blobUri;
359+
if (!string.IsNullOrEmpty(blobUri.Query))
360+
{
361+
UriQueryParamsCollection queryParamsCollection = new UriQueryParamsCollection(blobUri.Query);
362+
if (queryParamsCollection.ContainsKey(Constants.SnapshotParameterName))
363+
{
364+
_snapshot = System.Web.HttpUtility.ParseQueryString(blobUri.Query).Get(Constants.SnapshotParameterName);
365+
}
366+
if (queryParamsCollection.ContainsKey(Constants.VersionIdParameterName))
367+
{
368+
_blobVersionId = System.Web.HttpUtility.ParseQueryString(blobUri.Query).Get(Constants.VersionIdParameterName);
369+
}
370+
}
371+
372+
_clientConfiguration = new BlobClientConfiguration(
373+
pipeline: options.Build(authentication),
374+
tokenCredential: tokenCredential,
375+
clientDiagnostics: new StorageClientDiagnostics(options),
376+
version: options.Version,
377+
customerProvidedKey: options.CustomerProvidedKey,
378+
transferValidation: options.TransferValidation,
379+
encryptionScope: options.EncryptionScope,
380+
trimBlobNameSlashes: options.TrimBlobNameSlashes);
381+
382+
_clientSideEncryption = options._clientSideEncryptionOptions?.Clone();
383+
_blobRestClient = BuildBlobRestClient(blobUri);
384+
385+
BlobErrors.VerifyHttpsCustomerProvidedKey(_uri, _clientConfiguration.CustomerProvidedKey);
386+
BlobErrors.VerifyCpkAndEncryptionScopeNotBothSet(_clientConfiguration.CustomerProvidedKey, _clientConfiguration.EncryptionScope);
387+
}
388+
321389
/// <summary>
322390
/// Initializes a new instance of the <see cref="BlobBaseClient"/>
323391
/// class.
@@ -574,6 +642,42 @@ private void SetNameFieldsIfNull()
574642
}
575643
}
576644

645+
#region internal static accessors for Azure.Storage.DataMovement.Blobs
646+
/// <summary>
647+
/// Get a <see cref="BlobBaseClient"/>'s <see cref="TokenCredential"/>
648+
/// for passing the authorization when performing service to service copy
649+
/// where OAuth is necessary to authenticate the source.
650+
/// </summary>
651+
/// <param name="client">The BlobServiceClient.</param>
652+
/// <returns>The BlobServiceClient's HttpPipeline.</returns>
653+
internal static TokenCredential GetTokenCredential(BlobBaseClient client) =>
654+
client.ClientConfiguration.OAuthTokenCredential;
655+
656+
/// <summary>
657+
/// Get a <see cref="BlobBaseClient"/>'s <see cref="BlobClientOptions"/>
658+
/// for creating child clients.
659+
/// </summary>
660+
/// <param name="client">The BlobServiceClient.</param>
661+
/// <returns>The BlobServiceClient's BlobClientOptions.</returns>
662+
internal static BlobClientOptions GetClientOptions(BlobBaseClient client) =>
663+
new BlobClientOptions(client.ClientConfiguration.Version)
664+
{
665+
// We only use this for communicating diagnostics, at the moment
666+
Diagnostics =
667+
{
668+
IsDistributedTracingEnabled = client.ClientConfiguration.ClientDiagnostics.IsActivityEnabled
669+
}
670+
};
671+
672+
/// <summary>
673+
/// Get a <see cref="BlobBaseClient"/>'s <see cref="BlobClientOptions"/>
674+
/// for creating child clients.
675+
/// </summary>
676+
/// <param name="client">The BlobServiceClient.</param>
677+
/// <returns>The BlobServiceClient's BlobClientOptions.</returns>
678+
internal static bool GetUsingClientSideEncryption(BlobBaseClient client) => client.UsingClientSideEncryption;
679+
#endregion protected static accessors for Azure.Storage.DataMovement.Blobs
680+
577681
///// <summary>
578682
///// Creates a clone of this instance that references a version ID rather than the base blob.
579683
///// </summary>

sdk/storage/Azure.Storage.Blobs/src/BlockBlobClient.cs

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -351,14 +351,14 @@ protected static BlockBlobClient CreateClient(Uri blobUri, BlobClientOptions opt
351351
return new BlockBlobClient(
352352
blobUri,
353353
new BlobClientConfiguration(
354-
pipeline,
355-
null,
356-
new StorageClientDiagnostics(options),
357-
options.Version,
358-
options.CustomerProvidedKey,
359-
options.TransferValidation,
360-
null,
361-
options.TrimBlobNameSlashes));
354+
pipeline: pipeline,
355+
sharedKeyCredential: null,
356+
clientDiagnostics: new StorageClientDiagnostics(options),
357+
version: options.Version,
358+
customerProvidedKey: options.CustomerProvidedKey,
359+
transferValidation: options.TransferValidation,
360+
encryptionScope: null,
361+
trimBlobNameSlashes: options.TrimBlobNameSlashes));
362362
}
363363

364364
private static void AssertNoClientSideEncryption(BlobClientOptions options)

sdk/storage/Azure.Storage.Blobs/src/BlobClientConfiguration.cs renamed to sdk/storage/Azure.Storage.Blobs/src/Shared/BlobClientConfiguration.cs

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,16 +4,29 @@
44
using System;
55
using System.Collections.Generic;
66
using System.Text;
7+
using Azure.Core;
78
using Azure.Core.Pipeline;
89
using Azure.Storage.Blobs.Models;
910
using Azure.Storage.Shared;
1011

1112
namespace Azure.Storage.Blobs
1213
{
14+
/// <summary>
15+
/// Provides the configurations to connecting to the Blob Service and to create the Blob Clients
16+
/// </summary>
1317
internal class BlobClientConfiguration : StorageClientConfiguration
1418
{
19+
/// <summary>
20+
/// The versions of Azure Blob Storage supported by this client
21+
/// library. For more, see
22+
/// <see href="https://docs.microsoft.com/en-us/rest/api/storageservices/versioning-for-the-azure-storage-services">
23+
/// Versioning for Azure Storage Services</see>.
24+
/// </summary>
1525
public virtual BlobClientOptions.ServiceVersion Version { get; internal set; }
1626

27+
/// <summary>
28+
/// The encryption key to be used with client provided key server-side encryption.
29+
/// </summary>
1730
public virtual CustomerProvidedKey? CustomerProvidedKey { get; internal set; }
1831

1932
public virtual TransferValidationOptions TransferValidation { get; internal set; }
@@ -22,6 +35,24 @@ internal class BlobClientConfiguration : StorageClientConfiguration
2235

2336
public bool TrimBlobNameSlashes { get; internal set; }
2437

38+
public BlobClientConfiguration(
39+
HttpPipeline pipeline,
40+
TokenCredential tokenCredential,
41+
ClientDiagnostics clientDiagnostics,
42+
BlobClientOptions.ServiceVersion version,
43+
CustomerProvidedKey? customerProvidedKey,
44+
TransferValidationOptions transferValidation,
45+
string encryptionScope,
46+
bool trimBlobNameSlashes)
47+
: base(pipeline, tokenCredential, clientDiagnostics)
48+
{
49+
Version = version;
50+
CustomerProvidedKey = customerProvidedKey;
51+
TransferValidation = transferValidation;
52+
EncryptionScope = encryptionScope;
53+
TrimBlobNameSlashes = trimBlobNameSlashes;
54+
}
55+
2556
public BlobClientConfiguration(
2657
HttpPipeline pipeline,
2758
StorageSharedKeyCredential sharedKeyCredential,

sdk/storage/Azure.Storage.Common/src/Shared/Constants.cs

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
// Copyright (c) Microsoft Corporation. All rights reserved.
22
// Licensed under the MIT License.
33

4+
using System;
45
using System.Collections.Generic;
56

67
namespace Azure.Storage
@@ -231,6 +232,7 @@ internal static class Block
231232
internal static class Page
232233
{
233234
public const int PageSizeBytes = 512;
235+
public const int MaxPageBlockBytes = 4 * Constants.MB; // 4MB
234236
}
235237

236238
internal static class Container

0 commit comments

Comments
 (0)