Skip to content

Commit 41248f6

Browse files
Update DataMovement READMEs (Azure#37590)
* starting point readme * re-added missed extensions samples * handling failures * fix readme verification * fixes * spelling * add samples to validation * update snippets base package * blobs samples generated
1 parent 2ca8d4a commit 41248f6

File tree

5 files changed

+938
-166
lines changed

5 files changed

+938
-166
lines changed

sdk/storage/Azure.Storage.DataMovement.Blobs/README.md

Lines changed: 127 additions & 60 deletions
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,11 @@ We guarantee that all client instance methods are thread-safe and independent of
6464

6565
## Examples
6666

67-
### Examples using BlobContainerClient extension methods to upload and download directories.
67+
This section demonstrates usage of Data Movement for interacting with blob storage.
68+
69+
### Extensions on `BlobContainerClient`
70+
71+
For applicatons with preexisting code using Azure.Storage.Blobs, this package provides extension methods for `BlobContainerClient` to get some of the benefits of the `TransferManager` with minimal extra code.
6872

6973
Instantiate the BlobContainerClient
7074
```C# Snippet:ExtensionMethodCreateContainerClient
@@ -139,95 +143,158 @@ DataTransfer tranfer = await container.StartDownloadToDirectoryAsync(localDirect
139143
await tranfer.WaitForCompletionAsync();
140144
```
141145

142-
### Examples using BlobContainerClient extension methods to upload and download directories.
146+
### Initializing Blob Storage `StorageResource`
143147

144-
Create Instance of TransferManager with Options
145-
```C# Snippet:CreateTransferManagerWithOptions
146-
// Create BlobTransferManager with event handler in Options bag
147-
TransferManagerOptions transferManagerOptions = new TransferManagerOptions();
148-
TransferOptions options = new TransferOptions()
148+
Azure.Storage.DataMovement.Blobs exposes a `StorageResource` for each type of blob (block, page, append) as well as a blob container. Storage resources are initialized with the appropriate client object from Azure.Storage.Blobs.
149+
150+
```C# Snippet:ResourceConstruction_Blobs
151+
StorageResource containerResource = new BlobStorageResourceContainer(blobContainerClient);
152+
StorageResource blockBlobResource = new BlockBlobStorageResource(blockBlobClient);
153+
StorageResource pageBlobResource = new PageBlobStorageResource(pageBlobClient);
154+
StorageResource appendBlobResource = new AppendBlobStorageResource(appendBlobClient);
155+
```
156+
157+
Blob `StorageResource` objects can be constructed with optional "options" arguments specific to the type of resource.
158+
159+
```C# Snippet:ResourceConstruction_Blobs_WithOptions_VirtualDirectory
160+
BlobStorageResourceContainerOptions virtualDirectoryOptions = new()
149161
{
150-
MaximumTransferChunkSize = 4 * Constants.MB,
151-
CreateMode = StorageResourceCreateMode.Overwrite,
162+
DirectoryPrefix = "blob/directory/prefix"
163+
};
164+
165+
StorageResource virtualDirectoryResource = new BlobStorageResourceContainer(
166+
blobContainerClient,
167+
virtualDirectoryOptions);
168+
```
169+
170+
```C# Snippet:ResourceConstruction_Blobs_WithOptions_BlockBlob
171+
BlockBlobStorageResourceOptions leasedResourceOptions = new()
172+
{
173+
SourceConditions = new()
174+
{
175+
LeaseId = leaseId
176+
}
152177
};
153-
TransferManager transferManager = new TransferManager(transferManagerOptions);
178+
StorageResource leasedBlockBlobResource = new BlockBlobStorageResource(
179+
blockBlobClient,
180+
leasedResourceOptions);
154181
```
155182

156-
Start Upload from Local File to Block Blob
183+
When resuming a transfer, a credential to Azure Storage is likely needed. Credentials are not persisted by the transfer manager. When using `BlobStorageResources.TryGetResourceProviders()` to recreate a `StorageResource` for resume, the returned provider can create the resource with a credential specified by the calling code. This allows for workflows like scoping generation of a Shared Access Signature to the given resource path. Your application should provide its own mechanism for getting the appropriate credential, represented by `GenerateMySasCredential()` in the sample below.
184+
185+
```C# Snippet:RehydrateBlobResource
186+
StorageResource sourceResource = null;
187+
StorageResource destinationResource = null;
188+
if (BlobStorageResources.TryGetResourceProviders(
189+
info,
190+
out BlobStorageResourceProvider blobSrcProvider,
191+
out BlobStorageResourceProvider blobDstProvider))
192+
{
193+
sourceResource ??= await blobSrcProvider?.MakeResourceAsync(
194+
GenerateMySasCredential(info.SourcePath));
195+
destinationResource ??= await blobSrcProvider?.MakeResourceAsync(
196+
GenerateMySasCredential(info.DestinationPath));
197+
}
198+
```
199+
200+
### Upload
201+
202+
An upload takes place between a local file `StorageResource` as source and blob `StorageResource` as destination.
203+
204+
Upload a block blob.
205+
157206
```C# Snippet:SimpleBlobUpload
158207
DataTransfer dataTransfer = await transferManager.StartTransferAsync(
159208
sourceResource: new LocalFileStorageResource(sourceLocalPath),
160209
destinationResource: new BlockBlobStorageResource(destinationBlob));
161210
await dataTransfer.WaitForCompletionAsync();
162211
```
163-
Apply Options to Block Blob Download
164-
```C# Snippet:BlockBlobDownloadOptions
165-
await transferManager.StartTransferAsync(
166-
sourceResource: new BlockBlobStorageResource(sourceBlob, new BlockBlobStorageResourceOptions()
167-
{
168-
DestinationConditions = new BlobRequestConditions(){ LeaseId = "xyz" }
169-
}),
170-
destinationResource: new LocalFileStorageResource(downloadPath2));
171-
```
172212

173-
Start Directory Upload
213+
Upload a directory as a specific blob type.
214+
174215
```C# Snippet:SimpleDirectoryUpload
175-
// Create simple transfer directory upload job which uploads the directory and the contents of that directory
176216
DataTransfer dataTransfer = await transferManager.StartTransferAsync(
177217
sourceResource: new LocalDirectoryStorageResourceContainer(sourcePath),
178218
destinationResource: new BlobStorageResourceContainer(
179-
container,
180-
new BlobStorageResourceContainerOptions() { DirectoryPrefix = "sample-directory2" }),
219+
blobContainerClient,
220+
new BlobStorageResourceContainerOptions()
221+
{
222+
// Block blobs are the default if not specified
223+
BlobType = BlobType.Block,
224+
DirectoryPrefix = optionalDestinationPrefix,
225+
}),
181226
transferOptions: options);
227+
await dataTransfer.WaitForCompletionAsync();
228+
```
229+
230+
### Download
231+
232+
A download takes place between a blob `StorageResource` as source and local file `StorageResource` as destination.
233+
234+
Download a block blob.
235+
236+
```C# Snippet:SimpleBlockBlobDownload
237+
DataTransfer dataTransfer = await transferManager.StartTransferAsync(
238+
sourceResource: new BlockBlobStorageResource(sourceBlobClient),
239+
destinationResource: new LocalFileStorageResource(downloadPath));
240+
await dataTransfer.WaitForCompletionAsync();
182241
```
183242

184-
Start Directory Download
185-
```C# Snippet:SimpleDirectoryDownload
186-
DataTransfer downloadDirectoryJobId2 = await transferManager.StartTransferAsync(
187-
sourceDirectory2,
188-
destinationDirectory2);
243+
Download a container which may contain a mix of blob types.
244+
245+
```C# Snippet:SimpleDirectoryDownload_Blob
246+
DataTransfer dataTransfer = await transferManager.StartTransferAsync(
247+
sourceResource: new BlobStorageResourceContainer(
248+
blobContainerClient,
249+
new BlobStorageResourceContainerOptions()
250+
{
251+
DirectoryPrefix = optionalSourcePrefix
252+
}),
253+
destinationResource: new LocalDirectoryStorageResourceContainer(downloadPath));
254+
await dataTransfer.WaitForCompletionAsync();
189255
```
190256

191-
Simple Logger Sample for Transfer Manager Options
192-
```C# Snippet:SimpleLoggingSample
193-
// Create BlobTransferManager with event handler in Options bag
194-
TransferManagerOptions options = new TransferManagerOptions();
195-
TransferOptions transferOptions = new TransferOptions();
196-
transferOptions.SingleTransferCompleted += (SingleTransferCompletedEventArgs args) =>
197-
{
198-
using (StreamWriter logStream = File.AppendText(logFile))
199-
{
200-
logStream.WriteLine($"File Completed Transfer: {args.SourceResource.Path}");
201-
}
202-
return Task.CompletedTask;
203-
};
257+
### Blob Copy
258+
259+
A copy takes place between two blob `StorageResource` instances. Copying between to Azure blobs uses PUT from URL REST APIs, which do not pass data through the calling machine.
260+
261+
Copy a single blob. Note the change in blob type on this copy from block to append.
262+
263+
```C# Snippet:s2sCopyBlob
264+
DataTransfer dataTransfer = await transferManager.StartTransferAsync(
265+
sourceResource: new BlockBlobStorageResource(sourceBlockBlobClient),
266+
destinationResource: new AppendBlobStorageResource(destinationAppendBlobClient));
267+
await dataTransfer.WaitForCompletionAsync();
204268
```
205269

206-
Simple Failed Event Delegation for Container Transfer Options
207-
```C# Snippet:FailedEventDelegation
208-
transferOptions.TransferFailed += (TransferFailedEventArgs args) =>
209-
{
210-
using (StreamWriter logStream = File.AppendText(logFile))
211-
{
212-
// Specifying specific resources that failed, since its a directory transfer
213-
// maybe only one file failed out of many
214-
logStream.WriteLine($"Exception occured with TransferId: {args.TransferId}," +
215-
$"Source Resource: {args.SourceResource.Path}, +" +
216-
$"Destination Resource: {args.DestinationResource.Path}," +
217-
$"Exception Message: {args.Exception.Message}");
218-
}
219-
return Task.CompletedTask;
220-
};
270+
Copy a blob container.
271+
272+
```C# Snippet:s2sCopyBlobContainer
273+
DataTransfer dataTransfer = await transferManager.StartTransferAsync(
274+
sourceResource: new BlobStorageResourceContainer(
275+
sourceContainer,
276+
new BlobStorageResourceContainerOptions()
277+
{
278+
DirectoryPrefix = sourceDirectoryName
279+
}),
280+
destinationResource: new BlobStorageResourceContainer(
281+
destinationContainer,
282+
new BlobStorageResourceContainerOptions()
283+
{
284+
// all source blobs will be copied as a single type of destination blob
285+
// defaults to block blobs if unspecified
286+
BlobType = BlobType.Block,
287+
DirectoryPrefix = downloadPath
288+
}));
221289
```
222290

223291
## Troubleshooting
224292

225-
All Azure Storage services will throw a [RequestFailedException][RequestFailedException]
226-
with helpful [`ErrorCode`s][error_codes].
293+
***TODO***
227294

228295
## Next steps
229296

230-
Get started with our [Blob DataMovement samples][samples].
297+
***TODO***
231298

232299
## Contributing
233300

0 commit comments

Comments
 (0)