Commit 0abe583
DataMovement Blobs Initial Merge (Azure#32483)
* Create packages for DM Common and Blobs
* Making Test packages for DM Common and Blobs; Added Readme, Changelog, BreakingChanges stubs
* WIP - Added BlobDirectoryUploadOptions and StorageTransferStatus
* Initial creation of filesystem scanner for DMLib (Azure#21492)
* Filesystem scanner refactored to non-static implementation (Azure#21715)
* Created filesystem scanner for DM Common
* Modifed scanner to properly handle missing permissions; added test cases for filesystem scanner
* Tests remade using native temp files; Scanner now throws errors that precede first successful yield
* Changed Posix compatibility dependency
* Edited versioning and READMEs to adhere to pipelines
* Refactored scanner to non-static implementation; provided user configurable options for error handling
* Removed test dependencies from central package list
* Refactored scanner to non-static implementation; provided user configurable options for error handling
* Removed test dependencies from central package list
* Scanner will only work on one path for now
* Capitalization on FileSystemScanner
* Changed scanner to internal
* Refactored FS scanner to use factory model and work better with mocking (Azure#21894)
* Refactored FS scanner to use factory model and work better with mocking
* Rename class/simplify factory implementation
* Return folders as well; preview of ctor logic changes (only throw if path nonexistent/malformed)
* Changed parameter name for scan (continueOnError), re-exported API
* More exported API changes
* DMLib Skeleton start (Azure#22336)
* WIP - Removed DataMovement Blobs package, consildate to one package
* WIP - Storage Transfer Jobs
* WIP - remove dm blobs
* WIP - Added TraansferItemScheduler
* Ran exportapis
* WIP - Resolve package conflicts
* Addressed most PR comments
* Ran export-api script
* Made job for each specific operation for blobs
* Added specific copy directory jobs, added option bags for copy scenarios
* Ran ExportApi script
* Update comments in StorageTransferManager
* Rename BlobUploadDirectoryOptions -> BlobDirectoryUploadOptions
* Run ExportAPI
* PR Comments
* Merge fix
* WIP
* Directory Upload and Download basic tests work
* Test recordings test
* Rerecord tests
* WIP - not all ListBlobs/GetBlobs tests for DirectoryClient pass
* WIP - blobtransfermanager
* WIP - Moving configuations for DM Blobs
* WIP - blobtransferjobs
* Updated storage solution file
* WIP - pathScanner tests
* WIP - champion scenarios
* WIP - champ scenarios
* WIP - small changes
* WIP'
* WIP
* WIP
* Create packages for DM Common and Blobs
* Making Test packages for DM Common and Blobs; Added Readme, Changelog, BreakingChanges stubs
* WIP - Added BlobDirectoryUploadOptions and StorageTransferStatus
* Initial creation of filesystem scanner for DMLib (Azure#21492)
* Filesystem scanner refactored to non-static implementation (Azure#21715)
* Created filesystem scanner for DM Common
* Modifed scanner to properly handle missing permissions; added test cases for filesystem scanner
* Tests remade using native temp files; Scanner now throws errors that precede first successful yield
* Changed Posix compatibility dependency
* Edited versioning and READMEs to adhere to pipelines
* Refactored scanner to non-static implementation; provided user configurable options for error handling
* Removed test dependencies from central package list
* Refactored scanner to non-static implementation; provided user configurable options for error handling
* Removed test dependencies from central package list
* Scanner will only work on one path for now
* Capitalization on FileSystemScanner
* Changed scanner to internal
* Refactored FS scanner to use factory model and work better with mocking (Azure#21894)
* Refactored FS scanner to use factory model and work better with mocking
* Rename class/simplify factory implementation
* Return folders as well; preview of ctor logic changes (only throw if path nonexistent/malformed)
* Changed parameter name for scan (continueOnError), re-exported API
* More exported API changes
* DMLib Skeleton start (Azure#22336)
* WIP - Removed DataMovement Blobs package, consildate to one package
* WIP - Storage Transfer Jobs
* WIP - remove dm blobs
* WIP - Added TraansferItemScheduler
* Ran exportapis
* WIP - Resolve package conflicts
* Addressed most PR comments
* Ran export-api script
* Made job for each specific operation for blobs
* Added specific copy directory jobs, added option bags for copy scenarios
* Ran ExportApi script
* Update comments in StorageTransferManager
* Rename BlobUploadDirectoryOptions -> BlobDirectoryUploadOptions
* Run ExportAPI
* PR Comments
* Merge fix
* Merge main update
* WIP
* Builds here without Azure.Storage.DataMovement.Blobs
* Builds - DMLib common, DMlib blobs, DMlib samples
* Added back in blobs tests
* BlobTransferScheduler updated, logger updated, plan file updated
* API generates
* Rerun some tests, attempting to fix some parallel start problems
* Resolve bad merge conflicts
* DMLib builds but Blobs.Tests does not build
* Conversion from internal job to job details
* Run exports api update
* Update logger information
* Changed threadpool method to use inherit TransferScheduler
* Remove previous implementation of getting job details, and combine into one
* Removing mistake of committed files
* Update to Job Plan header
* Updating manager detail of API
* Add abstract resumeJob to base storagetransfermanager
* Update event arguments to have individual ones for each case, update progress handler to basic handler, update copy method
* Removed base DM models, made base event args, made protected ctor StorageTransferManager
* Changed Directory Download to DownloadTo, added overwrite options, updated internal pipeline transfer for directoryclient
* change string to uri for local paths, remove unncessary things from blob job properties
* WIP - changing job details out, added more champ scenarios regarding progress tracking
* Updating Resume API, correcting event arg names, correctly linked internal deep copy for directory client
* Readded upload directory tests with working json files, changed uploadDirectory API return type, Mild changes to some APIs, renamed part files
* WIP
* Cannot catch exception properly, tear downs upload call
* Addressing Arch board comment
* Some fixes from merging from main
Remove test dependency on AesGcm for datamovement
* WIP
* Renamed Experimental to DataMovement
* Fixed channel blocklist issue
* WIP - changing event handler in uploader to trigger block status
* Working commit block handler
* WIP
* Changes to Download and APIs regarding download
* Copy Service Job cleanup
* WIP - API changes to StorageResource and Controller
* WIP
* WIP - Aligning blobs API usage
* WIP - Added dependenices to Azure.Storage.DataMovement.Test
* WIP - Updated APIs to include checkpointing
* WIP - ConsumeableStream -> GetConsumerableStream
* WIP - make old API structure internal; todo: remove all old APIs
* WIP - Remade API for blobs DM, removed CopyMethod
* WIP -Update to StorageTransfer event args name
* WIP - Removed factory calls, made dervived storage resource types public
* Merged BlobDataController to main controller, renamed DataController to Transfermanager, removed ListType from StorageResource
* WIP - Added Checkpointer API, removed unnecessary -1 enum values, updated job plan header bytes
* WIP - removed options from respective derived storage resource calls, added options bag to blob storage resources
* WIP - renamed CommitListTYpe to clearer type
* WIP - Update to Copy Options api in blockblob storage, and samples
* WIP - Updated APIs
* WIP - Updated APIs to include offset streams
* WIP - Rename writetooffsetoptions with storageresource prefixed
* WIP - copy to and from and update to mmp job plan file
* Added over the concurrency tuner
* Remove ConfigureAwait from samples
* WIP - changes to MMF, service to service copy and adding method to pass the token credential or bearer token to storage resource
* WIP - fixes to event handler, removable of complete transfer check api
* WIP - fix to closing stream when reading from local, setting blocklist order before commiting
* WIP - tests
* WIP - Remove unnecessary APIs and old code
* Removing more unnecessary changes and test recordings for old tests
* More removal of old test recordings
* Removing BlobFolderClient / BlobVirtualDirectoryClient
* Ran Export APIs, moved DataTransferExtensions to DataTransfer
* ApiView Comments addressed
* Renamed from Blobs.DataMovement to DataMovement.Blobs
* Ran ExportApis
* Updating assemblyinfo datamovement blobs namespace
* Move over Storage Resource tests; Made some API corrections
* Remove suppression on editorconfig
* Added API for creation of blobs storage resource, max chunk size, more tests, fixes
* Changed GetStorageResources to return a base class of storage resource; fixed bugs with append / sequential operations; Updated copy status handler for async copy
* PR Comments - reverted necessary config files, moved constants to a separate file, rremvoed globalsupression files
* Export APIs
* PR Comments - removed merge mistakes, updated some xml comments, change some option bags, removed blobstorageresourcefactory, removed more globalsupression files
* PR Comments - Move unnecessary return xml removed and removed localfilefactory
* PR Comments - Removing leftover folder models from BlobVirtualFolderClient
* Updating GetProperties comment XML, removing first value from cpu monitor reading, adding try block to delete file when failed download chunks occur
* Fix to directory, and some test changes to use DataTransfer awaitcompletion
* Update to tests and adding discovered length optimization
* Ignore some tests for now, to push recording in a separate PR
* Update readmes
* Ignore more tests
* Ignore more local directory tests
* Temporarily remove nuget package link; readd when link works when package is released
* Update snippets to include length
Co-authored-by: Rushi Patel <84736613+rnpmsft@users.noreply.github.com>1 parent bb2e13f commit 0abe583
File tree
176 files changed
+21810
-22
lines changed- sdk/storage
- Azure.Storage.Blobs/src
- Azure.Storage.Common
- tests
- Azure.Storage.DataMovement.Blobs
- api
- samples
- src
- tests
- Azure.Storage.DataMovement
- api
- src
- ConcurrencyTuner
- JobPlanModels
- Models
- tests
- SessionRecords
- LocalDirectoryStorageResourceTests
- LocalFileStorageResourceTests
- Azure.Storage.Files.DataLake/src
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
176 files changed
+21810
-22
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| 10 | + | |
10 | 11 | | |
11 | 12 | | |
12 | 13 | | |
| |||
21 | 22 | | |
22 | 23 | | |
23 | 24 | | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
24 | 32 | | |
25 | 33 | | |
26 | 34 | | |
| |||
240 | 248 | | |
241 | 249 | | |
242 | 250 | | |
243 | | - | |
| 251 | + | |
244 | 252 | | |
245 | 253 | | |
246 | 254 | | |
| |||
263 | 271 | | |
264 | 272 | | |
265 | 273 | | |
266 | | - | |
| 274 | + | |
267 | 275 | | |
268 | 276 | | |
269 | 277 | | |
| |||
290 | 298 | | |
291 | 299 | | |
292 | 300 | | |
293 | | - | |
| 301 | + | |
294 | 302 | | |
295 | 303 | | |
296 | 304 | | |
| |||
313 | 321 | | |
314 | 322 | | |
315 | 323 | | |
316 | | - | |
| 324 | + | |
317 | 325 | | |
318 | 326 | | |
319 | 327 | | |
320 | 328 | | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
321 | 389 | | |
322 | 390 | | |
323 | 391 | | |
| |||
574 | 642 | | |
575 | 643 | | |
576 | 644 | | |
| 645 | + | |
| 646 | + | |
| 647 | + | |
| 648 | + | |
| 649 | + | |
| 650 | + | |
| 651 | + | |
| 652 | + | |
| 653 | + | |
| 654 | + | |
| 655 | + | |
| 656 | + | |
| 657 | + | |
| 658 | + | |
| 659 | + | |
| 660 | + | |
| 661 | + | |
| 662 | + | |
| 663 | + | |
| 664 | + | |
| 665 | + | |
| 666 | + | |
| 667 | + | |
| 668 | + | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
| 673 | + | |
| 674 | + | |
| 675 | + | |
| 676 | + | |
| 677 | + | |
| 678 | + | |
| 679 | + | |
| 680 | + | |
577 | 681 | | |
578 | 682 | | |
579 | 683 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
351 | 351 | | |
352 | 352 | | |
353 | 353 | | |
354 | | - | |
355 | | - | |
356 | | - | |
357 | | - | |
358 | | - | |
359 | | - | |
360 | | - | |
361 | | - | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
362 | 362 | | |
363 | 363 | | |
364 | 364 | | |
| |||
Lines changed: 31 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| 7 | + | |
7 | 8 | | |
8 | 9 | | |
9 | 10 | | |
10 | 11 | | |
11 | 12 | | |
12 | 13 | | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
13 | 17 | | |
14 | 18 | | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
15 | 25 | | |
16 | 26 | | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
17 | 30 | | |
18 | 31 | | |
19 | 32 | | |
| |||
22 | 35 | | |
23 | 36 | | |
24 | 37 | | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
25 | 56 | | |
26 | 57 | | |
27 | 58 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
| 4 | + | |
4 | 5 | | |
5 | 6 | | |
6 | 7 | | |
| |||
231 | 232 | | |
232 | 233 | | |
233 | 234 | | |
| 235 | + | |
234 | 236 | | |
235 | 237 | | |
236 | 238 | | |
| |||
0 commit comments