Skip to content

Commit 1ddfc5d

Browse files
authored
Switched to using credential types instead of strings. Support for Azurite (Azure#19319)
* Switched to using credential types instead of strings. Support for washington * Updated some docs * Removed unused method * renamed constant * Fix exception message * Fixed readme samples
1 parent f0b4e81 commit 1ddfc5d

File tree

10 files changed

+112
-117
lines changed

10 files changed

+112
-117
lines changed

sdk/storage/azure-storage-blob-nio/CHANGELOG.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,8 @@
11
# Release History
22

33
## 12.0.0-beta.4 (Unreleased)
4-
4+
- Added support for Azurite
5+
- Change FileSystem configuration to accept an endpoint and credential types instead of a string for the account name, key, and token
56

67
## 12.0.0-beta.3 (2021-02-10)
78
- Added support for FileSystemProvider.checkAccess method

sdk/storage/azure-storage-blob-nio/README.md

Lines changed: 15 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -146,20 +146,21 @@ Create a `FileSystem` using the [`shared key`](#get-credentials) retrieved above
146146
Note that you can further configure the file system using constants available in `AzureFileSystem`.
147147
Please see the docs for `AzureFileSystemProvider` for a full explanation of initializing and configuring a filesystem
148148

149-
<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L39-L43 -->
149+
<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L40-L45 -->
150150
```java
151151
Map<String, Object> config = new HashMap<>();
152-
String stores = "<your_container_name>,<another_container_name>"; // A comma separated list of container names
153-
config.put(AzureFileSystem.AZURE_STORAGE_ACCOUNT_KEY, "<your_account_key>");
152+
String stores = "<container_name>,<another_container_name>"; // A comma separated list of container names
153+
StorageSharedKeyCredential credential = new StorageSharedKeyCredential("<account_name", "account_key");
154+
config.put(AzureFileSystem.AZURE_STORAGE_SHARED_KEY_CREDENTIAL, credential);
154155
config.put(AzureFileSystem.AZURE_STORAGE_FILE_STORES, stores);
155-
FileSystem myFs = FileSystems.newFileSystem(new URI("azb://?account=<your_account_name"), config);
156+
FileSystem myFs = FileSystems.newFileSystem(new URI("azb://?endpoint=<account_endpoint"), config);
156157
```
157158

158159
### Create a directory
159160

160161
Create a directory using the `Files` api
161162

162-
<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L47-L48 -->
163+
<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L49-L50 -->
163164
```java
164165
Path dirPath = myFs.getPath("dir");
165166
Files.createDirectory(dirPath);
@@ -169,7 +170,7 @@ Files.createDirectory(dirPath);
169170

170171
Iterate over a directory using a `DirectoryStream`
171172

172-
<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L52-L54 -->
173+
<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L54-L56 -->
173174
```java
174175
for (Path p : Files.newDirectoryStream(dirPath)) {
175176
System.out.println(p.toString());
@@ -180,7 +181,7 @@ for (Path p : Files.newDirectoryStream(dirPath)) {
180181

181182
Read the contents of a file using an `InputStream`. Skipping, marking, and resetting are all supported.
182183

183-
<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L58-L61 -->
184+
<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L60-L63 -->
184185
```java
185186
Path filePath = myFs.getPath("file");
186187
InputStream is = Files.newInputStream(filePath);
@@ -193,7 +194,7 @@ is.close();
193194
Write to a file. Only writing whole files is supported. Random IO is not supported. The stream must be closed in order
194195
to guarantee that the data is available to be read.
195196

196-
<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L65-L67 -->
197+
<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L67-L69 -->
197198
```java
198199
OutputStream os = Files.newOutputStream(filePath);
199200
os.write(0);
@@ -202,15 +203,15 @@ os.close();
202203

203204
### Copy a file
204205

205-
<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L71-L72 -->
206+
<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L73-L74 -->
206207
```java
207208
Path destinationPath = myFs.getPath("destinationFile");
208209
Files.copy(filePath, destinationPath, StandardCopyOption.COPY_ATTRIBUTES);
209210
```
210211

211212
### Delete a file
212213

213-
<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L76-L76 -->
214+
<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L78-L78 -->
214215
```java
215216
Files.delete(filePath);
216217
```
@@ -219,7 +220,7 @@ Files.delete(filePath);
219220

220221
Read attributes of a file through the `AzureBlobFileAttributes`.
221222

222-
<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L80-L81 -->
223+
<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L82-L83 -->
223224
```java
224225
AzureBlobFileAttributes attr = Files.readAttributes(filePath, AzureBlobFileAttributes.class);
225226
BlobHttpHeaders headers = attr.blobHttpHeaders();
@@ -229,7 +230,7 @@ Or read attributes dynamically by specifying a string of desired attributes. Thi
229230
to retrieve any attribute will always retrieve all of them as an atomic bulk operation. You may specify "*" instead of a
230231
list of specific attributes to have all attributes returned in the map.
231232

232-
<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L85-L85 -->
233+
<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L87-L87 -->
233234
```java
234235
Map<String, Object> attributes = Files.readAttributes(filePath, "azureBlob:metadata,headers");
235236
```
@@ -238,15 +239,15 @@ Map<String, Object> attributes = Files.readAttributes(filePath, "azureBlob:metad
238239

239240
Set attributes of a file through the `AzureBlobFileAttributeView`.
240241

241-
<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L89-L90 -->
242+
<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L91-L92 -->
242243
```java
243244
AzureBlobFileAttributeView view = Files.getFileAttributeView(filePath, AzureBlobFileAttributeView.class);
244245
view.setMetadata(Collections.EMPTY_MAP);
245246
```
246247

247248
Or set an attribute dynamically by specifying the attribute as a string.
248249

249-
<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L94-L94 -->
250+
<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L96-L96 -->
250251
```java
251252
Files.setAttribute(filePath, "azureBlob:blobHttpHeaders", new BlobHttpHeaders());
252253
```

sdk/storage/azure-storage-blob-nio/src/main/java/com/azure/storage/blob/nio/AzureFileSystem.java

Lines changed: 17 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33

44
package com.azure.storage.blob.nio;
55

6+
import com.azure.core.credential.AzureSasCredential;
67
import com.azure.core.http.HttpClient;
78
import com.azure.core.http.policy.HttpLogDetailLevel;
89
import com.azure.core.http.policy.HttpPipelinePolicy;
@@ -59,12 +60,12 @@ public final class AzureFileSystem extends FileSystem {
5960
/**
6061
* Expected type: String
6162
*/
62-
public static final String AZURE_STORAGE_ACCOUNT_KEY = "AzureStorageAccountKey";
63+
public static final String AZURE_STORAGE_SHARED_KEY_CREDENTIAL = "AzureStorageSharedKeyCredential";
6364

6465
/**
6566
* Expected type: String
6667
*/
67-
public static final String AZURE_STORAGE_SAS_TOKEN = "AzureStorageSasToken";
68+
public static final String AZURE_STORAGE_SAS_TOKEN_CREDENTIAL = "AzureStorageSasTokenCredential";
6869

6970
/**
7071
* Expected type: com.azure.core.http.policy.HttpLogLevelDetail
@@ -121,11 +122,6 @@ public final class AzureFileSystem extends FileSystem {
121122
*/
122123
public static final String AZURE_STORAGE_DOWNLOAD_RESUME_RETRIES = "AzureStorageDownloadResumeRetries";
123124

124-
/**
125-
* Expected type: Boolean
126-
*/
127-
public static final String AZURE_STORAGE_USE_HTTPS = "AzureStorageUseHttps";
128-
129125
static final String AZURE_STORAGE_HTTP_CLIENT = "AzureStorageHttpClient"; // undocumented; for test.
130126
static final String AZURE_STORAGE_HTTP_POLICIES = "AzureStorageHttpPolicies"; // undocumented; for test.
131127

@@ -136,8 +132,6 @@ public final class AzureFileSystem extends FileSystem {
136132

137133
static final String PATH_SEPARATOR = "/";
138134

139-
private static final String AZURE_STORAGE_BLOB_ENDPOINT_TEMPLATE = "%s://%s.blob.core.windows.net";
140-
141135
static final Map<Class<? extends FileAttributeView>, String> SUPPORTED_ATTRIBUTE_VIEWS;
142136
static {
143137
Map<Class<? extends FileAttributeView>, String> map = new HashMap<>();
@@ -157,7 +151,7 @@ public final class AzureFileSystem extends FileSystem {
157151
private FileStore defaultFileStore;
158152
private boolean closed;
159153

160-
AzureFileSystem(AzureFileSystemProvider parentFileSystemProvider, String accountName, Map<String, ?> config)
154+
AzureFileSystem(AzureFileSystemProvider parentFileSystemProvider, String endpoint, Map<String, ?> config)
161155
throws IOException {
162156
// A FileSystem should only ever be instantiated by a provider.
163157
if (Objects.isNull(parentFileSystemProvider)) {
@@ -168,7 +162,7 @@ public final class AzureFileSystem extends FileSystem {
168162

169163
// Read configurations and build client.
170164
try {
171-
this.blobServiceClient = this.buildBlobServiceClient(accountName, config);
165+
this.blobServiceClient = this.buildBlobServiceClient(endpoint, config);
172166
this.blockSize = (Long) config.get(AZURE_STORAGE_UPLOAD_BLOCK_SIZE);
173167
this.putBlobThreshold = (Long) config.get(AZURE_STORAGE_PUT_BLOB_THRESHOLD);
174168
this.maxConcurrencyPerRequest = (Integer) config.get(AZURE_STORAGE_MAX_CONCURRENCY_PER_REQUEST);
@@ -215,7 +209,7 @@ public FileSystemProvider provider() {
215209
@Override
216210
public void close() throws IOException {
217211
this.closed = true;
218-
this.parentFileSystemProvider.closeFileSystem(this.getFileSystemName());
212+
this.parentFileSystemProvider.closeFileSystem(this.getFileSystemUrl());
219213
}
220214

221215
/**
@@ -374,32 +368,27 @@ public WatchService newWatchService() throws IOException {
374368
throw LoggingUtility.logError(logger, new UnsupportedOperationException());
375369
}
376370

377-
String getFileSystemName() {
378-
return this.blobServiceClient.getAccountName();
371+
String getFileSystemUrl() {
372+
return this.blobServiceClient.getAccountUrl();
379373
}
380374

381375
BlobServiceClient getBlobServiceClient() {
382376
return this.blobServiceClient;
383377
}
384378

385-
private BlobServiceClient buildBlobServiceClient(String accountName, Map<String, ?> config) {
386-
// Build the endpoint.
387-
String scheme = !config.containsKey(AZURE_STORAGE_USE_HTTPS)
388-
|| (Boolean) config.get(AZURE_STORAGE_USE_HTTPS)
389-
? "https" : "http";
379+
private BlobServiceClient buildBlobServiceClient(String endpoint, Map<String, ?> config) {
390380
BlobServiceClientBuilder builder = new BlobServiceClientBuilder()
391-
.endpoint(String.format(AZURE_STORAGE_BLOB_ENDPOINT_TEMPLATE, scheme, accountName));
381+
.endpoint(endpoint);
392382

393383
// Set the credentials.
394-
if (config.containsKey(AZURE_STORAGE_ACCOUNT_KEY)) {
395-
builder.credential(new StorageSharedKeyCredential(accountName,
396-
(String) config.get(AZURE_STORAGE_ACCOUNT_KEY)));
397-
} else if (config.containsKey(AZURE_STORAGE_SAS_TOKEN)) {
398-
builder.sasToken((String) config.get(AZURE_STORAGE_SAS_TOKEN));
384+
if (config.containsKey(AZURE_STORAGE_SHARED_KEY_CREDENTIAL)) {
385+
builder.credential((StorageSharedKeyCredential) config.get(AZURE_STORAGE_SHARED_KEY_CREDENTIAL));
386+
} else if (config.containsKey(AZURE_STORAGE_SAS_TOKEN_CREDENTIAL)) {
387+
builder.credential((AzureSasCredential) config.get(AZURE_STORAGE_SAS_TOKEN_CREDENTIAL));
399388
} else {
400389
throw LoggingUtility.logError(logger, new IllegalArgumentException(String.format("No credentials were "
401390
+ "provided. Please specify one of the following when constructing an AzureFileSystem: %s, %s.",
402-
AZURE_STORAGE_ACCOUNT_KEY, AZURE_STORAGE_SAS_TOKEN)));
391+
AZURE_STORAGE_SHARED_KEY_CREDENTIAL, AZURE_STORAGE_SAS_TOKEN_CREDENTIAL)));
403392
}
404393

405394
// Configure options and client.
@@ -452,12 +441,12 @@ public boolean equals(Object o) {
452441
return false;
453442
}
454443
AzureFileSystem that = (AzureFileSystem) o;
455-
return Objects.equals(this.getFileSystemName(), that.getFileSystemName());
444+
return Objects.equals(this.getFileSystemUrl(), that.getFileSystemUrl());
456445
}
457446

458447
@Override
459448
public int hashCode() {
460-
return Objects.hash(this.getFileSystemName());
449+
return Objects.hash(this.getFileSystemUrl());
461450
}
462451

463452
Path getDefaultDirectory() {

sdk/storage/azure-storage-blob-nio/src/main/java/com/azure/storage/blob/nio/AzureFileSystemProvider.java

Lines changed: 28 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -81,8 +81,8 @@
8181
* {@link FileSystemProvider}.
8282
* <p>
8383
* The scheme for this provider is {@code "azb"}, and the format of the URI to identify an {@code AzureFileSystem} is
84-
* {@code "azb://?account=<accountName>"}. The name of the Storage account is used to uniquely identify the file
85-
* system.
84+
* {@code "azb://?endpoint=<endpoing>"}. The endpoint of the Storage account is used to uniquely identify the
85+
* filesystem.
8686
* <p>
8787
* An {@link AzureFileSystem} is backed by an account. An {@link AzureFileStore} is backed by a container. Any number of
8888
* containers may be specified as file stores upon creation of the file system. When a file system is created,
@@ -96,8 +96,8 @@
9696
* types. Any entries not listed here will be ignored. Note that {@link AzureFileSystem} has public constants defined
9797
* for each of the keys for convenience.
9898
* <ul>
99-
* <li>{@code AzureStorageAccountKey:}{@link String}</li>
100-
* <li>{@code AzureStorageSasToken:}{@link String}</li>
99+
* <li>{@code AzureStorageSharedKeyCredential:}{@link com.azure.storage.common.StorageSharedKeyCredential}</li>
100+
* <li>{@code AzureStorageSasTokenCredential:}{@link com.azure.core.credential.AzureSasCredential}</li>
101101
* <li>{@code AzureStorageHttpLogDetailLevel:}{@link com.azure.core.http.policy.HttpLogDetailLevel}</li>
102102
* <li>{@code AzureStorageMaxTries:}{@link Integer}</li>
103103
* <li>{@code AzureStorageTryTimeout:}{@link Integer}</li>
@@ -110,15 +110,13 @@
110110
* <li>{@code AzureStoragePutBlobThreshold:}{@link Long}</li>
111111
* <li>{@code AzureStorageMaxConcurrencyPerRequest:}{@link Integer}</li>
112112
* <li>{@code AzureStorageDownloadResumeRetries:}{@link Integer}</li>
113-
* <li>{@code AzureStorageUseHttps:}{@link Boolean}</li>
114113
* <li>{@code AzureStorageFileStores:}{@link String}</li>
115114
* </ul>
116115
* <p>
117116
* Either an account key or a sas token must be specified. If both are provided, the account key will be preferred. If
118117
* a sas token is specified, the customer must take care that it has appropriate permissions to perform the actions
119-
* demanded of the file system in a given workflow, including the initial connection check specified above. Furthermore,
120-
* it must have an expiry time that lasts at least until the file system is closed as there is no token refresh offered
121-
* at this time. The same token will be applied to all containers.
118+
* demanded of the file system in a given workflow, including the initial connection check specified above. The same
119+
* token will be applied to all operations.
122120
* <p>
123121
* An iterable of file stores must also be provided; each entry should simply be the name of a container. The first
124122
* container listed will be considered the default file store and the root directory of which will be the file system's
@@ -160,7 +158,7 @@ public final class AzureFileSystemProvider extends FileSystemProvider {
160158
*/
161159
public static final String CACHE_CONTROL = "Cache-Control";
162160

163-
private static final String ACCOUNT_QUERY_KEY = "account";
161+
private static final String ENDPOINT_QUERY_KEY = "endpoint";
164162
private static final int COPY_TIMEOUT_SECONDS = 30;
165163
private static final Set<OpenOption> OUTPUT_STREAM_DEFAULT_OPTIONS =
166164
Collections.unmodifiableSet(new HashSet<>(Arrays.asList(StandardOpenOption.CREATE,
@@ -198,7 +196,7 @@ public String getScheme() {
198196
/**
199197
* Constructs a new FileSystem object identified by a URI.
200198
* <p>
201-
* The format of a {@code URI} identifying a file system is {@code "azb://?account=<accountName>"}.
199+
* The format of a {@code URI} identifying a file system is {@code "azb://?endpoint=<endpoint>"}.
202200
* <p>
203201
* Once closed, a file system with the same identifier may be reopened.
204202
*
@@ -213,22 +211,22 @@ public String getScheme() {
213211
*/
214212
@Override
215213
public FileSystem newFileSystem(URI uri, Map<String, ?> config) throws IOException {
216-
String accountName = extractAccountName(uri);
214+
String endpoint = extractAccountEndpoint(uri);
217215

218-
if (this.openFileSystems.containsKey(accountName)) {
219-
throw LoggingUtility.logError(this.logger, new FileSystemAlreadyExistsException("Name: " + accountName));
216+
if (this.openFileSystems.containsKey(endpoint)) {
217+
throw LoggingUtility.logError(this.logger, new FileSystemAlreadyExistsException("Name: " + endpoint));
220218
}
221219

222-
AzureFileSystem afs = new AzureFileSystem(this, accountName, config);
223-
this.openFileSystems.put(accountName, afs);
220+
AzureFileSystem afs = new AzureFileSystem(this, endpoint, config);
221+
this.openFileSystems.put(endpoint, afs);
224222

225223
return afs;
226224
}
227225

228226
/**
229227
* Returns an existing FileSystem created by this provider.
230228
* <p>
231-
* The format of a {@code URI} identifying an file system is {@code "azb://?account=&lt;accountName&gt;"}.
229+
* The format of a {@code URI} identifying an file system is {@code "azb://?endpoint=&lt;endpoint&gt;"}.
232230
* <p>
233231
* Trying to retrieve a closed file system will throw a {@link FileSystemNotFoundException}. Once closed, a
234232
* file system with the same identifier may be reopened.
@@ -241,11 +239,11 @@ public FileSystem newFileSystem(URI uri, Map<String, ?> config) throws IOExcepti
241239
*/
242240
@Override
243241
public FileSystem getFileSystem(URI uri) {
244-
String accountName = extractAccountName(uri);
245-
if (!this.openFileSystems.containsKey(accountName)) {
246-
throw LoggingUtility.logError(this.logger, new FileSystemNotFoundException("Name: " + accountName));
242+
String endpoint = extractAccountEndpoint(uri);
243+
if (!this.openFileSystems.containsKey(endpoint)) {
244+
throw LoggingUtility.logError(this.logger, new FileSystemNotFoundException("Name: " + endpoint));
247245
}
248-
return this.openFileSystems.get(accountName);
246+
return this.openFileSystems.get(endpoint);
249247
}
250248

251249
/**
@@ -1133,29 +1131,29 @@ void closeFileSystem(String fileSystemName) {
11331131
this.openFileSystems.remove(fileSystemName);
11341132
}
11351133

1136-
private String extractAccountName(URI uri) {
1134+
private String extractAccountEndpoint(URI uri) {
11371135
if (!uri.getScheme().equals(this.getScheme())) {
11381136
throw LoggingUtility.logError(this.logger, new IllegalArgumentException(
11391137
"URI scheme does not match this provider"));
11401138
}
11411139
if (CoreUtils.isNullOrEmpty(uri.getQuery())) {
11421140
throw LoggingUtility.logError(this.logger, new IllegalArgumentException("URI does not contain a query "
1143-
+ "component. FileSystems require a URI of the format \"azb://?account=<account_name>\"."));
1141+
+ "component. FileSystems require a URI of the format \"azb://?endpoint=<account_endpoint>\"."));
11441142
}
11451143

1146-
String accountName = Flux.fromArray(uri.getQuery().split("&"))
1147-
.filter(s -> s.startsWith(ACCOUNT_QUERY_KEY + "="))
1144+
String endpoint = Flux.fromArray(uri.getQuery().split("&"))
1145+
.filter(s -> s.startsWith(ENDPOINT_QUERY_KEY + "="))
11481146
.switchIfEmpty(Mono.error(LoggingUtility.logError(this.logger, new IllegalArgumentException(
1149-
"URI does not contain an \"" + ACCOUNT_QUERY_KEY + "=\" parameter. FileSystems require a URI "
1150-
+ "of the format \"azb://?account=<account_name>\""))))
1151-
.map(s -> s.substring(ACCOUNT_QUERY_KEY.length() + 1))
1147+
"URI does not contain an \"" + ENDPOINT_QUERY_KEY + "=\" parameter. FileSystems require a URI "
1148+
+ "of the format \"azb://?endpoint=<endpoint>\""))))
1149+
.map(s -> s.substring(ENDPOINT_QUERY_KEY.length() + 1)) // Trim the query key and =
11521150
.blockLast();
11531151

1154-
if (CoreUtils.isNullOrEmpty(accountName)) {
1155-
throw LoggingUtility.logError(logger, new IllegalArgumentException("No account name provided in URI"
1152+
if (CoreUtils.isNullOrEmpty(endpoint)) {
1153+
throw LoggingUtility.logError(logger, new IllegalArgumentException("No account endpoint provided in URI"
11561154
+ " query."));
11571155
}
11581156

1159-
return accountName;
1157+
return endpoint;
11601158
}
11611159
}

0 commit comments

Comments
 (0)