[purview] add scanning client (Azure#17780)

* generate azure-purview-scanning

* basic head smoke test

* regenerate

* add get datasources smoke test

* add dev requirements

* regen with new public swagger and request builder ordering

* use powershell preparer for tests

* fix setup.py

* regenerate

* regenerate and add packaging files

* fix 2.7 tests (hopefully)

* add tuesday as release date

* add scanning to ci

* switch to new aka link for llc quickstart

* remove low level client reference

* remove more llc references

* AzurePurviewScanningClient -> PurviewScanningClient

* remove pkgutil init info from core and rest folders

Author: iscai-msft

sdk/purview/azure-purview-scanning/CHANGELOG.md

@@ -0,0 +1,5 @@
+# Release History
+
+## 1.0.0b1 (2021-05-11)
+
+- This is the initial release of the Azure Purview Scanning library.

sdk/purview/azure-purview-scanning/MANIFEST.in

@@ -0,0 +1,5 @@
+recursive-include tests *.py
+recursive-include samples *.py *.md
+include *.md
+include azure/__init__.py
+include azure/purview/__init__.py

sdk/purview/azure-purview-scanning/README.md

@@ -0,0 +1,172 @@
+# Azure Purview Scanning client library for Python
+
+Azure Purview Scanning is a fully managed cloud service whose users can scan your data into your data estate (also known as your **catalog**). Scanning is a process by which the catalog connects directly to a data source on a user-specified schedule.
+
+- Scan your data into your catalog
+- Examine your data
+- Extract schemas from your data
+
+**Please rely heavily on the [service's documentation][scanning_product_documentation] and our [client docs][request_builders_and_client] to use this library**
+
+[Source code][source_code] | [Package (PyPI)][scanning_pypi] | [API reference documentation][scanning_ref_docs]| [Product documentation][scanning_product_documentation]
+
+## Getting started
+
+### Prerequisites
+
+- Python 2.7, or 3.6 or later is required to use this package.
+- You must have an [Azure subscription][azure_subscription] and a [Purview][purview_resource] to use this package.
+
+#### Create a Purview Resource
+
+Follow [these][purview_resource] instructions to create your Purview resource
+
+### Install the package
+
+Install the Azure Purview Scanning client library for Python with [pip][pip]:
+
+```bash
+pip install azure-purview-scanning
+```
+
+### Authenticate the client
+
+To use an [Azure Active Directory (AAD) token credential][authenticate_with_token],
+provide an instance of the desired credential type obtained from the
+[azure-identity][azure_identity_credentials] library.
+
+To authenticate with AAD, you must first [pip][pip] install [`azure-identity`][azure_identity_pip] and
+[enable AAD authentication on your Purview resource][enable_aad]
+
+After setup, you can choose which type of [credential][azure_identity_credentials] from azure.identity to use.
+As an example, [DefaultAzureCredential][default_azure_credential]
+can be used to authenticate the client:
+
+Set the values of the client ID, tenant ID, and client secret of the AAD application as environment variables:
+AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_CLIENT_SECRET
+
+Use the returned token credential to authenticate the client:
+
+```python
+from azure.purview.scanning import PurviewScanningClient
+from azure.identity import DefaultAzureCredential
+
+credential = DefaultAzureCredential()
+client = PurviewScanningClient(endpoint="https://<my-account-name>.scanning.purview.azure.com", credential=credential)
+```
+
+## Key concepts
+
+### Client
+
+This package offers request builders so you can build http requests and send these requests to the service using the `send_request` method.
+For more information on how to use request builders and our clients, see [here][request_builders_and_client].
+
+## Examples
+
+The following section shows you how to initialize and authenticate your client, then list all of your data sources.
+
+- [List All Data Sources](#list-all-data-sources "List All Data Sources")
+
+### List All Data Sources
+
+```python
+from azure.purview.scanning import PurviewScanningClient
+from azure.identity import DefaultAzureCredential
+from azure.purview.scanning.rest import data_sources
+from azure.core.exceptions import HttpResponseError
+
+credential = DefaultAzureCredential()
+client = PurviewScanningClient(endpoint="https://<my-account-name>.scanning.purview.azure.com", credential=credential)
+
+request = data_sources.build_list_all_request()
+
+response = client.send_request(request)
+try:
+    response.raise_for_status()
+    json_response = response.json()
+
+    assert len(json_response['value']) == json_response['count']
+    for value in json_response['value']:
+        print(value)
+
+except HttpResponseError as e:
+    print(e)
+```
+
+## Troubleshooting
+
+### General
+
+The Purview Scanning client will raise exceptions defined in [Azure Core][azure_core] if you call `.raise_for_status()` on your responses.
+
+### Logging
+
+This library uses the standard
+[logging][python_logging] library for logging.
+Basic information about HTTP sessions (URLs, headers, etc.) is logged at INFO
+level.
+
+Detailed DEBUG level logging, including request/response bodies and unredacted
+headers, can be enabled on a client with the `logging_enable` keyword argument:
+
+```python
+import sys
+import logging
+from azure.identity import DefaultAzureCredential
+from azure.purview.scanning import PurviewScanningClient
+
+# Create a logger for the 'azure' SDK
+logger = logging.getLogger('azure')
+logger.setLevel(logging.DEBUG)
+
+# Configure a console output
+handler = logging.StreamHandler(stream=sys.stdout)
+logger.addHandler(handler)
+
+endpoint = "https://<my-account-name>.scanning.purview.azure.com"
+credential = DefaultAzureCredential()
+
+# This client will log detailed information about its HTTP sessions, at DEBUG level
+client = PurviewScanningClient(endpoint=endpoint, credential=credential, logging_enable=True)
+```
+
+Similarly, `logging_enable` can enable detailed logging for a single `send_request` call,
+even when it isn't enabled for the client:
+
+```python
+result = client.send_request(request, logging_enable=True)
+```
+
+## Next steps
+
+For more generic samples, see our [client docs][request_builders_and_client].
+
+## Contributing
+
+This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit [cla.microsoft.com][cla].
+
+When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.
+
+This project has adopted the [Microsoft Open Source Code of Conduct][code_of_conduct]. For more information see the [Code of Conduct FAQ][coc_faq] or contact [opencode@microsoft.com][coc_contact] with any additional questions or comments.
+
+<!-- LINKS -->
+
+[source_code]: https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/purview/azure-purview-scanning/azure/purview/scanning
+[scanning_pypi]: https://aka.ms/azsdk/python/purviewscanning/pypi
+[scanning_ref_docs]: https://aka.ms/azsdk/python/purviewscanning/ref-docs
+[scanning_product_documentation]: https://azure.microsoft.com/services/purview/
+[azure_subscription]: https://azure.microsoft.com/free/
+[purview_resource]: https://docs.microsoft.com/azure/purview/create-catalog-portal
+[pip]: https://pypi.org/project/pip/
+[authenticate_with_token]: https://docs.microsoft.com/azure/cognitive-services/authentication?tabs=powershell#authenticate-with-an-authentication-token
+[azure_identity_credentials]: https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/identity/azure-identity#credentials
+[azure_identity_pip]: https://pypi.org/project/azure-identity/
+[default_azure_credential]: https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/identity/azure-identity#defaultazurecredential
+[request_builders_and_client]: https://aka.ms/azsdk/python/protocol/quickstart
+[enable_aad]: https://docs.microsoft.com/azure/purview/create-catalog-portal#add-a-security-principal-to-a-data-plane-role
+[python_logging]: https://docs.python.org/3.5/library/logging.html
+[cla]: https://cla.microsoft.com
+[code_of_conduct]: https://opensource.microsoft.com/codeofconduct/
+[coc_faq]: https://opensource.microsoft.com/codeofconduct/faq/
+[coc_contact]: mailto:opencode@microsoft.com

sdk/purview/azure-purview-scanning/azure/__init__.py

@@ -0,0 +1 @@
+__path__ = __import__('pkgutil').extend_path(__path__, __name__)  # type: ignore

sdk/purview/azure-purview-scanning/azure/purview/__init__.py

@@ -0,0 +1 @@
+__path__ = __import__('pkgutil').extend_path(__path__, __name__)  # type: ignore

sdk/purview/azure-purview-scanning/azure/purview/scanning/__init__.py

@@ -0,0 +1,19 @@
+# coding=utf-8
+# --------------------------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License. See License.txt in the project root for license information.
+# Code generated by Microsoft (R) AutoRest Code Generator.
+# Changes may cause incorrect behavior and will be lost if the code is regenerated.
+# --------------------------------------------------------------------------
+
+from ._azure_purview_scanning_client import PurviewScanningClient
+from ._version import VERSION
+
+__version__ = VERSION
+__all__ = ['PurviewScanningClient']
+
+try:
+    from ._patch import patch_sdk  # type: ignore
+    patch_sdk()
+except ImportError:
+    pass

sdk/purview/azure-purview-scanning/azure/purview/scanning/_azure_purview_scanning_client.py

@@ -0,0 +1,104 @@
+# coding=utf-8
+# --------------------------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License. See License.txt in the project root for license information.
+# Code generated by Microsoft (R) AutoRest Code Generator.
+# Changes may cause incorrect behavior and will be lost if the code is regenerated.
+# --------------------------------------------------------------------------
+
+from copy import deepcopy
+from typing import TYPE_CHECKING
+
+from azure.core import PipelineClient
+from azure.purview.scanning.core.rest import HttpResponse, _StreamContextManager
+from msrest import Deserializer, Serializer
+
+if TYPE_CHECKING:
+    # pylint: disable=unused-import,ungrouped-imports
+    from typing import Any, Dict
+
+    from azure.core.credentials import TokenCredential
+    from azure.purview.scanning.core.rest import HttpRequest
+
+from ._configuration import PurviewScanningClientConfiguration
+
+
+class PurviewScanningClient(object):
+    """Creates a Microsoft.Scanning management client.
+
+    :param credential: Credential needed for the client to connect to Azure.
+    :type credential: ~azure.core.credentials.TokenCredential
+    :param endpoint: The scanning endpoint of your purview account. Example: https://{accountName}.scan.purview.azure.com.
+    :type endpoint: str
+    """
+
+    def __init__(
+        self,
+        credential,  # type: "TokenCredential"
+        endpoint,  # type: str
+        **kwargs  # type: Any
+    ):
+        # type: (...) -> None
+        base_url = '{Endpoint}'
+        self._config = PurviewScanningClientConfiguration(credential, endpoint, **kwargs)
+        self._client = PipelineClient(base_url=base_url, config=self._config, **kwargs)
+
+        self._serialize = Serializer()
+        self._deserialize = Deserializer()
+        self._serialize.client_side_validation = False
+
+    def send_request(self, http_request, **kwargs):
+        # type: (HttpRequest, Any) -> HttpResponse
+        """Runs the network request through the client's chained policies.
+
+        We have helper methods to create requests specific to this service in `azure.purview.scanning.rest`.
+        Use these helper methods to create the request you pass to this method. See our example below:
+
+        >>> from azure.purview.scanning.rest import build_get_request
+        >>> request = build_get_request(key_vault_name)
+        <HttpRequest [GET], url: '/azureKeyVaults/{keyVaultName}'>
+        >>> response = client.send_request(request)
+        <HttpResponse: 200 OK>
+
+        For more information on this code flow, see https://aka.ms/azsdk/python/protocol/quickstart
+
+        For advanced cases, you can also create your own :class:`~azure.purview.scanning.core.rest.HttpRequest`
+        and pass it in.
+
+        :param http_request: The network request you want to make. Required.
+        :type http_request: ~azure.purview.scanning.core.rest.HttpRequest
+        :keyword bool stream_response: Whether the response payload will be streamed. Defaults to False.
+        :return: The response of your network call. Does not do error handling on your response.
+        :rtype: ~azure.purview.scanning.core.rest.HttpResponse
+        """
+        request_copy = deepcopy(http_request)
+        path_format_arguments = {
+            'Endpoint': self._serialize.url("self._config.endpoint", self._config.endpoint, 'str', skip_quote=True),
+        }
+        request_copy.url = self._client.format_url(request_copy.url, **path_format_arguments)
+        if kwargs.pop("stream_response", False):
+            return _StreamContextManager(
+                client=self._client._pipeline,
+                request=request_copy,
+            )
+        pipeline_response = self._client._pipeline.run(request_copy._internal_request, **kwargs)
+        response = HttpResponse(
+            status_code=pipeline_response.http_response.status_code,
+            request=request_copy,
+            _internal_response=pipeline_response.http_response
+        )
+        response.read()
+        return response
+
+    def close(self):
+        # type: () -> None
+        self._client.close()
+
+    def __enter__(self):
+        # type: () -> PurviewScanningClient
+        self._client.__enter__()
+        return self
+
+    def __exit__(self, *exc_details):
+        # type: (Any) -> None
+        self._client.__exit__(*exc_details)

sdk/purview/azure-purview-scanning/azure/purview/scanning/_configuration.py

@@ -0,0 +1,70 @@
+# coding=utf-8
+# --------------------------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License. See License.txt in the project root for license information.
+# Code generated by Microsoft (R) AutoRest Code Generator.
+# Changes may cause incorrect behavior and will be lost if the code is regenerated.
+# --------------------------------------------------------------------------
+
+from typing import TYPE_CHECKING
+
+from azure.core.configuration import Configuration
+from azure.core.pipeline import policies
+
+from ._version import VERSION
+
+if TYPE_CHECKING:
+    # pylint: disable=unused-import,ungrouped-imports
+    from typing import Any
+
+    from azure.core.credentials import TokenCredential
+
+
+class PurviewScanningClientConfiguration(Configuration):
+    """Configuration for PurviewScanningClient.
+
+    Note that all parameters used to create this instance are saved as instance
+    attributes.
+
+    :param credential: Credential needed for the client to connect to Azure.
+    :type credential: ~azure.core.credentials.TokenCredential
+    :param endpoint: The scanning endpoint of your purview account. Example: https://{accountName}.scan.purview.azure.com.
+    :type endpoint: str
+    """
+
+    def __init__(
+        self,
+        credential,  # type: "TokenCredential"
+        endpoint,  # type: str
+        **kwargs  # type: Any
+    ):
+        # type: (...) -> None
+        if credential is None:
+            raise ValueError("Parameter 'credential' must not be None.")
+        if endpoint is None:
+            raise ValueError("Parameter 'endpoint' must not be None.")
+        super(PurviewScanningClientConfiguration, self).__init__(**kwargs)
+
+        self.credential = credential
+        self.endpoint = endpoint
+        self.api_version = "2018-12-01-preview"
+        self.credential_scopes = kwargs.pop('credential_scopes', ['https://purview.azure.net/.default'])
+        kwargs.setdefault('sdk_moniker', 'purview-scanning/{}'.format(VERSION))
+        self._configure(**kwargs)
+
+    def _configure(
+        self,
+        **kwargs  # type: Any
+    ):
+        # type: (...) -> None
+        self.user_agent_policy = kwargs.get('user_agent_policy') or policies.UserAgentPolicy(**kwargs)
+        self.headers_policy = kwargs.get('headers_policy') or policies.HeadersPolicy(**kwargs)
+        self.proxy_policy = kwargs.get('proxy_policy') or policies.ProxyPolicy(**kwargs)
+        self.logging_policy = kwargs.get('logging_policy') or policies.NetworkTraceLoggingPolicy(**kwargs)
+        self.http_logging_policy = kwargs.get('http_logging_policy') or policies.HttpLoggingPolicy(**kwargs)
+        self.retry_policy = kwargs.get('retry_policy') or policies.RetryPolicy(**kwargs)
+        self.custom_hook_policy = kwargs.get('custom_hook_policy') or policies.CustomHookPolicy(**kwargs)
+        self.redirect_policy = kwargs.get('redirect_policy') or policies.RedirectPolicy(**kwargs)
+        self.authentication_policy = kwargs.get('authentication_policy')
+        if self.credential and not self.authentication_policy:
+            self.authentication_policy = policies.BearerTokenCredentialPolicy(self.credential, *self.credential_scopes, **kwargs)

sdk/purview/azure-purview-scanning/azure/purview/scanning/_version.py

@@ -0,0 +1,9 @@
+# coding=utf-8
+# --------------------------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License. See License.txt in the project root for license information.
+# Code generated by Microsoft (R) AutoRest Code Generator.
+# Changes may cause incorrect behavior and will be lost if the code is regenerated.
+# --------------------------------------------------------------------------
+
+VERSION = "1.0.0b1"