[Monitor][Ingestion] Update samples (Azure#28466)

Added a samples README to give users more information
regarding on how to set up environment variables and run
the samples.

Additional samples for uploading file contents and uploading
data from a pandas DataFrame were added.

Signed-off-by: Paul Van Eck <paulvaneck@microsoft.com>

Author: pvaneck

sdk/monitor/azure-monitor-ingestion/README.md

@@ -136,12 +136,16 @@ except HttpResponseError as e:
 To upload logs with custom error handling, you can pass a callback function to the `on_error` parameter of the `upload` method. The callback function will be called for each error that occurs during the upload and should expect one argument that corresponds to an `UploadLogsError` object. This object contains the error encountered and the list of logs that failed to upload.
 
 ```python
+# Example 1: Collect all logs that failed to upload.
 failed_logs = []
 def on_error(error):
     print("Log chunk failed to upload with error: ", error.error)
-    # Collect all logs that failed to upload.
     failed_logs.extend(error.failed_logs)
 
+# Example 2: Ignore all errors.
+def on_error_pass(error):
+    pass
+
 client.upload(rule_id=rule_id, stream_name=os.environ['LOGS_DCR_STREAM_NAME'], logs=body, on_error=on_error)
 ```
 

sdk/monitor/azure-monitor-ingestion/samples/README.md

@@ -0,0 +1,92 @@
+---
+page_type: sample
+languages:
+  - python
+products:
+    - azure
+    - azure-monitor
+urlFragment: ingestion-azuremonitor-samples
+---
+
+# Azure Monitor Ingestion client library Python samples
+
+This library allows you to send data from virtually any source to supported built-in tables or to custom tables that you create in Log Analytics workspaces. The following code samples show common scenarios with the Azure Monitor Ingestion client library.
+
+|**File Name**|**Description**|
+|-------------|---------------|
+|[sample_send_small_logs.py][sample_send_small_logs] and [sample_send_small_logs_async.py][sample_send_small_logs_async]|Send a small number of logs to a Log Analytics workspace.|
+|[sample_custom_error_callback.py][sample_custom_error_callback] and [sample_custom_error_callback_async.py][sample_custom_error_callback_async]|Use error callbacks to customize how errors are handled during upload. |
+|[sample_upload_file_contents.py][sample_upload_file_contents] and [sample_upload_file_contents_async.py][sample_upload_file_contents_async]|Upload the contents of a file to a Log Analytics workspace.|
+|[sample_upload_pandas_dataframe.py][sample_upload_pandas_dataframe] and [sample_upload_pandas_dataframe_async.py][sample_upload_pandas_dataframe_async]|Upload data in a pandas DataFrame to a Log Analytics workspace.|
+
+## Prerequisites
+
+- Python 3.7 or later
+- An [Azure subscription][azure_subscription]
+- An [Azure Log Analytics workspace][azure_monitor_create_using_portal]
+- A [Data Collection Endpoint (DCE)][data_collection_endpoint]
+- A [Data Collection Rule (DCR)][data_collection_rule]
+
+## How to run the samples
+
+### Install the dependencies
+
+To run the samples, you need to install the following dependencies:
+```bash
+pip install azure-monitor-ingestion azure-identity pandas
+```
+
+To run the async samples, you need an asynchronous HTTP framework like `aiohttp`:
+
+```bash
+pip install aiohttp
+```
+
+### Set up authentication
+
+We use [azure-identity][azure_identity]'s [DefaultAzureCredential][azure_identity_default_azure_credential] to authenticate. Ensure that your service principal or managed identity has the `Monitoring Metrics Publisher` role assigned on the Data Collection Rule resource. If you are using a service principal, set the following environment variables:
+
+```bash
+AZURE_TENANT_ID="your Azure AD tenant (directory) ID"
+AZURE_CLIENT_ID="your Azure AD client (application) ID"
+AZURE_CLIENT_SECRET="your Azure AD client secret"
+```
+
+### Set up additional environment variables
+
+Change and set the following environment variables to match your configuration:
+
+```bash
+DATA_COLLECTION_ENDPOINT="your data collection endpoint"
+LOGS_DCR_RULE_ID="your data collection rule immutable ID"
+LOGS_DCR_STREAM_NAME="your data collection rule stream name"
+```
+
+### Run the samples
+
+Navigate to the directory that the samples are saved in, and follow the usage described in the file. For example, `python sample_send_small_logs.py`.
+
+## Next steps
+
+To learn more about Azure Monitor, see the [Azure Monitor service documentation][azure_monitor_docs] and the [Logs Ingestion API overview][azure_monitor_logs_ingestion_overview].
+
+
+<!-- Sample links -->
+[sample_send_small_logs]: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/monitor/azure-monitor-ingestion/samples/sample_send_small_logs.py
+[sample_send_small_logs_async]: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/monitor/azure-monitor-ingestion/samples/async_samples/sample_send_small_logs_async.py
+[sample_custom_error_callback]: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/monitor/azure-monitor-ingestion/samples/sample_custom_error_callback.py
+[sample_custom_error_callback_async]: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/monitor/azure-monitor-ingestion/samples/async_samples/sample_custom_error_callback_async.py
+[sample_upload_file_contents]: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/monitor/azure-monitor-ingestion/samples/sample_upload_file_contents.py
+[sample_upload_file_contents_async]: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/monitor/azure-monitor-ingestion/samples/async_samples/sample_upload_file_contents_async.py
+[sample_upload_pandas_dataframe]: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/monitor/azure-monitor-ingestion/samples/sample_upload_pandas_dataframe.py
+[sample_upload_pandas_dataframe_async]: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/monitor/azure-monitor-ingestion/samples/async_samples/sample_upload_pandas_dataframe_async.py
+
+<!-- External links -->
+[azure_identity]: https://pypi.org/project/azure-identity/
+[azure_identity_default_azure_credential]: https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/identity/azure-identity#defaultazurecredential
+[azure_monitor_create_using_portal]: https://docs.microsoft.com/azure/azure-monitor/logs/quick-create-workspace
+[azure_monitor_docs]: https://docs.microsoft.com/azure/azure-monitor/
+[azure_monitor_logs_ingestion_overview]: https://learn.microsoft.com/azure/azure-monitor/logs/logs-ingestion-api-overview
+[azure_subscription]: https://azure.microsoft.com/free/
+[data_collection_endpoint]: https://learn.microsoft.com/azure/azure-monitor/essentials/data-collection-endpoint-overview
+[data_collection_rule]: https://learn.microsoft.com/azure/azure-monitor/essentials/data-collection-rule-overview

sdk/monitor/azure-monitor-ingestion/samples/async_samples/sample_custom_error_callback_async.py

@@ -1,9 +1,35 @@
+# -------------------------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License. See License.txt in the project root for
+# license information.
+# --------------------------------------------------------------------------
+
 """
-Usage: python sample_custom_error_callback_async.py
+FILE: sample_custom_error_callback_async.py
+
+DESCRIPTION:
+    This sample demonstrates how to use error callbacks to customize how errors are handled during upload.
+
+    Note: This sample requires the azure-identity library.
+
+USAGE:
+    python sample_custom_error_callback_async.py
+
+    Set the environment variables with your own values before running the sample:
+    1) DATA_COLLECTION_ENDPOINT - your data collection endpoint
+    2) LOGS_DCR_RULE_ID - your data collection rule immutable ID
+    3) LOGS_DCR_STREAM_NAME - your data collection rule stream name
+
+    If using an application service principal for authentication, set the following:
+    1) AZURE_TENANT_ID - your Azure AD tenant (directory) ID
+    2) AZURE_CLIENT_ID - your Azure AD client (application) ID
+    3) AZURE_CLIENT_SECRET - your Azure AD client secret
 """
+
 import asyncio
 import os
 
+from azure.core.exceptions import HttpResponseError
 from azure.identity.aio import DefaultAzureCredential
 from azure.monitor.ingestion import UploadLogsError
 from azure.monitor.ingestion.aio import LogsIngestionClient
@@ -29,17 +55,24 @@ async def send_logs():
     failed_logs = []
 
     # Sample callback that stores the logs that failed to upload.
-    async def on_error(error: UploadLogsError) -> None:
+    async def on_error_save(error: UploadLogsError) -> None:
         print("Log chunk failed to upload with error: ", error.error)
         failed_logs.extend(error.failed_logs)
 
     # Sample callback that just ignores the error.
-    async def on_error_pass(*_) -> None:
+    async def on_error_pass(_) -> None:
         pass
 
+    # Sample callback that raises the error if it corresponds to a specific HTTP error code.
+    # This aborts the rest of the upload.
+    async def on_error_abort(error: UploadLogsError) -> None:
+        if isinstance(error.error, HttpResponseError) and error.error.status_code in (400, 401, 403):
+            print("Aborting upload...")
+            raise error.error
+
     client = LogsIngestionClient(endpoint=endpoint, credential=credential, logging_enable=True)
     async with client:
-      await client.upload(rule_id=rule_id, stream_name=os.environ['LOGS_DCR_STREAM_NAME'], logs=body, on_error=on_error)
+      await client.upload(rule_id=rule_id, stream_name=os.environ['LOGS_DCR_STREAM_NAME'], logs=body, on_error=on_error_save)
 
       # Retry once with any failed logs, and this time ignore any errors.
       if failed_logs:

sdk/monitor/azure-monitor-ingestion/samples/async_samples/sample_send_small_logs_async.py

@@ -1,6 +1,31 @@
+# -------------------------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License. See License.txt in the project root for
+# license information.
+# --------------------------------------------------------------------------
+
 """
-Usage: python sample_send_small_logs_async.py
+FILE: sample_send_small_logs_async.py
+
+DESCRIPTION:
+    This sample demonstrates how to send a small number of logs to a Log Analytics workspace.
+
+    Note: This sample requires the azure-identity library.
+
+USAGE:
+    python sample_send_small_logs_async.py
+
+    Set the environment variables with your own values before running the sample:
+    1) DATA_COLLECTION_ENDPOINT - your data collection endpoint
+    2) LOGS_DCR_RULE_ID - your data collection rule immutable ID
+    3) LOGS_DCR_STREAM_NAME - your data collection rule stream name
+
+    If using an application service principal for authentication, set the following:
+    1) AZURE_TENANT_ID - your Azure AD tenant (directory) ID
+    2) AZURE_CLIENT_ID - your Azure AD client (application) ID
+    3) AZURE_CLIENT_SECRET - your Azure AD client secret
 """
+
 import asyncio
 import os
 

sdk/monitor/azure-monitor-ingestion/samples/async_samples/sample_upload_file_contents_async.py

@@ -0,0 +1,77 @@
+# -------------------------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License. See License.txt in the project root for
+# license information.
+# --------------------------------------------------------------------------
+
+"""
+FILE: sample_upload_file_contents_async.py
+
+DESCRIPTION:
+    This sample demonstrates how to upload the contents of a file to a Log Analytics workspace. The file is
+    expected to contain a list of JSON objects.
+
+    Note: This sample requires the azure-identity library.
+
+USAGE:
+    python sample_upload_file_contents_async.py
+
+    Set the environment variables with your own values before running the sample:
+    1) DATA_COLLECTION_ENDPOINT - your data collection endpoint
+    2) LOGS_DCR_RULE_ID - your data collection rule immutable ID
+    3) LOGS_DCR_STREAM_NAME - your data collection rule stream name
+
+    If using an application service principal for authentication, set the following:
+    1) AZURE_TENANT_ID - your Azure AD tenant (directory) ID
+    2) AZURE_CLIENT_ID - your Azure AD client (application) ID
+    3) AZURE_CLIENT_SECRET - your Azure AD client secret
+"""
+
+import asyncio
+from datetime import datetime
+import json
+import os
+
+from azure.core.exceptions import HttpResponseError
+from azure.identity.aio import DefaultAzureCredential
+from azure.monitor.ingestion.aio import LogsIngestionClient
+
+
+async def upload_file_contents() -> None:
+    endpoint = os.environ['DATA_COLLECTION_ENDPOINT']
+    credential = DefaultAzureCredential()
+
+    client = LogsIngestionClient(endpoint=endpoint, credential=credential, logging_enable=True)
+
+    # Update this to point to a file containing a list of JSON objects.
+    FILE_PATH ="../../test-logs.json"
+
+    async with client:
+        # Option 1: Upload the file contents by passing in the file stream. With this option, no chunking is done, and the
+        # file contents are uploaded as is through one request. Subject to size service limits.
+        with open(FILE_PATH, "r") as f:
+            try:
+                await client.upload(
+                    rule_id=os.environ['LOGS_DCR_RULE_ID'],
+                    stream_name=os.environ['LOGS_DCR_STREAM_NAME'],
+                    logs=f)
+            except HttpResponseError as e:
+                print(f"File stream upload failed: {e}")
+
+        # Option 2: Upload the file contents by passing in the list of JSON objects. Chunking is done automatically, and the
+        # file contents are uploaded through multiple requests.
+        with open(FILE_PATH, "r") as f:
+            logs = json.load(f)
+            try:
+                await client.upload(
+                    rule_id=os.environ['LOGS_DCR_RULE_ID'],
+                    stream_name=os.environ['LOGS_DCR_STREAM_NAME'],
+                    logs=logs)
+            except HttpResponseError as e:
+                print(f"List upload failed: {e}")
+
+    await credential.close()
+
+
+if __name__ == '__main__':
+    asyncio.run(upload_file_contents())

sdk/monitor/azure-monitor-ingestion/samples/async_samples/sample_upload_pandas_dataframe_async.py

@@ -0,0 +1,81 @@
+# -------------------------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License. See License.txt in the project root for
+# license information.
+# --------------------------------------------------------------------------
+
+"""
+FILE: sample_upload_pandas_dataframe_async.py
+
+DESCRIPTION:
+    This sample demonstrates how to upload data stored in a pandas dataframe to Azure Monitor. For example,
+    a user might have the result of a query stored in a pandas dataframe and want to upload it to a Log Analytics
+    workspace.
+
+    Note: This sample requires the azure-identity and pandas libraries.
+
+USAGE:
+    python sample_upload_pandas_dataframe_async.py
+
+    Set the environment variables with your own values before running the sample:
+    1) DATA_COLLECTION_ENDPOINT - your data collection endpoint
+    2) LOGS_DCR_RULE_ID - your data collection rule immutable ID
+    3) LOGS_DCR_STREAM_NAME - your data collection rule stream name
+
+    If using an application service principal for authentication, set the following:
+    1) AZURE_TENANT_ID - your Azure AD tenant (directory) ID
+    2) AZURE_CLIENT_ID - your Azure AD client (application) ID
+    3) AZURE_CLIENT_SECRET - your Azure AD client secret
+"""
+
+import asyncio
+from datetime import datetime
+import json
+import os
+
+import pandas as pd
+
+from azure.core.exceptions import HttpResponseError
+from azure.identity.aio import DefaultAzureCredential
+from azure.monitor.ingestion.aio import LogsIngestionClient
+
+
+async def upload_dataframe() -> None:
+
+    # Set up example DataFrame.
+    data = [
+        {
+            "Time": datetime.now().astimezone(),
+            "Computer": "Computer1",
+            "AdditionalContext": "context-2"
+        },
+        {
+            "Time": datetime.now().astimezone(),
+            "Computer": "Computer2",
+            "AdditionalContext": "context"
+        }
+    ]
+    df = pd.DataFrame.from_dict(data)
+
+    # If you have a populated dataframe that you want to upload, one approach is to use the DataFrame `to_json` method
+    # which will convert any datetime objects to ISO 8601 strings. The `json.loads` method will then convert the JSON string
+    # into a Python object that can be used for upload.
+    body = json.loads(df.to_json(orient='records', date_format='iso'))
+
+    endpoint = os.environ['DATA_COLLECTION_ENDPOINT']
+    credential = DefaultAzureCredential()
+
+    client = LogsIngestionClient(endpoint=endpoint, credential=credential, logging_enable=True)
+    async with client:
+        try:
+            await client.upload(
+                rule_id=os.environ['LOGS_DCR_RULE_ID'],
+                stream_name=os.environ['LOGS_DCR_STREAM_NAME'],
+                logs=body)
+        except HttpResponseError as e:
+            print(f"Upload failed: {e}")
+    await credential.close()
+
+
+if __name__ == '__main__':
+    asyncio.run(upload_dataframe())