|
| 1 | +# Azure Monitor Ingestion client library for Java |
| 2 | + |
| 3 | +The Azure Monitor Ingestion client library is used to send custom logs to [Azure Monitor][azure_monitor_overview]. |
| 4 | + |
| 5 | +This library allows you to send data from virtually any source to supported built-in tables or to custom tables |
| 6 | +that you create in Log Analytics workspace. You can even extend the schema of built-in tables with custom columns. |
| 7 | + |
| 8 | +## Getting started |
| 9 | + |
| 10 | +### Prerequisites |
| 11 | +- A [Java Development Kit (JDK)][jdk_link], version 8 or later. |
| 12 | +- [Azure Subscription][azure_subscription] |
| 13 | +- A [Data Collection Endpoint][data_collection_endpoint] |
| 14 | +- A [Data Collection Rule][data_collection_rule] |
| 15 | +- A [Log Analytics workspace][log_analytics_workspace] |
| 16 | + |
| 17 | +### Include the package |
| 18 | + |
| 19 | +[//]: # ({x-version-update-start;com.azure:azure-monitor-ingestion;current}) |
| 20 | +```xml |
| 21 | +<dependency> |
| 22 | + <groupId>com.azure</groupId> |
| 23 | + <artifactId>azure-monitor-ingestion</artifactId> |
| 24 | + <version>1.0.0-beta.1</version> |
| 25 | +</dependency> |
| 26 | +``` |
| 27 | + |
| 28 | +### Create the client |
| 29 | + |
| 30 | +An authenticated client is required to upload logs to Azure Monitor. The library includes both synchronous and asynchronous forms of the clients. To authenticate, |
| 31 | +the following examples use `DefaultAzureCredentialBuilder` from the [com.azure:azure-identity](https://search.maven.org/artifact/com.azure/azure-identity) package. |
| 32 | + |
| 33 | +#### Authenticating using Azure Active Directory |
| 34 | +You can authenticate with Azure Active Directory using the [Azure Identity library][azure_identity]. |
| 35 | +To use the [DefaultAzureCredential][DefaultAzureCredential] provider shown below, or other credential providers provided with the Azure SDK, please include the `azure-identity` package: |
| 36 | +[//]: # ({x-version-update-start;com.azure:azure-identity;dependency}) |
| 37 | +```xml |
| 38 | +<dependency> |
| 39 | + <groupId>com.azure</groupId> |
| 40 | + <artifactId>azure-identity</artifactId> |
| 41 | + <version>1.5.2</version> |
| 42 | +</dependency> |
| 43 | +``` |
| 44 | +Set the values of the client ID, tenant ID, and client secret of the AAD application as environment variables: AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_CLIENT_SECRET. |
| 45 | + |
| 46 | +#### Synchronous Logs Ingestion client |
| 47 | + |
| 48 | +```java readme-sample-createLogsIngestionClient |
| 49 | +DefaultAzureCredential tokenCredential = new DefaultAzureCredentialBuilder().build(); |
| 50 | + |
| 51 | +LogsIngestionClient client = new LogsIngestionClientBuilder() |
| 52 | + .endpoint("<data-collection-endpoint>") |
| 53 | + .credential(tokenCredential) |
| 54 | + .buildClient(); |
| 55 | +``` |
| 56 | + |
| 57 | +#### Asynchronous Logs Ingestion client |
| 58 | + |
| 59 | +```java readme-sample-createLogsIngestionAsyncClient |
| 60 | +DefaultAzureCredential tokenCredential = new DefaultAzureCredentialBuilder().build(); |
| 61 | + |
| 62 | +LogsIngestionAsyncClient asyncClient = new LogsIngestionClientBuilder() |
| 63 | + .endpoint("<data-collection-endpoint>") |
| 64 | + .credential(tokenCredential) |
| 65 | + .buildAsyncClient(); |
| 66 | +``` |
| 67 | +## Key concepts |
| 68 | + |
| 69 | +### Data Collection Endpoint |
| 70 | + |
| 71 | +Data Collection Endpoints (DCEs) allow you to uniquely configure ingestion settings for Azure Monitor. [This |
| 72 | +article][data_collection_endpoint] provides an overview of data collection endpoints including their contents and |
| 73 | +structure and how you can create and work with them. |
| 74 | + |
| 75 | +### Data Collection Rule |
| 76 | + |
| 77 | +Data collection rules (DCR) define data collected by Azure Monitor and specify how and where that data should be sent or |
| 78 | +stored. The REST API call must specify a DCR to use. A single DCE can support multiple DCRs, so you can specify a |
| 79 | +different DCR for different sources and target tables. |
| 80 | + |
| 81 | +The DCR must understand the structure of the input data and the structure of the target table. If the two don't match, |
| 82 | +it can use a transformation to convert the source data to match the target table. You may also use the transform to |
| 83 | +filter source data and perform any other calculations or conversions. |
| 84 | + |
| 85 | +For more details, refer to [Data collection rules in Azure Monitor](https://docs.microsoft.com/azure/azure-monitor/essentials/data-collection-rule-overview). |
| 86 | + |
| 87 | +### Log Analytics Workspace Tables |
| 88 | + |
| 89 | +Custom logs can send data to any custom table that you create and to certain built-in tables in your Log Analytics |
| 90 | +workspace. The target table must exist before you can send data to it. The following built-in tables are currently supported: |
| 91 | + |
| 92 | +- [CommonSecurityLog](https://docs.microsoft.com/azure/azure-monitor/reference/tables/commonsecuritylog) |
| 93 | +- [SecurityEvents](https://docs.microsoft.com/azure/azure-monitor/reference/tables/securityevent) |
| 94 | +- [Syslog](https://docs.microsoft.com/azure/azure-monitor/reference/tables/syslog) |
| 95 | +- [WindowsEvents](https://docs.microsoft.com/azure/azure-monitor/reference/tables/windowsevent) |
| 96 | + |
| 97 | +## Examples |
| 98 | + |
| 99 | +- [Upload custom logs](#upload-custom-logs) |
| 100 | +- [Upload custom logs with max concurrency](#upload-custom-logs-with-max-concurrency) |
| 101 | + |
| 102 | +### Upload custom logs |
| 103 | + |
| 104 | +```java readme-sample-uploadLogs |
| 105 | +DefaultAzureCredential tokenCredential = new DefaultAzureCredentialBuilder().build(); |
| 106 | + |
| 107 | +LogsIngestionClient client = new LogsIngestionClientBuilder() |
| 108 | + .endpoint("<data-collection-endpoint") |
| 109 | + .credential(tokenCredential) |
| 110 | + .buildClient(); |
| 111 | + |
| 112 | +List<Object> logs = getLogs(); |
| 113 | +UploadLogsResult result = client.upload("<data-collection-rule-id>", "<stream-name>", logs); |
| 114 | +System.out.println("Logs upload result status " + result.getStatus()); |
| 115 | +``` |
| 116 | + |
| 117 | +### Upload custom logs with max concurrency |
| 118 | + |
| 119 | +If the in input logs collection is too large, the client will split the input into multiple smaller requests. These |
| 120 | +requests are sent serially, by default, but by configuring the max concurrency in `UploadLogsOptions`, these requests |
| 121 | +can be concurrently sent to the service as shown in the example below. |
| 122 | + |
| 123 | +```java readme-sample-uploadLogsWithMaxConcurrency |
| 124 | +DefaultAzureCredential tokenCredential = new DefaultAzureCredentialBuilder().build(); |
| 125 | + |
| 126 | +LogsIngestionClient client = new LogsIngestionClientBuilder() |
| 127 | + .endpoint("<data-collection-endpoint") |
| 128 | + .credential(tokenCredential) |
| 129 | + .buildClient(); |
| 130 | + |
| 131 | +List<Object> logs = getLogs(); |
| 132 | +UploadLogsOptions uploadLogsOptions = new UploadLogsOptions() |
| 133 | + .setMaxConcurrency(3); |
| 134 | +UploadLogsResult result = client.upload("<data-collection-rule-id>", "<stream-name>", logs, uploadLogsOptions, |
| 135 | + Context.NONE); |
| 136 | +System.out.println("Logs upload result status " + result.getStatus()); |
| 137 | +``` |
| 138 | + |
| 139 | +## Troubleshooting |
| 140 | + |
| 141 | +### Enabling Logging |
| 142 | + |
| 143 | +Azure SDKs for Java offer a consistent logging story to help aid in troubleshooting application errors and expedite |
| 144 | +their resolution. The logs produced will capture the flow of an application before reaching the terminal state to help |
| 145 | +locate the root issue. View the [logging][logging] wiki for guidance about enabling logging. |
| 146 | + |
| 147 | +## Next steps |
| 148 | +More samples can be found [here][samples]. |
| 149 | + |
| 150 | +## Contributing |
| 151 | + |
| 152 | +This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License |
| 153 | +Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. |
| 154 | +For details, visit [https://cla.microsoft.com](https://cla.microsoft.com). |
| 155 | + |
| 156 | +When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the |
| 157 | +PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this |
| 158 | +once across all repos using our CLA. |
| 159 | + |
| 160 | +This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). |
| 161 | +For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact |
| 162 | +[opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments. |
| 163 | + |
| 164 | +<!-- LINKS --> |
| 165 | +[azure_identity]: https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/identity/azure-identity |
| 166 | +[azure_monitor_overview]: https://docs.microsoft.com/azure/azure-monitor/overview |
| 167 | +[azure_subscription]: https://azure.microsoft.com/free |
| 168 | +[cla]: https://cla.microsoft.com |
| 169 | +[coc]: https://opensource.microsoft.com/codeofconduct/ |
| 170 | +[coc_faq]: https://opensource.microsoft.com/codeofconduct/faq/ |
| 171 | +[coc_contact]: mailto:opencode@microsoft.com |
| 172 | +[data_collection_endpoint]: https://docs.microsoft.com//azure/azure-monitor/essentials/data-collection-endpoint-overview |
| 173 | +[data_collection_rule]: https://docs.microsoft.com/azure/azure-monitor/essentials/data-collection-rule-overview |
| 174 | +[DefaultAzureCredential]: https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/identity/azure-identity/README.md#defaultazurecredential |
| 175 | +[jdk_link]: https://docs.microsoft.com/java/azure/jdk/?view=azure-java-stable |
| 176 | +[log_analytics_workspace]: https://docs.microsoft.com//azure/azure-monitor/logs/log-analytics-workspace-overview |
| 177 | +[logging]: https://docs.microsoft.com//azure/developer/java/sdk/logging-overview |
| 178 | +[samples]: https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/monitor/azure-monitor-ingestion/src/samples/java/com/azure/monitor/ingestion |
| 179 | + |
| 180 | + |
0 commit comments