Missing Output to Log Analytics Workspace in latest Version

[Gill-Bates]

Thanks a lot for your effort to keep this tool running!

I am collecting the O365 Logs for over 10,000 Users and put them into a Log Analytics workspace. I would really appreachiate to have this feature in your latest Rust-Version.

[ddbnl]

Hiya,

No worries, thanks for using it :)

I can add the interface, I just wasn't sure which ones were actually being used. I'll have time to work on it monday or tuesday, so you can expect it in the coming days. I'll update here when it's available!

[ddbnl]

A new version is available that supports the Azure Log Analytics interface. Due to the full rewrite some small changes might be required:

• Command line arguments need full syntax
• OMS interface had a shared key config option which is insecure, instead the key should be passed as a command line argument "--oms-key"
• Container available in github container registry if you want it (see README)
• Schedule option has been removed, since it's probably better to use cron/task scheduler, or run the container on a schedule

New syntax would look like:

OfficeAuditLogCollector --tenant-id "11111111-1111-1111-1111-1111111111111" --client-id "111111111-1111-1111-11111111111111111111" --secret-key "12345" --config logAnalytics.yaml --oms-key "12345"

Let me know how it works for you. It's a full rewrite but I tested it as much as I could by myself.

Release:
https://github.com/ddbnl/office365-audit-log-collector/releases/tag/v2.3

[Gill-Bates]

@ddbnl I have tested your recent version against my LogAnalytics-Workspace and must confirm, that no Tables inside Log Analytics workspace appear.

However, I have noticed the following points:

• It takes two minutes for the data to be visible in the LogAnalytics workspace. This should be mentioned as a comment in the config. This is an Azure behaviour and affects all applications.
• The log file is not generated as required. My config.yaml looks like this:

  log: 
    path: '/root/nextgen/collector.log'
    debug: True
  collect:
    workingDir: '/root/nextgen' 
  

Console Output

./LINUX-OfficeAuditLogCollector --tenant-id "*****" --client-id "*****" --secret-key "*****" --config config.yaml
SEND
Blobs found: 315
Blobs successful: 315
Blobs failed: 0
Blobs retried: 0
Logs saved: 26747

The Custom Log Table are still missing:

[ddbnl]

@Gill-Bates,

I'll take a look asap. I can at least confirm the logging issue, I'll fix that and link the new release here when it's done.

In regards to log analytics, do you also have the output defined in the config, and the log analytics key as a command line arg? The config (based on your post) would be:

log: 
  path: '/root/nextgen/collector.log'
  debug: True
collect:
  workingDir: '/root/nextgen' 
output:
  azureLogAnalytics:
    workspaceId: 11111111-1111-1111-1111-1111111111111

And the command would be:

./LINUX-OfficeAuditLogCollector --tenant-id "*****" --client-id "*****" --secret-key "*****" --oms-key "12345" --config config.yaml

You can get the log analytics key as follows using AZ CLI:

az monitor log-analytics workspace get-shared-keys --resource-group my-rg --workspace-name my-oms --query "primarySharedKey"

If you already had all that then I will try to reproduce.

Thanks for testing!

[Gill-Bates]

@ddbnl i can confirm that I have configured the Collector as described above. For me it looks like that the Logs are fetched correctly. But when shifting them to the log analytics workspace, it will fail.

[ddbnl]

So far I have not been able to reproduce, for me the tables show up. My test is creating a log analytics workspace (just next, next, finish) and putting the workspace ID in the config. Then getting the shared key:

az monitor log-analytics workspace get-shared-keys --resource-group my-rg --workspace-name my-oms --query "primarySharedKey"

And putting the result in the '--oms-key' parameter. This gives the following result:

The result config is (also to enable logging):

log:
  path: './log.txt'
  debug: True
collect:
  contentTypes:
    Audit.General: True
    Audit.AzureActiveDirectory: True
    Audit.Exchange: True
    Audit.SharePoint: True
    DLP.All: True
output:
  azureLogAnalytics:
    workspaceId: 11111-1111111-11111111-111111111-11111111

I have released a new version with fixed logging and also extended logging, hopefully we'll be able to capture the error:

https://github.com/ddbnl/office365-audit-log-collector/releases/tag/v2.3.1

Let me know what it does for you.

[ddbnl]

I just fixed a bug that caused issues with the 'hoursToCollect' parameter. If you were using it, you might want to try with the latest release.

https://github.com/ddbnl/office365-audit-log-collector/releases/tag/v2.3.2

It also added an interactive interface that can be used for testing. If you have the new release, you can run the command as you did before, but add the '--interactive' command line parameter. This allows you to run the collector and immediately see the logs for any errors.