Many System Center Orchestrator administrators longed for centralized logging of Orchestrator runbook events and results. With no native centralized logging facility, many administrators used a simple SQL database as a centralized logging repository. While not ideal, this method is functional for a single data center.
Think about the challenges of implementing this strategy on a global basis. You would need either have to configure your Orchestrator runbook servers to log events to a SQL instance in a remote data center, or perhaps configure SQL replication to centralize the events after they were logged locally.
In this article, we will look at establishing a centralized runbook progress and result logging strategy for the Azure Automation Hybrid Runbook Worker using the Microsoft Operations Management Suite (OMS) that is quick, easy and searchable…no SQL database required.
In this post:
- Hybrid Runbook Worker job logging in the Azure Portal
- Native Logging on the Hybrid Worker
- Why custom central logging for the Hybrid Worker?
- Basic Custom Logging Implementation for Hybrid Worker
- Logging runbooks events with a child runbook
Let us get down to business.
Hybrid Runbook Worker job logging in the Azure Portal
Enter the Azure Automation Hybrid Runbook Worker (hybrid worker) role, which enables Azure Automation customers to execute (run) runbooks on designated servers in their data centers around the world. The back-end infrastructure is Microsoft Azure, so implementing a global automation infrastructure is much less work than with Orchestrator.
However, the desire for detailed logging of activities on the hybrid runbook worker remains. Certainly the Jobs log (shown below) in Azure Automation shows results, inputs and outputs of runbook jobs. However, it does not provide detailed progress logging of your choosing at key points along the way.
With that in mind, the desire / need for centralized progress logging at our discretion remains.
Native runbook event logging on the Hybrid Worker
One bit of good news is that the hybrid worker includes some native logging to the Windows Event Log. You can find the log in the Windows Event Viewer under Applications and Services Logs > Microsoft-Automation > Operational.
More good news. You can easily add this log for collection by OMS in the Settings area on the Data tab. In the ‘Collect events from the following event logs’ window, type “Microsoft-Automation/Operational” and click on the plus sign (+) to add the log for collection. Then check the boxes for all three event levels.
However, the bad news is, once this data arrives in our OMS subscription, you can see the formatting of the details in this log is less than ideal. Using the search interface, you can retrieve the contents of this log with the following query:
The result reveals that in the ParameterXml within the event lies the job status, but only the value (Succeeded, Failed, etc.) rather than a well-formatted name value pair we could use to extract the result into a custom field for better search. As a result, this native logging is not as useful as it could be, but can still serve to reduce the need for automation admins to use the Azure portal.
Moreover, this still does not address our need for ad-hoc progress logging hosted in a central repository.
Why custom central logging for the Hybrid Worker?
Even with the aforementioned logging, the need to log events within a runbook at key junctures remains, and this is where logging to the Windows Event Log with a custom source, with an event description detailing runbook progress or results, with output collected and forwarded to OMS. This gives us central logging without the need for a SQL database.
You can log this data to any local Windows Event Log, even the System log. One approach might be to use a couple of custom sources, such as one for logging runbook progress messages and another reporting the runbook result, both in an easily searchable format in OMS once the data has been collected and uploaded.
Basic Custom Logging Implementation for Hybrid Worker
I will describe a couple of simple approaches you could take for custom logging locally on the hybrid worker.
First, in order to write messages with a custom Windows event log source, the providers must exist. You can create custom event sources with the New-EventLog cmdlet. If you wanted, you could simply create these providers each time you deploy a new hybrid worker.
For runbook progress messages, I will write to the System event log with a source of OMSAutomation-Status. To create this new provider, simply run the following one-liner on the hybrid worker.
New-EventLog -LogName System -Source OMSAutomation-Status
For the final runbook result message, I will again write to the System log with a source of OMSAutomation-Result. To create this new provider, simply run the following one-liner on the hybrid worker.
New-EventLog -LogName System -Source OMSAutomation-Result
This approach ensures I can quickly and easily search runbook messages by event source. Then, any time you want to log a status message within your runbook, you would simply add a Write-EventLog message at the appropriate point in your runbook. Since the System log is already being collected in OMS, you do not have to configure any additional log collection in OMS.
Let’s assume that every time a runbook begins I log a message in the following format from within my PowerShell runbook on hybrid workers:
Write-EventLog -EventId 1 -LogName System -Message "Runbook $RBName is now beginning" -Source OMSAutomation-Status
Then in OMS I can find every message showing a runbook beginning with the following wildcard search:
Type=Event EventLog=System Source=OMSAutomation-Status “*now beginning “
Add | measure count() by Source and you can get a count of how many runbooks have run on hybrid workers.
Type=Event EventLog=System Source=OMSAutomation-Status “*now beginning ” | measure count() by Source
Add the computer name to filter results to a specific hybrid worker, as shown here
Type=Event EventLog=System Source=OMSAutomation-Status “*now beginning ” Computer=”VMBR01.contoso.corp” | measure count() by Source
If I wanted to find ever event where a specific runbook ran, I would simply expand the search based on my message syntax, for example.
Type=Event EventLog=System Source=OMSAutomation-Status “Runbook Hello-World is now beginning”
And finally, a search for all successful completions of a specific runbook
Type=Event EventLog=System Source=OMSAutomation-Result “Runbook Hello-World completed successfully” Computer=”VMBR01.contoso.corp” Source=OMSAutomation-Result
or any runbook, as pictured below
Type=Event EventLog=System Source=OMSAutomation-Result “*successfully” Computer=”VMBR01.contoso.corp” Source=OMSAutomation-Result
As you can see, logging inline from within your runbook is simple, quick, easy and searchable in OMS!
Logging runbooks events with a child runbook
We could take what some might deem a slightly more elegant approach and instead call a child runbook to log the event. One advantage of this method is that we can accept the event source as an input, and check that the source exists before attempting to write the event. This means there would be no need to create the custom event sources manually. On the downside, this approach might mean slightly longer execution time, resulting in billing for a few more runbook minutes per month versus the simple inline method described above.
Here is a simple, generic and effective Windows event logging script borrowed from my friend Tao Yang. This script accepts event log name, ID, source, type and message as parameters. Before attempting to log the event, it checks for your event source on the hybrid worker. If the source does not exist, it creates the new source, then logs the event.
$ExistingEventSource = Get-Eventlog -LogName $EventLog -Source $EventSource -ErrorAction SilentlyContinue
#The event source does not exist, create it now
New-EventLog -LogName $EventLog -Source $EventSource
Write-EventLog -LogName $EventLog -source $EventSource -EventId $EventID -EntryType $EntryType -Message $Message
Here is a sample usage of this runbook you can past into your Azure Automation runbooks destined for hybrid workers. This assumes you have added the above sample runbook to your Azure Automation subscription with the name Write-WindowsEvent.
.\Write-WindowsEvent.ps1 -Message 'Test message' -EntryType 'Information' -EventLog 'System' -EventSource OMSAutomation -EventID 50001
As you can see, Azure Automation, the new hybrid worker and OMS open the door to easy and flexible implementation of a global, hybrid runbook automation infrastructure that transcends the limitations of System Center Orchestrator. Get started with your free subscription to OMS at http://microsoft.com/oms.