How to Onboard Non-Predefined or Custom Log source in EventLog Analyzer

How to Onboard Non-Predefined or Custom Log source in EventLog Analyzer

Objective

EventLog Analyzer offers predefined support for various log sources. Predefined support includes built-in parsing, reports (as per market requirement), Detection Rules & Alerts. EventLog Analyzer offers extensive capability to onboard and to create parsing rules, reports, Detections rules & Alerts for Non-Predefined supported log sources that includes non predefined log source, Supported log sources with unsupported versions and In-house custom applications. This article offers you a detailed information on how to onboard a non-predefined log source or custom log source/device/application in EventLog Analyzer.

Prerequisites

  1. Availability of License:
For Build version below 13000,
    1. Syslog device(s): Onboarding one log source consumes one Device license.
    2. Syslog Application(s) hosted in Windows device: Onboarding one application, will consume one Application license.
    3. Syslog Applications(s) hosted in Linux device: No additional license required for onboarding application. It will consume the same Device license as consumed by onboarded Linux/Unix log source.
    4. Log Import: EventLog Analyzer/Log360 consumes one Application license for each import and each unique log format identified during log file import. Importing the same file twice with different formats will consume two licenses.
For Build version 13000 and above,
    1. Syslog device(s): Onboarding one log source consumes one Logsources license
    2. Syslog Application(s) hosted in Windows device: No additional license required for onboarding application. It will consume the same license as consumed by Windows log source.
    3. Syslog Applications(s) hosted in Linux device: No additional license required for onboarding application. It will consume the same Logsources license as consumed by onboarded Linux/Unix log source. 
    4. Log Import: No additional license required for importing the data. It will consume the same Logsources or Endpoint license as consumed by source log source.
  1. Device Onboarding: The log source must comply with either one of the following options for onboarding them
    1. Syslog forwarding in BSD format or CEF log format.
    2. Log file to be printed on a location in unencrypted human readable format.
  2. Parsing: Knowledge about event types, fields to be extracted from the Log source.
  3. Reports: Knowldege of event types, Reports required & field values created in parsing for the log source.
  4. Detection Rules & Alerts: Knowledge about rules to be created for the log source.

Steps to follow

Complete Onboarding of a log source involves the following process,

I. Adding Log source

Log source can be added in three ways,
A. Syslog forwarding: If the Log source has capabilities to forward the events as syslog.
B. Log Import: If the Log source does not have capabilities to forward the events as syslog and the logs are printed in a specific location in Windows or Linux/Unix device.
C. Windows Application & events are recorded in Event Viewer: Log source mentioned here is an Application Hosted in Windows device and the events are recorded in Event Viewer.

A. Syslog Forwarding: If the Log source has capabilities to forward the events as syslog, you can proceed with the following steps based on the cases mentioned below.

Case A: Log Source mentioned here is a physical device(Non Windows or Linux OS).
Case B: Log source mentioned here is an Application Hosted in Windows device and the events are not recorded in Event Viewer but can be forwarded as syslog.
Case C: Log source mentioned here is an Application Hosted in Linux/Unix device.

Please find the details of the case mentioned below.
Case A: Log Source mentioned here is a physical device(Non Windows or Linux OS).
  1. Ensure sufficient license is available to onboard the device. (Build below 13000 - Device license ; Build 13000 and above - Log Sources)
  2. In EventLog Analyzer/Log360 User interface, navigate to Settings >> System Settings >> Listener Ports and ensure the syslog listener ports are configured in EventLog Analyzer/Log360 application as per requirement.
  3. Syslog listener ports must be allowed in the EventLog Analyzer/Log360 server’s local firewall, security applications, and network firewall to ensure proper log flow from the log source to EventLog Analyzer/Log360. The log flow direction is always from the log source to the EventLog Analyzer/Log360 application.

    For build 13000 and above, you can forward the syslogs to agents which can indeed collect the logs and forward them to EventLog Analyzer/Log360 application. Refer Associate devices to an agent for more details.
  4. If you are forwarding syslogs to EventLog Analyzer server, the local log collector will collect the logs. The incomming traffic of the local log collector can be validated using Log Receiver, Built-in packet capture tool or using any packet capture tool.

    For build 13000 and above, you can either forward the syslog packets to Agent(remote log collector) or EventLog Analyzer server(local log collector).  Log Receiver, Built-in packet capture tool can capture the syslog packets that are collected on respective instance(EventLog Analyzer server or log processor) only. For Agent based syslog collection, you can use any packet capture tool to track the packets.
  5. Once the log packet is captured by agent or log collector of EventLog Analyzer/Log360 over the port listened by the Application/agent, the device will be auto added in the Application User Interface.
  6. You can proceed with Creating Custom Parsing rules to extract fields. If you do not find the logs getting captured, refer Related articles at the end of this page for more details.
Case B: Log source mentioned here is an Application Hosted in Windows device and the events cannot be recorded in Event Viewer but can be forwarded as syslog.
Scenario 1: To collect logs from both the host Windows machine events and the hosted Application.
  1. Ensure license is available.

    For Build version below 13000, you need a Windows server/Workstation(based on OS) and Application license to perform this action.  
    For Build version 13000 and above, you need license to onboard Windows device(Windows Server - Log sources ; Windows Workstation - Endpoints).
  2. Add Windows device in Settings >> Log Source Configuration >> Devices >> Windows. Refer Adding Windows Devices for detailed steps.
  3. In EventLog Analyzer/Log360 User interface, navigate to Settings >> System Settings >> Listener Ports and ensure the syslog listener ports are configured in EventLog Analyzer/Log360 application.
  4. Syslog listener ports are to be enabled in EventLog Analyzer/Log360 server's local firewall/Security Applications and network firewall. The Direction of the log flow will be Log Source to EventLog Analyzer/Log360 application.
  5. In EventLog Analyzer/Log360 user interface, navigate to Settings >> Log Source Configuration >> Application >> General Application >> Syslog Application >> and add the respective device as Syslog Application.
    If the log format is CEF, you can consider adding the device under Settings >> Log Source Configuration >> Application >> Security Application and add the respective device as CEF Format
  6. Configure syslog service to forward the events to EventLog Analyzer/Log360 to collect the logs.
  7. Application events that are forwarded as syslog will be parsed as Syslog Application or CEF Format Log type. You can Create parsing and Reports as per requirement for the respective log type.
  8. Windows events that are collected will be parsed as Windows Log type.
Scenario 2: To collect only the Application's logs, without including Windows event logs.
You can configure syslog service to forward the logs to EventLog Analyzer/Log360 to have it added as a syslog device under Settings >> Log source configuration >> Devices >> Syslog device.
  1. Ensure the license is available to perform log collection.
    To add the log source as syslog device,
    1. For build below 13000, you need a Device license.
    2. For build 13000 and above, you need a Log source license.
  2. In EventLog Analyzer/Log360 User interface, navigate to Settings >> System Settings >> Listener Ports and ensure the syslog listener ports are configured in EventLog Analyzer/Log360 application.
  3. Syslog listener ports are to be enabled in EventLog Analyzer/Log360 server's local firewall/Security Applications and network firewall. The Direction of the log flow will be Log Source to EventLog Analyzer/Log360 application.
  4. Configure syslog service for forward the logs to EventLog Analyzer/Log360 server. Check with respective vendor for syslog configuration.
  5. The device will be auto added in user interface under Settings >> Log source configuration >> Devices >> Syslog device.
  6. You can proceed to create Custom Log format, Custom Reports and Rules.

    Note: Once you add an application log source hosted on a Windows device as a syslog device in EventLog Analyzer/Log360, the same device cannot be added again as a Windows device. If you want to audit both Windows events and the application hosted in same device that forwards events as syslogs, but the device is already added as a syslog device, follow the steps below.
1. Navigate to Settings >> Log source configuration >> Devices >> Syslog device.
2. Place the cursor on top of device name and Choose Update device.
3. Update the Log Source Type as Syslog Application or CEF format. Choose Update to apply the changes.
4. The device will be updated as Syslog Application or CEF format. You can find it in the Settings >> Log Source configuration >> Application tab. Syslog Application type will be under General Application and CEF format type will be under Security Applications.
NOTE: When you switch the log type, older logs will still be available with fields parsed according to the previous log type and can be accessed from the Search or Reports tab, depending on log retention settings. New logs will be parsed using the newly selected log type. If you created custom parsing rules for the previous log type, you will need to recreate them for the new log type.
5. You can proceed to add the device for Windows event log collection. Refer Adding Windows Devices for detailed steps.
You can proceed to configure custom log parsing/format, custom reports and rules as per requirement.

Case C:  Log source mentioned here is an Application Hosted in Linux/Unix device.
If any particular applications' logs or audit logs needs to be forwarded, then the following configurations needs to be done in Linux devices under rsyslog.conf (or) syslog.conf
  1. Under the MODULES section, check whether the "$ModLoad imfile" is included. (This module "imfile" converts any input text file into a syslog message,which can then be forwarded to the EventLog Analyzer Server.)
  2. The following directives contain the details of the external log file:
    $InputFileName <Monitored_File_Absolute_Path>
    $InputFileStateFile <State_Filename>
    $InputFileSeverity <Severity >
    $InputFileFacility <Facility >
    $InputRunFileMonitor
  3. To forward the logs we must provide this line:
    <Facility>.<Severity> @Host-Ip:Port
    Refer: How to add Unix/Linux device in EventLog Analyzer
Example:
$InputFileName /var/log/sample.log
$InputFileStateFile sample
$InputFileSeverity info
$InputFileFacility local6
local6.info @eventloganalyzer-Server:514
Here /var/log/sample.log is the external file to be forwarded and eventlog analyzer-server is the EventLog Analyzer server or Agent installed machine's hostname.
 
B. Log import: If the Log source does not have capabilities to forward the events as syslog and the logs are printed in a specific location in Windows or Linux/Unix device.

You can import the logs from Local path, Remote path and Cloud path(AWS S3 bucket). Scheduled log collection is available only in Remote path and Cloud path(AWS S3 bucket).
Refer following documents to know the steps for importing the log files.
How to import logs in EventLog Analyzer from a Remote path (Recommended for scheduled log collection)
How to import logs in EventLog Analyzer from S3 bucket (Recommended for scheduled log collection)
C. Windows Application & events are recorded in Event Viewer: Log source mentioned here is an Application Hosted in Windows device and the events are recorded in Event Viewer.
EventLog Analyzer collects Application, Security and System Events by default from Event Viewer. If the log source events are recorded in the same event source (Application, Security or System), you can skip to Creating Custom Parsing rules to extract fields or Create Custom Reports as per requirement.

If the events are recorded under Application and Services logs, you can Configure Event Source File or add registry key manually to collect the events. Please find the following steps.
Configure event source file from User Interface:
Prerequisite:
  1. The credentials offered for the machine should be valid. If you are performing agent based log collection, Configure Event Source file will be auto synced.
  2. Bidirectional communication between Agent & server is needed for this action. The target device must be reachable from EventLog Analyzer/Log360 application.
  3. Remote registry service should be enabled.
  4. For agentless log collection, the user should have admin priviledges or full control to perform registry changes.
Steps:
  1. Open Event Viewer of respective Windows Host machine.
  2. Identify the event source file location, right click the file and select Properties.
  3. Note down the Full name. Sample image below.
In the above image the Full Name is Microsoft-Windows-PowerShell/Operational
  1. In EventLog Analyzer/Log360 user interface, Go to Settings >> Log source configuration >> Devices >> Windows
  2. Click the Configure Event Source Files icon for the respective device.
  3. In the Event Source File dialog box, search for the Full name of event source file that you have noted, select the type of event source file.
  4. Click Configure

NOTE: Configuring through the Configure Event Source file will automatically apply the necessary registry changes on the Windows host machine. If sufficient ports or permissions are not provided, you also have the option to update the registry manually.

Adding registry value manually:
  1. Open Event Viewer of respective Windows Host machine.
  2. Identify the event source file location, right click the file and select Properties.
  3. Note down the Full name. Sample image below.
In the above image the Full Name is Microsoft-Windows-PowerShell/Operational
4. Open the Registry Editor and navigate to Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\EventLog
  1. Right click EventLog and choose New >> Key
  2. Provide the Event Source file Full name as Key name.
To configure event source files for multiple devices via Group Policy, refer How to configure an event source for multiple devices via Group Policy 

II. Creating Custom Parsing rules to extract fields

    1. Case A: Log source onboarded is in list of Predefined support Log Format, but the parsing is unavailable for the log source version/Application hosted in log source.
    2. Case B: The log source onboarded is not available in Predefined support Log format.
Case A: Log source onboarded is in list of Predefined support Log Format, but the parsing is unavailable for the log source version/Application hosted in log source.
You can extract the custom fields for the predefined format to have the parsing utilized at its best. Refer the following documents to extract fields.
Case B: The log source onboarded is not available in Predefined support Log format.
Refer the following documents to create Custom Log format and extract fields.
 

III. Create Custom Reports

Based on the custom fields extracted, you can create custom reports as per requirement. Follow the below procedure to create custom reports
  1. Identify and list down your requirements of having the reports.
  2. Using Search tab, build the criteria to sort down the required events.
  3. Once you build a criteria that matches the events for the report, use save as option to save the criteria as Custom Report.
  1. Select the report type, Devices, Report Group(to group the report under one roof) and save the report.
Refer the following documents to create a custom report.
 

IV. Create Custom Detection Rules & Alerts

You can create custom Detection Rules & Alerts as per requirement.
Refer the following help documents to learn how to create Rules.
For Build below 13000,
For Build 13000 and above,
 
 


      New to M365 Manager Plus?
      New to M365 Manager Plus?
        New to RecoveryManager Plus?
        New to RecoveryManager Plus?
          New to Exchange Reporter Plus?
          New to Exchange Reporter Plus?
            New to SharePoint Manager Plus?
            New to SharePoint Manager Plus?
                See all integrations
                New to ADManager Plus?
                See all integrations
                New to ADManager Plus?

                  New to ADSelfService Plus?

                  Start your free trial
                  Start your free trial

                      Related Products