Objective

This article offers you the steps to extract new fields from log data in EventLog Analyzer. It will guide you through the process of defining and extracting custom fields to enhance log analysis and reporting.

Prerequisites

• Access to EventLog Analyzer console as Administrator or custom roles (Log Search and Manage Custom Log Format

• Knowledge of the log data structure and the fields you want to extract. You can check on respective vendor site to learn more.
  

Steps to follow

1. Log in to EventLog Analyzer.

2. Navigate to Search tab and Search the logs from which fields need to be extracted. Click Create Additional Fields button to view the available fields and extract fields.

Note: Alternatively, you can also extract additional fields while importing the log file.

3. You can view the extracted field details in the Event Information Window. If the required value is not parsed, you can extract further fields by clicking the Extract Additional Fields button.

4. There are three methods by which custom fields can be specified.

1. Regex method

2. Delimiter method

3. JSON method

Regex Method

Regex (Regular Expressions) parsing in EventLog Analyzer refers to the use of regular expressions to extract specific, meaningful data from raw, unstructured log entries. Log data often comes in varying formats and can contain a lot of noise. Regex provides a powerful way to define patterns that identify and isolate the relevant information within these logs.

1. Provide a rule name.

2. Select and click the word(s) in the message, to be extracted as a field.

3. You can use the Auto Identify option to identify the fields automatically.

4. Provide a name for this field. Optionally, specify the prefix and suffix to the field value. Refer How to add Prefix and Suffix to know more.

5. Click on Create Pattern to generate a parser rule pattern.

6. A parser rule pattern is created using the field definition. You can edit the generated pattern manually if you are familiar with regular expressions.

7. Validate link is used to test the generated pattern against the previous search results. You can manually check the suitability of the pattern by analyzing the 'Matched Log Messages' and 'Unmatched Log Messages' displayed.

8. Click on Choose another pattern to choose a pattern from the list of patterns generated by the application.

9. If you would like to apply this pattern only for selective events, enable the check box for "Apply this Pattern only when" to build the criteria for which these fields can be applied.

NOTE: Enabling "Apply this Pattern only when" offers you a flexibility to apply the parser rule only when the criteria meets.

10. Save the pattern to extract the field(s) from the upcoming logs.

​

A delimiter parser in an event log analyzer, such as ManageEngine EventLog Analyzer, is a component or feature that allows the system to dissect log entries based on specific characters or strings that act as separators between different data fields within a single log line. 

1. Provide a rule name.

2. You can use the Delimiter to extract fields using delimiters such as Space, Comma, Tab, or Pipe. You can also use the text box in the drop down to enter a Custom Delimiter, which can split the field values.

3. If you would like to apply this pattern only for selective events, enable the check box for "Apply this Pattern only when" to build the criteria for which these fields can be applied.

NOTE: Enabling "Apply this Pattern only when" offers you a flexibility to apply the parser rule only when the criteria meets.

4. Click Save rule to save the created rule and have them applied for upcoming logs.

JSON Method

ManageEngine EventLog Analyzer supports JSON as a log parsing method. This feature allows users to define rules for extracting fields from logs that are formatted in JSON.

• For JSON format logs, users can extract fields by specifying the JSON path to target specific fields in the log. Each field is defined by assigning a field name and mapping it to its corresponding JSON path. It's important to note that the JSON path should accurately reference the key or nested structure within the JSON log.

• JSON field extraction can be configured only for the File Import custom log format. Users can create JSON extraction rules in the following two ways:

With a Sample Log:

• Provide a valid JSON log.

• Select the required fields from the JSON tree.

• Assign field names for the corresponding JSON paths.

• If you would like to apply this pattern only for selective events, enable the check box for "Apply this Pattern only when" to build the criteria for which these fields can be applied.

NOTE: Enabling "Apply this Pattern only when" offers you a flexibility to apply the parser rule only when the criteria meets.

• Choose Save Rule to save the fields extracted.

Without a Sample Log:

• Manually specify the JSON path to target specific fields.

• Ensure the JSON path matches the log structure.

• If you would like to apply this pattern only for selective events, enable the check box for "Apply this Pattern only when" to build the criteria for which these fields can be applied.

• Choose Save Rule to save the fields extracted.

 Validation and confirmation: 

• Verify that the new fields are extracted by searching the field name under Add/Remove Field option.

• Ensure that the extracted data is displayed correctly and matches the expected values.

 Tips

• Newly extracted fields will be applicable only for new logs collected. Old logs will not contain the parsed value.

• If you have extracted the field however it does not apply for the log source below are the possible causes.

• The parser pattern does not match with all the logs/log sources. Try adding prefix/suffix to obtain precise results.

• The actual log event varies based on OS version or event type, create multiple parser rules to extract fields from them.

• Newly extracted fields will be mapped to respective Log format which can be seen under Settings >> Admin Settings >> Custom Log formats.

• You can create a new log format under Custom Log format and create custom parser rules as per requirement to parse non predefined supported or custom log sources. Custom log format feature is available from Build 12435 and above. Kindly review your current version once and Upgrade to the latest version of EventLog Analyzer to have this feature.

• New fields can be used in Reports, Search and Alerts tab and can be used for custom reports, Custom Alerts, Custom Action/Rule creation.

• You can add Prefix/sufix in Regix to build the pattern easily.

• You can also include the prefix and/or suffix of a field value to improve precision. To include a prefix and/or suffix, click on the icon in the right corner of the Fields table and select the required option. Click Apply.

• For instance, consider the message : Successful Network Logon: User Name: sylvian Domain: ADVENTNET Logon ID: (0x0,0x6D51131) Logon Type: 3 Logon Process: NtLmSsp Authentication Package: NTLM Workstation Name: SYLVIAN Logon GUID: - Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 192.168.113.97 Source Port: 0 22873

• The prefix Logon Type can be a static value as most of the logs will have the exact word as Logon Type where as Source Network Address can be dynamic as the logs may have different word(s) like, Source IP Address, Source Address, but with the same pattern.

• If the prefix and suffix are defined with exact match, the field extraction will be precise.

• Note: An open attribute will not have a prefix or suffix.

Related topics and articles

How to Extract Fields in Log Import in EventLog Analyzer