Objective

Creating Custom Log format option is available in EventLog Analyzer from build 12435 onwards, which offers you a flexibility to create your own Log formats. You can create multiple custom parsing rules within them to extract multiple fields as per requirement. This feature offers you a robust functionality to extract fields for Non-Predefined log sources or in-house application using which you can create custom Alerts, Rules, Reports and log analysis. This article offers you a step by step instruction for utilizing this feature to the fullest.

Prerequisites

• Access to EventLog Analyzer user interface as Administrator or Custom role to Manage Custom Log Parsing Rules.

• Sample logs from the source systems (e.g., servers, applications) for which the parsing has to be created.

• Knowledge of the log source, event types and field values for which parser rules has to be created.

Steps to follow

EventLog Analyzer supports Custom Log Formats to parse fields from logs collected via syslog or log import.

Custom Log format can be created in two ways,

I. During Log import Configuration - Refer How to Extract Fields during Log Import in EventLog Analyzer

II. Custom Log Format Settings: Refer below

1. Log in to EventLog Analyzer

2. Navigate to Settings >> Admin Settings >> Custom Log Format

3. Choose + Add Log Format

4. Enter a name for log format for easy identification that you would like to create and select the type of custom log collection that you are opting for - Syslog or Import. Save the option.

5. You will be able to see the following columns specified: Refer image.

• Actions - Offers you an option to Delete the Format

• Format Name - Name of the format that you have created (Name will be followed by (_custom) for easy differentiation between predefined and custom log format

• Created By - Technician who created this log format

• Type - Log collection type

• Custom Rule Count - Number of parser rules created

• Logs - View log option to see the logs collected that are associated with the respective log format.

• Rules - Manage Parser Rules option to create parser rules.

6. Parser rules can be created in three ways.

• In Custom Log Format settings (Recommended if you are yet to onboard the log source).

• Using Search tab (Recommended if you have already onboarded a log source)

• During Log import (Recommended during log import configuration)

NOTE: Parser rules will be mapped to the Log format for which the log source is associated to. During Log import or post onboarding a syslog device you can select the log format that you have created in Custom Log Format settings.

7. To create parser rules using Search tab refer, How to Extract New/Custom Fields from Log Data Using Log Search in EventLog Analyzer

8. To create parser rules during log import, refer How to Extract Fields during Log Import in EventLog Analyzer

9. To create parser rules in Custom Log Format, Refer following steps.

• Choose Manage Parser Rules for the respective Custom Log format for which parser rules/fields are to be extracted.

• Choose + Add Parser Rule,

• Paste the sample eventlog and Offer a Rule name.

• There are three methods by which custom fields can be specified.

1. Regex method

2. Delimiter method

3. JSON method

 Regex Method:

1. Provide a rule name.

2. Select and click the word(s) in the message, to be extracted as a field.

3. You can use the Auto Identify option to identify the fields automatically.

4. Provide a name for this field. Optionally, specify the prefix and suffix to the field value. Refer How to add Prefix and Suffix in the TIPS at the end of the page to know more.

5. Click on Create Pattern to generate a parser rule pattern.

6. A parser rule pattern is created using the field definition. You can edit the generated pattern manually if you are familiar with regular expressions.

7. Validate link is used to test the generated pattern against the current and previous search results. You can manually check the suitability of the pattern by analyzing the 'Matched Log Messages' and 'Unmatched Log Messages' displayed.

8. Click on Choose another pattern to choose a pattern from the list of patterns generated by the application.

9. You can define any existing field matching criteria to apply the pattern for this specific log type.

10. If you would like to apply this pattern only for selective events, enable the check box for "Apply this Pattern only when" to build the criteria for which these fields can be applied.

NOTE: Enabling "Apply this Pattern only when" offers you a flexibility to apply the parser rule only when the criteria meets.

11. Save the pattern to extract the field(s) from the upcoming logs.

 Delimiter Method:

1. Provide a rule name.

2. You can use the Delimiter to extract fields using delimiters such as Space, Comma, Tab, or Pipe. You can also use the text box in the drop down to enter a Custom Delimiter, which can split the field values.

3. If you would like to apply this pattern only for selective events, enable the check box for "Apply this Pattern only when" to build the criteria for which these fields can be applied.

NOTE: Enabling "Apply this Pattern only when" offers you a flexibility to apply the parser rule only when the criteria meets.

4. Click Save rule to save the created rule and have them applied for upcoming logs.

JSON Method: 

1. For JSON format logs, users can extract fields by specifying the JSON path to target specific fields in the log. Each field is defined by assigning a field name and mapping it to its corresponding JSON path. It's important to note that the JSON path should accurately reference the key or nested structure within the JSON log.

2. JSON field extraction can be configured only for the File Import custom log format. Users can create JSON extraction rules in the following two ways:

With a Sample Log:

• Provide a valid JSON log.

• Select the required fields from the JSON tree.

• Assign field names for the corresponding JSON paths.

Without a Sample Log:

• Manually specify the JSON path to target specific fields.

• Ensure the JSON path matches the log structure.

3. If you would like to apply this pattern only for selective events, enable the check box for "Apply this Pattern only when" to build the criteria for which these fields can be applied.

NOTE: Enabling "Apply this Pattern only when" offers you a flexibility to apply the parser rule only when the criteria meets.

4. Save the pattern to extract the field(s) from the upcoming logs.  

Validation and testing

• Open Search tab, select the device, set the calendar date and time as per log and perform search. Verify that the new fields are extracted by searching the field name under Add/Remove Field option.

• Ensure that the extracted data is displayed correctly and matches the expected values.

 Tips

• Custom Format feature is available from build 12435 and above. If you are in a lower version (End of Support Version), you will not have an option create a new format and you may see a message as "Go to search to create custom parser rule". You can either extract custom fields using Search tab or Upgrade EventLog Analyzer to latest version to have this functionality.

• Logs imports will be parsed based on the Log format selected. If a log format is created new, application will remember the format and will be used during Auto identification.

• Newly extracted fields will be applicable only for new logs collected. Old logs will not contain the parsed value.

• If you have extracted the field however it does not apply for the log source below are the possible causes.

• The parser pattern does not match with all the logs/log sources. Try adding prefix/suffix to obtain precise results.

• The actual log event varies based on OS version or event type, create multiple parser rules to extract fields from them.

• Newly extracted fields will be mapped to respective Log format which can be seen under Settings >> Admin Settings >> Custom Log formats.

• You can  create a new log format during import  using the following steps.

• Choose Automatically Identify drop down

• Enter a new Log Format name and choose + option near the text box to add it.

• New fields can be used in Reports, Search and Alerts tab and can be used for custom reports, Custom Alerts, Custom Action/Rule creation.

• You can  add Prefix/sufix  in Regix to build the pattern easily.

• You can also include the prefix and/or suffix of a field value to improve precision. To include a prefix and/or suffix, click on the icon in the right corner of the Fields table and select the required option. Click Apply.

• For instance, consider the message :
  Successful Network Logon: User Name: sylvian Domain: ADVENTNET Logon ID: (0x0,0x6D51131) Logon Type: 3 Logon Process: NtLmSsp Authentication Package: NTLM Workstation Name: SYLVIAN Logon GUID: - Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 192.168.113.97 Source Port: 0 22873

• The prefix Logon Type can be a static value as most of the logs will have the exact word as Logon Type where as Source Network Address can be dynamic as the logs may have different word(s) like, Source IP Address, Source Address, but with the same pattern.

• If the prefix and suffix are defined with exact match, the field extraction will be precise.

Note: An open attribute will not have a prefix or suffix.

Related topics and articles

How to Extract New/Custom Fields from Log Data Using Log Search in EventLog Analyzer

How to Extract Fields during Log Import in EventLog Analyzer