Objective

Prerequisites

1. Understanding of the duration of log management needed by the organization.
2. Understanding of the duration of logs searched by organization.
3. Access to EventLog Analyzer as an Administrator.
   

Steps to follow

1. Case 1: Live Logs
2. Case 2: Archives 

Case 1: Live Logs 

Live logs are stored in two formats in Elasticsearch: Indexed format and zipped Indexed format.

Indexed Format: The duration of logs collected is less than 32 days old.

Zipped Indexed Format: The duration of logs collected is older than 32 days.

1. All the event logs that are collected and proceeded by the Parsing Engine.
2. Processed Threat feeds and Dark Web monitoring data by the Threat Engine.
3. Event logs that are flagged by Alerts Engine.
4. Event logs that are flagged by Correlation Engine.
5. Technician User Activities performed in the application.

1. Current Storage Size: Current Storage Size defines the number of days the collected raw logs will be retained in the database. The logs collected earlier than the configured value will be deleted. The default value is 32 days.
2. Correlation Retention Period: Correlation Retention Period defines the number of days the formatted log data will be retained in the database. The logs formatted earlier than the configured value will be deleted. The default value is 90 days.
3. Alert Retention Period: Alert retention period defines the number of days that the alerts will be retained in the database. The alerts raised earlier than the specified number of days will be deleted. The default retention period is 90 days.
4. Audit Retention Period: Audit Retention Period defines the number of days that the Audit data for External APIs and technicians will be retained in the database. Audit data that is earlier than the configured value will be deleted. The default value is 90 days.
   

 Case 2: Archives 

a. Format: 

All the archive files will be stored in .gz format which contains compressed and encrypted (optional) flat files.

Event logs can be stored either as raw logs or raw logs with parsed fields. Default value would be raw logs with parsed fields.

b. Type of logs stored:

All the event logs that are collected will be stored in the archives as Raw logs or Raw logs with parsed fields based on the log type selected in Archive settings.

c. Policy-based archiving:

EventLog Analyzer provides policy-based log archiving which features a Default Config archive policy which applied to the log sources. User can create custom policy using the + Create New Policy option available in the archive settings to create new policies for selective devices any number of times. When you configure multiple archive policies for the same log source, device-based archiving will work based on the priority of the policies configured. The policy that is on the top has the highest priority.

d. Duration of Data stored:

The default value of Archive during installation will be set as Forever, which means the archive will not be deleted until a manual intervention is done. User can customize the duration based on your requirement by modifying Retain Archive Logs For value using the templates, or choosing the Custom option to enter the desired retention value based on days, weeks, months, or years.

1. Retain Archive Logs for: Set the desired log term duration that you would like to set for the device.
2. Archive Zip Location: Set the location where you would like to store the Zipped archives. The options available will be to store them in the Local, Shared location, or S3 bucket.
3. More Options >> Temp File Location: Set the location where you would like to allocate the flat file creation and zipping process.
4. Note: You should retain the files in the directly attachable storage with specifications recommended in the System Requirements.
5. Flat File Encryption: Used to Enable/Disable the Flat File Encryption option.

Tips

1. Set higher retention for Alerts, Correlation, and Technician audit in Retention settings as the preauditing and security requirement information will be available only in Elasticsearch.
2. Configure Live logs based on usability. Storage consumption can be minimized by minimizing the duration set for live logs.
3. Create multiple archive policies to store device logs based on auditing and security needs for flexible storage needs.
4. Use the System Resource Calculator to calculate the estimated storage size based on log flow and log storage retention requirement.
5. It's recommended that you use a high performing disk with less OS load to store Live logs and archive temp files.