Troubleshooting: Disk space issues in EventLog Analyzer
Issue description
EventLog Analyzer server might run out of storage due to misconfiguration and other known factors. This article offers troubleshooting steps to resolve when your disk or drive where the application is installed is full and help you to resolve the concern.
Possible causes
Low System Requirements: Offering low disk storage and setting up higher storage retention for live or archive data which is beyond the storage size offered. Offering low specification hardware resources (like CPU, RAM) can cause performance issue which leads to formation of CachedRecord files (temp raw logs) or Out of Memory issue which indeed affects the performance of log injection however, also consumes disk space.
Misconfiguration: Misconfigurations correlation rules with higher observable duration, unreachable mail server etc can create temp files can consume storage.
Log folders size getting exceeded(less likely to happen): Application's logs folder created large log files due to errors in the application/configuration
Prerequisites
- Ensure system requirements are met: RAM, CPU, disk space & speed, and VM configuration as per suggested System Requirement or System Resource Calculator
- Ensure you are or near to latest version to have optimal performance. Refer Upgrade to the latest version of EventLog Analyzer for latest PPM version and service pack upgrade.
- Access to EventLog Analyzer server as administrator.
- Know the EventLog Analyzer edition (Bundled/Common ES vs Standalone/Local ES).
- Access to EventLog Analyzer Admin Settings.
Resolution
Step 1: Identify Storage Consumption
Use system tools or manually check folder sizes of the following to validate which folder consumes much storage and preform steps 2 actions accordingly:
Step 2: Reduce Storage consumption.
- Folder location : EventLog Analyzer/archive
This is the default location where EventLog Analyzer archive logs are stored. To know the exact location, you can check them in Settings >> Admin Settings >> Archives >> Settings(top right corner). (Image below)
To free up the storage, you can perform either of the following:
- Migrate Archive data to different locations (preferred by most users)
- Setup multiple archive policy to store archive in multiple locations (Most Recommended & Preferred by most users)
Reduce Archive Retention:
The retention for EventLog Analyzer archives is set as "Forever" by default which can be adjusted as per requirement in Archive Settings.
Reducing the archive retention settings can reduce the storage consumption. Please follow the below steps to reduce archive settings.
NOTE: Reducing the archive retention will take effect within 12 hours for data cleanup which once deleted cannot be restored. Archive files of the devices which you have deleted earlier should be removed manually. You can also restart the EventLog Analyzer service to have the settings to start taking effect in 60 minutes.
a. Navigate to Settings >> Admin Settings >> Archives >> Settings(top right corner)
b. Adjust the value of "Retain Archive Logs For" to required value.
c. Choose Save to apply the changes.
Migrate Archive data to different location:
To reduce the storage occupied by archives, you can migrate the existing Archive zip files to different location which indeed free up the archive space.
Refer following Knowledge Base articles for more details.
How to migrate the EventLog Analyzer Archive ZIP file from one location to another in a Windows instance
Setup multiple archive policy to store archive in multiple locations:
You can configure multiple archive policy to store archive in various location. Creating archive policy offers you option to store select log source's logs in a specific location with required log retention. By creating multiple archive policy, you can store the data in different location as per requirement.
Example: By Creating 3 archive policy(3 new archive policy and 1 default policy), you achieve the following.
Domain Controller & Database logs can be stored in a restricted file server for 10 years, Network device logs can be stored for 5 years in File server, Workstation logs can be stored for 3 year in NAS storage and All other archives to be stored for 1 year in S3 bucket.
Domain Controller & Database logs can be stored in a restricted file server for 10 years, Network device logs can be stored for 5 years in File server, Workstation logs can be stored for 3 year in NAS storage and All other archives to be stored for 1 year in S3 bucket.
Refer How to create new archive policy in EventLog Analyzer for more details.
Best Practice: If the disk is full, you can consider migrating the data to different location(s) and set up multiple archive policy with required log retention based on your organisation's log term log storage needs.
ii. Folder location: EventLog Analyzer/ES/data or elasticsearch/ES/data
This is the default location where EventLog Analyzer live data is stored which is the data that you are able to view in the User interface in a click of a button. The default value is set for 32 days. Disk Space consumed in this location is based on the number of logs injected/collected and the duration of log storage in the live logs.
To reduce the storage consumed, you can perform the following.
Reduce the live logs storage retention:
It is always recommended to have the live logs stored based on the most frequently searched logs or based on the scheduler's duration.
Example: If you prefer to search a week of data always, or past 10 days of data or usually like to export a month of data, based on your requirement, you need to set up the live logs storage retention.
Example: If you prefer to search a week of data always, or past 10 days of data or usually like to export a month of data, based on your requirement, you need to set up the live logs storage retention.
This value can be adjusted in Settings >> Admin Settings >> Retention Settings. Reduce the value of Current Storage Size to reduce the storage.
NOTE: Applying the changes in retention will take effect by 12:00 AM on the same day. You can load archive anytime to view the old data.
Migrate live logs to different location:
You can migrate the live logs to different drive to reduce the storage on the current directory.
Live logs storage always needs a disk with high IOPS. Refer the following document to perform the migration.
How to migrate live data (ES Data) from one location to another - Windows
Best Practice: If you have a high log flow, you may reduce the live log retention as per the standard duration of search or scheduled export duration. Migrate the data to different directly attached storage drive as of installation directory with higher storage space based on the log retention set.
iii. Folder location: EventLog Analyzer/ES/archive
This is the default location where the following data are stored.
- Correlation engine data
- Alerts recorded
- Technician audit data (least expected to occupy the storage)
- Zipped live logs (Live logs will be zipped in this location that are exceeding the default "Zip index after" period) - (More likely to occupy space when Current Storage Size value in Retention Settings is more than 32 days.
To reduce the storage, you can perform the following.
Reduce the occurrence of events:
Reducing the occurrence of events by fine tuning the correlation and alert profile can help in reducing the storage.
Recommendations to fine tune:
- Understand the use case and validate if it is an Alert or Correaltion event.
- Build the criteria for Alert or Actions(correction) which can match the event perfectly.
- Set up threshold limit wherever needed. You can use Adaptive thershold in Alert to have the alert recorded based on thershold set by ML base line.
For more details, refer How to edit Correlation rule in EventLog Analyzer or How to create Alert profile in EventLog Analyzer
Reduce the retention value:
Reducing the retention value in Settings >> Admin Settings >> Retention Settings will have a direct impact.
You can reduce the storage of the following entities, reduce the respective user interface value for the mentioned entity as per the following.
Zipped live logs can be reduced by lowering the Current Storage Size value to 32 days or less.
Correlation/Detection data can be reduced by lowering the Detection/Correlation Retention Period.
Alert data can be reduced by lowering the Alert Retention Period.
Technician Audit data can be reduced by lowering the Audit Retention Period.
NOTE: Applying the changes in retention will take effect by 12:00 AM on the same day.
iv. Folder Location: EventLog Analyzer/data (This folder is less likely to consume the storage)
This folder serves as temporary storage for storing the processed Alerts, Correlation/Rule Detection and Notification before being processed. If you find this folder consuming the storage, you can perform the following.
a. If NotificationDump folder consumes storage check if Email/SMS server is properly configured and if you are getting notification for any configured alert.
b. For other sub folders check if the application installed server is allocated with sufficient resource as per recommended System Requirements.
c. Reducing the occurrence of events by fine tuning the correlation and alert profile can help in reducing the storage.
Recommendations to fine tune:
- Understand the use case and validate if it is an Alert or Correlation event.
- Build the criteria for Alert or Actions(correction) which can match the event perfectly.
- Set up threshold limit wherever needed. You can use Adaptive threshold in Alert to have the alert recorded based on threshold set by ML base line.
For more details, refer How to edit Correlation rule in EventLog Analyzer or How to create Alert profile in EventLog Analyzer
v. Folder Location: EventLog Analyzer/ES/CachedRecord
CachedRecord folder are the logs collected but unprocessed. This could be due to resource usage by Java to process the data or insufficient hardware resources at that time. Please do not delete those files as it would lead to data loss.
Those files will automatically be processed when the product gets enough room to process the logs. As long as the files in that folder get processed and go empty, without keep accumulating, there is nothing to be concerned about.
If you see the files getting accumulated in this folder, Refer CachedRecord - Troubleshooting for more details.
vi. Folder Location: EventLog Analyzer/ReportHistory
This folder servers as the location of Scheduled Export repository where all the reports that are exports will be stored in this location. The exports can be found under unique ID(SchedulerID) which can be safely moved or removed directly from the location.
NOTE: Once you move/remove the export file, the file will be no longer accessible from the application. You will be not be able to download the file from the user interface from export history.
NOTE: Once you move/remove the export file, the file will be no longer accessible from the application. You will be not be able to download the file from the user interface from export history.
vii. Folder Location: EventLog Analyzer/logs
This is the default location where EventLog Analyzer application logs are printed. Identify which folder or file name consumes the storage and feel free to reach out the support team for analysis on the application log growth if you find any single log file size more than 1GB.
Expected file or folder which can consume storage: Serverout, Archive(archive location of log files where all the older logs will be retained for certain duration of time)
The Archive folder within the Logs directory can consume significant storage. You may manually remove older files to free up space.
viii. Folder Location: EventLog Analyzer/pgsql
This is the default location of built-in Postgresql database where all the configuration are stored. This location is less likely to consume storage however, if you notice the storage consumed abnormally, feel free to contact support to validate this further with the details of which sub folder or file consumes the storage.
If the issue persists, feel contact support team with the related details and troubleshooting/Analysis performed.
If the issue persists, feel contact support team with the related details and troubleshooting/Analysis performed.
Tips
- Plan to upgrade the application to the latest version to ensure seamless optimize performance. Refer Upgrade to the latest version of EventLog Analyzer for latest PPM version and service pack upgrade.
- Configure Log Collection Filter to filter out noisy unwanted events and manage the storage effectively.
- Regularly audit and prune unused device log collections to free up the license & storage.
- Configure multiple archive policies with priority to optimize storage.
Related topics and articles
How to reach support
- EventLog Analyzer build number and product version
- Which folder or sub folder consumes storage.
- Troubleshooting and actions performed so far.
New to ADSelfService Plus?
Related Articles
Troubleshooting guide: EventLog Analyzer UI is unresponsive
Overview This document outlines the common causes and recommended steps to resolve the issue when the EventLog Analyzer UI becomes unresponsive. Possible causes Insufficient system resources High CPU or memory usage on the server. Low disk space in ...Unable to start EventLog Analyzer
Issue description This issue occurs when the EventLog Analyzer service fails to start, or when users are unable to access the web client through the browser (typically on ports 8400 or 8445). Users may experience one or more of the following ...How to reduce the number of cached records in EventLog Analyzer
Issue description Cached record files are unprocessed log files that can accumulate within the EventLog Analyzer local directory when the indexing process is disrupted. When the number of cached record files exceeds 100, an email notification will be ...How to get notified about unprocessed log files in EventLog Analyzer
Objective EventLog Analyzer provides email notifications when unprocessed log files accumulate and form cached records. This helps administrators detect potential issues in log ingestion or processing early, enabling proactive troubleshooting to ...Troubleshooting guide: No data available in a Compliance Report
Overview This document provides a technical explanation and resolution guide when there is no data being displayed under compliance reports in ManageEngine EventLog Analyzer. Compliance reports include regulatory standards such as PCI-DSS, HIPAA, ISO ...