Real World Audit Examples: Product Weaknesses
Having just gone through a real world audit last year and trying to use this product to produce the reports the auditors required we found it
severely lacking in several areas. See below for the main issues we faced:
- NTLM events were not even collected by the product until the last release of 2015. This was not documented anywhere and meant that reports were effectively useless for audit as you might be missing huge amounts of logon data. In response to community outcries this was eventually added.
- If you need to restore events (we archive everything older than 30 days) once restored they will again be archived after 2 days! This is not documented anywhere in the product (but confirmed with support). This caused hundreds of reports to be inaccurate for use as again, there is no indication anywhere that this is occurring. This is still an issue in the latest release.
- Reports can be run across time-frames even though the events in those time frames have been archived and need to be restored before being reflected in the reports. This has been addressed in the latest build and now when you run a report across a time frame it will prompt you to restore archived events.
- No ability to run a report on a specific file/folder making detailed quick reports impossible. Only workaround to run report against a whole share and filter thereafter which is very time consuming.
- The product resolves IP addresses to DNS names in events ehrn the event is collected. This is actually against the entire idea of audit as you should only event be reporting on the data in the event. If there is no hostname in the event raw data you should never resolve it to a DNS name as you have no guarantee that the hostname that will resolve for a given IP address at the time you process the event was the hostname for that IP Address at the time the event was generated. Also, it does not account for common scenarios where the DC might be in one DNS domain and the AuditPlus server in a different on. If have seen the IP address in the event resolve to different hostnames depdning on the DNS suffixes used to resolve it.
- The biggest weakness of this product in my opinion is the lack of a data warehouse. When dealing with auditors, they will often want reports run against long time periods (e.g. 6 months). This means you have to restore large amounts of data from .csv files. This seems like a very poor method of data management. Archived data should have the option to archive to a DataBase and then reports could search that DB directly and could be backed up with DB technologies. This would also remove the need to restore archived events whilst keeping space usage under control. The method of archiving data out to a flat file and importing back in is both cumbersome and old fashioned in my opinion.
As such, I doubt we will be renewing it next year and will look at something such as Splunk which is much more robust. The fact was that our reports simply were not accurate when we compared them with other auditing tools in our environment. We bought this product for it's easy user interface etc. but that is no use if the data is not accurate.