Causes of Transaction Count Mismatch

1. Sampling Factor: APM Insight tracks only a subset of requests to reduce overhead. The Sampling Factor in the agent configuration profile is a 1-in-N counter, not a percentage — a value of 3 means the agent instruments 1 in every 3 requests of the same kind. Newly created agent configuration profiles default to 3, so some sampling is active out of the box. Setting the factor to 1 tracks every request and is what aligns the counts to a 99% to 100% match.
   
2. Smart Sampling (Java agent only): When enabled, the Java agent automatically increases the sampling factor based on the real-time CPU and memory consumption of the JVM. Under load the effective factor can become a multiple of the configured value, so the tracked share of requests varies with load rather than staying fixed.
   
3. Configuration Issues: Misconfigurations in the agent configuration profile or application settings can lead to incorrect data collection. This includes filters, exclusions, and incorrect threshold settings.
4. Network Latency: Delays or losses in data transmission between the application and the APM Insight agent can affect transaction reporting.
5. Data/Time Aggregation: Differences in how data is aggregated and reported by our agent compared to the application can cause discrepancies. APM tools often use different time windows or aggregation methods, resulting in these differences. There could be maximum of 1 or 2 minutes difference between data collection and reporting times.
6. Calculation Differences: Custom time ranges set in Applications Manager may include an extra minute or use  different rounding rules, affecting the transaction counts.
   

Troubleshooting Steps

1. Set the Sampling Factor to 1: In the agent configuration profile, set Sampling Factor to 1 so that every request of every kind is tracked. Compare the counts over a controlled period before deciding whether to restore a higher value. 
   
1. 

   Do not enter 100. The field is a 1-in-N counter measured in transactions, not a percentage. A value of 100 tracks only 1 in 100 requests. 1 is the value that tracks everything.
2. Note: Sampling is applied when the request starts, before its outcome is known. Requests that are skipped by sampling are therefore not recorded even if they later fail, which is why counts of errored requests can also fall short of the application's own logs.
2. Disable Smart Sampling (Java agent only): Smart Sampling raises the sampling factor automatically under CPU or memory pressure and can drop transactions. In the agent configuration profile for the Java instance, uncheck the Enable smart sampling checkbox and save. The profile is pushed to the agent from the server, so no agent restart or file edit is required.
   
1. 

   Setting smart.sampler.enabled=false in <agent_extracted_path>/apminsight.conf has no effect. Sampling and tracing behaviour is delivered to the agent from the server-side agent configuration profile, which overrides the local file. Always use the Enable smart sampling checkbox in the profile.
2. Smart Sampling exists for the Java agent only. It is not applicable to the .NET, .NET Core, Node.js, PHP, Python or Ruby agents, and the checkbox is not shown for them.

3. Compare Data Collection Methods: Understand the aggregation and filtering methods in both systems. For example, ensure that no transactions are being filtered out or preprocessed in a way that affects the counts.

4. Analyze Network Traffic: Monitor for packet loss and network latency.
5. Review Logs and Metrics: Look for discrepancies and missing entries.
6. Test with Controlled Transactions: Perform controlled tests to compare reported counts. For example, execute a known number of transactions and compare the counts reported by the agent and the application. This can help isolate the source of the mismatch.
7. Adjust for Time Aggregation and Calculation Differences: Ensure matching time aggregation settings and custom ranges. For example, be aware that custom time ranges in Applications Manager might include an extra minute or use different rounding rules. Adjust the custom ranges to match the application's reporting periods.
   

Sampling Controls by Agent Type

Transaction Counts vs. Trace Snapshots

1. Transaction counts and metrics are recorded for every tracked request. With Sampling Factor 1 and Smart Sampling disabled, counts should align with the application's own numbers.
2. Trace snapshots (the detailed drill-down under the Traces tab) are captured only for requests that meet a trace condition — response time above the Trace Threshold, an error, a slow SQL statement, a distributed trace, or an external-component count above its threshold. In addition, only the most significant trace per transaction is retained per collection window. A fast, successful request therefore has a count but no trace snapshot, by design. Lowering the Trace Threshold widens trace capture but increases agent and server overhead, and is not recommended on production systems without review.

Additional Considerations

Conclusion

Transaction count mismatches can be resolved by setting the Sampling Factor to 1, disabling Smart Sampling on Java agents, ensuring complete instrumentation, and aligning data collection methods. By systematically troubleshooting and understanding time aggregation and calculation differences, a 99% to 100% match is achievable, ensuring accurate performance monitoring. Note that this applies to transaction counts; trace snapshots remain condition-based and are not captured for every request.