Since AMPS is used in a wide variety of applications that range from extremely latency-sensitive parts of order processing all the way to large-scale but low volume telemetry collection, there is no single set of values that makes sense for all AMPS applications.
Rather than providing one-size-fits-all numbers, 60East recommends that you benchmark the counters that make sense for your application against a test run that provides the performance and scale that the application needs and then consider that initial data to be the baseline normal for the application in production. Determine the normal variation over a set of test runs, and scale that variation by some reasonable factor to set alerting thresholds. When problems emerge, adjust the numbers to be sure that the monitoring catches the problems. Periodically re-evaluate the baseline and the variation in numbers to see if it's necessary to adjust the baseline or the tolerances.
Also bear in mind how the application uses AMPS: metrics that may be normal for certain days, or certain times of the day, may not be normal at other times. Likewise, the operating environment for the instance may affect the expected numbers.
For example, consider an application that receives a large amount of data from a set of publishers at the beginning of the day then has a restricted set of processes that analyze the data throughout the day. For that application, you may want to monitor the number of active connections. You would allow a high value at the beginning of the day with large numbers of messages entering AMPS. You would configure the monitoring so that later in the day the monitoring allows a smaller number of clients with low numbers of messages entering AMPS and high numbers of messages leaving AMPS.
As another example, an AMPS instance that is the sole tenant on a high-end system dedicated to low-latency trading may have very different performance characteristics and a much narrower acceptable range of variance in performance counters than a view server instance that is co-located on a virtual machine that runs several other AMPS instances.
To help determine reasonable baseline values, 60East support is happy to assist with analyzing the metrics captured in the statistics database during normal operation or test runs. With this information and information on the environment and service-level guarantees for the application, 60East can work with you on initial baseline numbers.
Keywords: monitoring, admin, Galvanometer, ITRS, DataDog, Geneos, Grafana, Prometheus