Tuning Heap Memory Performance
Long garbage collection (GC) cycles are one of the most common causes of issues in a Terracotta cluster because a full GC event pauses all threads in the JVM. Servers disconnecting clients, clients dropping servers, and timed-out processes are just some of the problems long GC cycles can cause. Having a clear understanding of how your application behaves with respect to creating garbage, and how that garbage is being collected, is necessary for avoiding or solving these issues.
Printing and Analyzing GC Logs
The most effective way to gain that understanding is to create a profile of GC in your application by using tools made for that purpose. Consider using JVM options to generate logs of GC activity:
-verbose:gc-Xloggc:<filename> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps Apply an appropriate parsing and visualization tool to GC log files to help analyze their contents.
Observing GC Statistics With jstat
One way to observe GC statistics is by using the Java utility jstat. The following command will produce a log of GC statistics, updated every ten seconds:
jstat -gcutil <pid> 10 1000000
An important statistic is the Full Garbage Collection Time. The difference between the total time for each reading is the amount of time the system was paused. A jump of more than a few seconds will not be acceptable in most application contexts.
Solutions to Problematic GC
Once your application's typical GC cycles are understood, consider one or more of the following solutions:
Maximizing BigMemory to eliminate the drag GC imposes on performance in large heaps.
BigMemory opens up off-heap memory for use by Java applications, and off-heap memory is not subject to GC.
Increase nodes' tolerance of inactivity in other nodes due to GC cycles.
Tuning the GC parameters to change the way GC runs in the heap.
If running multi-core machines and no collector is specifically configured, consider -XX:+UseParallelGC and -XX:+UseParallelOldGC.
If running multiple JVMs or application processes on the same machine, tune the number of concurrent threads in the parallel collector with -XX:ParallelGCThreads=<number>.
Another collector is called Concurrent Mark Sweep (CMS). This collector is normally not recommended (especially for Terracotta servers) due to certain performance and operational issues it raises. However, under certain circumstances related to the type of hosting platform and application data usage characteristics, it may boost performance and may be worth testing with.
If running on a 64-bit JVM, and if your JDK supports it, use
-XX:+UseCompressedOops.
This setting can reduce substantially the memory footprint of object pointer used by the JVM.
Refactoring clustered applications that unnecessarily create too much garbage.
Ensuring that the problem node has enough memory allocated to the heap.