Performance issues are one of the biggest challenges to expect when designing and implementing web applications. Performance problems can disrupt your business, which can result in short and long term loss of revenue.

The ColdFusion support team at Adobe, has dealt with several performance related issues with ColdFusion and a couple of them, around JDK 1.8 as well.

Based on our experience with customers, the major performance issues could be categorized as – CPU hikes, Website crashes, Processing of slow requests, Memory (for example, OutOFMemory, Memory leaks) issues, Error 503/Service unavailable error, Slow performance while running database queries, SecureRandom seed generation on some linux servers, Network latency and likewise.

ColdFusion is a Java-based application server. Any Java-related change directly impacts ColdFusion. With introduction of Java 1.8, ColdFusion had to be optimized for Java 1.8. Even after the optimization, there could be few parameters, that might cause the performance hit on the ColdFusion server.

Let’s discuss about these performance issues, method to trace them, and their possible resolutions, in a bit more detail.

CPU Hike: Hikes in CPU usage are the most common performance issue, which we experienced. Usually, the absence of load and performance testing fails to predict the impact on existing CPU utilization. CPU hike can occur due to various reasons, such as: –

● Out of memory issues
● Excessive Garbage collection
● Slow database query processing
● Network latency
● Linux random number generation
● Security scanner

OutOfMemory
This is the first parameter we must look at, when CPU surges/spikes are seen in your ColdFusion application. We can check the ColdFusion logs for OutOfMemory entries. There are two possible scenarios further.

● OutOfMemoryError: Heap – Generally OutOfMemory:Heap would not only happen because application usage is higher than the upper limit provided. But, it can also happen, because a lower value of heap, than actual usage could slow down the jvm. OutOfMemory could be because GC is not able to claim memory. This could happen because of strong references to stale objects or aggressive load so that before GC cleans up OOM is thrown. The default value for Maximum JVM Heap Size is 1GB in ColdFusion (2016 release). Based on your application’s memory usage, we can update the maximum heap value.You can change the value in ColdFusion Administrator or in jvm.config (ColdFusionXXXX/instance_name/bin).

● OutOfMemoryError: Metaspace: A new flag is available in Java 1.8 (MaxMetaspaceSize), allowing you to limit the amount of native memory used for class metadata.
In metaspace, most allocations for the class metadata are now allocated out of native memory. By default class metadata allocation is limited by the amount of available native memory. Garbage collection of the dead classes and classloaders is triggered once the class metadata usage reaches the “MaxMetaspaceSize”. Proper monitoring & tuning of the Metaspace will obviously be required in order to limit the frequency or delay of such garbage collections. Excessive Metaspace garbage collections may be a symptom of classes, classloaders memory leak or inadequate sizing for your application. If you don’t specify this flag, the Metaspace will dynamically resize depending of the application demand at runtime.

Excessive Garbage collection:
Extra load on a server triggers increased GC and causes CPU spikes. There are four types of Garbage collectors. We must figure out the one, that best suits your application.

For more information on Garbage collections please refer to below documentation:
http://www.oracle.com/webfolder/technetwork/tutorials/obe/java/gc01/index.html

By default, ColdFusion uses parallel GC. You can change the values in jvm.config (ColdFusionXXXX/instance_name/bin):

-XX:+UseConcMarkSweepGC
-XX:+UseParallelGC
-XX:+UseSerialGC
-XX:+UseG1GC – This is recommended when heap size is large (At least more than 4GB)

For detailed investigation of memory leaks or out of memory errors, a heap dump analysis can be very useful. Add the following jvm arguments in jvm.config(ColdFusionXXXX/instance_name/bin) to obtain heap dump:
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath= <path_to_dump_file>

If you have JDK installed, run the following command from jdkbin directory:
jmap -dump:format=b,file=dump.hprof  where pid is the ColdFusion process id.
You can use Eclipse Memory Analyzer Tool (MAT) to review heap dumps.

Slow Database query processing:
ColdFusion logs (Application, Exception, error log) sometimes indicate whether your queries timeout or not. You can then identify slow queries and fix them.

CF does closes the connections after the timeout. CF reuses the idle/unclosed connections. When required unless the connection is still busy executing some query. If the query execution is taking too long, it has to be a problem with either the application or the database. The idle connections are being re-used, as and when required.

Technical details about the timeout:
We take two parameters in admin for this – Timeout and interval.
CF closes a maximum of 5 timed out connections at each interval. Say for example, If we have 20 open connections with timeout being set to 10 and interval being set to 5, then CF will close:-
0 connections after 5 mins
5 connections after 5 more mins
5 more connections after 5 more mins
5 more connections after 5 more mins
5 more connections after 5 more mins

So to close all the connections (as per above calculation), CF will take at least 25 mins to close, all open connections. The maximum limit of closing 5 timeout connections is not configurable and is by design.

The optimized value for timeout can be set 5 and interval to 1. You can configure them further, as per your application requirement. You can change the database timeout value in CF administrator in Advanced settings of Data & Services > Datasources to optimize idle/unclosed connections.

Network latency
If the application code resides and being accessed from a shared drive in ColdFusion Application, network latency can cause slow request processing, resulting in performance issues. This can even cause a server to crash/unresponsive. Its highly recommended to check your internal Network throughput. You can also refer to the information available on below blog:
http://blogs.coldfusion.com/source-code-deployed-on-network-path-identifying-network-latency/

You may try below :
Add the jvm arguments below to speed up the the processing of cfm pages on network/shared location:
-Dsun.io.useCanonPrefixCache=true -Dsun.io.useCanonCaches=true.

Note that 30 sec is default timeout.

This enables canonical cache that caches the canonical path of a file. This helps, when there are a lot of threads waiting to get path from WinNTFileSystem. While accessing files from a network drive, each “getCanonicalPath” would end up going to network and would become quite expensive task. Enabling this cache means that for same file, JVM would never go back to disk (till the time it is in cache) to find its path.

Linux random number generation:
Random number generation and server startup is slow on Unix platforms for some of the servers. This could be because, /dev/random is used in Unix platforms for random number generation.
java.security.SecureRandom is designed to be crypto secure. It provides strong and secure random numbers. SecureRandom should be used when high-quality randomness is important and is worth consuming CPU. We can add the below jvm argument, to get rid of performance issue due to random number generation:
-Djava.security.egd=file:/dev/./urandom

Security scanner:
If you see CPU spikes at some specific time of the day/week, this could be due to a third party security scanner interfering with your ColdFusion application. The scanner hits the server monitoring port 5500 (by default) with 0.0.0.0, which goes to infinite loop and causes server crash.

To fix this issue, we need to modify the jetty.xml at ColdFusionXXXXcfusionlib. Change the Server monitoring IP address from 0.0.0.0 to 127.0.0.1 and restart ColdFusion.

Code Cache:
If your program has high codecache memory set via -XX:ReservedCodeCacheSize, you can limit it by disabling code cache flushing. If flushing is disabled, the JIT does not compile methods after the codecache fills up and hence there won’t be CPU hikes. You can add the following jvm argument. This can be used to flush code cache.

XX:-UseCodeCacheFlushing
You can also disable tieredcompilation with below argument:
-XX:-TieredCompilation (Applicable only with Java 1.8. Java versions less than 8 doesn’t have tiered compilation enabled by default.)

Service unavailable error:
503 – Service unavailable is a generic error. Whenever we get this error, the first thing we should check is, whether ColdFusion is started and running or not. In case you experience intermittent 503’s, then its time to investigate the less responsive server, which might be dropping requests. This could be because of Long GC pauses or any reason that could delay response from ColdFusion server. The ColdFusion connector tuning can help us to overcome service unavailable error. Below blog post can be used to tune ColdFusion connector and avoid such errors.
http://blogs.coldfusion.com/coldfusion-11-iis-connector-tuning/

We have also seen some issues because of bugs in few specific update level of java. The best practice would be to keep your ColdFusion Java updated to latest version. Use the below blog to keep your java up to date.
http://blogs.coldfusion.com/installing-and-troubleshooting-java-updates-in-coldfusion/

ColdFusion thread dumps:
ColdFusion thread dumps can be used to analyze New, Runnable, Blocked, Waiting, Timed_Waiting and Running threads.
The issues such as Thread race, Deadlock, Hang IO calls, GC/OutOfMemory exceptions, Infinite Loop can be determined using the thread dumps. Following Blog can be used to take thread dump on a ColdFusion server:
http://blogs.coldfusion.com/taking-thread-dumps-from-coldfusion-server-programmatically/
If you are on CF11 update 12 and CF2016, you can skip copying threaddump.jar. And just use takethreaddump.cfm file to capture the thread dump.

Another issue we have seen in one or two cases, If the performance is impacted by XML parsing, the jvm argument below can fix it:
-Dcom.sun.xml.bind.v2.bytecode.ClassTailor.noOptimize=true

The other causes of performance issue include:
Lack of proper database SQL tuning & capacity planning
Application specific performance problems
Lack of proper data caching
Excessive data caching
Excessive logging

In case the above steps does not resolve the issue, please feel free to contact Adobe support (https://helpx.adobe.com/support/coldfusion.html) for analysis of the issue.

Some key points to remember:-

*Please note that, any changes made to jvm.config, would require a ColdFusion service restart.

Jvm.config: ColdFusionXXXX/instance_name/bin
ColdFusion logs: ColdFusionXXXX/instance_name/logs

Reference:
https://dzone.com/articles/java-8-permgen-metaspace
https://dzone.com/articles/top-10-causes-java-ee