June 7, 2019
Solving problems calling out of CF via https, by updating JVM
Comments
(5)
June 7, 2019
Solving problems calling out of CF via https, by updating JVM
ColdFusion troubleshooter
Legend 126 posts
Followers: 90 people
(5)

(Originally written Jun 2019, updated Oct 2021, Jul 2021)

Sometimes the solution to a problem is not what many may suggest it to be. And the right solution may even be easier than one most commonly suggested.

Over the years I’ve had clients present to me (or have seen people write on the web ) that they were getting errors when trying to do a cfhttp call to an https URL (like to a payment gateway or AP), or it could be other tags like cfmail or cfftp,  or just admin settings like datasource connections or mail servers, etc.: anything calling out from CF to some other server via https/tls. The problem is that either when they changed “nothing” or perhaps changed CF versions, suddenly they get failures calling out to other servers.

The good news is that there is generally a simple solution.

TLDR: And I have found that in most cases the solution is NOT necessarily about importing certs or tweaking JVM args but is much simpler: they probably just needed to update the JVM that CF is using. I help you here to know how to do that, and what options you have.

Importing certs may well have worked for such problems in the past (and may be needed still in some unique cases), but in many cases, that’s NOT the only or even best solution. And if a cert DOES need to be imported, do be careful to import it into the CORRECT location (the JVM CF is using). And same with tweaking JVM args: at least be careful with what you’re doing, to make sure it makes sense for the version of the JVM that your CF is using.

Let me explain all these things.

Common error messages

If you have code trying to call out from CF to another server via https, you may at some point find CF reporting errors like these:

I/O Exception: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

or

I/O Exception: peer not authenticated

Or others. And even if not in a CF error, you might find these in the saved output of a CF scheduled task (calling out to an https URL).

Or the more generic error you may see in the output of a cfhttp could just be just a less-helpful:

Connection Failure

Still other errors you may see include:

  • I/O Exception: Could not generate secret
  • I/O Exception: Couldn’t kickstart handshaking
  • SSL Handshake failed

If you get any of these errors, it COULD be that you just need to update the JVM, as I discuss in this post. (Though it should be said that sometimes, you may get these errors for some different problem.)

What might be the real cause, when “nothing changed”?

It’s of course very frustrating when this happens and you feel, “nothing changed” on your end.

First, a very common reason for this sort of problem is simply that the destination server being called has changed what SSL or TLS versions or protocols it supports. And it may now be using a version of those which the JVM that your CF is relying on DOES NOT YET SUPPORT. And that JVM CF is relying on may not have been updated in years, as I will explain below.

Second, the destination server may well have changed something about its server certificates, such that the client certs (or root certs) stored within the CF JVM can no longer work with those newer certs. These may be certificates that you did install in the past, or they merely be the ones built into the java certificate store (cacerts, in the lib/security folder of the JVM that CF is using), but now they “don’t work”, again because the server you’re calling has changed, and the JVM (and perhaps default certs) that it uses no longer work.

What if the problem comes and goes?

This can drive you batty, where the failing URL “sometimes works” and “sometimes fails”. In this case, often the problem is that the URL you’re calling is being served by a load balancer on the “other end”, and one of the destination servers (in that cluster) has NOT yet had whatever change was done. So SOME of your requests end up on a server WITH some change (affecting you, that fail) and some of your requests end up on a server WITHOUT the change (and “work” for you).

Why updating the JVM that CF uses may help

Anyway, often the problem is simply that the JVM which underlies CF is just old (perhaps very old). Over the years, the JVM has been updated to support more and more recent protocols (and indeed to drop support for older ones), and to have more and more updated certs by default.

This could happen with ANY version of CF, since CF began running atop Java with CF6. Even with the VERY latest CF version, the JVM it’s using might not have been updated since you installed it months or years ago.

[Update in Mar 2021]: For example, this can happen even to those running CF2021. With the original installer that came out in November 2020, it came out of the box running Java 11.0.1–which is strange, as the Java 11 version current at that time was 11.0.9. Anyway, folks have had these sort of errors, and updating to that 11.0.9 or any later Java 11 version since then generally made such problems go away. As another update in Oct 2021: good news on this front is that Adobe refreshed the CF2021 installer in Sept ’21, and that now DOES include Java 11.0.11. While it was again not the latest available then or as of this update I’m writing, which is 11.0.12, at least 11.0.11 is still 2+ years more updated than the 11.0.1 which CF2021 originally came with.]

When I first wrote this post in 2019, the issue was that it was a client running a still older CF version. In their particular case, it was CF11 (but this problem can happen with CF2018, 2016, and other CF versions). Like many folks, they had left CF running with the original JVM that THAT CF installer came with (in 2014), which was Java 1.8.0_25. That was 5 years old, at the time of this writing, and the latest Java 8 version (in June 2019) was 1.8.0_212 (and Oracle is still updating Java 8 and 11, as “long term support” releases of Java, even as of this update to my post in Oct 2021. More on that in a moment)

So all we did in that last case (as I have done with many folks) was to simply install the current latest available Java update (of the latest Java version supported by the version of CF they were running), then told CF to use that, then restarted CF, and now their https calls via cfhttp and scheduled tasks were working. (If that solution doesn’t suffice for you, keep reading.)

What JVM version can I update to?

I have a post where I offer a table with each recent CF version what JVM versions that CF version supports (which may vary based on what CF updates you have applied to that CF version).

How to go about updating the JVM CF uses: questions you may have

When I say “all we did was to simply update” the Java that underlies CF, you may have lots of questions about that. 🙂 Here’s good news, I have done various posts that help with this, including one announcing some recent Java updates and where I addressed typical questions you may have about updating the JVM, including:

  • where to download the latest Java updates, from Adobe (and why)
  • where to find more about HOW TO update the JVM CF that uses
  • what versions of CF support what versions of Java
  • why you should NOT use Java 12, even though it’s “more recent” than Java 11
  • how to recover if you make a mistake trying to update the JVM CF that uses

(I should add that even updating to a slightly older Java update than the very latest one possibly could solve this problem as well, for those who “don’t want to be on the bleeding edge” of latest Java updates. But it’s generally best to be on the latest available update of the Java version supported by your version of CF, and Adobe does recommend that.)

Whether you’ll be updating to that specific Java update or not, check do out that other post for answers to those and some other common questions about updating the JVM that CF users.

What if you really DO need to import certs? Be careful about where you do!

I’ve shared here mostly how it is that for most people they do NOT need to import any certificates into CF’s java certificate store (cacerts).

But some people WILL need to do that. Maybe the server you’re calling is one that only allows calling into it if the http client (in this case, CF) has implemented a special client cert that the server owner has provided. OK, that sometimes IS necessary. Why then may the error above happen when you feel you DID import the cert?

I can tell you that a very common problem is that someone HAS updated the JVM that CF uses (see above), but now they have found instructions on the web showing how to use the java keytool to import the cert into cacerts file found in the lib/security…only the instructions tell you to point it at the cacerts in the /jre/lib/security directly under the CF folder. But if someone has told CF to use a new JVM in a new location (again, see above), then CF is NO LONGER paying attention to that cacerts in its jre/lib/security folder.

In that case, the simple solution is to import the cert (if indeed it’s really needed) into the lib/security/cacerts of whatever JVM you are using. Look at the CF Admin and its “java and JVM” page, to see what the java home (first field) points to. THAT is the JVM CF is using (also found in the java.home line of CF’s jvm.config file.)

Note that if the JVM is a Java 8 JRE, the folder will be in jre/lib/security, while in a JDK (and for Java 11, that’s all there is), you will find the cacerts in the JVM’s lib/security folder.

What about cacerts in other places within CF?

You may also discover that there are lib/security/cacerts folders in other places within, particularly its jnibridge and jetty folders (under CF’s cfusion folder, or under any sibling “instance” folder, if you are running CF Enterprise and created multiple instances.). Those are NOT certs you need to worry about. Those are for the JVM’s implemented in support of the process of INSTALLING the features that use those two folders, namely .NET integration (for the jnibridge folder) and Solr operations and CFHTMLtoPDF (for the jetty folder). You can safely ignore those.

What you may not need to bother with

So really, the JVM update may be all most need, and not any other approaches. But what are the other approaches that some may suggest, that you may not need?

First, you may NOT need to import any certificates after all. You will often see resources on the web proposing that as the solution. Indeed, one of the resources my client had found (which led them to ask if they needed to go about importing a new cert) was this blog post of years ago from the venerable hass.de site.

He was explaining why importing certs may be needed to solve some SSL problems.  Again, sometimes it may be. Again, for the client I helped which prompted this post, they were using an older CF with an older JVM from years ago. It may well be that such an old JVM had expired certificates and root certs, leading to a need for some (at some time in the past) to do such cert imports, before a newer jvm included the better certs.   (Indeed, sometimes people have been importing certs by rote for years every time they did update JVMs thinking they needed them, when instead all they needed was to update the JVM to get the better certs and root certs it would have by default.)

But again just updating Java worked for my client and I hope it may for some of my readers.

And that hass.de blog did indeed mention in passing that “an update to latest Java sometimes also help, too.” That would be easy to miss amid all else in the post.  One reason I point this out here is that sadly that hass.de site no longer allows comments, so I couldn’t add this clarification there. (I get it that more and more sites are not permitting comments. I see the pros AND cons.)

A JVM arg that may or may not help

Second, the commonly suggested Dhttps.protocols arg may not help or be needed: again you may see resources on the web proposing adding those as a solution for some cf ssl issues. But it is not the right solution for this more common problem (of the destination server changing what it supports, and that being something that the JVM CF is running on does NOT support).

This Dhttps.protocols arg would be for limiting what protocols CF/the JVM should use. But the SSL negotiation between CF and the destination server (such as in a CFHTTP call or scheduled task URL) should cause CF to use whatever the best protocol the destination DOES support. But again, if the ones that the destination DOES support are not ones that the JVM underlying CF supports, you will get the error. As such, this jvm arg won’t help, if you’re asking your OLD JVM to support a protocol it doesn’t know about.

That said, this jvm arg CAN be helpful when instead the problem is that the server you are calling out to wants to use a higher TLS version than will work for your JVM version (and either you can’t update your JVM, or there’s some other issue).

For example, as of my update to this post in Oct 2021, I have helped some people running even CF2021 (and Java 11.0.12) solve such https/tls problems by adding:

-Dhttps.protocols=TLSv1.2

or

-Djdk.tls.client.protocols=TLSv1.2

You might even be told to try -Dhttps.protocols=TLSv1.1,TLSv1,TLSv1.2, and that may have been right for older JVMs, but something to note that as of Java 11.0.11, the JVM does not support calling out to those TLS versions by default. That deserves its own new section here, which I add next.

As for adding or changing such JVM args, you’d do that on java.args line of CF’s jvm.config file (being careful to save a backup of the file first, to recover if you make any mistake), and then restart CF for the change to take effect. (While you can also modify such args in the CF Admin “Java and JVM” page, a risk there is that if you make a mistake, and CF won’t restart, then you won’t be able to use the Admin to “undo” your change.) Again, I discuss this in my post, CF911: ‘Help! I’ve updated the JVM which ColdFusion uses, and now it won’t start!’

But again doing this jvm arg change about protocols should not be the FIRST step folks try, if they have not yet updated the JVM as discussed in the rest of the post here. That generally makes far more sense to do, for a number of reasons.

JVMs since Apr 2021 remove TLS v1.1 and 1.0, by default

[Update: Oct 2021]

I’m adding this as a new section in Oct 2021.

I mentioned in passing at the end of the last section that another issue can cause problems (in calling out of CF using https/tls): recent Java updates (updates to Java 11 and 8 since April 2021) don’t support calling out to servers that don’t yet support TLS v1.2. Specifically, the JVM regards as “disabled” the older TLS v.1.1 or 1.0 versions and prevents CF calling to a server using those.

For more on that change, and how you can change your JVM configuration to allow talking to such “dirty old TLS versions”, at your own risk–if you feel you must, see my blog post on the JVM updates at that time.

A JVM arg that may help, with TLS mail problems at least

Before wrapping up, if you may confirm that your JVM is up to date and STILL get problems specifically with sending out email from CF (when you have told the CF Admin or CFMAIL tag to “use tls”), you may find yourself getting an error in the mail.log, saying “javax.mail.MessagingException: Could not convert socket to TLS; nested exception is: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target“.

And from that error you may again think (or find people asserting) that “you need to import a certificate!”. But again maybe you don’t.

You may just need to tell the JVM (underlying CF) to “trust the mail server you are sending to”. If you just have one that you are setting in the “mail server” field of the CF Admin Mail page (or in the Server attribute of a cfmail), you could tell the JVM to trust that server  (like smtp.example.com) with a JVM arg like:

-Dmail.smtp.ssl.trust=smtp.example.com

You can even throw caution to the wind (with due concern over the security implications) by using:

-Dmail.smtp.ssl.trust=*

But this is now getting far afield of the original problem. But I’ll add just one more, for good measure:

A JVM change that may break accessing LDAP servers

As long as I’m updating this post (in Oct 2021), I’ll add this one more new section, about one more JVM change (and jvm arg) related which could break CF’s ability to access an LDAP server. I don’t think it’s TLS-related, but as long as I’m helping people with recent JVM changes and JVM args to consider…

Note that Java 11 (11.0.1) added a new sort of verification of LDAP servers, and if somehow you want to disable that rather than fix whatever is amiss, that can be done with this JVM arg:

-Dcom.sun.jndi.ldap.object.disableEndpointIdentification=true

Bottom line: Try updating the Java that CF uses

My goal here was just to get the word out that for those errors offered above (and perhaps other failures to call out of CF to https pages), you may not need to bother importing certs. You may need to just update the JVM. It’s worth a shot. And you can easily revert back to the previous JVM version, if you follow best practices in updating the JVM that CF users, as discussed in resources I point out in that other post. Or see also a resource page I have on updating Java, and the page also covers updating CF, the CF web server config, and more..

Hope all this may help some readers.


For more blog posts from Charlie Arehart, see his posts here as well as his posts at carehart.org. And follow him on Twitter and other social media as carehart.

5 Comments
2021-04-14 18:05:35
2021-04-14 18:05:35

I want to point out here that Pete Freitag did a more recent post on a related topic, How to Resolve Java HTTPS Exceptions. While his opening and closing points reinforce may main point here (how updating the JVM may be the most common solution), he does point out still a few more points that folks may want to consider, especially if somehow updating the JVM does not help.

Like
2021-03-25 20:39:06
2021-03-25 20:39:06

In case it helps others, on ACF 2016 (Update 17) we started seeing “I/O Exception: Unsupported or unrecognized SSL message” calling a 3rd Party API. Updating the JVM to the latest (281) fixed the issue.

Like
(1)
>
davequested
's comment
2021-03-25 21:13:35
2021-03-25 21:13:35
>
davequested
's comment

Thanks for that, Dave. But can you clarify: did you really ONLY apply the CF update? Or did anything else change, perhaps about that 3rd party API? It would just seem curious that the CF update ALONE should call for updating the JVM. Still, let’s all take it under advisement, especially if you confirm that’s all that changed as far as you know. 🙂

Like
2019-06-09 23:45:31
2019-06-09 23:45:31

Hi Charlie,

thanks you for your article. You speak of “some unique cases”. I’d say it depend’s on when you get the exception. If you try to connect to some internal server it likely is the case that the cert is not created by some trusted autority and needs to be imported. If you try to access a public facing website and get the exception, it likely uses some recent cert that was not part of your original Java install.

Like
(2)
(1)
>
Bardnet
's comment
2019-06-10 00:07:25
2019-06-10 00:07:25
>
Bardnet
's comment

Right, thanks Bernhard. I was torn about elaborating on the other cases. Fair point that that’s the most likely one.

I figured that for anyone who might find this jvm update alone didn’t help, they’d easily find other discussions of importing certs.

I will add that those who do that should beware that IF someone has indeed already changed the jvm CF uses, they need to be sure to import the cert into the lib/security /cacerts of THAT jvm, not the one in CF’s jre/lib–as most resources would presume to tell folks.

Like
Add Comment