Apache crashes while serving multiple thread requests using CF2018

February 8, 2019
Newbie 1 posts
Followers: 0 people
6

Apache crashes while serving multiple thread requests using CF2018

Newbie 1 posts
Followers: 0 people
February 8, 2019

Hi Team,

We encountered an interesting (rather annoying) bug with Apache & CF2018, in one of the Proof of Concept Servers (CF2018 with Apache2.4). The scenario, which I am going to describe here is working well in our production servers which are not using CF2018 or Apache2.4 (and for years now).

Few reports are scheduled to run one after another and the Apache crashes with Segment Fault, before the next report is run. ColdFusion doesn’t crash though. This is how the log looks like, for Apache & Apache connector. The issue occurs, irrespective of the status of SELinux being enforced/disabled.

/httpd/error.log

[Thu Feb 07 16:52:25.147757 2019] [core:notice] [pid 3063] AH00052: child pid 3436 exit signal Segmentation fault (11)

[Thu Feb 07 16:52:25.147840 2019] [core:notice] [pid 3063] AH00052: child pid 3445 exit signal Segmentation fault (11)

[Thu Feb 07 16:52:55.307720 2019] [core:notice] [pid 3063] AH00052: child pid 3444 exit signal Segmentation fault (11)

[Fri Feb 08 03:00:25.332926 2019] [core:notice] [pid 3063] AH00052: child pid 4118 exit signal Segmentation fault (11)

[Fri Feb 08 07:29:01.457578 2019] [core:notice] [pid 3063] AH00052: child pid 4219 exit signal Segmentation fault (11)

/wsconfig/1/mod_jk.log

[Thu Feb 07 16:52:24 2019] [3445:140618548705024] [warn] ajp_get_endpoint::jk_ajp_common.c (3705): Unable to get the free endpoint for worker cfusion from 1 slots

[Thu Feb 07 16:52:24 2019] [3436:140618548705024] [warn] ajp_get_endpoint::jk_ajp_common.c (3705): Unable to get the free endpoint for worker cfusion from 1 slots

[Thu Feb 07 16:52:54 2019] [3444:140618548705024] [warn] ajp_get_endpoint::jk_ajp_common.c (3705): Unable to get the free endpoint for worker cfusion from 1 slots

[Fri Feb 08 03:00:24 2019] [4118:140618548705024] [warn] ajp_get_endpoint::jk_ajp_common.c (3705): Unable to get the free endpoint for worker cfusion from 1 slots

[Fri Feb 08 07:29:00 2019] [4219:140618548705024] [warn] ajp_get_endpoint::jk_ajp_common.c (3705): Unable to get the free endpoint for worker cfusion from 1 slots

If a single report is scheduled to run, it runs without any issues.

It sounds very similar to this issue

https://coldfusion.adobe.com/discussion/2549262/

I would be grateful, if someone from Adobe or the Community could assist in this.

Thank You,

Annie

Comments (6)
2019-02-14 16:17:18
2019-02-14 16:17:18

OK, thanks for the clarification, Annie. (I will assume by “no errors” in the update log that you mean not just that you “didn’t see any at the bottom”, but specifically that you had looked at the table near the top, tracking successes and errors, and it shows there were zero fatal or non-fatal errors.)

And I missed that you said that changing the heartbeat_interval DID help solve things, as a workaround. That’s interesting.

Hope someone at Adobe may come to a solution for you soon, as it could affect others. I’ve not heard this elsewhere yet, myself.

(There seems quite a few different issues with the latest update–affecting some though not all. But since it was an update across all 3 supported CF versions, they will likely have their hands full addressing such issues in the coming hours and days.)

Like
(1)
>
Charlie Arehart
's comment
2019-02-18 12:19:04
2019-02-18 12:19:04
>
Charlie Arehart
's comment

Your assumption is right. There are 0 FatalErrors and 0 NonFatalErrors.

Someone from Adobe said that the bug is fixed in the CF2018, update 2. But its still happening and the workaround helps. There’s a bug raised with them. Hope there’s a solution for this.

Thanks for your prompt response.

Annie

Like
2019-02-14 14:08:44
2019-02-14 14:08:44

Annie, did you upgrade the apache connector for cf, as was indicated in this update’s tech note? It would be easy to miss.

And did you confirm that there are no errors in the update log created for you in the cfusion/hf-updates and the subdolder for this update?

These things may show this is not a “bug” in the update as it may seem.

Like
(1)
>
Charlie Arehart
's comment
2019-02-14 16:11:27
2019-02-14 16:11:27
>
Charlie Arehart
's comment

Hi Charlie,

There were no errors in the update log and the apache connector was upgraded and the SELinux context was applied as usual to the mod_jk.conf and workers.properies. The apache started as usual, and the application seems to be working fine. CFAdmin is accessible. On changing the heartbeat_interval=0, the apache stopped crashing.

Annie

Like
2019-02-14 12:00:39
2019-02-14 12:00:39

Even after installing the ColdFusion2018 update 2, the apache crashes the above mentioned scenario. Hence we had to change the heartbeat_interval=0 in the workers.properties.

Could Adobe CF technical team, look into this issue?

Like
2019-02-12 16:19:43
2019-02-12 16:19:43

Thanks for the Adobe Support Team to quick response over e-mail. Their solution was to update the workers.properties with the following.

heartbeat_interval=0

Updating workers.properties file solved the issue. However it would be great if you would let us know when this bug will be solved? And its related to Performance Monitoring Tool. Looking for an update on this please.

Like
Add your comment