Troubleshooting Timeout exceeded for HTTPS POST?

Good Morning,

Please can I ask for some advice on a effective way of troubleshooting a Timeout exceeded error. The system API that is being used is experiencing "timeout exceeded" errors when connecting to a SugarCRM REST API using HTTPS. The errors appear to occur after about 10 seconds, despite the fact that both the connection idle timeout and response timeout values are configured to be 30 seconds.

One important point is that the errors do not appear consistently. You can have an run of requests with no timeout errors, then you may get one or two errors, followed by another series with no errors. The timeouts are very random with no obvious pattern.

The error message that is being observed is

[2018-09-21 07:10:31.908] ERROR org.mule.extension.http.internal.request.HttpRequester [Grizzly-IdleTimeoutFilter-IdleCheck]: Error sending HTTP request to...

[2018-09-21 07:10:31.909] ERROR org.mule.runtime.core.internal.exception.OnErrorContinueHandler [Grizzly-IdleTimeoutFilter-IdleCheck]:

********************************************************************************

Message : HTTP PUT on resource '...' failed: Timeout exceeded.

Element XML : <http:request method="PUT" doc:name="Request" doc:id="f8d07982-7d3c-4914-9cc9-5681c80c10a5" config-ref="SugarCRM_HTTP_Request_configuration" path="${sugar.api.path.contact}/{contactId}">

:

:

</http:request>

Error type : HTTP:TIMEOUT

Payload Type : org.mule.runtime.core.internal.streaming.bytes.ManagedCursorStreamProvider

--------------------------------------------------------------------------------

Root Exception stack trace:

java.util.concurrent.TimeoutException: Timeout exceeded

at com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.timeout(GrizzlyAsyncHttpProvider.java:433)

at com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$3.onTimeout(GrizzlyAsyncHttpProvider.java:281)

at org.glassfish.grizzly.utils.IdleTimeoutFilter$DefaultWorker.doWork(IdleTimeoutFilter.java:402)

at org.glassfish.grizzly.utils.IdleTimeoutFilter$DefaultWorker.doWork(IdleTimeoutFilter.java:381)

at org.glassfish.grizzly.utils.DelayedExecutor$DelayedRunnable.run(DelayedExecutor.java:158)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

The corresponding configuration reference is:

<http:request-config name="SugarCRM_HTTP_Request_configuration"

doc:name="SugarCRM HTTP Request configuration" doc:id="1e94b60b-f27f-4a5b-8d8d-dbf31712430c"

basePath="${sugar.api.path.base}" responseTimeout="${sugar.api.responseTimeout}">

<http:request-connection protocol="HTTPS"

host="${sugar.api.host}" port="${sugar.api.port}" />

</http:request-config>

And the configuration values are:

- sugar.api.responseTimeout=30000 (30 seconds)

- Use persistent connections = true

- Connection idle timeout = 30000 (default as defined in source code = 30 seconds)

I am hesitant to use org.mule.service.http.impl.service.HttpMessageLogger in DEBUG mode as it exposes a lot of sensitive data. Is there a way of limiting the logging to only expose the basic information that does not include the payload or sensitive headers?

So far, I have not picked up anything using javax.net.debug=ssl.

The concerning part is the approx 10 second failure for the timeout, even though all values are set to 30 seconds.

Any pointers would be greatly appreciated.

Thank you.