HTTP keep alive management

[bgranvea]

In my application I have random connection errors like this:

ClickHouseStatementImpl - Error during connection to ru.yandex.clickhouse.settings.ClickHouseProperties@785d9e37, reporting failure to data source, message: localhost:8123 failed to respond
org.apache.http.NoHttpResponseException: localhost:8123 failed to respond
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)

(I've modified the code to print the stacktrace in ClickHouseStatementImpl)

I think the root cause is that in some rare cases, Apache Http client doesn't detect a connection closed by the server (half-closed socket). I have a Wireshark capture which confirms this.

I've seen that the server returns how long it will keep the connection open but it doesn't seem to be used client side:

POST /?compress=1&extremes=0&database=default HTTP/1.1
Content-Length: 55
Content-Type: text/plain; charset=UTF-8
Host: xxxxxx:8123
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.5.2 (Java/1.8.0_152)

select timezone() FORMAT TabSeparatedWithNamesAndTypes;HTTP/1.1 200 OK
Date: Mon, 14 Jan 2019 08:04:41 GMT
Connection: Keep-Alive
Content-Type: text/tab-separated-values; charset=UTF-8
X-ClickHouse-Server-Display-Name: xxxxxxxxx
Transfer-Encoding: chunked
Keep-Alive: timeout=3
(...)

and

private ConnectionKeepAliveStrategy createKeepAliveStrategy() {
        return new ConnectionKeepAliveStrategy() {
            @Override
            public long getKeepAliveDuration(HttpResponse httpResponse, HttpContext httpContext) {
                // in case of errors keep-alive not always works. close connection just in case
                if (httpResponse.getStatusLine().getStatusCode() != HttpURLConnection.HTTP_OK) {
                    return -1;
                }
                HeaderElementIterator it = new BasicHeaderElementIterator(
                        httpResponse.headerIterator(HTTP.CONN_DIRECTIVE));
                while (it.hasNext()) {
                    HeaderElement he = it.nextElement();
                    String param = he.getName();
                    //String value = he.getValue();
                    if (param != null && param.equalsIgnoreCase(HTTP.CONN_KEEP_ALIVE)) {
                        return properties.getKeepAliveTimeout();
                    }
                }
                return -1;
            }
        };
    }

So, shouldn't it be better to parse 'Keep-Alive: timeout=3' to get the keep-alive duration, and if it fails, use properties.getKeepAliveTimeout() ?

[enqueue]

@bgranvea, @ylltw01 how would you design a test for this?

[ylltw01]

@enqueue i am sorry for that，i made a mistake，please ignore

[AlexMAS]

Hi,

Sometimes I get the same exception:

Caused by: org.apache.http.NoHttpResponseException: clickhouse-server:8123 failed to respond
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
	at ru.yandex.clickhouse.ClickHouseStatementImpl.getInputStream(ClickHouseStatementImpl.java:614)

Versions:

• clickhouse-server v19.15.3.6
• clickhouse-jdbc v0.2

Could you explain the reason and how to fix it?

[esukanovic]

I'm using latest clickhouse-jdbc version 0.2.3 and have Hikari CP configured to use ClickHouseDataSource with all the properties like socketTimeout stc.

"Failed to respond" exception happens ocassionally and it has nothing to do with timeouts since I'm logging all the query/call durations and can see that it's clearly less than timeout period even in case of failure.
Is it possible that Hikari CP is not aware of connections being closed on server side?

Anyone using it and not having this issue?

UPDATE: After configuring connectionTestQuery & validationTimeout for Hikari the issue seems to be solved.
As part of getting connection from pool Hikari now checks whether connection is alive and closes it if necessary.
I still sometimes see "Failed to respond" error in logs but now it's just for the connection test query during 'isConnectionAlive' check.

[zhicwu]

I think the root cause is that in some rare cases, Apache Http client doesn't detect a connection closed by the server (half-closed socket). I have a Wireshark capture which confirms this.

Yes, I was able to reproduce the issue by shutting down WireMockServer before client get any response. Retry can mitigate the issue but it won't work for non-idempotent operation like insertion.

So, shouldn't it be better to parse 'Keep-Alive: timeout=3' to get the keep-alive duration, and if it fails, use properties.getKeepAliveTimeout()?

keepAliveTimeout is deprecated since 0.2.5(see details in #515).

[lewis-ing]

Hi @esukanovic .do you figure it,today i met same error and using Hikari CP for datasource pool,but i use it for mybatis.

[chengbinghan]

i meet this when java insert client server momery not enough