ESP8266 WiFiClientSecure connection drops immediately on GET, error not decoded

[dalbert2]

Basic Infos

• This issue complies with the issue POLICY doc.
• I have read the documentation at readthedocs and the issue is not addressed there.
• I have tested that the issue is present in current master branch (aka latest git).
• I have searched the issue tracker for a similar issue.
• [N/A] If there is a stack dump, I have decoded it.
• I have filled out all fields below.

Platform

• Hardware: [ESP-12E]
• Core Version: [Core 5.0.1 Espressif8266 2.6.2]
• Development Env: [Platformio]
• Operating System: [Windows]

Settings in IDE

• Module: [AIThinker]
• Flash Mode: [qio]
• Flash Size: [4MB]
• lwip Variant: [v2 Lower Memory]
• Reset Method: [nodemcu]
• Flash Frequency: [40Mhz]
• CPU Frequency: [160MHz]
• Upload Using: [SERIAL]
• Upload Speed: [921600] (serial upload only)

Problem Description

I am experiencing a problem with the ESP8266 WiFiClientSecure where it connects to my site, but drops the connection the moment the GET request is sent. The exact same code will work for other sites (e.g. www.google.com/index.html) but not this specific site. I have tested using fingerprint, trusted root certificate, and inscure modes; all have the same result: successful SSL connection, but connection lost the moment any HTTP GET is sent. The site works properly when tested with openssl (openssl s_client -crlf -connect mysite.com:443) or from any browser.

The SSL error returned (296) is not decoded successfully so debugging is difficult. Any guidance would be appreciated; the site is open so you can issue the Additional error handling is requested to provide useful feedback as to why a connection is dropped. More documentation will help too (I will be happy to write it once this is solved).

MCVE Sketch

#include <Arduino.h>
#include <ESP8266WiFi.h>
#include <ESP8266HTTPClient.h>
#include <WiFiClientSecure.h>

char ssid[] = "myssid";
char psk[]  = "mykey";

void ICACHE_FLASH_ATTR setup()
{	
	Serial.begin(115200);

	// Start WiFI network
	WiFi.mode(WIFI_STA);
	delay(2000);
	WiFi.begin(ssid,psk); 
    while (WiFi.status() != WL_CONNECTED) {
    	delay(100);
  	}
  	delay(1000);

	WiFiClientSecure client;
	client.setInsecure();

	HTTPClient http;
	// works
	//http.begin(client, "https://www.google.com/index.html");

	// doesn't work but works with: openssl s_client -crlf -connect mysite.com:443
	http.begin(client, "https://www.mysite.com/index.html");

	int httpResult = http.GET();
	if (httpResult == HTTP_CODE_OK) {
		Serial.println(http.getString());
	} else {
		Serial.printf("HTTP error: %d\r\n", httpResult);
	}
	char sslErrorMsg[80];
	int sslError = client.getLastSSLError(sslErrorMsg, sizeof(sslErrorMsg));
	if (sslError) {
		Serial.printf("SSL error: %d: %s\r\n", sslError, sslErrorMsg);
	}
	http.end();
}

void ICACHE_FLASH_ATTR loop()
{
	yield();
}

Debug Messages

 BSSL:Connected!
          [HTTP-Client] connected to www.mysite.com:443
                                                         [HTTP-Client] sending request header
             -----
                  GET /index.html HTTP/1.1
Host: www.mysite.com
User-Agent: ESP8266HTTPClient
Accept-Encoding: identity;q=1,chunked;q=0.1,*;q=0
Connection: keep-alive
Content-Length: 0

-----
     [HTTP-Client][returnError] error(-5): connection lost
                                                          HTTP error: -5
SSL error: 296: Unknown error code.
[HTTP-Client][end] tcp is closed

[earlephilhower]

Please enable full SSL debugging in the menu and capture the output. It's pretty chatty and may point to the exact issue in the logs.

[earlephilhower]

Also, if you're really on core 2.6.2, can you please upgrade to the latest release, 2.7.4. It should be available thru the normal channels and has quite a few bugfixes from 2.6.x.

[dalbert2]

I'll give it a try; 2.6.2 is the newest core that's available through platformio.
I overwrote the 2.6.2 files with 2.7.4 and there was no improvement; here's the output (now showing SDK and Core versions; the only other change was the addition of that code to display the version info:

	Serial.print("SDK: ");
	Serial.println(system_get_sdk_version());
	Serial.print("Core: ");
	Serial.println(ESP.getCoreVersion());

Thanks again!

[earlephilhower]

Looks like it connects and then the other end times out. Are you running at 160MHZ? OTW, the SSL handshake may take too long and GOOG will drop the connection.

[earlephilhower]

Also, if 2.6.2 is really the latest PIO release we need to figure out why that is and fix it...2.7.x came out months ago.

[dalbert2]

I'm running at 160MHz and I don't think it's that sort of problem because I can connect to many other sites (google, yahoo, etc.). My site isn't just broken because I can connect to it with browsers and from the CLI with openssl.
So I think this is an incompatibility between BearSSL and my site (running apache2), but the unsupported error code makes it hard to figure out.

As you suggest, the disconnect is instant once the server receives the full GET request. If I hold off on sending the final \r\n, it will keep the connection open until it receives it, but then the disconnect is instant before any response is received.

[earlephilhower]

I'm not sure what the exact error is w/SSL, but from a quick glance I see that when bit 8 is set (256) then BSSL is saying the remote side sent a SSL fatal alert:
https://bearssl.org/gitweb/?p=BearSSL;a=blob;f=inc/bearssl_ssl.h;h=e91df47556905512f8c8850a27e280c384d8b149;hb=dda1f8a0c46e15b4a235163470ff700b2f13dcc5#l219

If it's your own site, can you enable debugging on the webserver and see if there's anything in the logs. Once the SSL handshake finishes (BSSL:Connected!) comms are very simple using AES encrypted packets. So if it was a SSL-specific thing, I'd expect the handshake to fail and not after connection.

Another thing we can add is HTTP client debugging (you can just enable everything in the debug menu) which would verify that it's not a HTTP-level issue vs. a SSL one.

[dalbert2]

Good eyes! I saw the error codes but they appeared to be an enum rather than a bitmap and I didn't look closely enough; the 256 is added to the error code. Unfortunately that still leaves me with an error code of 40 which is also un-mapped.

I'm sorry for the dumb question, you've mentioned the debug menu several times, and I've spent quite a bit of time looking for it, can you give me a pointer? From what I've seen, all debugging in PlatformIO is enabled by build flag defines in platformio.ini

[dalbert2]

From the apache error log:

[Tue Oct 27 21:26:45.477215 2020] [ssl:error] [pid 10333] [client 72.81.142.2:61447] AH02261: Re-negotiation handshake failed
[Tue Oct 27 21:26:45.477300 2020] [ssl:error] [pid 10333] SSL Library Error: error:14094153:SSL routines:ssl3_read_bytes:no renegotiation

I think this suggests a problem with incompatible cipher suites...I'll dig into that.

[earlephilhower]

[Tue Oct 27 21:26:45.477215 2020] [ssl:error] [pid 10333] [client 72.81.142.2:61447] AH02261: Re-negotiation handshake failed
[Tue Oct 27 21:26:45.477300 2020] [ssl:error] [pid 10333] SSL Library Error: error:14094153:SSL routines:ssl3_read_bytes:no renegotiation

I think that's your culprit. SSL3 was deprecated many years ago because it was busted: https://tools.ietf.org/html/rfc7568 . BearSSL doesn't support it, either.

If that's the case and your server is still configured to try SSLv3, this is expected behavior...

[earlephilhower]

BearSSL has never had SSL3 support, and it can't negotiate it ever. It was written well after the SSL3 deprecation. So if your apache isn't configured to use it, then maybe it's a false lead. I think it's time to start digging thru the BSSL sources https://bearssl.org/gitweb/?p=BearSSL;a=summary and see where the undocumented error level 40 is coming from...

Also, if you have wireshark, it's quick work to verify the actual TLS level and cipher used with it, just for sanity's sake. I don't think it can decode anything to do w/renegotiations, though.

[dalbert2]

There is an error code 40: BR_ERR_X509_EXTRA_ELEMENT (Decoding error: extraneous element.)

[dalbert2]

Note that in WiFiClientSecureBearSSL.cpp, method getLastSSLError() should probably handle the 256 and 512 values that can apparently be added to other error enumeration codes.

[earlephilhower]

Agreed about the errors. I didn't think of looking in the X509 decoder, either.

So that would lead me to believe there's something fishy with the cert exchange. I'm a little fuzzy on multi-tenant SSL, is your server shared with multiple SNI names (hence the initial handshake and then another handshake)?

[dalbert2]

I've submitted query to Thomas Pornin (author of BearSSL) to see if he has any ideas; I dug around in the X509 code, but it is not for the faint of heart.

[dalbert2]

I seem to have solved the issue. In my apache ssl configuration, when I moved the following two lines out of the Directory sections and into the global (VirtualHost) section, the renegotiation handshake failure stopped and my HTTP requests were processed properly.

        SSLVerifyClient optional
        SSLVerifyDepth 1