Large download fails with WiFiClientSecure and maybe WiFiClient

[ZinggJM]

I could successfully download bitmap files up to ~86kB size through WiFiClientSecure, but download of larger files do not terminate. If client.read(buffer, amount) returns zero, every next call does the same forever, although client.connected() still returns true. I tried workaround with client.available() and delay(), but found no safe method. The amount that works seems dependent on available heap size.

example: https://github.com./ZinggJM/GxEPD2_32/tree/master/examples/GxEPD2_32_Spiffs_Loader

[devyte]

Issue template has been completely ignored. Also, the requirement is for a MCVE sketch, which should be included in the issue, and not linked. Emphasis on the M (minimal), the linked sketches are rather big.
Please open a new issue, fill in the required fields, and include a MCVE sketch.
Closing.

[ZinggJM]

/*
    HTTP over TLS (HTTPS) example sketch

    This example demonstrates how to use
    WiFiClientSecure class to access HTTPS API.
    We fetch and display the status of
    esp8266/Arduino project continuous integration
    build.

    Limitations:
      only RSA certificates
      no support of Perfect Forward Secrecy (PFS)
      TLSv1.2 is supported since version 2.4.0-rc1

    Created by Ivan Grokhotkov, 2015.
    This example is in public domain.
*/

#include <ESP8266WiFi.h>
#include <WiFiClientSecure.h>

const char* ssid = "........";
const char* password = "........";

//const char* host = "api.github.com.";
const char* host = "raw.githubusercontent.com";
const int httpsPort = 443;

// Use web browser to view and copy
// SHA1 fingerprint of the certificate
//const char* fingerprint = "35 85 74 EF 67 35 A7 CE 40 69 50 F3 C0 F6 80 CF 80 3B 2E 19";
const char* fingerprint     = "cc aa 48 48 66 46 0e 91 53 2c 9c 7c 23 2a b1 74 4d 29 9d 33";

void setup() {
  Serial.begin(115200);
  Serial.println();
  Serial.print("connecting to ");
  Serial.println(ssid);
  WiFi.mode(WIFI_STA);
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());

  // Use WiFiClientSecure class to create TLS connection
  WiFiClientSecure client;
  Serial.print("connecting to ");
  Serial.println(host);
  if (!client.connect(host, httpsPort)) {
    Serial.println("connection failed");
    return;
  }

  if (client.verify(fingerprint, host)) {
    Serial.println("certificate matches");
  } else {
    Serial.println("certificate doesn't match");
  }

  //String url = "/repos/esp8266/Arduino/commits/master/status";
  //String url = "/prenticedavid/MCUFRIEND_kbv/master/extras/bitmaps/betty_1.bmp"; // works
  String url = "/prenticedavid/MCUFRIEND_kbv/master/extras/bitmaps/test.bmp"; // hangs
  Serial.print("requesting URL: ");
  Serial.println(url);

  client.print(String("GET ") + url + " HTTP/1.1\r\n" +
               "Host: " + host + "\r\n" +
               "User-Agent: BuildFailureDetectorESP8266\r\n" +
               "Connection: close\r\n\r\n");

  Serial.println("request sent");
  while (client.connected()) {
    String line = client.readStringUntil('\n');
    if (line == "\r") {
      Serial.println("headers received");
      break;
    }
  }
  //  String line = client.readStringUntil('\n');
  //  if (line.startsWith("{\"state\":\"success\"")) {
  //    Serial.println("esp8266/Arduino CI successfull!");
  //  } else {
  //    Serial.println("esp8266/Arduino CI has failed");
  //  }
  //  Serial.println("reply was:");
  //  Serial.println("==========");
  //  Serial.println(line);
  //  Serial.println("==========");
  //  Serial.println("closing connection");
  uint32_t count = 0;
  while (client.connected())
  {
    if (client.available())
    {
      int16_t v = client.read();
      count++;
      if (0 == count % 1000)
      {
        Serial.print("got "); Serial.print(count); Serial.println(" so far...");
        delay(1); // avoid WDT
      }
    }
    else delay(1); // avoid WDT
  // hangs after 48000
  }
  Serial.print("terminated, received "); Serial.println(count);
}

void loop() {
}

[d-a-v]

With debug activated,

...
15:09:11.731 -> ssl->need_bytes=16432 > 11259
...
15:09:12.062 -> :er -14 0x00000000
(stops here)

-14 is lwIP's "Connection reset", meaning that the server has closed the TCP session.
FreeHeap is ~28KB

I tried with another https server of mine and the same file:
the connection is lagging like hell but is it not reset by my server (which is not github dealing with billions of clients, hence without aggressive timeout like above).

So this is an interesting problem with TCP, not specifically with ssl.
TCP is not smooth (even with "large bandwidth" or v1.4), and is awaken by its retransmit timeout every some-tens of seconds.

Thanks for the MCVE

[d-a-v]

I forgot to enable SSL messages.
This a SSL problem. Please try with BearSSL instead of axTLS.

[ZinggJM]

Thank you, I had a hard time to find the WiFiClientSecure example (originally, and again), until I got the idea to search for https in the examples. That said means I am a newbie user of SSL in a program, and don't know about BearSSL. Is there an example somewhere?

[d-a-v]

Sure, in File>Examples>ESP8266WiFi you have examples with BearSSL::WiFiClientSecure:

BearSSL_CertStore.ino
BearSSL_MaxFragmentLength.ino
BearSSL_Server.ino
BearSSL_ServerClientCert.ino
BearSSL_Validation.ino

However, the HTTPSRequest.ino example (ESP8266WiFi/examples/HTTPSRequest/HTTPSRequest.ino) that you use does not work as-is when replacing WiFiClientSecure by BearSSL::WiFiClientSecure.

I will try and dig that out, maybe @earlephilhower can help with that precise example.

[ZinggJM]

Aha. I see these examples on GitHub, but not in my installation. So I need to update. But Boards Manager says 2.4.1 installed...

[d-a-v]

You need to install the git version, 2.4.2 is not released yet.

[d-a-v]

@ZinggJM BearSSL works flawlessly with your image, quick and fast.
See the example updated (link) to update your sketch.
You will need client->setInsecure() which is not currently the default.

[ZinggJM]

@d-a-v thank you, I will try to use it. I need to find out if it is enough to download the actual ESP8266WIFI library. And I can only use it in my library after 2.4.2 is released. Do you think the issue in axTLS will be resolved? Thank you.

[devyte]

Issues traced to axtls won't be fixed. The bearssl integration is meant to replace axtls. Right now, axtls is still the default, and will remain so as long as bearssl is considered experimental, but axtls will be deprecated and then retired.

[jpswade]

This suggests that the ESP8266WiFi/examples/HTTPSRequest/HTTPSRequest.ino example should be updated to use bearssl or removed, is that correct?

[d-a-v]

Yes, an update is needed.

[cboehm-it]

@d-a-v @devyte Could you help for me for setting up a post request?
I tried to enhance the example but it does not work with BearSSL. The fingerprint check works well.

String postData = "grant_type=refresh_token&refresh_token=" + currentRefreshToken + "&client_id=" + clientId + "&client_secret=" + clientSecret;

client.print(String("POST ") + url + " HTTP/1.1\r\n" +
"Host: " + host + "\r\n" +
"User-Agent: BuildFailureDetectorESP8266\r\n" +
"Content-Type: application/x-www-form-urlencoded\r\n" +
"Content-Length: " + postData.length() + "\r\n\r\n" + postData +
"Connection: close\r\n\r\n");

How can I get the json Response from the service? Is there any documentation for this?

[ZinggJM]

I got confused, as I just set up the ESP8266 package on a new notebook. I now see examples for BearSSL. I then noticed these are also on an other notebook I had updated some time ago. So maybe I was just not aware that BearSSL was already part of the official package for some time. But the example I started with - HTTPSRequest - is still the old one. The link from @d-a-v points to a newer version in his fork.

So it looks like I can use BearSSL::WiFiClientSecure, and just need to add client.setFingerprint(fingerprint); to make it work.

[ZinggJM]

client.setFingerprint((uint8_t*)fingerprint); // ("present the other hand, please")
seems not enough to make client.connect(host, httpsPort) work with BearSSL.

[ZinggJM]

This is what I get from BearSSL_Validation

Connecting to C4A2760E
.....
WiFi connected
IP address:
192.168.4.220

If there are no CAs or insecure options specified, BearSSL will not connect.
Expect the following call to fail as none have been configured.
Trying: api.github.com.:443...*** Can't connect. ***

This is absolutely insecure, but you can tell BearSSL not to check the
certificate of the server. In this mode it will accept ANY certificate,
which is subject to man-in-the-middle (MITM) attacks.
Trying: api.github.com.:443...Connected!

HTTP/1.1 200 OK

The SHA-1 fingerprint of an X.509 certificate can be used to validate it
instead of the while certificate. This is not nearly as secure as real
X.509 validation, but is better than nothing.
Trying: api.github.com.:443...Connected!

HTTP/1.1 200 OK

It is also possible to accept any self-signed certificate. This is
absolutely insecure as anyone can make a self-signed certificate.
First, try and connect to a badssl.com self-signed website (will fail):
Trying: self-signed.badssl.com:443...*** Can't connect. ***

Now we'll enable self-signed certs (will pass)
Trying: self-signed.badssl.com:443...*** Can't connect. ***

The server certificate can be completely ignored and its public key
hardcoded in your application. This should be secure as the public key
needs to be paired with the private key of the site, which is obviously
private and not shared. A MITM without the private key would not be
able to establish communications.
Trying: api.github.com.:443...*** Can't connect. ***

A specific certification authority can be passed in and used to validate
a chain of certificates from a given server. These will be validated
using BearSSL's rules, which do NOT include certificate revocation lists.
A specific server's certificate, or your own self-signed root certificate
can also be used. ESP8266 time needs to be valid for checks to pass as
BearSSL does verify the notValidBefore/After fields.
Try validating without setting the time (should fail)
Trying: api.github.com.:443...*** Can't connect. ***

Try again after setting NTP time (should pass)
Waiting for NTP time sync: .
Current time: Sat Oct 20 18:30:41 2018
Trying: api.github.com.:443...*** Can't connect. ***

[ZinggJM]

aha, not only signed/unsigned difference, but also format: byte array instead of ascii HEX representation, maybe.

[ZinggJM]

This helps to connect, but the original issue got even worse.

[d-a-v]

@ZinggJM With latest master, large downloads with HTTPS are still failing ?
Have you checked your heap and enabled debug options ?
I think the ascii fingerprint representation will be re-added for release 2.5.0.

[ZinggJM]

Boards Manager says "esp8266 by ESP8266 Community version 2.4.2 installed"
installed on 18.10.2018 using Boards Manager with "additional urls":
"http://arduino.esp8266.com/stable/package_esp8266com_index.json"
I am not familiar with heap check and debug option; will try to find out.
I use Board: "LOLIN(Wemos) D1 R2 & mini", as I am used to, with Wemos D1 mini V2.2.0 or mini pro.
Sketch uses 390664 bytes (37%) of program storage space. Maximum is 1044464 bytes.
Global variables use 49628 bytes (60%) of dynamic memory, leaving 32292 bytes for local variables. Maximum is 81920 bytes.
Uploading 394816 bytes from C:\Users\ZinggJ\AppData\Local\Temp\arduino_build_942110/GxEPD_WiFi_Example.ino.bin to flash at 0x00000000
................................................................................ [ 20% ]
................................................................................ [ 41% ]
................................................................................ [ 62% ]
................................................................................ [ 82% ]
.................................................................. [ 100% ]

Yes, large downloads with HTTPS are still failing with BearSSL, and "large" is much smaller than with axTLS.

This is the only debug output I see when I enable Debug port and Debug level SSL+TLS_MEM+HTTP_Client

SDK:2⸮1
SDK:2.2.1(cfd48f3)/Core:2.4.2/lwIP:2.0.3(STABLE-2_0_3_RELEASE/glue:arduino-2.4.1-13-g163bb82)/BearSSL:6d1cefc
scandone
state: 0 -> 2 (b0)
state: 2 -> 3 (0)
state: 3 -> 5 (10)
add 0
aid 6
cnt

connected with C4A2760E, channel 1
dhcp client start...
.6.6.6ip:192.168.4.220,mask:255.255.255.0,gw:192.168.4.2
.3
WiFi connected
192.168.4.220

downloading file "logo200x200.bmp"
connecting to raw.githubusercontent.com
requesting URL: https://raw.githubusercontent.com/ZinggJM/GxEPD2/master/extras/bitmaps/logo200x200.bmp
request sent
HTTP/1.1 200 OK

headers received
File size: 5662
Image Offset: 62
Header size: 40
Bit Depth: 1
Image size: 200x200
waited for available 771 ms
bytes read 5662
loaded in 3274 ms
Power On : 10
update : 1
update : 1
Power Off : 1

[ZinggJM]

Debug output with all enabled.
GxEPD_WiFi_Example_Debug.txt

[ZinggJM]

@ZinggJM BearSSL works flawlessly with your image, quick and fast.
See the example updated (link) to update your sketch.
You will need client->setInsecure() which is not currently the default.

How did you achieve this?

[ZinggJM]

Now I tried the example: BearSSL_MaxFragmentLength.ino

Connecting to https://tls.mbed.org
No MFLN attempted
Unable to connect

Connecting to https://tls.mbed.org
MFLN supported: yes
Unable to connect

Default SSL: 43824 bytes used
1024 byte MFLN SSL: 43824 bytes used

it looks like

client.setInsecure();

does not work.

[ZinggJM]

I also tried BearSSL_MaxFragmentLength.ino with client.setFingerprint(fp);
const uint8_t fp[20] = {0xeb, 0xd9, 0xdf, 0x37, 0xc2, 0xcc, 0x84, 0x89, 0x00, 0xa0, 0x58, 0x52, 0x24, 0x04, 0xe4, 0x37, 0x3e, 0x2b, 0xf1, 0x41};
result also Unable to connect.
So I have to give up trying to use BearSSL for now.

[d-a-v]

@ZinggJM Your debug log shows ":oom". This means you are running Out Of Memory.
When this happens, things may (usually will) not work anymore.
Read this - subsection memory, then sort it out first.

[d-a-v]

@ZinggJM You also need to work with latest git core.
BearSSL was experimental and not default in core-2.4.2.
Improvements happened since then and is expected to be stable and default in next release 2.5.0.

[ZinggJM]

@ZinggJM Your debug log shows ":oom". This means you are running Out Of Memory.
When this happens, things may (usually will) not work anymore.
Read this - subsection memory, then sort it out first.

Yes, out of memory seems to be the cause with my test, although I had 32292 bytes for local variables, as reported. This example has a 15k graphics buffer for an e-paper display.

In the meantime I found out that the GxEPD2_32_Spiffs_Loader works with BearSSL.

Maybe the "trick" with the separate stack space just needs too much RAM to be practical.

[ZinggJM]

I tried to update my MCVE sketch to use with BearSSL to show the issue. But my MCVE sketch works.
So I need to analyze my issue with download to draw on e-paper from start.

[ZinggJM]

Using BearSSL with GxEPD2_32_Spiffs_Loader resolves the large download issue for this example, and thus for this issue.
The 15k buffer is not the reason for my issue with my GxEPD_WiFi_Example. Needs further analysis and most likely is not related with BearSSL. So I close this issue (and may open a new one if I find out a reason related to the ESP8266 package).
Thank you for all your support.

[ZinggJM]

I had used this method (from class Client):
virtual int read(uint8_t *buf, size_t size) = 0;
If I use single byte reads instead, checking for available, it works.
virtual int available() = 0; virtual int read() = 0;

[playground]

read() works for me but seems like the response does not match with the one I get from Postman. I have posted the issue here https://www.esp32.com/viewtopic.php?f=19&t=20565&p=75332#p75332. I'm still looking for a solution. Thanks for your help.