HttpClient: HTTPS request without specifying certificate fingerprint?

[tuxedo0801]

Hi there,

I have to push some data from ESP8266 to an IoT Cloud system. But this requires HTTPS.

In the examples I found, I always need to specify the fingerprint of the https-certficate.

This binds my source directly to the cloud system. If certificate changes, the request fails I guess...
So I have to update my sketch with every certificate-change, which is not that easy for my use-case.

Is there a possibility to accept whatever certificate is running behind https?

For my use-case it would be okay to use http instead of https, but the cloud system does only provide https. So I have to stick to it.

br,
Alex

[SnakeDoc]

Not the best solution, but could you proxy the requests through a service/server you control? I suppose you'd have the same issue once your service/server's certificate expires though...

[tuxedo0801]

This issue relates to #1851 ...

Maybe I add a php-script inbetween to decouple https-request from esp8266, but this is more a workaround than a real solution.

[tuxedo0801]

Found this:

https://github.com./esp8266/Arduino/blob/master/libraries/ESP8266WiFi/examples/HTTPSRequestCACert/HTTPSRequestCACert.ino

Have to check if this is a valid solution for my use-case.

[tuxedo0801]

Currently I'm testing the solution provided in #1851, but I would like to see, that I can make https-calls, without specifying a certificate-finger-print and without providing the root-ca-certificate to do certificate validation. Just "accept all certificates and ignore the complete certificate-check", like you can do on any other desktop-programming-language ...

Is this in general possible? Or is this technically on the esp8266 not possible?

[JamesGKent]

you might be able to do it if you set the methods that check the cert to return as if they matched at all times, from a brief look you need to:

• replace the return with a return true at line 485 in wificlientsecure.cpp
• replace return with return true at line 518 in wificlientsecure.cpp
• replace return with return true at line 546 in wificlientsecure.cpp

I would suggest that to do what you want you'll want to move the returns closer to the start of each function, but after checking that you have an ssl context, so i'd suggest just adding return true before each of these lines:

• 465
• 494
• 524
• 554 (needed because call to verify cert chain will fail)

can always give this a try.

[tuxedo0801]

Sounds like copy&paste&modify of Wificlientsecure.cpp/.h to create a quick'n'easy workaround...

Did someone test this already?

[ghost]

I just had the same problem. Just ignore the result of the client.verify() method, or just dont run it at all. It's up to you what to do if the certificate verification fails.
Works for me. No need to change the source code of WiFiClientSecure.cpp

[tuxedo0801]

Finally, I have some feedback:

When I use this sample as a starting point: https://github.com./esp8266/Arduino/blob/master/libraries/ESP8266WiFi/examples/HTTPSRequest/HTTPSRequest.ino

and just ignore lines 52-56:

  if (client.verify(fingerprint, host)) {
    Serial.println("certificate matches");
  } else {
    Serial.println("certificate doesn't match");
  }

then https communication is working in principle. My Code:

void sendMessageHTTPS(String msg) {

  
  // Use WiFiClientSecure class to create TLS connection
  WiFiClientSecure client;
  Serial.print("connecting to : '");
  Serial.print(host);
  Serial.println("'");
  
  if (!client.connect(host, httpsPort)) {
    Serial.println("connection failed");
    return;
  }

//  if (client.verify(fingerprint, host)) {
//    Serial.println("certificate matches");
//  } else {
//    Serial.println("certificate doesn't match");
//  }

  Serial.print("requesting URL: '");
  Serial.print(url);
  Serial.println("'");

  client.print(String("POST ") + url + " HTTP/1.1\r\n" +
          "Host: " + host + "\r\n" +
          "Connection: close\r\n" +
          "Accept: */*\r\n" +
          "User-Agent: Mozilla/4.0 (compatible; esp8266 Lua; Windows NT 5.1)\r\n" +
          "Content-Type: application/json;charset=utf-8\r\n" +
          "Authorization: Bearer "+cloudDeviceToken+"\r\n" + 
          "Content-Length: "+msg.length()+"\r\n" +
          "\r\n" + 
          msg + "\r\n");

  Serial.println("request sent");
  unsigned long timeout = millis();
  while (client.available() == 0) {
    if (millis() - timeout > 5000) {
      Serial.println(">>> Client Timeout !");
      client.stop();
      return;
    }
  }
  
  // Read all the lines of the reply from server and print them to Serial
  while(client.available()){
    String line = client.readStringUntil('\r');
    Serial.print(line);
  }
}

but I face some ESP crashes after a few https requests. Still need to investigate what goes wrong.

[tuxedo0801]

decoded stacktrace:

Exception 29: StoreProhibited: A store referenced a page mapped with an attribute that does not permit stores
Decoding 29 results
0x402167d2: memset at /Users/igrokhotkov/e/newlib-xtensa/xtensa-lx106-elf/newlib/libc/machine/xtensa/../../../../.././newlib/libc/machine/xtensa/memset.S line 133
0x402231d6: more_comps at crypto/bigint.c line 672
0x40223368: alloc at crypto/bigint.c line 672
0x402235e9: regular_multiply at crypto/bigint.c line 672
0x4020eaa8: esp_yield at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266/core_esp8266_main.cpp line 87
0x40223368: alloc at crypto/bigint.c line 672
0x4022409f: bi_barrett at crypto/bigint.c line 1289
0x402242f4: bi_mod_power at crypto/bigint.c line 1414
0x40223238: trim at crypto/bigint.c line 672
0x40225425: RSA_public at crypto/rsa.c line 252
:  (inlined by) RSA_encrypt at crypto/rsa.c line 286
0x40220e9c: send_client_key_xchg at ssl/tls1_clnt.c line 409
0x40221331: do_clnt_handshake at ssl/tls1_clnt.c line 123
0x40208200: ClientContext::_consume(unsigned int) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\libraries\ESP8266WiFi\src/WiFiClientSecure.cpp line 609
:  (inlined by) ClientContext::read(char*, unsigned int) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\libraries\ESP8266WiFi\src\include/ClientContext.h line 253
0x4020eb20: __yield at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266/core_esp8266_main.cpp line 96
0x40220ca4: do_handshake at ssl/tls1.c line 2005
:  (inlined by) basic_read at ssl/tls1.c line 1481
0x40221080: do_client_connect at ssl/tls1_clnt.c line 168
0x401004d8: malloc at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266\umm_malloc/umm_malloc.c line 1664
0x40220e0c: ssl_read at ssl/tls1.c line 2005
0x4020868d: SSLContext::connect(ClientContext*, char const*, unsigned int) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\libraries\ESP8266WiFi\src/WiFiClientSecure.cpp line 609
:  (inlined by) WiFiClientSecure::_connectSSL(char const*) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\libraries\ESP8266WiFi\src/WiFiClientSecure.cpp line 321
0x40207cf2: WiFiClient::connect(IPAddress, unsigned short) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\libraries\ESP8266WiFi\src/WiFiClient.cpp line 324
0x402088ad: WiFiClientSecure::connect(char const*, unsigned short) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\libraries\ESP8266WiFi\src/WiFiClientSecure.cpp line 313
0x402038c4: sendMessageHTTPS(String) at C:\dev\git-repos\TestFW\Test/Test.ino line 489
0x4020dec3: String::changeBuffer(unsigned int) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266/WString.cpp line 519
0x4020df0f: String::reserve(unsigned int) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266/WString.cpp line 519
0x4020df41: String::copy(char const*, unsigned int) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266/WString.cpp line 519
0x4020e048: String::operator=(String const&) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266/WString.cpp line 519
0x40203c74: loop at C:\dev\git-repos\TestFW\Test/Test.ino line 584
0x4020eaf4: loop_wrapper at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266/core_esp8266_main.cpp line 124
0x4010070c: cont_norm at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266/cont.S line 109

Hmm ...

[tuxedo0801]

@igrr
Any ideas what's going wrong?

[tuxedo0801]

Initially, the request was sent every 5 sec. It fails every 5-8 times, then I get stacktrace, device restarts.

I changed interval from 5sec to 60sec. This did not change anything. Crashed after 5 requests.

Beside the https-request, the device is serving a small webpage (which has not been accessed during my tests), queries NTP time and reads power-measurement data from HLW8012 chip via interrupt.
Everything works unless I activate HTTPS.
HTTP worked quite well, was extremly stable (no crash so far). And according to the stacktrace, it has something to do with https/rsa/crypto ...

Would be great if someone can shed some light on this issue.

[tuxedo0801]

switches from 2.4.0-rc1 to rc2... looks better now. already running with >20requests without any crash.

[cimba007]

Any chance to get this working with HTTPClient? My responses could become lager and I don't want to deal with chunked responses manually...

[odelot]

a noob question: this fingerprint changes over time?

I know you are trying not to use them, but I have a valid one and I was thinking to hardcode it on my device.

[romkey]

@odelot That's a good question.

Yes, fingerprints do change over time. They'll change any time the web server's SSL certificate changes.

So if a web server's SSL certificate expires on March 21, 2018 then it'll need a new certificate then and the new certificate will have a new fingerprint.

The server's certificate might easily change before the expiration date, too.

It's even possible for a web server which is load balanced by multiple servers to use different certificates on different servers, so you might see different fingerprints for the same HTTPS endpoint. That's not a likely scenario - it's probably the result of an odd or mis-configuration - but it's possible.

Using the fingerprint to verify the server isn't ideal, but the alternative is maintaining a list of certificates for the root certificate authorities - this is what more capable computers do. Unfortunately these need to be updated too, and their size well exceeds the storage available on an ESP8266.

Unfortunately I don't have a good alternative to hardcoding the fingerprint on your device.

[MikeS159]

@odelot @romkey
There is an example here which lets you use the root certificates fingerprint. Root certs have much longer life spans, since they are hard coded into browser and operating systems, making them hard to update. The root certificate for GitHub (used in the example) is valid until 10 November 2031, 00:00:00 GMT.

To get a root certificate you can view it in a browser (click the green padlock), view certificate, select the root certificate and export it as a .der file. I wrote a little python script to format .der files as a hex string you can past straight into the CACert.ino file.

[davidverweij]

@tuxedo0801 , I have a nearly identical error (stacktrace) - what do you mean by 2.4.0-rc2? My NodeMCU v1.0 crashes at the first attempt.

switches from 2.4.0-rc1 to rc2... looks better now. already running with >20requests without any crash.

[davidverweij]

Apologies - after fiddeling with the variables I found I needed to add the 'https://' to the host adress in order for it to work. I was attempting a POST request a custom Google Cloud functions https endpoint. Somehow I could not find documentation on a Firebase/store request but not Cloud Functions. (e.g. .cloudfunctions.net). I did had a near similar error - so perhaps that might help? Cheers!

[devyte]

Closing due to bearssl merge (axtls will be deprecated).

[C0rn3j]

https://github.com./esp8266/Arduino/tree/master/libraries/ESP8266WiFi/examples

I can only find an example HTTPSRequest which still uses the fingerprint, am I looking in the wrong place?

[d-a-v]

@C0rn3j check BearSSL_Validation.ino in this examples repository.
also check this

[emitterpark]

To use https and ignore fingerprint validation, that worked for me.

Modified the verify function from the line 73 in ESP8266HTTPClient.cpp ,a different way.

bool verify(WiFiClient& client, const char* host) override
{
auto wcs = static_castaxTLS::WiFiClientSecure&(client);
//return wcs.verify(_fingerprint.c_str(), host);
wcs.verify(_fingerprint.c_str(), host);
return true;
}

Then in sketch when to begin client, Use a fake fingerprint, e.g. all Zeroes.

#include <ESP8266WiFi.h>
#include <ESP8266HTTPClient.h>

void setup() {

....
HTTPClient https;

// this goes to: bool HTTPClient::begin(String url, String httpsFingerprint) or char array
https.begin("https://your full url..", "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00");
...

}

[d-a-v]

You should not patch libraries but use their api (or propose patches/fixes):
What you need is available in this example
The improved API also prefers you give a WiFiClient as the first parameter

[emitterpark]

For I am not a programmer, Sorry for the anomalous way in recent.
As you recommended, I restored ESP8266HTTPClient.cpp

Then using HTTPClient wrapper in this way could send an insecure get request on port 443.
Thank you so much.

`#include <ESP8266WiFi.h>
#include <ESP8266HTTPClient.h>

const char *ssid = "my_ssid";
const char *pass = "my_password";

const char * host = "api.github.com.";
const uint16_t port = 443;
const char * path = "/";

void setup() {

Serial.begin(115200);

Serial.println("conntecting to network..");
WiFi.begin(ssid, pass);
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.print(".");
}
Serial.println("conntected to network..");

BearSSL::WiFiClientSecure client;
client.setInsecure();
HTTPClient https;

Serial.println("conntecting to server..");
if (https.begin(client, host, port, path)) {
int httpsCode = https.GET();
if (httpsCode > 0) {
Serial.println(httpsCode);
if (httpsCode == HTTP_CODE_OK) {
Serial.println(https.getString());
}
} else {
Serial.print("failed to GET");
}
} else {
Serial.print("failed to connect to server");
}

}
void loop() {

}`

[cyberphobia]

Here's a write-up that explains how to use HttpClient while checking against an actual certificate store (rather than specifying a certificate fingerprint):

https://medium.com/@dfa_31434/doing-ssl-requests-on-esp8266-correctly-c1f60ad46f5e

[codekoriko]

Here's a write-up that explains how to use HttpClient while checking against an actual certificate store (rather than specifying a certificate fingerprint):

https://medium.com/@dfa_31434/doing-ssl-requests-on-esp8266-correctly-c1f60ad46f5e

Thanks for this golden info :)

[nobilix]

Here's a write-up that explains how to use HttpClient while checking against an actual certificate store (rather than specifying a certificate fingerprint):

https://medium.com/@dfa_31434/doing-ssl-requests-on-esp8266-correctly-c1f60ad46f5e

yes it kinda works (I have only some problems with Let's Encrypt Wildcard Certs), but it runs 10x longer. In my case it was 20 seconds using BearSSL certificate store vs 2 seconds using direct fingerprint string.

[technorainbows]

Note from mtnr: This message is about esp32 and not applicable here

I found that simply by not setting the certificate you can get around the whole verify thing. This is because in the ssl_client library, which is used by WiFiClientSecure, if no certificate is provided it will default to not trying to verify it. See here

` // MBEDTLS_SSL_VERIFY_REQUIRED if a CA certificate is defined on Arduino IDE and
// MBEDTLS_SSL_VERIFY_NONE if not.

if (rootCABuff != NULL) {
    log_v("Loading CA cert");
    mbedtls_x509_crt_init(&ssl_client->ca_cert);
    mbedtls_ssl_conf_authmode(&ssl_client->ssl_conf, MBEDTLS_SSL_VERIFY_REQUIRED);
    ret = mbedtls_x509_crt_parse(&ssl_client->ca_cert, (const unsigned char *)rootCABuff, strlen(rootCABuff) + 1);
    mbedtls_ssl_conf_ca_chain(&ssl_client->ssl_conf, &ssl_client->ca_cert, NULL);
    //mbedtls_ssl_conf_verify(&ssl_client->ssl_ctx, my_verify, NULL );
    if (ret < 0) {
        return handle_error(ret);
    }
} else if (pskIdent != NULL && psKey != NULL) {
    log_v("Setting up PSK");
    // convert PSK from hex to binary
    if ((strlen(psKey) & 1) != 0 || strlen(psKey) > 2*MBEDTLS_PSK_MAX_LEN) {
        log_e("pre-shared key not valid hex or too long");
        return -1;
    }
    unsigned char psk[MBEDTLS_PSK_MAX_LEN];
    size_t psk_len = strlen(psKey)/2;
    for (int j=0; j<strlen(psKey); j+= 2) {
        char c = psKey[j];
        if (c >= '0' && c <= '9') c -= '0';
        else if (c >= 'A' && c <= 'F') c -= 'A' - 10;
        else if (c >= 'a' && c <= 'f') c -= 'a' - 10;
        else return -1;
        psk[j/2] = c<<4;
        c = psKey[j+1];
        if (c >= '0' && c <= '9') c -= '0';
        else if (c >= 'A' && c <= 'F') c -= 'A' - 10;
        else if (c >= 'a' && c <= 'f') c -= 'a' - 10;
        else return -1;
        psk[j/2] |= c;
    }
    // set mbedtls config
    ret = mbedtls_ssl_conf_psk(&ssl_client->ssl_conf, psk, psk_len,
             (const unsigned char *)pskIdent, strlen(pskIdent));
    if (ret != 0) {
        log_e("mbedtls_ssl_conf_psk returned %d", ret);
        return handle_error(ret);
    }
} else {
    mbedtls_ssl_conf_authmode(&ssl_client->ssl_conf, MBEDTLS_SSL_VERIFY_NONE);
    log_i("WARNING: Use certificates for a more secure communication!");
}`

[d-a-v]

@technorainbows Are you working on esp32 ?
esp8266 (this repository) does not use mbedtls.

[JeanVanDenderFlume]

Currently I'm testing the solution provided in #1851, but I would like to see, that I can make https-calls, without specifying a certificate-finger-print and without providing the root-ca-certificate to do certificate validation. Just "accept all certificates and ignore the complete certificate-check", like you can do on any other desktop-programming-language ...

Is this in general possible? Or is this technically on the esp8266 not possible?

Do you know if setInSecure with a valid certificate at the backend, not expired allow encryption of the connection line? I want toe accept several backends on several domains but i dont want to specifiy fingerprint or Cert Authority certificate with setCACert

[earlephilhower]

setInsecure still encrypts. You just can't be sure you're talking to the remote site you think you are since the certificate is not checked.

[Bharath-M-R]

Here's a write-up that explains how to use HttpClient while checking against an actual certificate store (rather than specifying a certificate fingerprint):

https://medium.com/@dfa_31434/doing-ssl-requests-on-esp8266-correctly-c1f60ad46f5e

very useful
Thanks

[thiebatgit]

Bonjour à tous,
Merci pour cette conversation. Grâce à vous et particulièrement à emitterpark que je remercie beaucoup, j'ai pu rétablir un sketch pour contacter l'api sms FREE et envoyer à nouveau des sms de l'esp8266 vers mon téléphone avec abonnement FREE sans aucun frais.
Encore une fois merci pour votre travail.
Thierry Roubaix France

[JaimeSilva]

Hey, I'm facing the same problem. ChatGTP getting me confused too as it's saying that ESP8266HTTPClient will make https work without certificate configuration in the esp itself (as opposed to the WiFiClientSecure).

Anyway, I am considering to make a view(for django) that will collect the certificate using python and send the current certificate to the esp. So everytime the esp will be turned on there will be a request to the webserver through HTTP, which will respond with the certificate and from there I can create an https connection.

But yes, for the original case this wouldn't be possible as the server is not accessible through HTTP only.