esp32's python dependency won't be installed by the board manager

[ianfixes]

Issue summary

via https://github.com./RobTillaart/RunningAverage/runs/1506360909?check_suite_focus=true

Installing board package esp32:esp32...                                        ✓
Installing board package esp8266:esp8266...                                    ✓
Compiling fillValue.ino for arduino:avr:uno...                                 ✓
Compiling fillValue.ino for arduino:sam:arduino_due_x...                       ✓
Compiling fillValue.ino for arduino:samd:arduino_zero_native...                ✓
Compiling fillValue.ino for arduino:avr:leonardo...                            ✓
Compiling fillValue.ino for adafruit:samd:adafruit_metro_m4...                 ✓
Compiling fillValue.ino for esp32:esp32:featheresp32:FlashFreq=80... 
Last command:  $  /usr/local/bin/arduino-cli --format json compile --fqbn esp32:esp32:featheresp32:FlashFreq=80 --warnings all --dry-run /github/home/Arduino/libraries/RunningAverage/examples/fillValue/fillValue.ino
Error during build: exec: "python": executable file not found in $PATH         ✗

[per1234]

I believe the situation is that the ESP32 boards platform expects Python to be installed and in the path, accessible as python (e.g., espressif/arduino-esp32#4017). So they intentionally don't install it. We're accustomed to Arduino boards platforms installing all thier dependencies, but the authors of the Espressif platforms took a different path in the case of the Python dependency.

It's unfortunate to be forced to configure the action to install arbitrary platform dependencies. The workaround I found for doing this on the user end of things when I ran into this issue with the ESP8266 platform was to use a custom entrypoint wrapper script that installed Python in the container:
arduino-libraries/Arduino_ConnectionHandler@5694e24#diff-5e8968a2eeba10018e702f1a34161eef3e89f76406123286100b969fffdc416f

Since that time, I converted the arduino/compile-sketches action to a composite run steps action, meaning it runs in the GitHub Actions runner instead of a Docker container. This permits the user to easily install any dependencies in their workflow, so my philosophy is that it's reasonable to expect them to do so. But in the case of Docker container actions, it's not so user friendly.

[ianfixes]

Hey! You beat me to writing an upstream bug report in arduino-cli.

I see that the python dependencies were specified directly by the file pointed to by the esp8266 board manager URL: http://arduino.esp8266.com/stable/package_esp8266com_index.json

          "toolsDependencies": [
            {
              "packager": "esp8266",
              "version": "2.5.0-3-20ed2b9",
              "name": "xtensa-lx106-elf-gcc"
            },
            {
              "packager": "esp8266",
              "version": "2.5.0-3-20ed2b9",
              "name": "mkspiffs"
            },
            {
              "packager": "esp8266",
              "version": "3.7.2-post1",
              "name": "python"
            }
          ],
          "name": "esp8266"
        }
      ],
      "tools": [
        {
          "version": "3.7.2-post1",
          "name": "python3",
          "systems": [
            {
              "host": "x86_64-mingw32",
              "url": "https://github.com./earlephilhower/esp-quick-toolchain/releases/download/2.5.0-4/python3-3.7.2.post1-embed-win32v2a.zip",
              "archiveFileName": "python3-3.7.2.post1-embed-win32v2a.zip",
              "checksum": "SHA-256:f57cb2daf86176d2929e7c58990c2ac32554e3219d454dcac10e464ddda35bf2",
              "size": "6428926"
            },

(this is the most sensible snippet I could find in a long file... but just ctrl-f python in that URL and you'll get to the sections I'm referring to)

But I don't see any python dependencies listed in the URL that I have for the esp32 board manager: https://dl.espressif.com/dl/package_esp32_index.json

• If the board manager URL properly specified the python dependency, would arduino-cli honor it?
• Am I using the wrong URL (or an outdated one) for esp32?
• If the esp32 URL is correct and the problem is in fact the lack of dependency info, where would I report that? I assume they could just copy in the lines from the esp8266 board manager and have it work.

[per1234]

I see that the python dependencies were specified directly by the file pointed to by the esp8266 board manager URL:

The Windows installation of the ESP8266 platform provides a full Python installation, but for Linux and macOS it only installs a symlink pointing to the location where the system installation of Python is assumed to be (esp8266/Arduino#6931). If Python isn't at that location, then you will get the same error. This isn't an issue with Arduino CLI. It's simply a consequence of the configuration of the Espressif platforms.

But I don't see any python dependencies listed in the URL that I have for the esp32 board manager:

I think it's just a prerequisite to have Python installed as python and in the path if you want to use that platform.

If the board manager URL properly specified the python dependency, would arduino-cli honor it?

Yes, just as it would any other tools dependency.

Am I using the wrong URL (or an outdated one) for esp32?

I think so. The officially recommended ones are in the installation instructions here:
https://github.com./espressif/arduino-esp32/blob/master/docs/arduino-ide/boards_manager.md

If the esp32 URL is correct and the problem is in fact the lack of dependency info, where would I report that?

https://github.com./espressif/arduino-esp32/issues but it is my understanding that the platform is intentionally configured this way.

I assume they could just copy in the lines from the esp8266 board manager and have it work.

It would make it work on Windows systems that don't have Python installed and in the path, but it doesn't improve the situation much (if at all) for Linux and macOS because the symlink only helps if:

• You don't already have Python in the path
• You do have Python installed at the location the symlink points at.

[ianfixes]

In case you know the answer to this: is it sufficient to install just one python version or will I need both 2.x and 3.x to cover all the board dependencies?

Argh, and looking at espressif's github, I'm now nervous about which python packages I'll also need to pip install...

[ianfixes]

Based on acrandal/RevEng_PAJ7620#16 It looks like pyserial is the only package to worry about (at least for now).