Use esptool.py to handle sketch upload, make python available on Windows, too #5635
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add installation of python on Win32/Win64, and on all systems install esptool.py and pyserial.
Remove need for binary esptool-ck.exe by implementing the same logic as esptool-ck uses in Python. Only image creation is supported, and only in the Arduino standard mode (with its custom bootloader and ROM layout).
Remove all references to esptool-ck and use Python on Windows and Linux for all recipes where possible.
earlephilhower
force-pushed
the
esptool.py
branch
from
ef70c4e to
b630df3
Compare
Avoid ugly bash and CMD.exe tricks in platform.txt by using python to make the core_version header.
Need to make sure Python2 and Python3 compatible and paths are munged properly to avoid eaccidentally escaping things when calling esptool.py Able to compile, build a BIN and upload via esptool.py on a Windows machine without Python installed globally, only as part of the Arduino tools package.
earlephilhower
changed the title
|
Got a plain Windows with no Python installed to do a full build and upload using python in its own embedded directory, installed by the .JSON file. Also removed much of the BASH/CMD mess in platform.txt and moved to simpler, more maintainable python scripts (making core_version.h, others). Since Python can be used on Windows now, removed many of the separate Will need to update the JAVA plugin to use the perl esptool.py to handle uploads of SPIFFS images, but that's a different repo. |
|
Need to support |
added 2 commits
January 22, 2019 13:58
Add back in erase support by calling esptool.py twice (since it does not support chained operations like esptool-ck.exe).
961K doesn't seem to work with esptool, so make 460K the default upload speed and remove 961K. Even at this lower speed, esptool.py is much faster to upload (even before taking into account the compression when doing things like SPIFFS and code upload).
Closed
Arduino does not support a upload.#.cmd pattern, so we need to do everything in a single command line. Make it cleaner by introducing a Python wrapper script which will run the same executable with different sets of commands (since we need to erase a block w/a separate invocation from the real upload). Update boards.txt to use the new options format, placing the esptool command as "version" when there is no "erase_flash" or "erase_region" to be done to keep things simple.
Since esptool.py and pyserial are coming directly from github repos, there is no need to include them as a tool in package.json.
|
TODO: Add-back 921K, make it silently revert to 420K (with a note in the code that this goes away next major release) |
This was referenced Jan 24, 2019
To enable full backward compatibility, restore the 921k option for upload speed but silently change it to 460k in the upload.py script. Add error checking on upload.py
d-a-v
approved these changes
Feb 16, 2019
devyte
approved these changes
Feb 17, 2019
|
Did you restart get.py ? |
|
I have rebooted computer - it doesn't helps. |
|
No need to do that. |
|
I have restarted get.py with command |
|
@reaper7, @mcspr, anyone else interested in this: I've updated the ESP SPIFFS uploader to support esptool.py and done some basic testing. I'd appreciate if people could give it a try and if they find any issues, please comment on the PR in the other repo: |
|
@earlephilhower - personally I do not use spiffs partitions in my projects but I will try to check it out during the weekend |
1 task
|
On Windows 10 using the latest git version with a NodeMCU board after removing ActivePython to ensure the bundled python was used:
|
|
Thanks for the update, guys! |
|
Hi @earlephilhower, esptool.py maintainer here. Sorry I'm a bit late to the party on this one, but I'm glad that esptool.py is useful for the Arduino esp8266 project!
Just wanted to say, this was news to me. If anyone has any details about specific hardware that does this (hardware setup descriptions, output from esptool.py failing at 921600 and another tool succeeding at 921600) then please send issues this way! As of a few versions ago, esptool.py uses a heuristic to estimate the baud rate shift and try to account for inaccuracies between the host & the device's exact baud rate. This heuristic was more reliable than the old method for all of the hardware I tested with at the time, but maybe I didn't test enough hardware! |
|
Howdy @projectgus, Thanks for the tool. It's made build-test cycles much faster in the core here. For the baud rate issue, I am seeing the same kind of behavior across multiple "D1 Mini" (clone? hard to say, but probably) boards from 2 batches. It starts comms, seems to negotiate an 420K runs 100% of the time and seems to do so for everyone so far. The old esptool-ck.exe at 921k worked fine on this hardware/board (but had other issues and would often take multiple tries to get initial connections going, but once connected the upload did go through at 921k, or at least it said it did). I'm running the D1 minis on an X79 server motherboard off the internal INTC USB ports, and Ubuntu 18.04. If you'd like, I can remove the comment about not working at 921K above. Didn't mean to complain, at 420K it's still way faster than the uncompressed one, by 10x+ for mostly empty SPIFFS filesystems, in fact! |
|
Hi @earlephilhower , Thanks for the all these details!
I had one of these floating around and I can confirm this is the case. At 460800 all of the But esptool-ck passes at 921600. I don't have time to debug right now, but I'll open an issue to track this.
That's great to hear. No need to remove anything at all. I just appreciate knowing about the difference, so we can hopefully fix it. |
pfeerick
added a commit
to pfeerick/platform-espressif8266
that referenced
this pull request
Sep 8, 2019
As of esp8266/Arduino#5635 the esp8266/Arduino core has defaulted to 460800 baud for serial uploads and also silently limits the highest supported upload rate of 921600 to 460800. https://github.com./esp8266/Arduino/blob/77c4f5e5cfb2286d6862621bd78120f027d10a9f/tools/upload.py#L29 sed -i "s/\"speed\".*/\"speed\": 460800/g" *.json
maciekn
added a commit
to maciekn/vscode-esp8266fs
that referenced
this pull request
Jan 6, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Go from esptool-ck to esptool.py for the upload stage because the python version is actively maintained by Espressif, has compression, and will be needed for support of future SDKs.
On Windows installs its own, private copy of Python in the Arduino/tools directory.
On all OSes installs the latest pyserial release in the Arduino dir and uses it to simplify installation and ensure everyone is running the same code. It also installs a copy of esptool.py. No modifications are done, we use releases directly from the source.
For now esptool.exe (actually esptool-ck) and esptool.py get dumped into the same folder, but at some point it should be possible to completely remove esptool-ck.
Esptool-ck is still needed because our bootloader is different (and I think better) than the others that esptool.py depends on for single-file uploads.
No binary changes to the .BIN are done, esptool simply does the write_flash stage.
Runs under Linux (and probably Mac), but I need to boot up a Windows box w/passthrough to ensure the Windows python command line is good.
To test, be sure to run
python tools/get.pyto get the new tools.Note that esptool.py does NOT seem to work @921k, only 460K baud (even on boards that are known good at 921K). However, even with the lower baud rate it seems to run faster thanks to either better pipelining or the compression.
Fixes #4982