SPIFFSeditor file iterator not compatible with LittleFS

[sticilface]

Probably goes without saying that the clue is in the name, but still...

I needed to modify my file iterator to be able to call recursively for directory structures. As you can see below, the files list contains a directory which the editor displays as a file.

I have a working fix for esp8266 that looks like this

 void _fileListIterator(FS & fs, const char * dirName, std::function<void(File & f)> Cb ); 

void SPIFFSEditor::_fileListIterator(FS & fs, const char * dirName, std::function<void(File & f)> Cb )
{
    Dir dir = fs.openDir(dirName);
    while (dir.next()) {
      if (dir.isFile()) {
        File f =  dir.openFile("r");
        if (Cb && f) {
          Cb(f); 
          f.close(); 
        }
      } else {
        _fileListIterator(fs, dir.fileName().c_str() , Cb);
      }
    }
}

and finally.. the business end changed to...

void SPIFFSEditor::handleRequest(AsyncWebServerRequest *request){
  if(_username.length() && _password.length() && !request->authenticate(_username.c_str(), _password.c_str()))
    return request->requestAuthentication();

  if(request->method() == HTTP_GET){
    if(request->hasParam("list")){
      String path = request->getParam("list")->value();
      path = String();
      String output = "[";
      _fileListIterator(_fs, path.c_str(), [&output](File &f) { 
	        if (output != "[") output += ',';
	        output += "{\"type\":\"";
	        output += "file";
	        output += "\",\"name\":\"";
	        output += String(f.fullName());
	        output += "\",\"size\":";
	        output += String(f.size());
	        output += "}";
      });
      output += "]";
      request->send(200, "application/json", output);
      output = String();
    } ...

However, I've not used SPIFFS very much with ESP32 and have not developed on it in a very long time. How different are the FS APIs and does ESP32 support LittleFS now / is planning to?

I can make a pull request, should probably change it to FSEditor as well rather than SPIFFS..

Thoughts?

[lorol]

@sticilface
See a workaround also it in my fork,
The editor is almost an application than library and works with specific html part. (I have some particular additions, not good for a PR)
I am not sure of the best way to be made more universal. You can elaborate and PR as a cleaner solution. Just keep in mind other than listing, the creation of non existing new file on esp32 LITTLEFS needs folders of the path to be pre-made manually by mkdir in contrast of esp8266 way, deletion is tricky - folders again rmdir ... and/or you need to determine what editor should do with (empty) folders. You can run ESP_AsyncFSBrowser variant to see in practice in your browser /edit

[sticilface]

ah i see. A bit more work is involved, i also forgot the exclude list! I'm currently having a bit of a nightmare trying to bring all my ESPs up to date from late 2018 so i can't really work on it now. Is there a plan to bring littleFS to ESP32?

[Pablo2048]

https://github.com./lorol/LITTLEFS ;-)

[lorol]

and ... it is considered for the next generation of the core (2.x):
https://github.com./espressif/arduino-esp32/tree/esp32s2

[lorol]

Not easy to be made ideally compatible to SPIFFS, to esp8266 LittleFS implementation and the esp32 IDF / vfs concept

[sticilface]

mmm.. that's what I was thinking should be the goal. I found it very useful for development and tweaking things without having to upload new files constantly.

The other way.. might be to just make the FSEditor totally independent of the filesystem and it takes a linking class and uses callbacks to populate the data... this can be customised to each use case. Moving forward...

It would also allow it to be used with SD cards and the likes.

FSwebhandle = & webServer.addHandler(new FSEditor( SPIFFSEditorimpl, "user", "pass" ));

or

FSwebhandle = & webServer.addHandler(new FSEditor( LittleFSEditorimpl, "user", "pass" ));

now we are moving towards.. two platforms and two file systems, as it is with all the #ifdef blocks it is very difficult to work out what is going on for each platform.

[sticilface]

@sticilface
See a workaround also it in my fork,
The editor is almost an application than library and works with specific html part. (I have some particular additions, not good for a PR)
I am not sure of the best way to be made more universal. You can elaborate and PR as a cleaner solution. Just keep in mind other than listing, the creation of non existing new file on esp32 LITTLEFS needs folders of the path to be pre-made manually by mkdir in contrast of esp8266 way, deletion is tricky - folders again rmdir ... and/or you need to determine what editor should do with (empty) folders. You can run ESP_AsyncFSBrowser variant to see in practice in your browser /edit

My JS is rubbish. I've had a look at your fork. what are the changes in edit.htm that you have made?

I'm currently refactoring using templates in a mostly transparent way.

I'm also wondering if this just needs to be a operate lib. or a submodule

[sticilface]

@me-no-dev

I'm writing a version of this that should be much more flexible, be platform and FS independent just requiring an interface class...

I'm trying to allow multiple instances, so permit setting the URL of the editor in the constructor.

unfortunately the /edit is hardcoded into the html

here

ESPAsyncWebServer/src/edit.htm

Line 542 in f6fff3f

requests.add("POST", "/edit", formData, httpPostProcessRequest);

and

ESPAsyncWebServer/src/edit.htm

Line 554 in f6fff3f

requests.add("GET", "/edit", { edit: theUrl }, httpGetProcessRequest);

I really have no idea with JS unfortunately, and can't really test it either.

[sticilface]

Ok scratch that previous comment. I've managed to do it, i've now got an independent lib based on SPIFFSEditor that

• has independent implementations for each device/file format. My main thinking with this is to easily allow serving of an editor for files on SD cards, multiple partitions etc... Currently it does depend on fs::FS and fs::File especially with the exclude list stuff but i 'should' be able to migrate it out and then it can work with anything. I've not tackled ESP32 yet but should be easy. There is quite a bit of code duplication which I can maybe get rid of by using some inheritance.

• support for multiple instances of FSEditor so the following works in the same sketch

 server.addHandler(new FSEditor_ESP8266_SPIFFS("editSFS",  SPIFFS  ));
 server.addHandler(new FSEditor_ESP8266_LITTLEFS("/editLFS", LittleFS ));

I've actually written a lib called FSTools that is waiting for merge with the ESP8266 Core that allows some weird and wonderful FS stuff like converting between file system types and mounting two different partition types at the same time.

esp8266/Arduino#7696

• Modified the code to attempt to read an /edit.htm file from the file system then if it is not present then serve the static PROGMEM one.

Currently there is not "much" difference functionally between the implementations, and as i mentioned the ESP32 is missing. I could drop the dependence on File and FS which would allow some other weird uses such as virtual FS and maybe tag editing of the Preferences (nvs)??

I'd be interested on your thoughts?

https://github.com./sticilface/FSEditor

[stale]

[STALE_SET] This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[sticilface]

Any thoughts?

[stale]

[STALE_DEL] This stale issue has been automatically closed. Thank you for your contributions.

[embedded-creations]

This issue really should still be open. @sticilface your solution seems on the right track but I haven't had time to really dig into it. It would be nice if FSEditor had at least a basic README

[embedded-creations]

I'm finally evaluating @lorol's fork for LittleFS compatibility (and planning to add support for SD and multiple FS sources simultaneously). Some notes in case it helps anyone here:

One big change if you're used to the original SPIFFSEditor example: the editor and required files like javascript code are stored in the filesystem, so you need to use the Data Upload tool to load these before they'll work. Edit.htm can be stored in program flash (by commenting out #define EDFS), but you can't go back to loading Ace from the web.

If you want to try the SmartSwitch example, I got it to work by:

• flashing SmartSwitch example
• ESP8266 LittleFS Data Upload
• Find the network it serves, and add wifi credentials via the portal
• Login to the root page, password "switch"
• Now /edit works