Skip to content

SDFS/SD make F() strings appear in RODATA, not flash #6767

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
4 of 6 tasks
TD-er opened this issue Nov 13, 2019 · 9 comments · Fixed by #6787
Closed
4 of 6 tasks

SDFS/SD make F() strings appear in RODATA, not flash #6767

TD-er opened this issue Nov 13, 2019 · 9 comments · Fixed by #6787
Assignees
Milestone

Comments

@TD-er
Copy link
Contributor

TD-er commented Nov 13, 2019

Basic Infos

  • This issue complies with the issue POLICY doc.
  • I have read the documentation at readthedocs and the issue is not addressed there.
  • I have tested that the issue is present in current master branch (aka latest git).
  • I have searched the issue tracker for a similar issue.
  • If there is a stack dump, I have decoded it.
  • I have filled out all fields below.

Platform

  • Hardware: [ESP-12|ESP-01|ESP-07|ESP8285 device|other]
  • Core Version: [Core 2.6.0]
  • Development Env: [Platformio]
  • Operating System: [Windows]

Settings in IDE

  • Module: [Generic ESP8266 Module]
  • Flash Mode: [dio]
  • Flash Size: [4MB/1MB]
  • lwip Variant: [v2 Lower Memory]
  • Reset Method: [nodemcu]
  • Flash Frequency: [40Mhz]
  • CPU Frequency: [80Mhz]
  • Upload Using: [OTA|SERIAL]
  • Upload Speed: [115200] (serial upload only)

Problem Description

In my project, we have had log to SD present, but since it was disabled for a long time, I now made it compile again.
At first there is this strange thing when including SD.h, a lot of __FlashStringHelper related code would no longer compile.

For example I could not compile code like this anymore:

   static const __FlashStringHelper *gpMenu[8][3] = {
      F("⌂"),   F("Main"),          F("/"),              // 0
[...]
      F("🔧"), F("Tools"),         F("/tools"),         // 7
    };

Had to change it into separate functions with a switch statement returning F() macro defined strings.

After changing all of these, I ended up with this error:

c:/users/gijs/.platformio/packages/toolchain-xtensa/bin/../lib/gcc/xtensa-lx106-elf/4.8.2/../../../../xtensa-lx106-elf/bin/ld.exe: address 0x3fffe008 of .pio\build\debug_custom_ESP8266_4M1M\firmware.elf section `.bss' is not within region `dram0_0_seg'
c:/users/gijs/.platformio/packages/toolchain-xtensa/bin/../lib/gcc/xtensa-lx106-elf/4.8.2/../../../../xtensa-lx106-elf/bin/ld.exe: address 0x3fffe008 of .pio\build\debug_custom_ESP8266_4M1M\firmware.elf section `.bss' is not within region `dram0_0_seg'
collect2.exe: error: ld returned 1 exit status
*** [.pio\build\debug_custom_ESP8266_4M1M\firmware.elf] Error 1

Only when throwing out almost all code from my sketch, I was able to compile it wilt SD support enabled.
But I doubt it is useful, since you don't have any memory left.

Without SD support:

DATA:    [=====     ]  52.0% (used 42572 bytes from 81920 bytes)
PROGRAM: [=======   ]  69.0% (used 616020 bytes from 892912 bytes)
Creating BIN file ".pio\build\minimal_core_260_ESP8266_1M_OTA\firmware.bin" using ".pio\build\minimal_core_260_ESP8266_1M_OTA\firmware.elf"

With SD support:

DATA:    [========  ]  84.7% (used 69356 bytes from 81920 bytes)
PROGRAM: [=======   ]  70.9% (used 632760 bytes from 892912 bytes)
Creating BIN file ".pio\build\minimal_core_260_ESP8266_1M_OTA\firmware.bin" using ".pio\build\minimal_core_260_ESP8266_1M_OTA\firmware.elf"

That's an increase of over 26 kByte RAM usage.

I'm not entirely sure the issue with the __FlashStringHelper related code has anything to do with this, but maybe it does somehow prevent strings to be stored in flash?

@TD-er
Copy link
Contributor Author

TD-er commented Nov 14, 2019

Just an update.
Simply including the file SD.h is enough to run out of RAM.
A user of my project also did try compiling it in ArduinoIDE and got the same issue (running out of RAM)

@earlephilhower
Copy link
Collaborator

Can you provide a simple MCVE, @TD-er , which shows this? That would let me reproduce it locally and make fixing things much faster.

@TD-er
Copy link
Contributor Author

TD-er commented Nov 14, 2019

OK, took a while to come up with a proper MCVE.
But I think I got one and you need to do a bit yourself ;)

#include <Arduino.h>

#include <SD.h>

void setup() {
//  String foo = F("Lorem ipsum ...., euismod tortor.");

}

void loop() {

}

I used 5 paragraphs of Lorum Ipsum as one-liner in the flash string. (3229 characters, used 2 spaces between the paragraphs)

What happens is this: (core 2.6.0)

#include <SD.h>
String foo 

DATA:    [====      ]  38.1% (used 31216 bytes from 81920 bytes)
PROGRAM: [===       ]  26.5% (used 277132 bytes from 1044464 bytes)
#include <SD.h>
//String foo 

DATA:    [===       ]  34.2% (used 27984 bytes from 81920 bytes)
PROGRAM: [===       ]  26.2% (used 273892 bytes from 1044464 bytes)
//#include <SD.h>
String foo 

DATA:    [===       ]  34.0% (used 27844 bytes from 81920 bytes)
PROGRAM: [==        ]  24.3% (used 253744 bytes from 1044464 bytes)
//#include <SD.h>
//String foo 

DATA:    [===       ]  34.0% (used 27844 bytes from 81920 bytes)
PROGRAM: [==        ]  24.0% (used 250320 bytes from 1044464 bytes)

It looks like all flash strings are now in memory, as soon as you include SD.h

@earlephilhower
Copy link
Collaborator

Cool. That's very weird and I'll definitely look into it!

@TD-er
Copy link
Contributor Author

TD-er commented Nov 14, 2019

I just updated the order of the outputs, to make it more logical.

@TD-er
Copy link
Contributor Author

TD-er commented Nov 14, 2019

Just one thing that looks strange while browsing through the includes from SDFS.h.

The file assert.h does not have #ifndef... #define... #endif wrappers.

Probably nothing to do with this, but it also doesn't help compilation either I guess. (although it does include pgmspace.h )

@earlephilhower
Copy link
Collaborator

Ah, able to repro and narrow it down.

The issue is actually caused by including the real <SdFat.h> file from SdFat library!

For some reason he has this snippet in src/syscall.h

#ifdef ESP8266
// undefine F macro if ESP8266.
#undef F
#endif  // ESP8266

I'll pull it out (we already have some local changes for SPI compat) and do some testing...

@earlephilhower earlephilhower changed the title SDFS/SD does use a lot of RAM SDFS/SD make F() strings appear in RODATA, not flash Nov 15, 2019
@earlephilhower
Copy link
Collaborator

Updated the title, @TD-er , just to make it clear when the PR comes in what the root issue was.

@earlephilhower
Copy link
Collaborator

Also, about assert.h: That's coming from the GCC toolchain, basically, and I only added the pgmspace.h include and did minor tweaks to the #define. I think it's fine as-is, because that's what all other OSes and chips are doing (well, anything using GCC...)

earlephilhower added a commit to earlephilhower/ESP8266SdFat that referenced this issue Nov 16, 2019
When SdFat.h is included it was undefining the F() (flash string) macro.
This causes all strings to go into RAM, as seen in this issue:
esp8266/Arduino#6767

Remove this undef.  The F() is only used in a few print statements which
are F()-safe on the 8266 now, and in the SD.h and SDFS.h libs the
function that actually calls print(F(xxxx)) is never used.
earlephilhower added a commit to earlephilhower/Arduino that referenced this issue Nov 16, 2019
Fixes esp8266#6767 .  Remove the `undef F` from SysCall.h as it is not needed
nor used in the SD or SDFS libraries.  This puts F() strings back in
flash when using the SD/SDFS libs.
@devyte devyte added this to the 2.6.2 milestone Nov 16, 2019
earlephilhower added a commit that referenced this issue Nov 16, 2019
Fixes #6767 .  Remove the `undef F` from SysCall.h as it is not needed
nor used in the SD or SDFS libraries.  This puts F() strings back in
flash when using the SD/SDFS libs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants