inconsistent block size for spiffs in board.txt. and ld files #5412

apicquot · 2018-12-02T13:38:37Z

boardtxt.py is not consistent for generating spiffs block size in boards.txt and ld files:

it takes the flash size for boards.txt but the spiffs size for the lf files: there is currently a problem for boards with flash size >= 1M and spiffs size < 1M

impacted boards 2M (SPIFFS 128K), 2M (SPIFFS 256K) 2M (SPIFFS 512K)

Suggested change is to make block size dependable on spiffs size for board.txt:
if spiffs size < 1M block size=4096 else block size=8192

… be consistent with ld files spiffs < 1M has a 4096 block size and 8192 for all the other ones only impacted boards 2M (128K), 2M (256K) 2M (512K)

devyte · 2018-12-02T16:48:27Z

tools/boards.txt.py

    else:
        max_upload_size = 1024 * 1024 - reserved
        spiffs_start = (flashsize_kb - spiffs_kb) * 1024
+
+    if spiffs_kb < 1024:


Should this be < or <=?

apicquot · 2018-12-02T17:09:56Z

it should be < 1024: for example 2M (SPIFFS 1M) should sill have a block size of 8192.

devyte · 2018-12-02T17:22:38Z

@apicquot you mentioned a problem, but you didn't specify what. What did you encounter?

apicquot · 2018-12-02T17:28:48Z

Currently spiffs does not work for boards 2M (SPIFFS 128K), 2M (SPIFFS 256K) and 2M (SPIFFS 512K):
the ld files contain a blockssize of 4096 and board.txt a blocksize 8192. The SPIFFS system can not be found or SPIFFS.info(fs_info) and all other SPIFFS functions fail

devyte · 2018-12-02T22:38:10Z

@apicquot what about creating a SPIFFS image from the IDE for flashing? what decision is made for the block size there?

apicquot · 2018-12-03T14:48:14Z

@devyte the block size when flashing the spiffs from the ide (parameters passed to mkspiffs) is coming from board.txt. That is why it has to be the same value than in the ld files

d-a-v · 2018-12-03T15:11:08Z

While understandable, this is to me dangerous as-is, because users already using existing from some time 2M512 will have their spiffs reformated on boot after this patch.

You are basically stating that spiffs block size should be 4096 when spiffs size < 1MB.
Why not choosing if spiffs_kb < 512: instead of < 1024 ?

We can see below that it is 4096/0x1000 when size is < 512 and 0x2000 otherwise.
I reckon your patch is very useful and needed to make spiffs-blocksize calculated from a real understandable rule, but 1024 scares me.

tools/sdk/ld$ grep SPIFFS_block *.ld
eagle.flash.16m14m.ld:PROVIDE ( _SPIFFS_block = 0x2000 );
eagle.flash.16m15m.ld:PROVIDE ( _SPIFFS_block = 0x2000 );
eagle.flash.1m.ld:PROVIDE ( _SPIFFS_block = 0x0 );
eagle.flash.1m128.ld:PROVIDE ( _SPIFFS_block = 0x1000 );
eagle.flash.1m144.ld:PROVIDE ( _SPIFFS_block = 0x1000 );
eagle.flash.1m160.ld:PROVIDE ( _SPIFFS_block = 0x1000 );
eagle.flash.1m192.ld:PROVIDE ( _SPIFFS_block = 0x1000 );
eagle.flash.1m256.ld:PROVIDE ( _SPIFFS_block = 0x1000 );
eagle.flash.1m512.ld:PROVIDE ( _SPIFFS_block = 0x2000 );
eagle.flash.1m64.ld:PROVIDE ( _SPIFFS_block = 0x1000 );
eagle.flash.2m.ld:PROVIDE ( _SPIFFS_block = 0x0 );
eagle.flash.2m128.ld:PROVIDE ( _SPIFFS_block = 0x1000 );
eagle.flash.2m1m.ld:PROVIDE ( _SPIFFS_block = 0x2000 );
eagle.flash.2m256.ld:PROVIDE ( _SPIFFS_block = 0x1000 );
eagle.flash.2m512.ld:PROVIDE ( _SPIFFS_block = 0x2000 );
eagle.flash.4m.ld:PROVIDE ( _SPIFFS_block = 0x0 );
eagle.flash.4m1m.ld:PROVIDE ( _SPIFFS_block = 0x2000 );
eagle.flash.4m2m.ld:PROVIDE ( _SPIFFS_block = 0x2000 );
eagle.flash.4m3m.ld:PROVIDE ( _SPIFFS_block = 0x2000 );
eagle.flash.512k.ld:PROVIDE ( _SPIFFS_block = 0x0 );
eagle.flash.512k128.ld:PROVIDE ( _SPIFFS_block = 0x1000 );
eagle.flash.512k32.ld:PROVIDE ( _SPIFFS_block = 0x1000 );
eagle.flash.512k64.ld:PROVIDE ( _SPIFFS_block = 0x1000 );
eagle.flash.8m6m.ld:PROVIDE ( _SPIFFS_block = 0x2000 );
eagle.flash.8m7m.ld:PROVIDE ( _SPIFFS_block = 0x2000 );

apicquot · 2018-12-03T15:16:50Z

good point @d-a-v

changing to if spiffs_kb < 512 is a good idea

devyte

Per @d-a-v 's comments about < 512KB.

only impacts boards 2M (SPIFFS 128K) and 2M (SPIFFS 256K) 2M (SPIFFS 512K) remains with 8192 block size

….com/apicquot/Arduino into fixing_inconsistent_spiffs_blocksize

blocks size of 4096 for spiffs < 512 block size of 8192 for all other boards

….com/apicquot/Arduino into fixing_inconsistent_spiffs_blocksize

apicquot · 2018-12-04T00:23:53Z

indent is changed in package/package_esp8266com_index.template.json
board.txt.py likely to have been modified in master and file not generated ?

earlephilhower · 2018-12-04T02:44:27Z

#5429 will take care of the indent. platform....8266.json is being edited using string replacement not a JSON parser presently so there will be odd indent differences.

The SPIFFS blocksize change hasn't been tested by me, can't comment on it.

tools/boards.txt.py

earlephilhower

I'm OK w/a merge either w/the package/JSON or w/o it (spacing fix coming already).

tools/boards.txt.py

d-a-v

Thanks for adding this logic

TD-er · 2018-12-05T21:55:31Z

While understandable, this is to me dangerous as-is, because users already using existing from some time 2M512 will have their spiffs reformated on boot after this patch.

You are basically stating that spiffs block size should be 4096 when spiffs size < 1MB.
Why not choosing if spiffs_kb < 512: instead of < 1024 ?

We can see below that it is 4096/0x1000 when size is < 512 and 0x2000 otherwise.
I reckon your patch is very useful and needed to make spiffs-blocksize calculated from a real understandable rule, but 1024 scares me.

The tests I did with the 2M/512k layout simply did not work.
So I think it is very unlikely there are working setups running this layout with working SPIFFS.

Also a practical question about the block size.
Does any file take at least 1 block? (like the name suggests)
If so, then I would even opt for smaller blocks if possible.
On 128k SPIFFS there is only room for max. 32 (very small) files (more likely less due to filesystem overhead) with minimum filesize of 4k.

apicquot · 2018-12-05T23:12:00Z

2M 512K works great for me and so do the 2 new board configs 2M 128K and 2M 256K. I use the IDE for launching mkspiffs and setting up properly the parameters. As of the maximum number of files, my understanding is that they can more files than number of blocks, the minimum file size is the page size. I have 47 small files that works fine with the 128K so going below 4096 block size won't do any good. Hope it helps

devyte · 2018-12-06T00:32:09Z

@apicquot thanks for the info!

block size for board.txt now depends on spiffs size not flash size to…

12365ea

… be consistent with ld files spiffs < 1M has a 4096 block size and 8192 for all the other ones only impacted boards 2M (128K), 2M (256K) 2M (512K)

devyte approved these changes Dec 2, 2018

View reviewed changes

esp8266 deleted a comment from apicquot Dec 3, 2018

Merge branch 'master' into fixing_inconsistent_spiffs_blocksize

55ef182

devyte requested changes Dec 3, 2018

View reviewed changes

apicquot and others added 12 commits December 3, 2018 18:20

block size of 4096 for spiff size < 512 (instead of 1024)

6f54002

only impacts boards 2M (SPIFFS 128K) and 2M (SPIFFS 256K) 2M (SPIFFS 512K) remains with 8192 block size

Merge branch 'fixing_inconsistent_spiffs_blocksize' of https://github…

41459e1

….com/apicquot/Arduino into fixing_inconsistent_spiffs_blocksize

Merge branch 'master' into fixing_inconsistent_spiffs_blocksize

db681a4

block size of 4096 for all boads of flash size <= 1024

bbaf6d9

blocks size of 4096 for spiffs < 512 block size of 8192 for all other boards

Merge branch 'fixing_inconsistent_spiffs_blocksize' of https://github…

59894c4

….com/apicquot/Arduino into fixing_inconsistent_spiffs_blocksize

indent error

5510e88

indent issue

b64987c

indent issue

5d19803

indent issue

2739c1f

indent issue

8ad359d

indent issue

2403063

keep indent issue

f50df4e

earlephilhower requested changes Dec 4, 2018

View reviewed changes

tools/boards.txt.py Show resolved Hide resolved

earlephilhower approved these changes Dec 4, 2018

View reviewed changes

tools/boards.txt.py Show resolved Hide resolved

earlephilhower added this to the 2.5.0 milestone Dec 4, 2018

Merge branch 'master' into fixing_inconsistent_spiffs_blocksize

67098eb

d-a-v approved these changes Dec 4, 2018

View reviewed changes

d-a-v merged commit ee3b374 into esp8266:master Dec 4, 2018

d-a-v mentioned this pull request Apr 16, 2019

Make SPIFFS be an integer number of blocks #5989

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

inconsistent block size for spiffs in board.txt. and ld files #5412

inconsistent block size for spiffs in board.txt. and ld files #5412

apicquot commented Dec 2, 2018 •

edited by devyte

Loading

devyte Dec 2, 2018

apicquot commented Dec 2, 2018

devyte commented Dec 2, 2018

apicquot commented Dec 2, 2018 •

edited by d-a-v

Loading

devyte commented Dec 2, 2018

apicquot commented Dec 3, 2018

d-a-v commented Dec 3, 2018 •

edited

Loading

apicquot commented Dec 3, 2018 •

edited

Loading

devyte left a comment

apicquot commented Dec 4, 2018

earlephilhower commented Dec 4, 2018

earlephilhower left a comment

d-a-v left a comment

TD-er commented Dec 5, 2018

apicquot commented Dec 5, 2018

devyte commented Dec 6, 2018

inconsistent block size for spiffs in board.txt. and ld files #5412

inconsistent block size for spiffs in board.txt. and ld files #5412

Conversation

apicquot commented Dec 2, 2018 • edited by devyte Loading

devyte Dec 2, 2018

Choose a reason for hiding this comment

apicquot commented Dec 2, 2018

devyte commented Dec 2, 2018

apicquot commented Dec 2, 2018 • edited by d-a-v Loading

devyte commented Dec 2, 2018

apicquot commented Dec 3, 2018

d-a-v commented Dec 3, 2018 • edited Loading

apicquot commented Dec 3, 2018 • edited Loading

devyte left a comment

Choose a reason for hiding this comment

apicquot commented Dec 4, 2018

earlephilhower commented Dec 4, 2018

earlephilhower left a comment

Choose a reason for hiding this comment

d-a-v left a comment

Choose a reason for hiding this comment

TD-er commented Dec 5, 2018

apicquot commented Dec 5, 2018

devyte commented Dec 6, 2018

apicquot commented Dec 2, 2018 •

edited by devyte

Loading

apicquot commented Dec 2, 2018 •

edited by d-a-v

Loading

d-a-v commented Dec 3, 2018 •

edited

Loading

apicquot commented Dec 3, 2018 •

edited

Loading