8 November 2023
CycleGlide 5” Quad
BetaFlight 4.4.0, CLRacing F4S V1.6
This quad has fallen out of the sky on three occasions. On the first two incidents, I thought I may have ‘tumbled’ the gyros with flips and rolls, and the sensors stopped working. The third time it happened, the quad was flying straight and level and was only a few hundred feet away in direct line of site, so it was most likely not confused sensors or link loss. I figured the flight controller hardware itself was bad, and was preparing to swap out the flight controller for another F4 FC.
Before doing that hardware swap, I downloaded all the BetaFlight logs from the FC. I should have done this earlier. I found that for all three incidents, the log size was the same: 16,384 KB (exactly 16.0 MB). I was not sure if the BetaFlight logger saved in 16MB chunks, but I also saw in the log data that there were many other smaller log files of various sizes (411KB to 7370 KB), with nothing bigger than the three 16 MB files. This indicated to me that the logger probably does not save in set file size chunks. The 16MB files all contained ~165 seconds (2.8 minutes) of data, ending right before the quad just dropped. This “fall-out of the sky” issue was suddenly becoming to look like a SW problem and not HW.
I checked the SD card format; it was formatted in FAT ( FAT16 ). The largest file size in FAT16 is 2 GB, so these 16MB files should not be an issue for file size, and BetaFlight is supposed to deal with both FAT16 and FAT32. However, the BetaFlight website recommends that SD cards be formatted with FAT32 (versus FAT16), so I made that format change and went out to fly again. There were no problems this time; I flew around my front yard for almost 11 minutes. The log file for this flight is 31,085 KB, and contains 10:46 worth of data. This leads me to believe there is some issue with BF 4.4.0 (at least on this FC) that has a strange interaction with log file size on a FAT16-formatted memory card.
My next step was to review the source code for the logger code to see if it has any different settings when recording to FAT16 versus FAT32. The memory card hardware interface is done in io/asyncfatfs/asyncfatfs.c & .h and fat_standard.c & .h. The logger code grabs a large filespace (a supercluster, see comments in the code below) and then groups log data within a single FAT sector to allow for very fast logging. There are definitely different sizes to the log file attributes between FAT16 and FAT32; the FAT sector size (16 versus 32) may be the culprit here, but I have not traced the full differences yet.
Normal logs, examples:
7370 KB : recorded on FAT16, log duration as shown in BetaFlight BlackBox Explorer Legend : 0.00 - 2:30, time stamps and raw data shown for whole log (150 seconds) à 49.1 KB / sec,
31085 KB : recorded on FAT32, log duration as shown in BetaFlight BlackBox Explorer Legend : 0:00 - 10:46, time stamps and raw data shown for whole log (646 seconds) à 48.1 KB / sec
Incident logs:
16384 KB (05) : recorded on FAT16, log duration as shown in BetaFlight BlackBox Explorer Legend : 0:00 – 7:35, time stamps shown for 4:47 – 7:35 (168 seconds) à 97.52 KB / sec, no timestamps or raw data displayed before 4:47
16384 KB (08) : recorded on FAT16, log duration as shown in BetaFlight BlackBox Explorer Legend : 0:00 - 4:27, time stamps shown for 1:43 – 4:27 (164 seconds) à 99.9 KB / sec, no timestamps or raw data displayed before 1:43
16384 KB (10) : recorded on FAT16, log duration as shown in BetaFlight BlackBox Explorer Legend : 0:00 – 4:28, time stamps shown for 1:45 – 4:27 (162 seconds) à 101.1 KB / sec, no timestamps or raw data displayed before 1:45
This all leads me to believe that at somewhere between 150 seconds and 162 seconds of recording ( ~8MB at the 2 KHz rate I have set in my BlackBox), the logger is incorrectly setting a file value that suddenly causes the file size to jump and pointers to the write locations to become corrupted. I am guessing that a FAT32-sized variable is getting used somewhere as a global value. There is no issue when using a FAT32-formatted SD card, but in the FAT16 configuration, this incorrectly sized global value causes the logger to choke when crossing some file size boundary, inducing a cascading failure within the FC.
Bottom line: Format your SD card with FAT32 to avoid this issue.
IO/ASYNCFATFS/ASYNCFATFS.C:
/**
* This is a FAT16/FAT32 filesystem for SD cards which uses asynchronous
operations: The caller need never wait
* for the SD card to be ready.
*
* On top of the regular FAT32 concepts, we add the idea of a "super cluster".
Given one FAT sector, a super cluster is
* the series of clusters which corresponds to all of the cluster entries in that
FAT sector. If files are allocated
* on super-cluster boundaries, they will have FAT sectors which are dedicated to
them and independent of all other
* files.
*
* We can pre-allocate a "freefile" which is a file on disk made up of contiguous
superclusters. Then when we want
* to allocate a file on disk, we can carve it out of the freefile, and know that
the clusters will be contiguous
* without needing to read the FAT at all (the freefile's FAT is completely
determined from its start cluster and file
* size, which we get from the directory entry). This allows for extremely fast
append-only logging.
*/