Minimal USB Mass Storage emulation, future proof?

[grantb5]

What I need to do is make my USB device appear as a USB thumb drive for the exclusive purpose of copying over a small (a few kilobytes) configuration file to it. I like this method of transferring config info because it's generally platform independent, requires no host program -- user's can drag and drop the file.

So, I've been playing with the SI Labs AN282 Mass Storage dev kit on an 8051. The kit is overkill, BUT the USB drive does not show up on Win7/Win8. On XP it shows up as FAT16, but I'm not sure what the allocation block size is (8k? 16k? 32k?). Is there some format (FAT16 would be nice) that has a layout that will work on quite a few hosts?  Win XP/7/8, Linux, Mac, etc??  Preferably one where I don't have to license the format (FAT32 and NTFS are owned by MS, no?). The target hardware is an 8051 derivative.

Please recommend a royalty free format I can target (or other alternative if my idea is stupid in the first place). I have done HID and CDC before, but both require some kind of app on the host to send such config data to the target.

Thanks

[Barry Twycross]

FAT16 is about as universal as it comes. I'm surprised it didn't show up on Win 7/8. Which allocation block size are you thinking of? SCSI block size of 512 is most common. FAT cluster size seems to depend on the volume size.

[grantb5]

It's a curious demo board. If I format the device on Win7, FAT16, 16kb then it works on XP too. If I go the other way it says the drive is not formatted. Anyway I think what the demo is doing is reading what it can from the embedded SD card and providing that info to the PC. I don't need the SD card, just the emulation of it so hopefully I can learn something from it.

My CPU is an 8051 derivative so this is the closest demo I could find. I have an ARM demo somewhere that does mass storage as well which I'm sure works better, but hard to say how much I can transfer over from it... it would be nice if there was some universal USB device class that allowed for (simple host agnostic) application level code to access bidirectionally in a somewhat exclusive manner. Mass storage was the best I could come up with. At least it allows the user to drag and drop.

[Barry Twycross]

What a MSC driver should be doing is just serving whatever blocks are on the medium. It should not be interpreting the content of the blocks. That sounds like what you've got.

You can easily replace the SD card with some other block device, RAM disk is pretty simple. You can format it once and then just use that as your template.

CDC should be the sort of thing you want, but I don't think a host driver for it is very widespread.

[grantb5]

I'm kind of talking without thinking a bit here ... and will report more when I dig in deeper. CDC is my fallback position. What I wrote before ... I meant I was looking for a USER level USB interface not "application" level.  I've done some CDC, MIDI, HID but all of then require some kind of application ... Mass Storage allows for moving a small amount of data manually be the user.  I don't know HTML5 but was looking at some method, cross-platform as long as the host had a USB port (that supports mass storage of course).  Widest coverage possible. Most PCs and even tablets support mass storage. Many of them now support HTML5 in the browser as well, not that I know much about it.

Anyway back to the task at hand, the SI Labs demo is a glorified SDcard-to-USB adapter so I need to see how much of the conversation over USB originates from the 8051 or if it's passing on some of the requests to the card and then relaying the responses back to the PC. My device will not have an embedded SD card in it.

[Barry Twycross]

Strange I inerited a project very similar to this. (We just announced it yesterday.)

I had problems making the FAT volume show up on Win 7, it was working fine on Mac and Linux (not sure about Xp). Win 7 was very picky about the partition map in the MBR. That is it wanted an MBR which had a partition map (absolute sector zero), which pointed to an MBR which didn't have a partition map (absolute sector 1). The MBRs were totally identical otherwise apart from the partition map.

I have it working so the user can drag and drop a firmware file onto it for a software update. Its FAT12 and uses 2k clusters.

[Jan Axelson]

Yikes, Windows can be so much fun...

[grantb5]

Ah, cool product. Bought the thermostat some time ago but couldn't use it because someone ran goofy wiring in our house. I need to pull new wires to hook it up. The smoke detector looks neat too.

Well despite the original eval board demo flaws (see original post), after much hair pulling and rewriting, I was finally able to build a pretty solid little "drive" on an 8051 variant that runs on all the usual suspects (Win XP/7/8, Linux, Mac). It uses FAT16, 512 bytes/sector, 1 sector per cluster, 1 FAT, 1 reserved (boot) sector, 16 entries in the root, reports a size of 124kB.  It's actually only about 24k in size being that it's only a 64k flash micro to start with and it has other functions to perform, but so far it works great.  

For what I'm doing I couldn't see any advantage in using FAT12, but I did consider it early on. Really seemed like a goofy concept from the start in that it didn't really save that much disk space.

[Bret]

Actually, the only place an MBR can be is in sector zero.  The thing in sector one must have been a VBR (Volume Boot Record), which should have included a BPB (BIOS Parameter Block).

MBR's and VBR's usually have very different "looks" to them in the first several bytes of the sector (in addition to the fact that a VBR doesn't have a valid partition map).  I've actually had problems in the past with MBR's that at a cursory level looked like VBR's, and it caused me some grief.  I ending up having to modify the code to do a more thorough test of whether I was looking at an MBR or VBR to be able to properly handle the disk.

I don't know if that's what's actually causing the MS problem or not, but it certainly sounds likely.

[Barry Twycross]

Thanks for the terminology, I'd never heard the term VBR before.

I was hoping to make this thing non partitioned media so I wouldn't need an MBR, just a VBR. Much like floppies used to be. That worked fine with Macs and Linux, but it was Win 7 which didn't like it. After trying to make Win 7 happy, I got it to format the volume for me, and it added the MBR. The MBR looked very much like my VBR with a partition map, so that's what I went with. The added MBR is exactly the same as my VBR including the BPB, its only the partition map which is different. All the OSs seem happy with that.

PS: I inherited the project from a guy called Grant, he used CDC instead of MSC to get it work. For a while I wonder if grantb5 were the same Grant.

[Barry Twycross]

I went with FAT12 as people were very concerned about the overhead of using a FAT file system. So I tried my hardest to reduce the overhead. I got the overhead down to about 12k. (VBR, 1 FAT, 2 sectors of directory). Using FAT16 would use an extra sector for the FAT. In the end there's about 300k spare with our largest images, so I probably didn't need to be quite so aggressive at saving space.

[Bret]

Very interesting.  When a non-floppy is formatted without an MBR (sector 0 just contains a VBR), it's called a super-floppy.  MS used to be able to handle super-floppies correctly, but I guess they removed that functionality somewhere along the line.  What makes this even stranger (at least to me) is that newer versions of Windows won't mount more than one partition off of a removable disk (like USB) anyway, so requiring an MBR (whose primary purpose is to allow more than one partition) really doesn't make any sense at all.

Regarding the FAT12 decision, what file allocation system did they think might have a lot less overhead than FAT12?  From an overhead perspective, FAT12 is pretty efficient.  The code gets a little messy, though, since dealing with 12-bit "bytes" is a hassle on a machine based on multiples of 8 bits.

[grantb5]

I'm innocent! I did not work on that project.

The 12 vs 16 thing ... considering the size of the files the FAT probably points to, isn't niggling over a nibble kind of a waste of time?  The percentage of disk space saved seems pretty small to me, especially since you have to add the code to manage the goofy 12-bit fat in probably the same memory space.

[Barry Twycross]

FAT12 isn't that difficult to deal with, there are just a few functions where I care if its FAT12 or FAT16. Overall the difference is about 5 lines of code vs 10 lines and a couple of functions (which amount to another 12 lines). The code copes with either.

With FAT12, the FAT is actually 6 sectors, with FAT16 it'd be 9 sectors, another 1.5k.

Before I did the FAT, there was no file system overhead. The file was written directly into the flash (via CDC and S-Recs). So all of the file system was overhead and they were worried about it. My remit was to make the overhead as small as possible.

Besides, FAT12 is kind of nostalgic for me, when I started it was on DOS 2.0, which only did FAT12. The AT with DOS 3.0 and FAT16 had just been announced, but I never saw one.