Mass Storage Write (10) hangs at very last EP OUT data transfer

[bazz]

ARG! I'm implementing a device driver for the mass storage class on the Philips/NXP PDIUSBD12.. I'm hacking an old device.
It's been very challenging reverse engineering the device, the PDIUSBD12 is very picky.

I got to a point now where I can print to an LCD screen debug info and .. the USB device enumerates and begins communicating the standard basic commands READ INQUIRY, READ CAPACITY.. then it does a lot of READ (10)'s... then it starts heartbeating the TEST UNIT READY...

I'm on OSX Mavericks, and at this point, since I am pointing my LBA's directly to empty RAM.. OSX says "Do you want to initialize this drive" type message, and I say yes, and when I try to erase the drive as FAT32.. that's when I start to see the WRITE (10).. it tries to erase 0x100 512 byte sectors.. Since the PDIUSBD12 and I believe any other bulk EP max in this scenario is 64 bytes.. that's my max.. so I build up my sector thru many of these interrupts.. and after every sector I of course write it to memory, then repeat for all logical blocks.. anyways.. for some reason this will hang after the 7th transfer of the last logical block.. or more simply it doesn' transfer the last 64 byte block to me...

Now you might be wondering like.. maybe my logic was messed up and I was missing the first transfer and I was off... well.. perhaps not -- even if I consider this missing 64th block as if it was transferred, and send the CSW.. I get nothing... Even weirder is that if I stuff more data into the bulk IN buffer (like 1-2 more times) to the OSX host at the point of the missing last 64 bytes.. I will get a packet that is a bunch of 0's, definitely not a CBW.. and likely more erase data.. but if I follow the rabbit's hole any deeper I start getting REQUEST SENSE which my handler doesn't support the variable descriptor it seems to want...

So.. I'm at a total loss here... Any ideas anyone??

How could the host possibly want IN data at that point? What am I doing wrong 

I dont have access to a USB buss sniffer.. and I'm not very fortunate enough to have money lying around.. I would buy the popular one for $300-ish dollars Sayonara tax return...

[Jan Axelson]

The best starting point for mass storage may be to capture traffic from a working device and try to emulate it as closely as possible. This doesn't free you from the obligation to comply fully with the specs, but it's a place to start.

[bazz]

Does this look legit.. So the price was so freakin' attractive, and I read the reviews... albeit not very many (20+), 1 year long membership-ish, I went for it.. http://www.ebay.com/itm/New-Beagle-USB-12-Protocol-Analyzer-TOTALPHASE-/221721890486?pt=LH_DefaultDomain_2&hash=item339fa78eb6

I think it will be the real thing **fingers crossed**
I feel like the timing was just right... This auction was just put up and I found it with only hours to go and no bids.. I then bought it just at the 30 second mark and one person just had bidded on it.. Whew!

So I have it, theoretically speaking.. You might wonder why I bought this over using software like Wireshark.. Namely this feels easier and I can get at the core. Setting up Wireshark with OS X .. mystery.. And I need a break from USB for some days now.. Worked too hard one night you know how that goes.

I did manage to host a wireshark session on Ubuntu on my Macbook Pro, but it enumerates differently, so actually my device, because of not supporting the MODE SENSE command, never shows up as a drive or undergoes any READ commands, for instance... After I stall MODE SENSE, it's just a rinse repeat to that stage forever..

It's tough to completely mimick a USB flash drive since my device enumerates will all of the logical blocks as 0's, I'm not sure if I can get a flash drive to do that even by erasing the partition to free space.. I didn't notice the wireshark session actually write all 0 to the drive.. but anyways..I think I might buckle down and write my own FAT12/16/32 partition thingy.. but I'm a little lost, I'll try including a MBR in there..

Should I write a "FAT partition image generator" -- would anyone else find that useful??
  . specify SECTOR SIZE
  . specify 'drive' size
  . include MBR?
  . Create FAT image.. ??

Is there something like this already available?

[bazz]

Jan I'm just at that place... My code base is turning into a mess from all the experimentation.. I'm losing the organization I once had in trying to comply with the specs in the context of how the USB controller chip operates.. It's been really tough for me to put all the ducks in a row, so to speak.

If I know what's good for me, I'll go back a couple commits in the code base when things were more logical as far as sticking to SPEC.. and restart from there..

Once I get a scrollable debug screen and/or USB analyzer I should be able to make sense of my dilemma, and hopefully be able to compare to a working model.

[Jan Axelson]

The Total Phase analyzer should help. It can show low-level things a software analyzer can't.

[bazz]

Below, I have laid out a plan to communicate my debug data over separate bulk endpoints, but I must call upon your experience.
Can I do this while my device's USB interface is set to Mass Storage Device? With your experience, can you anticipate whether I will run into problems communicating to both the host's MSD driver and a host PC program at the same time [over different endpoints]?

My USB Controller (pdiusbd12) has another 2 endpoints I can use [and also the option of control EP vendor commands]. I was thinking of adding 2 more bulk endpoint descriptors, which I am planning to utilize during the mass storage operations.. I would forward all debug log messages to the endpoints via a PC program which regularly queries the device for debug information with 2 simple commands.

But, can a USB device communicate with a pc program and the host's MSD driver at the same time? Or would I be building something that isn't designed to work?

such a mechanism could go as follows:
There are 2 command --

Code: [Select]

#define CMD_QUERY_DATA_AMT 1
#define CMD_FETCH_DATA 2
 -- each command is sent over the special BULK OUT EP, followed by 1 IN transaction.
1) Host queries device for debug info by sending a CMD_QUERY_DATA_AMT over the BULK OUT EP. Can wrap this command inside a simple CBW-like structure for signature verification.
2) Device responds how much data it has available to send over the BULK IN EP. can wrap this USHORT value inside a simple CSW-like structure for signature verification.
3) Host, if device has data, sends a CMD_FETCH_DATA over BULK OUT EP to ask for that amount of data from device/
4) Device sends the amount of raw data requested over BULK IN EP, and updates its circular buffer marker.
5) Host prints debug info to console window

6) no matter what, at the end of the host loop, should delay some amount of time (10-100 ms) to avoid overwhelming the device.
7) goto 1.

---- what follows is
----- more personal data I don't expect ppl to reply on, but response is welcome --------
I need to find a good C implementation of circular buffer, any ideas? Perhaps it's time to make life easier in the world of CPP.

println() must be modified to utilize a circular buffer

in the println() -- if the new string size would cause the circular buffer to overflow upon itself:
1) [check if IRQ / INT_EINT0 is active .. ] disable INT_EINT0
2) enter a routine which polls the USB interrupt register for the special Bulk EP's we are concerned with
3) flush the buffer, and return
4) re-enable INT_EINT0
5) resume regular println() activity

[Jan Axelson]

If a device interface identifies itself as mass storage in the descriptors, the host computer will assign the mass-storage driver to the interface. On the PC, the interface appears as a drive, and host applications can use file functions to read and write to the drive.

If you can put your debug data into a file, a PC application will be able to read it using file functions.

To support a different mechanism for reading and writing data, the device will need another interface (HID, WinUSB, virtual serial port, etc), and the device firmware for that interface must have access to the debug data you want to send.

Or you could write a low-level mass-storage filter driver that supports new commands, but this option is more difficult and the filter driver would likely need to be digitally signed.

What is the debug information you want to request?

[bazz]

Well, I had low-level information regarding the implementation of my Mass storage driver printed out.. This could be anything from notification of which of the 13 cases I am currently experiencing.. to what SCSI command is being processed currently..

I was hoping that I could seamlessly have the USB process both kinds of data.. but as you said, it cannot be done without 2 interfaces.. There's no way I am going to select between 2 different interfaces constantly... So I just going to implement a more sophisticated debug log on the LCD screen, with a scroll feature.

[Barry Twycross]

I was going to say that you must have misplaced a packet somewhere, as my device does this with no hassle at all.

However, my device formats the media for the host. How big is your media? There are some sizes that are a little tricky for the host and I think I've seen initialization go wrong if the media is too small.

You really need an bus analyser to work out what's going on.

You might just be able to work out something is you used the logging USB family, or usbtracer. I think usbtracer in particular will tell you what transfers the host is lining up for the host controller. What controller is in you machine? What machine is it anyway?

[bazz]

I am on a Macbookpro 2009, OSX Mavericks 10.9.5
My drive is 8MB in size.

Do you have any tips on doing a format device-side (FAT) .. I should go FAT12 given my small capacity, right? I'm hoping you won't tell me I'll have to do it from scratch, I'm hoping there's a library. but I'll do what I'll have to.

Can I just be honest? The USB has beaten me.

[Barry Twycross]

I am on a Macbookpro 2009, OSX Mavericks 10.9.5

I'm pretty sure that'll be using the built in Intel UHCI.

My drive is 8MB in size.

That is a bit small, I don't think the partitioning software copes with it correctly. Mine's 4MB.

You could test this theory by lying and saying your media is larger. You can fake any requests to sectors past the end of your real media, but you should take notice of requests to the last couple of sectors in your virtual media, the system uses them to find out where the end of the disk really is.

Do you have any tips on doing a format device-side (FAT) .. I should go FAT12 given my small capacity, right? I'm hoping you won't tell me I'll have to do it from scratch, I'm hoping there's a library. but I'll do what I'll have to.

FAT12 should work. I wrote both in just in case I needed it.

I did it from scratch. I think I got the format from a disk image, OS X can successfully make a disk image that small. That and the Wikipedia article on FAT.

When the disc is that small you really don't want a partition table, just a naked volume. That works with OS X and most Windows before 7. Win 7 wants partitions, so I put in an extra sector in front with that. (Which may be called a VBR.) Then you need a sector for the MBR, which is a copy of the VBR without the partition table, then the FAT and the directory.

Can I just be honest? The USB has beaten me.

In this case, its not really USB, its the partitioning software, which is Core Media or something like that.

You might also get some traction if you posted to one of the Apple mailing lists. I'm not sure USB is the right place for it though.

[bazz]

Can you expound a little more on checking access to the last sectors of my "fake" virtual drive?
And do you just report the fake ones as being all zero?

If I manually preformat my memory as FAT, I can avert this problem altogether no? I mean avert having to fake the size of my disk..

[Barry Twycross]

Can you expound a little more on checking access to the last sectors of my "fake" virtual drive?

Just beware that the system will try accessing them, so requests shouldn't fail, at least reads.

And do you just report the fake ones as being all zero?

That would probably work.

If I manually preformat my memory as FAT, I can avert this problem altogether no?

Yes.

I mean avert having to fake the size of my disk..

I was suggesting faking the size of the disk would test whether the formatter is barfing on the small disk, or if you have a problem in your logic.

[bazz]

Ah, right. Thanks for straightening that out *feels stupid*

[bazz]

Made some progress today after a 5-day break period

this is a 2-part post, I'll post the latest news first!!
--------
I am using the Total Phase USB 12 USB Analyzer right now in this moment. It's worth the organization of the data, although I don't think a hardware level analyzer was necessary to spot this issue, I'm glad I have it/one. Trying to sift thru Apple's usbtracer command-line software looked like hell, and there's no manual I could find on it to make it usable. I hear it has limited output capabilities anyways.
----------- Part 1
Since I'm not yet well-versed in the Total Phase software, I did my analysis manually .. and since Total Phase doesn't provide Mass Storage "convenience" unless you fork lots of money for the higher-end models, I had to manually inspect the raw packet's CBW and CSW's .. It's not bad, given I'm already feeling the convenience from not having anything at all.. I'm kind of surprised they don't just provide the English translations.. but I guess they think that will motivate people to buy the higher end models... Dream on I say, at least for us hobbyists.. I'm ranting, so let me stay focused.

The problem lies in my driver, because the USB bus does indeed see the correct amount of packets. I'll take a look at that tomorrow when I'm feeling refreshed.

----------- Part 2

I [on a personal level, finally] learned how to incorporate on an initial level the FatFS library for embedded devices into my project. Fortunately, I was able to use the facilities of FatFS to format the drive. Yay, no need to create one absolutely manually!

I am able to successfully write a text file on the drive [with limited verification that this is operating 100% successfully, I am convinced].

Unfortunately, although the drive enumerates as before, and this time shows some updated information in Disk Utility (MBR partition scheme) for instance, I still run into that prior issue that spawned this whole thread -- the last transfer. See part 1 above in this post for updated details.