So I tracked my Mac problem (from the thread about the Mac kernel trace) down to this.
Apparently Windows XP (as well as an older version of Linux and perhaps other OS's) has a work around just for my device. After Set Configuration it does Clear Endpoint Halt. After this, my device works perfectly, including Windows Qual.
But, surprise! This doesn't happen on a new Linux and Max OS X 10.6.5. In this case, my device just goes away. According to my Beagle trace, the INs are "ORPHANED." After a thousand or so retries, the OS gives up and retries a reset, which always turns out the same.
I dug into the USB library code and added code to poke the Endpoint Mode to NAK, which did cause NAKs to occur, instead of the "ORPHANED" transfers. (Didn't make it work, yet, though.)
It is a HID mouse, full speed, mostly "normal" device and configs--single IN endpoint, somewhat long report desciptor and reports 13 bytes.
Does this ring any bells with anyone?
Someone must know something about this because the Clear Endpoint Halt is there. There is no reason I know of for it to be there other than as a workaround for a broken device. We even spellunked around in the Linux source and found it there--but only in an old version.
I could post the Beagle traces, desc's, code, or whatever, although I think I've got the gist of it here.
Obviously I have a support call in with my chip vendor, but last time it took two weeks for me to figure out and explain the problem to the tech support person well enough for him to say, yeh, that is a problem in the sample design and no, they aren't going to release a fix.