To Sergey (who asked) and anyone who wants my final thoughts on this matter:
Unfortunately, we never did 'solve' the problem other than to have the whole device power cycled. Long story, but given the our relationship with client and other vendors, best we could do in the time allotted.
The real problem seems to lie in the USB spec wherein the dataline must be 'quiet' for 2.5us to 2000us to detect the reset, but data can continue to fly on the datalines (to a different device on same hub) much more frequently than 2 seconds -- so the 'quiet time' may never be seen.
All the components (hub, target device, other hub device) are conforming to spec, it is the spec that fails. (Kinda like the 'tiny j' phenomena)
In our case, we had a second device on the hub which was driven by a USB-Serial driver in Linux. The driver (very inefficiently) was pinging the second device at a high rate (~20us) asking "got anything" to which the 2nd device responded "no". Meanwhile, the USB hub was waiting for 100uS of 'quiet' (meeting USB spec) and could not see it due to the ping rate of the second device. (if 2nd device was removed, everything worked perfectly.)
Fixes (in order of elegance and reverse order of practicality)
1) USB spec changes to declare by fiat that the 'quiet time' to detect a reset request is X, such that X is faster than space between potential communication packets.
2) The linux driver be changed to slow down the ping rate. Even a ping rate of 250ms would have been sufficient. (Vendor was unwilling to change the driver)
3) Force hub to operate at USB 1.1 instead of 2.0 (required an external EEPROM which was not on PCB)
4) Have linux box suspend pings for second device at critical time (linux box software vendor unwilling to change code)
5) Have linux box reset the power to the specific USB channel when stuck (stuck time was predictable but linux box software vendor unwilling to change code)
6) Have end customer power cycle entire device when USB stuck (stuck time was predictable. *sigh*)
That is the whole nut in a thingshell.
- Beethoven