A hardware protocol analyzer will show what is happening on the bus. If you don't have a hardware analyzer, use whatever debugging tools you have (monitor program, breakpoints, etc.) to find out what the device is doing and to isolate the problem.
For example, after the reset, does the host send a Get Port Status request, and if so, does the device ACK the Setup transaction and eventually respond to say that the reset is complete?
If yes, does the host send a Get Descriptor request, and if so, does the device respond by sending the number of bytes requested or the full descriptor, whichever is less?
And so on.
Once you know where the device is failing to respond appropriately, examine your firmware to see where it is going wrong. The firmware specifics will vary with the hardware and firmware.
One thing that appears to be different between the failed reset and the reset button is that the failed reset involves the Suspend state, so you might look there.
