MacBook OSX 10.9 USB subsystem crashing

[Renate]

I'm using a setup where an OTG device-under-test is being switched through relays from host to peripheral and back.
The device is an Android Nexus 9. It stands up to this just fine.
The problem is that I can crash the entire USB subsystem on my Mac by playing with this.
To the Mac, it looks like I am disconnecting and reconnecting a peripheral.
(The host path out of the Android does not go to the Mac.)

I'm not really beating on the Mac.
I'm basically connecting a peripheral for 5 seconds, then disconnecting it for 5 seconds, repeat.
If I do this about 5 times it will crash the entire USB system on the Mac, including the built-in keyboard.
I get a black screen with a movable mouse.

(The Android takes a lot more abuse, switching between peripheral and host every 5 seconds.
It doesn't have any problems.)

Are Mac USB stacks known to be problematic?
Does Yosemite (I don't have it yet) fix this?

I'll have to diagram this out.

[Jan Axelson]

Do you see the same thing if you physically detach and attach devices on the same port?

I don't think there is any USB specification that mandates performance for continual detach/attach. I don't know if anyone tests for it, or if they do, what amount of time they consider reasonable.

Five seconds to detach and attach a new device once is normal, but doing so repeatedly wouldn't be typical use.

Are you doing this just as a test, or is this something your device would do in normal use, and if so, why?

[Renate]

I'm actually not trying to beat on the Mac.
Using a PC it mostly works fine.
I haven't really tried physically disconnection it repeatedly.
In essence, this whole project is avoiding that.

I'm using a relay matrix to switch the direction of data on USB OTG.
It switches the data pair and the Id pin from ground to open.
In normal operation, it only switches whenever you have a need, maybe every few minutes.

I'm just saying that when I ran into the problem I tried switching it at 5 second intervals.
You'd think the tricky part would be on the Android side.
I give it 500 ms of "dead time", "break-before-make".
It doesn't mind this in the least.
It works fine until the Mac locks up.

I just wanted to know if the Mac USB stack has a bad reputation.
I'd do a dmesg or something, but when it crashes it is unaccessible.

Still, if I were writing a stack, beating on it would be a good test.

[Barry Twycross]

I'm not surprised. Relays tend to be very noisy, you may well be producing more plug/unplugs that you expect. Fast plug/unplug cycles are a problem for most Mac OS versions.

There are solid state switched which do a much better job of switching without the noise.

[Renate]

I'm betting that the little reed relays I'm using have less bounce than a person haphazardly inserting a USB plug.
Some of the drivers that I've looked at have 30 mS settling time for just this reason.

In any case, no matter how mean I am being to the Mac, why should the whole computer lock up?
My PC seems to handle this without a problem.

Fast plug/unplug cycles are a problem for most Mac OS versions.

Oh, well that's the answer I wanted.
I'll have to see if a $25 Raspberry Pi is more resilient than my over-priced (company-supplied) MacBook.

[Barry Twycross]

I'm betting that the little reed relays I'm using have less bounce than a person haphazardly inserting a USB plug.
Some of the drivers that I've looked at have 30 mS settling time for just this reason.

In any case, no matter how mean I am being to the Mac, why should the whole computer lock up?
My PC seems to handle this without a problem.

Fast plug/unplug cycles are a problem for most Mac OS versions.

Oh, well that's the answer I wanted.
I'll have to see if a $25 Raspberry Pi is more resilient than my over-priced (company-supplied) MacBook.

If you actually want to know the answer, and not just reinforce your prejudices, its because fast plug was never a user scenario.

When the first Mac OS USB was developed, QA did produce a lot of bugs which came down to fast plugging. As we had better things to do and it wasn't an realistic user case, those bugs were put aside to fix actual user issues. When things quietened down sufficiently, I spent about 6 months hardening the USB stack against fast plugging, it was quite an effort but the stack became quite robust.

Then things were ported to OS X, and I stopped working on the implementation. Sometime later I did work on the OS X implementation but QA wasn't generating those bugs anymore, so there was no particular reason to fix anything. There were plenty of other things to do.

As for structural differences as to why Mac OS is more vulnerable to fast plugging, Windows P&P manager is very single threaded. Everything happens sequentially, this makes life quite simple for the driver, it also means that it takes a while before a device is available to the user. Mac OS has always used a multithreaded approach, each device is enumerated in parallel with every other device. This makes enumeration for the user much faster. It also means that of you do do fast plugs you can have multiple instances of the same device running though the same code at once. It makes for some very amusing problems in concurrency (aka "asynchronous funnies" as it was called at one place I worked).

The biggest driving force in debugging fast plugging was when Steve was using a mechanical switched based KVM switch for his "Stevenotes". Its well know that he had multiple hot spares of his demo machines in case one crashed on stage. With the wrong switch he could switch to a hot spare only to have it crash as well. That got quite a lot of issues cleaned up before better KVM switches were found for use on stage.

Also, a settling time of 30ms sounds just about perfect to cause maximum havoc. USB has several characteristic times of around 10s of ms, like 20ms debounce, 10 or 50ms resets. 30ms will put a disconnect right in a critical part of the enumeration state machine.

Its not my problem anymore though.

[Jan Axelson]

Agree with Barry that continual, fast detach/reattach isn't "normal" user behavior, and if you need to do multiple detach/reattach for testing or whatever, the best solution is probably to increase the cycle time.

[Renate]

Well, this problem was detected switching every 60 seconds.
I don't think that that can be considered "fast".
The bit about every 5 seconds was trying conservatively to replicate the problem.

Contact bounce on mechanical insertion is a serious situation that has to be addressed by anybody writing device drivers.
Contact bounce on reed relays is in the single millisecond range.

Your anecdotes were interesting.

[Renate]

Just a postscript on this.

Although I had tried this setup both connecting through the Thunderbolt display and connecting directly to the laptop,
I had not tried without the Thunderbolt display connected at all.
This seems to work quite reliably.
Unfortunately, I usually have to run this with the Thunderbolt display connected.

USB through the Thunderbolt display seems to also have its own idiosyncrasies.

[Bret]

Based on Barry's experiences when he was working at Apple, and your particular experiences as well, it sounds like Apple probably does have some USB stack issues.  It also sounds like it could be more of an underlying architecture issue surrounding real-time tasks/threads than a pure USB issue, though.  Hard to say.  I know Windows has problems with their real-time device handling (mice and keyboards) as well, though at least in may case it's usually that the mouse and keyboard stop responding for long periods of time (sometimes several minutes) rather than the entire OS locking up and needing to reboot (and I should also point out that it usually has nothing to do with plugging and unplugging anything -- it just does it randomly for no apparent reason).

In any case, I can certainly understand how disconnect/reconnect times in the sub-100ms range could cause problems inside a device, but it really shouldn't cause problems in the software drivers.  Anything slower than about 100 milliseconds (and certainly 5 or 60 seconds) shouldn't even be a concern.

[Barry Twycross]

Just a postscript on this.

Although I had tried this setup both connecting through the Thunderbolt display and connecting directly to the laptop,
I had not tried without the Thunderbolt display connected at all.
This seems to work quite reliably.
Unfortunately, I usually have to run this with the Thunderbolt display connected.

Am I understanding this right that connecting the Thunderbolt display causes the issue to happen, even if the USB connection is via port on the laptop (nothing to do with the display)? That would be very strange.

USB through the Thunderbolt display seems to also have its own idiosyncrasies.

You could say that. If its the one with the EHCI in it, its probably the only Thunderbolt connected EHCI that will ever be supported. The latencies introduced by Thunderbolt really don't play well with EHCI architecture, which is very dependent on host memory latency. XHCI architecture was designed from the ground up to cope with such latencies, and makes a much better controller for use in a Thunderbolt device.

[Tsuneo]

The device is an Android Nexus 9.
...
Well, this problem was detected switching every 60 seconds.

I believe a problem should lie in the Android side, too, not just on the Mac side.
I found very short "detach" interval on Nexus7 (2013, LTE, Android 4.4.4), when it swaps its USB configuration between camera (PTP) and media device (MTP). On the swap, Nexus7 soft-detaches USB lines first, and after around 100 us, it soft-attaches in the new configuration.

This short detach may occur on your test.
Get a clue on a hardware bus analyzer.

Tsuneo