Maxing out on USB devices on MAC

[Renate]

I ran into this problem with having too many devices on a Mac, specifically a Mac Mini.
I've seen this mentioned elsewhere without a solution or an explanation.
This happens when I connect over 16 to 20 devices, depending on exact topology.
It refuses to enumerate any additional devices.
The error in the system.log is "the USB hardware ran out of device slots"

Yes, I know that this is a bit unusual usage, but I inherited this.
We're trying to run 28 devices on 4 USB 3.0 7 port hubs plugged into the 4 USB 3.0 ports on a Mac Mini.
I can connect all 4 hubs into a USB 2.0 hub and plug this single cable into the Mac Mini with the same results.
I can also plug this single cable into a MacBook Pro and it will also only enumerate 20 of the 28 devices.
(The error in the system log shows failing enumeration but doesn't mention "device slots".)
I can plug this single cable through an OTG adaptor into an Android and it successfully enumerates all 28 devices.

Experimentation shows me that a keyboard or a mouse counts the same as a USB 3.0 device in this limit.
The hubs are all powered and have port indicator LEDs.
This does not appear to be either a power limit or a 127 address error.

My next experiment would be to try a Thunderbolt to USB adapter and see if devices on that contribute to the limit.

Would anybody have any idea if this limit is in the driver, the firmware or the hardware?

[Jan Axelson]

I am not a Mac expert but am hoping that Barry will weigh in on this. In the meantime:

One reason for failed enumeration is lack of bandwidth. If the devices use interrrupt or isochronous transfers and attempt to reserve more bandwidth than is available, some devices won't enumerate. However, the fact that you can enumerate all of the devices on an Android host suggests it's probably not that.

Yes, each USB keyboard and mouse, which may be hard-wired, is a USB device, as is each physical hub. In fact, one physical hub might contain multiple hub devices. However, the theoretical maximum number of devices on a bus is 127, and if bandwidth isn't a problem, I would expect a host to be able to enumerate 30-40 attached devices.

A protocol analyzer might give more detail about where and why the enumeration is failing.

Is the limit always 20 even if you change the device types?

Have you tried tiering the hubs for example, using a tier of 5 hubs with each hub except the last connecting to 6 devices and a downstream hub? (The last hub will connect to the last 4 devices of the 28.) Even if that is not feasible for what you need to do, it could offer a clue.

[Renate]

Ah, I think that I've found the answer (and it's a big headache).

https://communities.intel.com/thread/52417
https://embedded.communities.intel.com/thread/7500

Intel XHCI controllers drivers can only support 96 endpoints.

If we count each of my devices as 3 (bulk in/out + control) * 20 = 60
Throw in the hubs, 4 x 7 port = 8 "hublets" + 1 x 4 port = 9 "hublets" * 4? = 36?

In any case, I'll try swapping out a device with only one interface with 2 EP for something complicated and see how that affects device limit.
If the device limit drops, then I'll know that it's actually an endpoint limit.

There is also the suggestion of switching the driver to EHCI for a twofold increase of endpoint limit at the sacrifice of SS to HS speed.

[Jan Axelson]

Oh my. This is a real restriction and makes the case for having multiple host controllers in PCs.

Unfortunately, the USB term "endpoint" is often used in different ways. From the math, it appears that 96 endpoints means 96 endpoint addresses, where for example, endpoint 1 IN and endpoint 1 OUT are each considered an endpoint, with the exception that bidirectional endpoint 0 is considered as one endpoint.

It's likely that your 7-port hubs actually each contain 2 hub devices, and each hub has a control endpoint and an interrupt endpoint. So that is 8 x 2 or 16 endpoints.

If you add the 4 ports on the root hub (assuming these count as part of the 96), that's another 8 endpoints.

The system probably has a USB keyboard and pointing device with 3 endpoints each.

That leaves 96 - 30 = 66 endpoints for devices.

If your devices each have 2 bulk endpoints, you would be able to enumerate 22 devices, which is close to the limit you're seeing.

Thank you for bringing this to our attention!




 

[Renate]

My totals were about right, but my math was awful.

The devices are iPhones, they have 4 endpoints each.
x 28 = 112 EPs.
There are 4 hubs, each with 2 EPs/half hub = 16 EPs
Total = 128 EPs

Of course that was my problem, 8 of the devices didn't work.
128 - (8 * 4) = 96

On a more amusing note, to test the limits of all this I tried to see how many EPs I could throw on a single bus and enumerate with other controllers.
It was interesting to try to figure out what was the cheapest price per EP.
For instance, I have a tiny Tenda USB WiFi adapter. It has 8 EPs but it costs around $8.
A cheap-o USB flash drive might be $3 but would only give you 3 EPs.

I tried programming an Atmel 32u4 to have lots of EPs.
Most things didn't seem to like interfaces with over 8 EPs.
I finally settled on 3 interfaces, 5 EPs each + control EP for 16 EPs total.
I'm not sure where the limits are.
In any case, with the 32u4 boards costing $16 I'm back to $1/EP!

I could get an Android host to enumerate the above 28 iPhones + hubs + 32u4 = 144 EPs.

Oh, I went and bought a Kanex Thunderbolt to USB 3.0 adapter and it solved my original problem.
I'd still be interested if there were an easy fix to switch the driver on the internal Intel controller to EHCI from XHCI.

[Barry Twycross]

I'm surprised it says ran out of device slots. As you've found the Intel XHCI has some limitations in hardware. There is a limit on the number of devices which can be enumerated at once. There is also a limit on the number of endpoints. I'd expect a different error between running out of endpoints and running out of devices. I wrote that code originally, its always possible I got it wrong, or someone else in the meantime.

The limits on the first generation XHCI hardware were considered severe enough that the EHCI controller was used for high speed traffic. There was some fancy multiplexing developed to swap devices between controllers. For the next generation of hardware, the limits were deemed sufficient to do without the EHCI and use just the XHCI.

You can tell which version of XHCI you have by looking at the USB in the System Profiler report. If it just says:

USB 3.0 High Speed Bus
USB 3.0 SuperSpeed Bus

You don't have the EHCI support. If it also says:

USB Hi-Speed Bus

You have EHCI support. If you have EHCI support, you can attach more devices to the EHCI by using high speed hubs, ones which are not superspeed capable. A superspeed hub as the high speed part attached to XHCI.

For XHCI, don't forget endpoint zero in your count of endpoints. I think it takes only one endpoint slot. The iPhone uses an unusually large number of endpoints. There are even more endpoints if the Phones have pictures taken on them, or if the hotspot is active.

[Jan Axelson]

Barry, very interesting!

Renate, Thunderbolt to USB as a solution, very creative!

[Renate]

Thanks Jan & Barry for your help.

What you pointed out about the iPhones, I hadn't considered.
I've never run much into (USB) configurations, they are mostly just 1.
The iPhone I was testing with has 4 configurations, varying between 3, 4, 6 endpoints.

Android devices seem to always have just one configuration (at a time).
The single configuration is different if the Android is in charge only (w/ADB), MTP or PTP.

[Barry Twycross]

Configurations are unusual as they're difficult to manage on Windows. The Mac APIs made using configurations a lot easier.

The iPhone configurations started as the powers that be wanted a good out of box experience. They wanted the user to be able to plug in a phone to a Windows machine which had no software installed on it, and have no annoying dialogs pop up. The idea was to have a default configuration which had in box drivers. In particular PTP.

That required a Windows driver to switch configurations for other interfaces to be used. It just so happened we'd just hired the guy who had implemented Windows USB. He managed to volunteer to write the Windows driver to manage the configurations. After the experience of writing the driver, he decided never to volunteer again. It was however very useful to be able to ship multiple configuration devices on all supported platforms. We used that on the iPods to support multiple power configurations.