I'm was able to figure out the issue on the linux side
0x75,0x08, // Report Size 1
0x95,0x20, // Report Count 1
My size Is apparently too big. If I change to 0x95,0x05, its fine.
So I guess I need to keep things as two devices and fix this windows bug. I tried to capture this in an image. Moving that section out of the main usage creates duplicate controllers.