Battery Charging Specification, Revision 1.2 says:
1.4.13 Standard Downstream Port
In this specification, a Standard Downstream Port (SDP) refers to a downstream port on a device that
complies with the USB 2.0 definition of a host or hub. An SDP expects a downstream device with a good
battery to draw less than 2.5 mA average when unconnected or suspended, up to 100 mA maximum when
connected and not configured and not suspended, and up to 500 mA maximum if so configured and not
suspended.
There is no mention of averaging over time.
Yes, it appears to say that if your charger uses a USB connection to a host system to charge its batteries and if you want the device to be certified by USB-IF, the charger should enumerate when attached to the host system. I don't know how many portable batteries comply with this requirement.
If you don't attempt certification, drawing >100mA briefly might not be an issue either though some SDPs are required to provide only 100mA.