Both speakers see the same mono output from the amp - so that suggests the problem is associated with the speaker - or the wiring to the speaker.
Have you checked and reflowed the solder joints to this speaker?
Have you gently jiggled the wires to see if something is not firmly connected?
Can you easily remove the speaker to see that the cone is not damaged?
And that there is no dirt trapped against the cone.
It is also possible that the distortion relates to either the cone sagging on its suspension so that the voice coil drags in the gap, or the frame of the speaker being slightly distorted by uneven tightening of the four screws, which also can cause the voice coil to drag on the gap. Opening and closing the lid of the unit while you work on it may be producing the variability you're observing. It's possible that distortion of the frame can be fixed by loosening off the speaker and carefully re-tightening it. If it's sag from aging, you may get temporary relief by rotating the speaker 180 degrees when you replace it.