Headset volume (Geek Warning)
It might not just be ambient noise.
Have telecoms check that they are not all on the same frame card.
When the switch is fully loaded it may have a lower gain on one card than the others causing (at peak call volumes both IB and OB) a drop in signal level across the whole card.
Do you use Mono or Bi Aural headsets because the next step (after turning the volume up on the turrets and checking the card) is to trial Bi Aural headsets.
The only way to check the headset turret itself I know would be to put a scope inline with a 1Khz test tone running down it then measure the signal level. I think you'll be looking at something like 20Mv on the peak of the Sine wave.
I'm not going to go too far into Audio theory here so if you want to drop me a line off list there are a couple of other options.
If anything works I'll then write up a post here.