This has happened to me a few times too-- just yesterday in fact. It's only happened when I've put them on speaker phone, but same thing as you-- I can hear them, but they can't hear me at all. I'll take it off speakerphone and they still can't hear me. It's only when I call them back without putting them on speaker that all is good.
ETA: I'm on a 4S