I can see why one could argue against using AWS in a generic situation, but what specifically makes it bad for voice/media servers?

Voice Calls which heavily depends on Kernel timing for encoding and decoding. Running a OS timer on a virtual machine will mess up things 1. When higher load is pumped 2. Conference audio mixing

Hence AWS or any other Virtual machine wasn't our first choice for Voice/Media Servers.

Yeah, that's a major issue with more loaded machines, especially with older/worse virtualization technologies. It seems tolerable on Twilio, and I've run asterisk and freeswitch on virtualized machines (with HVM) which work ok.

My reason for not wanting VoIP gateway on AWS is that I want to use it for AWS-hosted-app error reporting, so having dependencies on AWS is a problem.

Things word good when the throttle is not pushed on the VM. Also try our conference calls, you will see the difference :)