How Many Speakers Are Too Many?