Oligarchy

This week I received an email from moveon.org claiming that Pat Robertson said on ABC's "This Week" that the threat posed by liberal judges is "probably more serious than a few bearded terrorists who fly into buildings."
Pat Robertson replied to that on his website:
http://www.patrobertson.com/PressReleases/moveon.asp
In his defense, Robertson says he made this statement, "if we surrender our democracy to the tyranny of an oligarchy, we have made a terrible mistake."
From Wikipedia online:
Oligarchy is a form of government where most political power effectively rests with a small segment of society (typically the most powerful, whether by wealth, military strength, ruthlessness, or political influence). The word oligarchy is from the Greek for "few" and "rule".
http://en.wikipedia.org/wiki/Oligarchy
If one were to try, one might find it quite easy to see that the concentration of power going on now in Washington might be what Jefferson warned against. But I don't think that is what Robertson was suggesting. Is Robertson suggesting that Supreme Court justices are the oligarchy? If not, then who was he referring to?