Why Rankings – Including University Rankings – Are Crap

Every year, we make the same request of survey participants: Please tell us which schools you think are doing the best work in the blockchain space. And every year, the results indicate that everyone is simply sharing the schools that, if they were to be ranked highly, would most benefit the individual responder. Of course, I’m editorializing quite a bit, but the overwhelming majority of responses that we get follow the highly predictable pattern of single-school answers, clumped together by school, within 0-5 hours after a school tweets about our survey. There’s some amount of logic to the idea that students/faculty/alums think their school is doing great work (perhaps that’s why they chose that institution), but the shocking number of responses we get following this pattern, occasionally even suggesting that a school that has no discernable blockchain activity is the single best school in the space, implies to me that perhaps we are not incentivizing the right thing. In the future, I hope to experiment with other mechanics (Keynesian beauty contests, etc.), but for now, I consider this data to be highly skewed. What we’ve incentivized to date seems to be a popularity contest, not objective analysis.