What criteria should you use to judge a hackathon?

I was on the jury panel at the Angel Hack event over the weekend with others.  Over 150 attendees were at the event, and 50+ hacks were presented on the final day (Sunday). They ranged from the sublime to the trivial. The best part was there were attendees from over 10 different cities including a few that came from over 1000 kms away. Each team was given 2 minutes to present their hack and 1 min to answer questions.

The first  thing that struck me was most of the attendees were awake to present their hacks. In previous hackathons most of the presenters have been rather tired or sleepy so they tended to gloss over their work.

This is the 5th hackathon I have judged and I dont think I have a clear idea on what the criteria should be to judge a hackathon.

This time the winner was a product that’s been in the works for a few months, and the developers made some changes / modifications to their product over the weekend. So, really it was not a “new” hack over the weekend, but something they have been working on for a while.  The runners up (not announced) was a company that’s been in the works for a while. They were well thought-out ideas, fleshed-out products and good implementations.

That obviously ticked off a few developers who had built a new hack from scratch over the weekend (and it showed that their idea was a one weekend project), and I got 3-4 angry emails on why we chose to declare the mature product as a winner.

Did we know that the winners were “mature” and not “weekend hacks”? – we did and did not. Did, because we could make out that the products were well thought out, which is hard to do in one weekend. Did not, because we were not told that we had to only look at weekend hacks.

So what does a weekend hackathon really accomplish?

I think it provides an ability for developers to learn something new, try an idea and experiment. That’s it. Globally, according to Startup Weekend, fewer than 2% of these weekend hacks actually turn into a company, but many (dont know the %) of the developers get hired because of these events, many ideas are added to an existing product and many products are enhanced post the hackathon.

There will always be folks that keep working on their idea over several hackathons so their ideas will mature quite a bit and so will their products. The good part of this hackathon was I did not see a single team that had presented the product / idea before at any of the other hackathons. There were many rehashed ideas, but largely new teams.

I think the top 3 criteria for judging hackathons should be a) how unique & interesting is the idea given the constraints of the hackathon, b) how close to “product” has the hack been over the weekend and c) how creative have the developers been in their implementation

I think the key thing that hackathon organizers should do is to form a jury of 3 hackers / developers and maybe 1-2 other folks from the startup world (VC’s or generalists like me).

Our panel on Sunday was comprised of 1 designer and 1 developer. The rest were generalists (3). So it was obvious that we were going to be biased and look for how “big” the idea was, how well thought out the implementation was etc. If the goal of the hackathon was to look to turn weekend ideas into startups, then an even mix of generalists and hackers as jury members would make sense, else they should be weighted towards developers as jury panelists.

Do you think we should even have generalists as jury members? I think that 1 might be sufficient for most parts, but if they are not developers, what’s the point of having them on the  jury?

3 thoughts on “What criteria should you use to judge a hackathon?”

  1. Great post, I don’t think there should be any argument with judges on their decision, since there decision must be final. The judges were highly qualified and gave everyone excellent feedback.
    The only thing I felt is that the uniqueness of the app and the coding time spent should have been considered for a fairer experience.
    But none the less, thank you all for your valuable time, it was a real learning experience.

  2. It’s natural for people to be angry. I could potentially win this event without even coding a single line of code using the criterion that you have defined. Which I believe defeats the purpose of a hackathon. If you want to see well thought of products (and NOT hacks), then don’t think of calling it a hackathon, otherwise people are misled and you will be barraged with emails, no?

  3. Judges should have definetly seen the uniqueness of the project. The winning app at AngelHack Bangalore was no unique, there 100 of apps on the app market and the store that can do the same and they are way better. AngelHack is about converting your hack into a startup and get funding for the same. I don’t understand how is the winning app going to convert it into a startup, when there are 100’s of similar apps in the market. Apart from that there should have been a final round where the hacks could have been judges better.

Comments are closed.