Reddit
How Judging Works
Ahoy there, thanks so much for joining in the 10th Anniversary Ludum Dare! I gotta say, it was an amazing experience to see all 1400+ entries being submitted to the site!
The judging is pretty much the same as always. You judge entries, people judge you.
The big change this year is that now those who judge more entries will get more people judging their entry. I recommend starting judging a few entries now and keep judging them throughout the next three weeks to ensure your entry gets a good handful of ratings. Try to judge AT LEAST 25 entries. Please note that stuffing fake ratings is 100% uncool.
Also, coolness has changed: 50 ratings gets you bronze. 75 gets you silver. 100 gets you gold. Above that, you’re just awesome 
Be sure to make a post about your favorite entries to highlight the best-of-the-best!
Lastly, if you attempt to play an entry, but can’t, please just leave a comment to the user stating that you can’t judge their entry and explain why. “I don’t have a Mac” or “I can’t install Java” or whatever are all fine reasons. It will help them know how to create an entry that even more people can play next time!
Cheers!
-Phil
Woooaa Gold coolness is easyyy
I was expecting to rate 200/300 games! Will there be some platinium medals ? ^^ Oh wait! No! Adamantium medals! No wait! Chocolate Medals!
Bacon medals.
That’s awesome! I expect to judge as many game as possible, but some of them are too good that you don’t want to play anything else!
Also, I think the new rule is good, less people will stuff fake ratings, I guess.
Question: When we attempt to play an entry but cant due to platform limitations and we comment do we hit save or post comment. I’ve been just doing post comment since it doesnt “pad” the coolness rating (unless that changed this year) but I would REALLY like to clear games I can’t play off my list and the only way to do that would be to hit “save”. By the end of the first 150 I rated I had to really hunt and peck around all the games I couldnt quite remember If i already tried or not since they pretty much take up my entire front page “random” 20 now.
Yeah, you should probably click “Save” so it’s cleared from your queue.
Also please always leave a comment if you rate a game. Even if it’s just to say ‘hi, I tried your game’ it’s always appreciated to see that people are actually playing one’s game.
I agree with this entirely. It’s really cool how many comments I have on my game; I appreciate all of them!
Absolutely, if there’s one thing I’ve learned about game developers, they always like to hear other people’s opinions and thoughts about their game, myself included.
I agree w/Raptor. It’s not clear where the “Comments” below the rating go. Maybe call this “Ratings Feedback” or move it beside the ratings or somehow indicate that it’s part of the “feedback card” we’re filling out, and it won’t post to the section marked “Comments” directly below the box?
Now that I’ve got some free time, I’m trying to rate as many games as I can this time around! Doing a blog post on the best games is a great idea
Thank you for this new system. It’s excellent. I’ve already got as many ratings as I did for the whole competition last time.
Since I’m on Linux, I want to focus on rating the Linux entries, to show them some love. I probably *could* get Windows entries working, but I know they have a much better chance of being rated than Linux entries. Should I leave a comment saying I’m focusing on Linux entries, or would that be “gaming the system”? Thanks!
Your best bet there would be to use the search feature and type in “Linux” and just focus on those.
Sounds good. I probably don’t need to tell every single person who didn’t make a Linux version that they might have gotten one more rating if they had. It’s not that big a deal for most people, I’m sure.
unfortunately almost 50% of the games returned by this are “linux version maybe coming soon”, “XNA4 only, sorry linux & mac users” and “might work under WINE for linux”. IMHO search box would work a lot better if we could restrict it to searching the links only and not the description.
You could use this page, and Ctrl+F to find entries with linux links:
http://www.ludumdare.com/compo/ludum-dare-23/?action=misc_links
unfortunately that page does not remove ones i’ve already rated or ones I could not run, so it’s not every useful for that.
I’m going with a compromise of leaving a “sorry I’m on Linux” post on the false positives to remove them from my queue, and telling them it was a false positive. Maybe that will encourage them to not use the word “Linux” in the future unless they’ve got a version for it.
This new system is great. It’s fantastic to see the reviews come in when I rate a ton of other entries
Great work!
Any way for people that worked in teams for the jam to judge entries? I’d like to contribute to the judging, but the submission is under my partner’s account.
Not at present. Maybe your teammate will let you borrow his password, if you both want to help judging. If not, definitely check out some games and leave comments anyways.
It’s so rewarding reading all the comments that others post about your game, especially since everyone seems so positive and nice. Also that encourages me to play more games too.
I feel obliged to at least play the games of the peopel that have commented and played my game, which is a good thing
This! I’m so happy to get comments! This is my first LD entry, and I could not imagine I would get this kind of attention and comments!
I also try to rate all the game that my commenters did. I can’t play the Windows games, but that’s a minority.
Wait, I don’t have to whore out my game on IRC?
It would be great to have an torrent with all games available for download. It’s so sad that some games from former LDs aren’t available anymore.
My slogan: “Save the Games”!
Thank you so much for re-vamping the voting system. I already have more ratings than I did all last competition.
Also, I noticed that there no longer is a community stat for voting. Is that being worked out some other way, or did you guys drop it?
Community was a stat for people that did well in blogging/doing ancillary things with their game. Many people were scoring highly in that category, without even a single post. Not to mention, scores in community seemed to parallel other category results, so since it wasn’t working we removed it.
It’s sad that the community rating had to go. I really enjoyed thinking about how I could become more entrenched within the community.
Can you please update the Rules and Guide? Community is still listed as a category.
So what happens if 1/3 of the comments on my entry are like “I don’t have an Android so I can’t play the game” and those people used the “save” button to do so?
Does it effect rankings if they didn’t post any scores?
I personally never leave comment for rank. I comment separately with buttom comment form.
>The big change this year is that now those who judge more entries will get more people judging their entry. I recommend starting judging a few entries now and keep judging them throughout the next three weeks to ensure your entry gets a good handful of ratings. Try to judge AT LEAST 25 entries. Please note that stuffing fake ratings is 100% uncool.
Oh, this is not so good for people with bad internet connection, but, well, I have plenty time to fix it, so I will judge games in a week or something.
25 ratings over 20 days isn’t a big deal, and most of the games are under 2-3 megs so even 100 ratings in 20 days shouldn’t be an issue. (actually, a LOT of games are even under 1 meg, especially the html5/flash ones).
If you’re rating games at all though you’ll be fairly visible, looks like still around 60-70% have rated between 0 and 1 games so rate 2 games and you’re more visible than 900 entries.
In the time it takes you to play a game and post a rating, you could probably have downloaded the next game already. But yeah it would be nice if people put the download size on the entry page.
This is a great improvement as it will make people vote more, good job!
dealsparadise.gr…
[...]Ludum Dare » Blog Archive » How Judging Works[...]…
Hi.
Suese here. Writer of Tiny Civilization http://www.ludumdare.com/compo/ludum-dare-23/?action=preview&uid=8361
After speaking with people on chat and gathering people’s opinions as well as my own, I want to suggest some ideas here, post-mortem style:
1.Need category “Presentation”: much akin to plating in cooking competitions, this component is of big importants. Games that present themselves well, run without hassle etc, should get some favourable points for that category stand-alone. Some games can be brilliant without presenting well, and vice versa, presenting well but lacking substance.
2.Need category “Replay value”: this is definitely the corner-stone of GREAT games, this seperates the fun and flashy from the GREATS.
3. Need category “Completeness”: Lots of games are amazing but too short, feeling kind of like the story line was cut-off. Other games you sit there playing in amazement, wonder how so content was put into it in just a weekend.
4. Ditch “overall”, overall score should be the sum of stars or the average number of stars per category.
I would not be worried about changing all of the categories at once. The rest of the community will enjoy the need to adapt.
So to re-cap the new themes would be:
Fun
Innovation
Presentation
Replay-Value
Graphics
Music & Audio
Completeness
Overall = (Fun + Innovation + Presentation + Graphics + Replay-Value + Audio + Completeness ) / 7
Next, I think the star ratings are arbitrary and should be presented in a questionaire format that defines what each star-rating means. For example:
Music & Audio :
0 stars – No music/audio.
1 star – competitor put a minimal effort into this and it is bad.
2 stars – the competitor obviously put some effort into it but it’s not very good.
3 stars – the competitor put effort into it and it’s decent.
4 stars – music and audio were fitting and added to the game experience.
5 stars – extreme: anything beyond.
I think most people thirst for this kind of upgrade to the challenge.
Finally, this is on a personal note, I can’t stress enough that the Theme system is too wide-open and it’s also too easy to cheat and make changes after the competition. I suggest a few solutions:
o – Along with the theme, a second randomly selected modifier, some kind of artistic or technical limitation. See http://speedhack.allegro.cc/ Rule-o-matic voting system for my FAVORITE example of how to vote on a randomly-selected theme. This lends itself to extremely intriguing and innovative possibilities such the Theme ‘summer’ mixed with the modifier ‘Your game must use Conway’s game of life as a key mechanic’, or ‘Dungeons’ mixed with “Your game may only use 4 colours maximum on a 128×128 pixel screen”. The possibilities are endless and it also further ensures that people can’t grab a head start on their game due to the added level of uncertainty.
o – Create an upload bin so entries cannot be tampered with after submission.
o – Embellish the culture of the theme a bit more, take a few days to prepare a presentation before revealing the final theme to the public. I didn’t realize ‘tiny circle worlds’ were a genre of video game until 5 hours of sleep AFTER the competition. When the competition begins, we should at least have some background and perhaps some visuals about the culture behind the theme.
THANK YOU.
I don’t agree with the added modifier, though that could just be that both of your examples give very limited games. How many different games can really be made from the game of life? I’ve seen a few and they are normally just simple puzzle games with a little variation. I think that would kill the creativity and the spirit of the contest. The same with the culture of the theme: some people took it as literal small planets, and some people took it as a small subsection of the present world. If you put out a video saying “this is what a tiny world is” you are limiting the creativity of the designers. True, there are going to be people that make games very vague so they can fit any theme, but it is a self correcting system. If you see that, you vote them down and move on.
I agree with your proposal for a standard for voting practices, but I like the vague-ness of the theme system because I think it creates a very diverse game set. I would hate to play 1400 of the same game because the system limited the designers creativity.
I’ve seen action games that incorporate the game of life and I’ve seen puzzlers. This is of coarse just a tiny example, if you look at how rule-o-matic works the community still votes on the modifiers and they are chosen at random from amongst the best.
The problem with JUST a theme in my opinion is that it’s JUST too easy to get started or have something half-started before-hand and just modify it last-minute to fit the theme.
p.s. Artistic limitations are a big part of what makes the challenge fun. 48 hours, stick to a theme. Yes limiting, but not challenging enough for the creative mind. If my idea is too far of a stretch maybe the modifier could be a bonus, meant for hard-cores who want an added challenge.
added creative challenge*
I think GGJ way would be better then just force modifier on player. Set several possible modifiers and let author choose which one (if any) would be used.
I’ve earned my bronze medal! Going for gold.
This system is great. It addresses most of my misgivings about the rating system and the sort order biases. Good work!
I have some complaints left but I’m not even going to say ‘em because this is so much better!
I think the D=R-C system has some problems.
The formula appears to be: D = 50 + R – 5*SQRT[C'] where C’=Coolness, capped at 100
The term 5*SQRT[C'] is what earns us *bonus* ratings above those who choose to rate no games at all.
But the problem is that for CoolnessC bonus rating in return. Think of the economics of that… those who rate between 1 and 25 games – even though they are giving something, they will be taking more back from the community than they have contributed.
One consequence is that there’s a high risk that people with zero coolness will receive an absurdly low number of ratings.
I think the formula should be adjusted to D = 50 + R – C’/2.
This way, for every C games we rate, will be earning C/2 bonus ratings in return: ie, for every two we rate, one we earn back, and one we don’t. And the one we don’t will benefit the community.
Bad formatting. That should read:
But the problem is that for Coolness less than 25, the number of games we rate (C) are earning us more than C bonus rating in return.
Yeah, I was thinking about writing something similar. While I do love the new voting system because it gets me more victims… errr, players, there are tons of games with just one rate. There’s still 16 days for judging, but unless a lot more people will start voting at least a bit, we might end with half of the games with just five or less votes.
Yes. Having some games with very many votes and some with very few votes isn’t a good outcome.
It’d be like baking a cake where you measure your sugar to the exact gram, but then adding a couple of crudely measured handfuls of flour.
You have to use the same precision with all of your ingredients, otherwise the outcome will be only as precise as the crudest measurement.
The outcome in our case being the final rankings table, and how well it has stabilised. No matter how thoroughly game “A” has been rated, it will mean nothing if game “B” has accrued only one or two ratings. How can you possibly compare those two games and decide which should be higher placed than the other?
Well I guess we could always return to previous system with truly random game list for rating with hope that enough people would rate….
Or don’t put games with less then say five rates in final ladder. Five votes for the game is bare minimum, I’d rather see at least ten for game for some statistically meaningful results…
Between the old random system and the current system, for a short time we sorted games by least ratings, which I think was the best of the three systems.
I understand how the current system aims to encourage more voting by rewarding those who do. It’s just that the reward is so big, that it is amplifying the problems it is intending to solve.
To address this, and building upon my previous formula, we could increase the priority of games with R less than 20:
D = 50 + R – C’/2
where
if R is less than 20, then P = (20-R) else P=0
Then let C’ = C + P, capped at 100.
Any game with <20 votes that's not already showing up on the first page is because they're not bothering to rate any other games themselves.
We have to accept that not everyone will bother to rate games. At the moment, those people’s games all but ignored with just 1 or 2 votes.
You must realise that ALL games still need to be fairly rated in order to build a credible ladder. The system I’ve outlined should achieve this better than the current system, while fairly rewarding those who are rating games.
I think that the equation:
D = 50 + R – min(C,100)/2
is adequate – when this is in a system, it will naturally bring up games with few ratings. Whenever someone rates a game in this system, their own D rating decreases by 1/2, but someone else’s D rating increases by 1. So, there is a net increase of 1/2 in the system. This means that, as time goes on, the total of everyone’s D ratings will increase, and thus, the minimum D rating should increase as well.
Say the average number of games that each person rates is 10. This would mean that the average D rating would be 55, and since the disparity in D ratings should be relatively small, the minimum D rating might be close to 55 as well. Then, all games would get a minimum of around 5 ratings.
More generally, we just need something of the form:
D = R – B(C)
Where R is the number of ratings, B is the “bonus function”, taking coolness and yielding the number of extra ratings a user gets. The expression C – B(C) is the number of votes that a user generated (C) minus the number of votes they consumed (B(C)) – it’s the net change in the system affected by each user. It should be positive so that people who rated nothing get to use the extra votes generated by the system.
I think it’s a good system.
Those who do rate games get rewarded, and like any good leveling system you get rewards quickly at first so you get a taste for it and then have to work a bit harder for the next ones.
I have no problems with people who don’t rate *at all* not getting much feedback. As you said even a few ratings will get you a fair number of ratings back. But not doing it at all means they are not interested in that aspect of ludum dare. It should be however made clear why they don’t get much feedback.
I think there is a minimum number of votes to be in a category, so they would be ‘unrated’. But that might not even happen since people can also look at the list sorted by absolute number of ratings received.
We can’t expect everyone to have the time to rate many games, but anyone should be able to test 10 games if they really want to.
I love the new system!
It is a little disconcerting to see my own game on my rating list. :/
Susan
I think that’s very useful. You know you have rated enough games to put your game back in the list. And when it’s no longer there after a few minutes, you want to rate more games so you can see it again. I think it creates positive feedback loop that makes me rate more games and should be left as a feature.
Maybe it could have a special border around it to show that this is done on purpose.
I agree there aught to be a way to check your own standings easier. I keep searching my game up just to see how many ratings have been made, it’s a bit tedious.
Bookmark this link
http://www.ludumdare.com/compo/ludum-dare-23/?sortby=default&q=suese
Hi, I just wanted to note that in the beginning I was a little bit confused by having two comment boxes just next to each other (the one for the vote and the general comment).
Maybe the “Save” button could be relabeled to “Save Vote” so that it would be obvious, that your vote will only be saved once you click that button and it’s a button to save a comment.
I miss the old list, where I could see how many votes the most popular games have…I mean, I got a lot of ratings and I’m thinking “wow, this is the most I ever got, I wonder how everybody else is doing”
http://www.ludumdare.com/compo/ludum-dare-23/?action=misc_links
Am I correct in guessing that there’s no way to sort this page?…
Answered my own question: spreadsheet!
The coolness rule is a really really good idea
Nice judging gameplay!
I still don’t get it: what does N/A stand for? 0 or null (in an SQL-ish sense, I mean, if it’s 0 then it will influence the mean, if it’s null it will not)?
N/A means not included in the average. It doesn’t mean zero.
Then what if a game with no audio receives 100 N/A votes and one 5 star vote? I’m confuss
Then they score 5.00 for audio.
Except that in that situation they wouldn’t get a score for audio, because the single highest and single lowest ratings are deleted before the average is taken.
Soooo, Ludum 24 is coming to a close… when’s the next mini-LD?? I didn’t know where else to post this! =)
The single highest and lowest ratings are deleted before the average is taken…I like that! Although there goes my plan of creating a game for a platform that only me and my freind have, and us both rating it a 5 in everything.
Just kidding.
Does anybody know when the results will show up?