Icon Contest #15 Discussion

[Devine]

Not to diss any of the entrants or @The Panda , but the way the scores were handled is total nonsense.
A judge should not be able to determine a contest with a factor of 66 to 12.
The Panda picked the winners all by himself and that's not okay in something as subjective as an art contest. Realizing your opinion is in fact a little arbitrary, would be a good place to start when you're judging a contest like this. This is a community, Panda, don't take yourself so seriously... 66 to 12? There is no counterweight for this judging whatsoever and it's completely laughable. I would strongly suggest a recount, with a second judge. If that doesn't happen, my condolences to the entrants of this contest, especially to the losers. @Murlocologist Find a new judge. @Zombie

 

[The Panda]

Not to diss any of the entrants or @The Panda , but the way the scores were handled is total nonsense.
A judge should not be able to determine a contest with a factor of 66 to 12.
The Panda picked the winners all by himself and that's not okay in something as subjective as an art contest. Realizing your opinion is in fact a little arbitrary, would be a good place to start when you're judging a contest like this. This is a community, Panda, don't take yourself so seriously... 66 to 12? There is no counterweight for this judging whatsoever and it's completely laughable. I would strongly suggest a recount, with a second judge. If that doesn't happen, my condolences to the entrants of this contest, especially to the losers. @Murlocologist Find a new judge. @Zombie

I did what I had to do and peekay has life stuff happening so I'm not really sure what you want us too do here. I followed what murloc wanted, you want to question this contest go to him not to me. thanks. And, I'm not taking myself so seriously not sure what you mean by that sir. Ive judged icon contests in the past so I'm not really sure who your aiming at. If there was a mistake murloc would of told me before hand and I would of edited it so please elaborate.

 

[Devine]

I did what I had to do and peekay has life stuff happening so I'm not really sure what you want us too do here. I followed what murloc wanted, you want to question this contest go to him not to me. thanks. And, I'm not taking myself so seriously not sure what you mean by that sir. Ive judged icon contests in the past so I'm not really sure who your aiming at. If there was a mistake murloc would of told me before hand and I would of edited it so please elaborate.

You don't think that judging with a range of 12 to 66 out of 75 is a bit immodest when your individual scoring makes up 75% of the total? Where is your sense of community? Are you purposely trying to alienate people off this site? Also, I'm not putting you at fault here for the total, I applaud your efforts for judging, really, but I do have a massive problem with the way you defined the outcome of the contest. Even if there was a second judge, your judging would outway that of a more community minded judge in all scenarios. There is no need to give the loser 12 fucking points out of 75 here. What's the point even? You don't think that this person might take this personally and righteously so?

These were my scores for the judging of the last music contest 25% poll, 37.5% me, 37,5% Samuraid. The final result of the contest was not directly decided by my actions. I'm a professional musician, man, but I don't think it's my place to decide full on who wins and who loses.

​

Your opinion should never outweigh the opinion of your fellow judge by such factors. Imagine giving the winner 75 points and the loser 0, then you decide the outcome of the contest by default.

@Murlocologist if 2 judges make up 75% of the score, then 1 judge should make up 37,5% of the score. I would even diminish that further to 25%. Honestly, there should always be a second judge, or the results should not be made public. This is not a good example you want to set for this site's contests. I highly recommend recounting the points with a secondary judge, and a more modest judging by the panda.

 

[The_Silent]

Before we put too much blame on individual users: The weight of the judging versus the weight of votes was set back in the original contest post. And my guess is that the ratios was carried over from previous contests. That is the usual procedure, so lets keep a civil tone. I usually have no idea how much weight my judgings carry when I judge a contest. The judge shouldn't, it only encourages "strategical voting/rating".

I think it is a completely fair point that one person should not hold 75%+ power over the outcome of a contest. And that if we in the future gets more contest with only a single judge, that the balance is skewed more in the favor of voters than it is now. That could easily be added as a clause to the contests in the future.

No matter what, this is not a problem caused by anyone in particular as much as it is a general problem we haven't dealt properly with yet. I suggest a mod move the discussion to Site
Discussion. Then we can return to this contest when we've reached a conclusion.

 

[Devine]

I usually have no idea how much weight my judgings carry when I judge a contest. The judge shouldn't, it only encourages "strategical voting/rating".

I disagree. Equalizing the effect of both judges is the only way the results cannot be skewed for the opinion of some dude who decides he should give the winner 20 points more than the runner up. While the other judge gives 2 points in the opposite direction because his entire judging is less extreme in numbers. The weight of both judges should always be about equal. That being said; the mathematics behind this is too complex for me to find out a formula, and so the judges should have some kind of intrinsic decency and awareness of this matter.

I am speaking in a civil tone, and don't appreciate how you're patronizing me (and/or The Panda?) for expressing our opinions. If I come across strongly, it's because I find the situation ridiculous.

No matter what, this is not a problem caused by anyone in particular as much as it is a general problem we haven't dealt properly with yet. I suggest a mod move the discussion to Site
Discussion. Then we can return to this contest when we've reached a conclusion.

I agree.

 

[The_Silent]

Aim the discussion at the problem, not eachother. Decency, modesty, moral fiber, etc. is not relevant.

I never patronize.

 

[Devine]

Apologies, @The Panda, for any personal attack I gave you. My intent was not to vilify you.
I might add to that, that if there was a second judge, the problem I described might be solved and I might have been wrong to indicate you as a part of the problem. However, there was a more community friendly way of going about your final scores, and I do believe you could have still altered your scores to be less extreme, after finding out about the second judge (which I think you should have done).
That being said, the moderator in charge of the contest should maybe keep a vigil eye towards the decision powers of both judges comparably, and guide the 2 judges towards a more or less democratic score.

Let's discuss the real issue here then, which is the lacking of a second judge. I think the highest priority for fixing the issue is finding the second judge.
The standard might have been set too high for the quality of a judge while in search for one. There are many people who would be qualified, objective judges, even if they never made a single icon.
I suggest we look for one immediately and recount the scores. In any case this should be avoided at all cost for future reference.

Edit: @The_Silent, you were right. I forget myself.

 

[The Panda]

Apologies, @The Panda, for any personal attack I gave you. My intent was not to vilify you.
I might add to that, that if there was a second judge, the problem I described might be solved and I might have been wrong to indicate you as a part of the problem. However, there was a more community friendly way of going about your final scores, and I do believe you could have still altered your scores to be less extreme, after finding out about the second judge (which I think you should have done).
That being said, the moderator in charge of the contest should maybe keep a vigil eye towards the decision powers of both judges comparably, and guide the 2 judges towards a more or less democratic score.

Let's discuss the real issue here then, which is the lacking of a second judge. I think the highest priority for fixing the issue is finding the second judge.
The standard might have been set too high for the quality of a judge while in search for one. There are many people who would be qualified, objective judges, even if they never made a single icon.
I suggest we look for one immediately and recount the scores. In any case this should be avoided at all cost for future reference.

Edit: @The_Silent, you were right. I forget myself.

I was just stating a point, peekay is offline and murloc had to do a plan b because everyone wanted to see results. So therefore he took the 2nd judging into his own hands and finished it off.

 

[Murlocologist]

So what is the problem here? That the judge's review carries 75% of winner? 75% of judge's review to 25% of poll score is a forum contest convention, if you feel like it is not ''democratic'' enough, address your concern to the administration. Poll to judge ratio is a pattern not made up by the host.

You are free to post your criticism towards specific parts of judge's review you don't agree with, but not antagonizing the judge.

The other assigned judge did not deliver review on said time, after a week of waiting for activity response, The Panda remained as the only judge. If there were others interested in taking the judge role, they should have expressed so. After contest has minimum of one judge, other judges are optional. One judge for contest is relevant and preceded practice. Certainly it is better to have more than one judge, but not on expense of prolonging the contest indefinitely.

Judges are free to choose own approach of reviewing, be it reasonably biased or objective opinion, as long as in boundaries of common sense, is relevant. Assigned moderator is not superordinated to the judge and has no effect on how strict or biased review will be, judges are advised to give as objective and as consistent review they can.

 

[Devine]

So what is the problem here? That the judge's review carries 75% of winner? 75% of judge's review to 25% of poll score is a forum contest convention, if you feel like it is not ''democratic'' enough, address your concern to the administration. Poll to judge ratio is a pattern not made up by the host.

You are free to post your criticism towards specific parts of judge's review you don't agree with, but not antagonizing the judge.

The other assigned judge did not deliver review on said time, after a week of waiting for activity response, The Panda remained as the only judge. If there were others interested in taking the judge role, they should have expressed so. After contest has minimum of one judge, other judges are optional. One judge for contest is relevant and preceded practice. Certainly it is better to have more than one judge, but not on expense of prolonging the contest indefinitely.

Judges are free to choose own approach of reviewing, be it reasonably biased or objective opinion, as long as in boundaries of common sense, is relevant. Assigned moderator is not superordinated to the judge and has no effect on how strict or biased review will be, judges are advised to give as objective and as consistent review they can.

Of course I won't take it up with the administration. Do I look like the neighborhood watch? I'm not even an entrant. I'm not on THW to crusade twice every time someone messes up.
Maybe I should have offered to judge but I totally neglected to do so. Sorry guys, I had no idea what was going on! I guess you can all hold me responsible for the way this contest was handled and then we can have some closure.

Congratulations to the winners! (I'm actually unironic in this last statement, you guys all did an amazing job and I don't mean to rain on your parade)
Much love to all contestants! <3

 

[KILLCIDE]

Are you purposely trying to alienate people off this site? Also, I'm not putting you at fault here for the total, I applaud your efforts for judging, really, but I do have a massive problem with the way you defined the outcome of the contest

I'm confused as to why you would even put the blame on The Panda. You act like he knew what the situation was. He did his part by judging and went on with whatever he was doing prior to it.

I guess you can all hold me responsible for the way this contest was handled and then we can have some closure.

What a professional you are.

 

[Devine]

I'm confused as to why you would even put the blame on The Panda. You act like he knew what the situation was. He did his part by judging and went on with whatever he was doing prior to it. What a professional you are.

I was assuming he was informed, yes, but didn't alter his score. If that was incorrect I apologize.
Note that I had already apologized before your comment, and also note that your comment doesn't take away that nobody is taking any responsibility for this contest being very poorly handled.
(What I'm trying to point out by ironically taking the blame).

Edit: Nobody will take responsibility and nothing will happen because nobody cares.
And as the contest section is one of the last active forums on this site, I dare say that is a bit troublesome, but sure call me out for saying mildly offensive things.

 

[KILLCIDE]

I want to mention that contest judges have been an ongoing issue for a long time. As mentioned by The_Silent, this is something we should definitely discuss in Site Discussion as these three situations always happen:

• A user offers to judge a contest, but suddenly goes missing
• A user offers to judge a contest, but contestants feel they are not an expert enough to make such judgements
• No user offers to judge (we had this situation with the Zephyr contest, and a contestant offered to drop out to judge, but we decided to just go with the one judge we had)

With that in mind, hive contests have always been infamous for having extremely delayed results. My suggestion has always been to give a higher work-to-reward ratio.

 

[Devine]

I want to mention that contest judges have been an ongoing issue for a long time. As mentioned by The_Silent, this is something we should definitely discuss in Site Discussion as these three situations always happen:

• A user offers to judge a contest, but suddenly goes missing
• A user offers to judge a contest, but contestants feel they are not an expert enough to make such judgements
• No user offers to judge (we had this situation with the Zephyr contest, and a contestant offered to drop out to judge, but we decided to just go with the one judge we had)

With that in mind, hive contests have always been infamous for having extremely delayed results. My suggestion has always been to give a higher work-to-reward ratio.

You are absolutely right, here, though.
I would say that contestants should swallow their pride then and accept a "less than expert" judge. There's more than enough respected, well educated and active members who will take the assignment seriously.
When the situation calls for it, something can definitely be agreed upon with i.e. multiple "non-expert" judges, where the judgement of the "non-expert" users has slightly less weight.

My point is that contestants should always have a sense that everything was 100% fair and in good spirits, over the sense that everything was judged by experts. I think we need to underline the sense of community, the camaraderie and the idea of growing together in these contests to keep them alive and fun for everyone. Otherwise it will leave some people bitter and turned off. I feel very strongly about this, hence all the fuzz.
I mean... A guy had 12.5 points out of 100 in this contest. That's not fun. Do you reckon he'll enter next time?

 

[GhostThruster]

My point is that contestants should always have a sense that everything was 100% fair and in good spirits, over the sense that everything was judged by experts. I think we need to underline the sense of community, the camaraderie and the idea of growing together in these contests to keep them alive and fun for everyone. Otherwise it will leave some people bitter and turned off. I feel very strongly about this, hence all the fuzz.
I mean... A guy had 12.5 points out of 100 in this contest. That's not fun. Do you reckon he'll enter next time?

i feel like you're projecting your own idea of what a contest should be onto the actual purpose of a contest. keep in mind that contests, by design, are meant to be competitive and showcase the best of the entrants first and foremost.

that's not to say that they shouldn't be fun and encourage participation. but what you're doing is the opposite of fair. rather than give a user honest, accurate feedback, and award them the points they deserve in the spirit of fair competition, you want to sugarcoat and inflate their score for the sake of feelings. now i don't hate feelings, sometimes i even like them - i just don't think a judge's honest assessment should be compromised in order to make sure that every entrant feels happy.

in fact, an argument can be made that giving accurate, truthful feedback is ultimately more helpful for encouraging users to continue creating resources/improving their skills.
i don't like that you presume a user is embittered and turned off from further participation because they receive a lower score. while i know you have good intentions, i feel that's a very assumptive and patronizing view to have. for all you know, abovegame appreciates the feedback he received and now, more than ever, feels motivated to improve.

 

[Devine]

i feel like you're projecting your own idea of what a contest should be onto the actual purpose of a contest. keep in mind that contests, by design, are meant to be competitive and showcase the best of the entrants first and foremost.

that's not to say that they shouldn't be fun and encourage participation. but what you're doing is the opposite of fair. rather than give a user honest, accurate feedback, and award them the points they deserve in the spirit of fair competition, you want to sugarcoat and inflate their score for the sake of feelings. now i don't hate feelings, sometimes i even like them - i just don't think a judge's honest assessment should be compromised in order to make sure that every entrant feels happy.

in fact, an argument can be made that giving accurate, truthful feedback is ultimately more helpful for encouraging users to continue creating resources/improving their skills.
i don't like that you presume a user is embittered and turned off from further participation because they receive a lower score. while i know you have good intentions, i feel that's a very assumptive and patronizing view to have. for all you know, abovegame appreciates the feedback he received and now, more than ever, feels motivated to improve.

Of course I am projecting my own idea of what a contest should be. You are talking about art as if there is a 100% objective quality which can be measured. This is simply not the case. Therefore, multiple judges should be asked to judge. I am not making a case for judges making a dishonest assessment. I am making a case for all judges to have the same amount of say in the end result, and don't feel entitled to push ("project") their ideas into the results indefinitely. The way the contest is set up, one judge can overrule all other judges, simply by going more extreme in the fluctuation of ratings. This is a huge problem. There needs to be a more functional approach to these judge scores.

Let's say the judges rank the 5 players and give them all thorough feedback.
Then the first player for each judge gets 37.5 points (max), the second get 35 points, third gets 32.5, fourth 30, fifth 27.5. This way judges will always give the same amount of points and the power of 2 judges will be equally spread. Furthermore, there will be no numbers bashing like "12/75 points for the new guy (fuck him lol)". If a status quo is reached, that's where the poll comes in and the community decides. Grading the entries will go faster, will be more democratic and will engage the community more in a positive way. Plus it favors the guys who would definitely have lost and still will lose, it's just a little pat on the back saying "you did alright".

As a final note: Feedback is part of growing, so a judge must always be brutally honest in his assessment. Encouraging users to keep at their craft and stay in the community is equally important so a certain amount of tact is necessary. There is no need to bash an entry with 12 points out 75. It's simply anti-social. This is, in the end, a community for a 15 year old game and if we try to one off each other and or humiliate each other in our attempts to make art, it won't last much longer.

 

[GhostThruster]

You are talking about art as if there is a 100% objective quality which can be measured. This is simply not the case. Therefore, multiple judges should be asked to judge. I am not making a case for judges making a dishonest assessment. I am making a case for all judges to have the same amount of say in the end result, and don't feel entitled to push ("project") their ideas into the results indefinitely. The way the contest is set up, one judge can overrule all other judges, simply by going more extreme in the fluctuation of ratings. This is a huge problem. There needs to be a more functional approach to these judge scores.

my post was clearly directed at your point in regards to "community, camaraderie" etc., and not about multiple judges. in fact, i'd agree with you in general that the more judges there are, the better the assessment.

removing "fluctuations" comes with its own problems. what you basically want is for judges to rank entries, rather than score them - but this doesn't account for the magnitude of quality difference that the judge could otherwise stress. the first player might not be better than the second by the same amount that the second is better than the third.

you imply there is or has been number-bashing, as if anyone in the history of hive contests has said something akin to "X points for the new guy (fuck him lol)". as far as i know, you're complaining about a non-issue. and from experience, fixing what isn't broken leads to things that are actually broken.
i agree, we should encourage users to keep at their craft - with words, and honest feedback, not by giving dishonest scores. reserve scores for their intended purpose - differentiating quality, and quantifying the excellence of each entry. i think you're underestimating the average user's resolve and motivation, and over-exaggerating the impact of a low score.

 

[Devine]

removing "fluctuations" comes with its own problems. what you basically want is for judges to rank entries, rather than score them - but this doesn't account for the magnitude of quality difference that the judge could otherwise stress. the first player might not be better than the second by the same amount that the second is better than the third.

You are very right. But there might be ways to bypass this problem by allowing the judge to create one double interval where he'll jump 5 points instead of 2.5.
I could also argue that if the quality difference between 2 contestants is so big, then both judges will surely agree when they are judging. Then they will confirm each other's opinion, effectively letting the highest quality contestant win. If they disagree, which is also likely, then perhaps the case isn't so clear cut as you make it out to be. In this case the results of the judging will be tied and the community gets to decide through the results of the poll.

Spoiler: Quote in convo with Naze

Lets make an example here:
Judge 1, Judge 2
Player A, Player B, Player C

Judge 1's ratings are A 50 points, B 40 points, C 30 points.
Judge 2's ratings are B 50 points, A 45 points, C 40 points.

In this scenario player A will always win, while in reality we see that judge 2 was actually just as fond of player B as judge 1 was of player A (in relation player C).
He just used his own personal scoring system. This system is completely random, and has nothing to do with the actual quality in relation to all players, because if a certain judge decides he'll rule by a factor 5 and the other judge decides he'll rule by factor 10, he's basically making the other judge obsolete, and he gives himself more decision power.
My judging in the music contest was actually only worth 20% of the total score since I gave everyone 30 points by default and had the best entry have 40 points. Samuraid did a fairly similar thing, however if he had decided he gave everyone 10 points by default, and have the winner have 50 points, his scores would have overruled mine. But that does not mean his opinion would have mattered more than mine.

Back to the example:
We see that player A wins, but if the scoring system was actually standardized, the scores for A and B would be equal.

you imply there is or has been number-bashing, as if anyone in the history of hive contests has said something akin to "X points for the new guy (fuck him lol)". as far as i know, you're complaining about a non-issue. and from experience, fixing what isn't broken leads to things that are actually broken.
i agree, we should encourage users to keep at their craft - with words, and honest feedback, not by giving dishonest scores. reserve scores for their intended purpose - differentiating quality, and quantifying the excellence of each entry. i think you're underestimating the average user's resolve and motivation, and over-exaggerating the impact of a low score.

These scores are only dishonest in your head, as there is no way that a professional artist would ever rate 2 pieces from 66/75 to 12/75. There is only different value systems, and a different focus on different aspects of art. Professional artists don't give a shit about technical proficiency. They care about the idea getting across, and they understand how subjective this all is. Professional artists are master communicators, and they communicate functionally through their art.

There are only insecure artists who are trying to make a point, thinking that if someone is technically weak, he must be punished for it by an extremely low score. There's also people who want to give a fair, constructive rating, shifting the point of focus to the larger picture, and by doing so motivating the person to improve. What difference does it make if someone finishes last with 1 point or 100 points? The end result is the same for the contest, and you will not have pissed of a fairly new artist. You underestimate the negative impact of rejection, and let's be real here: giving a contestant 5 times less points than the winner is very much rejecting him. It's overkill and as I said before it should not happen in an amateur community drawing contest. What the hell?

 

[Heinvers]

Hey guys,

Your spawned discussion was transferred over to a new thread in Site Discussion section to keep the Results thread clean until you have reached some common ground.

Good day to you :]

 

[Sin'dorei300]

In essence all i wanted to say, @GhostThruster said it already. I couldn't have said it better, but i will add something.

This is your opinion, @Devine .
My opinion and anyone else's opinion can be totally different, or -similar in some aspects and -opposite in others...
We cannot change things based on individual opinion. The things are in the way they are based on a consensus, not randomly, and not because the mod, or I, or you, or anyone else wanted it so.
I would like to think that most of us agree that fairplay should be the basis for these contests. But this is just my approach, i'm aware that people here could see things different.
Regarding this contest -taken as a whole, the results came out as(many of us) expected. Viewed in detail -there are some exaggerations, but this is inevitable and depends on how everyone sees things.
We want more judges and, also, we want the judgement to be as quick as possible. But many of us forget that not all of us have the same age. Real-life demands differ from person to person and free time is conditioned by many factors.
We often want to do the promised things, but it doesn't always happen the way we want it to. It doesn't always depend on our will, other factors are involved too.
That's life and we can't(and shouldn't) turn off real life.
So, let's be more flexible when we "judge" others for their actions.

 

[The_Silent]

Here is an observation I made regarding the contest.

Votes almost didn't matter at all, except for the fact that San's had a 200% vote lead, so he was pushed up a single position.

Which not only leads me to conclude judges hold too much power in general, but that they most assuredly should not hold this much power singlehandedly.

I completely agree with the conclusion in the past that qualified judges should hold more power than random people on the internet, but I do think this is a bit extreme. It is the same reason we have moderators and reviewers, but I do think votes should hold a little more power than they do now, where they now are almost useless (do no difference). It took 17 people to push a single person up one position.

So I propose we do a threeway split when it comes to judges vs votes, so judges hold 66% of the power in competitions, but individual judges never hold more than 50% of the power so, basically: If we have 2 or more judges, they get to decide 66% of the outcome of the contest, but if we only have one judge, that that judge decides 50% and the votes decides the remaining 55%. So we, in the future, can prevent singular individuals from deciding the total outcome of a competition, while still acknowledging that some people are more qualified to judge than others. This should also ensure votes are relevant, no matter what, while qualified judges still hold the majority power. This should also, somewhat, combat favoritism.

 

[Sin'dorei300]

How can anyone ensure that votes are relevant, especially when it is known that only one judge is available and his opinion would count only 50%?

 

[Shar Dundred]

User polls can easily be extremely biased, with some people just voting for the PEOPLE, not the resources.
Happens too easily.

 

[The_Silent]

Well, making sure that the judges decisions is never overwhelming to a point where votes is meaningless is the aim. Right now it does seem like the judges often, singlehandedly, decides the outcome of the contests. I think it is important to remember that judges, like moderators, is just more qualified users that ensures that content lives up to a certain standard. But that any criticism you aim at the voters, at worst, can be aimed at the judges too.

I feel that preserving the public image that votes have power is very important, especially now 10+ years after wc3 was released. So I encourage a system where votes hold a somewhat significant power, while not a overwhelming one. Users is the lifeblood of the hiveworkshop. I am no wizard however, and can not pull any magic tricks out of my sleeve that will automatically ensure this.

 

[Kyrbi0]

Maaaaan... I've been having/preparing to have this kind of in-depth discussion of Contests for a long while... But I'm simply too busy now, moving & all that.

One question I think she be pre-eminent in our minds is "What is the point of a Contest?" I've already seen a few different answers to this question in this thread alone, but I have come to see it as the foundational issue.

 

[Shar Dundred]

I am quite sure we had such discussions many times in the past already.
At least I can remember discussing the point system in the past.

@The_Silent: You are correct, of course, when you say that what I said about any normal user
like myself can also be applied to a judge or moderator.

The problem is, to really find a fitting distribution of the points, you have to experiment with contests.
That means that some people could lose a contest because they had the "bad luck" to catch a
contest with unbalanced judging distribution.

Point is, no matter how you do it, there will always be someone who is unhappy with it. =P

 

[GhostWolf]

Luckily for you, the human civilization is pretty vast, and has a pretty vast history, both recent and old, so you can learn from other's experiences.

 

[Murlocologist]

Regardless of debates on contest specifications and poll to judge ratio, I just want to say how sad it is to see the amount of hostility that is directed towards Panda, something he in no way deserved. For his enthusiasm and good will to take role of contest judge, all the hours spent into writing detailed review and all the support he had given in the contest thread, he is given massive insults. Even if the review was flawed in some points, he did not make any major mistake and gave his most honest and objective opinion. He made detailed review, delivered it before poll even ended and than someone points finger at him saying he isn't suitable for judge because he had to show more ''sympathy'' for newcomer author...

There is absolutely no reason for Abovegame or anyone else to feel demotivated from making icons, lower score on contest means nothing and should never affect one to point of feeling bad about own work. Quite the contrary, people should make the most of the given honest feedback. Understanding that honest constructive criticism is far more valuable than empty praising is something that those who want to improve themselves will hopefully realize at some point.

Icon section is more than newcomer friendly area, we are full of patience and always try to encourage authors to improve with as much constructive criticism and suggestions needed to make icon approvable. I never sent any legit drawn icon to substandard section ever since I became reviewer, claims that newcomers are badly treated is a nonsense and furthermore an insult to all of us who do our best to support authors with constructive feedback in all forum areas.

 

[Devine]

@Sin'dorei300 I'm very well aware that what I am saying is my opinion. There are no truths in art and that is the whole reason I cannot consent with 1 person deciding an art contest result, and why giving the community back its voice is important.

@Murlocologist, you are right and I don't bare any ill will against you or The_Panda, regardless of how tactlessly I brought this up. For that, you have my sincerest apologies. The goal was to stir up a debate.
I will admit I wasn't on my best behavior earlier. Sorry.


I don't see how anyone can see any benefit in giving a user a super low score, other than to boost their own self esteem. I won't bring it up as a solid argument, since this is clearly a matter of different value systems.
As the result of my second proposal, found here below, it would be indirectly impossible to rate a user so low. I also think that people underestimate the community. The contest rules were conceptualized when Warcraft was a young game, when lots of young kids and teenagers were playing. In a scenario like that it makes a lot of sense to give less decision power to the community (risk of cheating etc). But let's be very real here: The game is old and so are we. You can treat the community as you would an adult.

Here is what I discussed with Naze earlier, bringing up 2 things I think are wrong with the way this was handled and how it can be prevented in the future.

1. A minimum of 2 judges. If a secondary "expert" judge cannot be found, at least 2 respected "semi suitable" users should take his place. Balance being along the lines of 35% (expert) (17.5 x2) users and 30% poll or something like that. Or like 35 - 20 - 20 - 25. Contest results should not be made public before a consensus is reached for a change of plan in judging. If all else fails, the host should take the matter to Site Discussion(?) and get to a consensus with the contestants. Communication with the community is important here. There is no point in solidifying this. The only important part is that the contestants are aware of and agree with the way the score is set.
   
   
   
2. A more formulaic decision system should be implemented where people are given points by the ranking each judge gives them. That means for instance -10% points of the judge's capacity for each rank you drop. Effectively confining the judges to scores like 30, 27, 24, 21... etc Less than expert judges would have less decision power with a series like 20, 18, 16, 14... The system is not flawless, but at least it gives back a sense of democracy to the judging and increases the significance of the polls and the involvement of the community.
   
   
   Example:
   Judge 1, Judge 2
   Player A, Player B, Player C
   
   Judge 1's ratings are A 50 points, B 40 points, C 30 points.
   Judge 2's ratings are B 50 points, A 45 points, C 40 points.
   
   In this scenario player A will always win, while in reality we see that judge 2 was actually just as fond of player B as judge 1 was of player A (in relation player C).
   He just used his own personal scoring system. This system is completely random, and has nothing to do with the actual quality in relation to all players, because if a certain judge decides he'll rule by a factor 5 and the other judge decides he'll rule by factor 10, he's basically making the other judge obsolete, and he gives himself more decision power.
   My judging in the music contest was actually only worth 20% of the total score since I gave everyone 30 points by default and had the best entry have 40 points. Samuraid did a fairly similar thing, however if he had decided he gave everyone 10 points by default, and have the winner have 50 points, his scores would have overruled mine. But that does not mean his opinion would have mattered more than mine.
   
   Back to the example:
   We see that player A wins, but if the scoring system was actually standardized, the scores for A and B would be equal.
   
   
   That way a judge cannot shanghai the contest by giving his number one 30 points and the runner up 10 points (and so bias is less of a factor here). In close calls, that would mean the poll would have increased significance and it can lead to some excitement.
   
   Judges should still justify their rankings by adding a review, discussing technicality, originality etc... which enables people to learn from the contest more.

I'm curious to see what the community thinks of these 2 concepts.

The problem is, to really find a fitting distribution of the points, you have to experiment with contests.
That means that some people could lose a contest because they had the "bad luck" to catch a
contest with unbalanced judging distribution.

Point is, no matter how you do it, there will always be someone who is unhappy with it. =P

You give the contestants a voice in how it's being handled. I'd say the idea probably would not fail, and I would say that if it did, the idea can be improved. It doesn't take many contests the get this idea on point. The community is changing and "expert" modders are becoming increasingly scarce. Something has to change or all contests will be like this, and then I doubt many people will join them anymore.

 

[Sin'dorei300]

At that point, i'll quote @Shar Dundred:

User polls can easily be extremely biased, with some people just voting for the PEOPLE, not the resources.
Happens too easily.

 

[Devine]

At that point, i'll quote @Shar Dundred:

Why is that such a big problem?
Art is interwoven with the person who made it, and in the mind it's nearly impossible to like art made by someone you dislike, or the other way around. If you dislike someone, you'll be more likely to focus on their flaws than on the good things. If you like the same person, you'll start noticing all the good things he does.
This means that letting the community decide is actually a less biased sample than when 1 judge can decide with all his own biases in place.
It's in the best interest of THW to increase the sense of community, to keep the site rolling. How does anyone even benefit from a dude, who gets alone with no one but the judges, winning a contest?
The more people are having a say, the less skewed the sample should be.

I don't think it hurts to have a guy who is nice to everyone have a super slight edge in a contest. People who are nice get further in life so why not in a hive contest? Out of 10 artists that hit it big in life, only 1 doesn't have super strong networking and people skills, I can assure you. The more famous they get, often time, the nicer they are behind the scenes.


In any case, the poll is irrelevant to this particular discussion.

 

[Sin'dorei300]

"Satisfaction" based on lying does not last long. I don't think it makes sense to lie to ourselves and to others on purpose of giving us a "friendly image". Does it??
And, in general, everything that's built on lies "stuffed in beautiful clothes" doesn't seem to last too long, nor a community.
History confirms it(oh, i'm starting to say big words).
"Friendly and nice" doesn't mean to lie to you because the truth "hurts".
Would i be a nicer person if, instead of telling the truth, i'd lie and false praise, if i'd wear a mask of lies??

 

[Direfury]

I've only got a few things to say. First off, hive contests tended to be popularity contests in the past. To a large degree, they still are. That in and of itself can be off-putting for many outer circle entrants. In some cases, I've seen objectively inferior work place far too highly, over something utterly trivial. For example, a sizeable portion of the community from X region, voting for one of the only modellers/texturers/modders also from X region because of that. Or voting off of screenshots, in regard to modelling contests. A model textured and wrapped by Murlocologist, and poorly animated by someone unskilled in that area would most likely win a paired contest, just because of the pictures.

Entries scores are weighted more in favour of the opinion of judges, who are, presumably, moderately to expertly skilled in their field, to offset the fact that any oaf can vote for anything for any reason, rather than voting off of merit and examination.


I've thought for a while that the Hive could benefit from having a Judging roster. Essentially, giving a Contest Judge position to any qualified and willing modders, same as having Contest/Resource moderators. Having a wider variety of people to call on for these contests would help speed judging along, and from what I understand, stop the raised issue from coming up again.

 

[Devine]

"Satisfaction" based on lying does not last long. I don't think it makes sense to lie to ourselves and to others on purpose of giving us a "friendly image". Does it??
And, in general, everything that's built on lies "stuffed in beautiful clothes" doesn't seem to last too long, nor a community.
History confirms it(oh, i'm starting to say big words).
"Friendly and nice" doesn't mean to lie to you because the truth "hurts".
I would be a nicer person if instead of telling the truth i would lie and false praise, if i'll wear a mask of lies??

Let's not get derailed from the actual issue, Sin. The polls are another story and could make a whole topic for themselves. I'm not advocating have 100% poll score at all, I'm okay with the 25%, so it is of little relevance to the topic. What I am proposing, however is, that if the judges do not come to a consensus (meaning a clear cut winner), the poll becomes the deciding factor. I would much more appreciate having feedback on the 2 propositions I posted earlier.

I've only got a few things to say. First off, hive contests tended to be popularity contests in the past. To a large degree, they still are. That in and of itself can be off-putting for many outer circle entrants. In some cases, I've seen objectively inferior work place far too highly, over something utterly trivial. For example, a sizeable portion of the community from X region, voting for one of the only modellers/texturers/modders also from X region because of that. Or voting off of screenshots, in regard to modelling contests. A model textured and wrapped by Murlocologist, and poorly animated by someone unskilled in that area would most likely win a paired contest, just because of the pictures.


Entries scores are weighted more in favor of the opinion of judges, who are, presumably, moderately to expertly skilled in their field, to offset the fact that any oaf can vote for anything for any reason, rather than voting off of merit and examination.

This is definitely a good argument against the polls. But, if the judges cannot come to a clear winner, what difference does it make if the poll is the tiebreaker? It proves that the whole result would be absolutely arbitrary anyway. I feel however, that in most cases it will not come to this.

Also not that art is made for people to be consumed, not for experts to review. It should move people one way or another, and if a contestant gets a vote, at least he got the emotional aspect right.

I've thought for a while that the Hive could benefit from having a Judging roster. Essentially, giving a Contest Judge position to any qualified and willing modders, same as having Contest/Resource moderators. Having a wider variety of people to call on for these contests would help speed judging along, and from what I understand, stop the raised issue from coming up again.

This is actually a great proposition. I don't know if it will have any actual impact, though. The whole problem is that there are not enough people who are both qualified and willing, which is why I'm suggesting enabling "less than expert judges" to have a say in the outcome, with less decision power than the actual "experts".

 

[Direfury]

Here's the thing. Generally, you couldn't force me to be a Judge.

However, if I were on that roster, and was informed that another modelling/animating Judge was needed, because the previous one were missing, I might fill the role if necessary. I figure it'd be the same for plenty of other modellers, like Grendel or Tauer.

 

[Devine]

Here's the thing. Generally, you couldn't force me to be a Judge.

However, if I were on that roster, and was informed that another modelling/animating Judge was needed, because the previous one were missing, I might fill the role if necessary. I figure it'd be the same for plenty of other modellers, like Grendel or Tauer.

I understand that, and that could totally work, but it doesn't take away that in some occasions there will be no second judge (or even a first judge entirely). The current music contest has no "expert" judge at all. The judging will be done by 2 or 3 (maybe even 4?) random hivers, probably, but that should be possible in my opinion. That's where the whole "less than expert" judge idea comes from. What's your opinion on this?

 

[Direfury]

My opinion is that, generally, the community is host to more than enough qualified contributors. I don't seem to remember Tauer, Trag and myself ever judging modelling contests, though I'd say we're all more than qualified.

So my opinion remains the same. Watch the Resource sections more closely, and you'll often find people fit to be Judges. People who might not check the forums as often, and would otherwise miss the opportunity to volunteer.

 

[Devine]

@Murlocologist can you please describe the difficulties you experienced finding a suitable icon judge?
Seems to me that most of the active members in the icon section are you, the panda, and the entrants of the contest. It would be only natural that you have trouble finding suitable judges, am I correct?

 

[Murlocologist]

Seems to me that most of the active members in the icon section are you, the panda, and the entrants of the contest. It would be only natural that you have trouble finding suitable judges, am I correct?

There was no problem with finding suitable judge as Panda offered his help at the very start of contest. My job is to make sure contest has minimum of one active judge. If contest had no judges at all, I would have taken the role and that is one of the reasons why I haven't given vote in poll. Contest notice was up for a month, anyone who cared for judging icon contest could have contacted me, as I additionally highlighted in that thread. One more person until the end of contest expressed willingness to judge, afterwards he went inactive, meaning back to one judge. People not being interested in judging contests is not my problem, nor I care for that. I made sure for contest to have plan a (2 judges) , plan b (one judge) and plan c (me as judge), we went for plan b and there it is.

 

[Devine]

Hmm well the whole thread is built upon the idea that 1 judge was enough to guarantee a non-biased result.
Perhaps plan B should have been to find a second judge after peekay didn't respond. Or finding 3 judges in the first place and if someone bails, there are still 2 judges?

 

[Direfury]

There shouldn't only be one Judge, because there's more than one aspect to creating a resource. For example, animation. My criteria would be wildly different than Grendels, because I place higher importance on good animations, and have the experience to pick apart custom sets. More generally, the same is true for Tauer, Trag or Frank, or whoever else. We all have our own specialties, and with that, comes more balanced, accurate results during judging.

It isn't often that you'll find two modders cut from the same cloth, so generally any two, or any three would suffice.

 

[Naze]

These discussions typically get messy, and this one is. Lots of things have been discussed and aren't being replied anymore, being left without a solution. So I'll narrow it down to these two:

[1st topic] Obligatory 2 judges. Amendment:

In case no two expert judges are available, we could pick 2 non-expert users in place of the expert judge, each of them with half of the judge decision power.​

[2nd topic] Changing final score calculation:

2 Judges: 66%; public poll: 33%​

Now please, this thread won't go anywhere if we don't explicitly reply agreeing/disagreeing to the propositions above (or suggest changes). When you voice your opinion, do it in a way that aims to bring us closer to a conclusion, not to simply raise a problem and then get away not replying to the rest of the post. This can also be decided in a poll if you guys think it's necessary.

I'll start.

[1st topic] Obligatory 2 judges. Amendment:

In case no two expert judges are available, we could pick 2 non-expert users in place of the expert judge, each of them with half of the judge decision power.​

I'm all for the obligatory 2 judges. The amendment of non-expert judges will mostly likely rarely happen, so it won't be a big of a problem. Also I think that two non-expert judges with half decision power are way better than only 1 expert judge anyway.

[2nd topic] Changing final score calculation:

2 Judges: 66%; public poll: 33%​

The Silent has a valid point with wanting to increase poll relevance, but as Direfury said, there is this evident problem of polls being a popularity constest rather than anything else. No one downloads all the entries. Even when it's a concept art contest people won't bother clicking the links to see each entry, and you don't have to fucking download anything. Yeah we can raise public poll participation a bit and see how it goes. And if we're to do it, it shouldn't be anything bigger than 33%, so yes I would also vote for this change. This isn't permanent anyway, if we aren't satisfied with it we could always change it back.

 

[Direfury]

2D art contests are generally easier for common users to judge, as they need only look at an image.

For other things, such as mapping, coding, modelling and hero contests, we should start encouraging voters to actually examine the entries, and judge for themselves. Perhaps counting votes that are backed up with even a brief review on atleast one or two things not evident from screenshots.

I've always had a problem with those mongrels that vote on models from pictures alone, despite the fact that some, or all entries are mostly/fully custom animated. People who would, as I mentioned before, see only a screenshot of a model textured by Murlocologist, but atrociously animated by whoever else, and vote for it. It masks glaring flaws in the entries, and unfairly inflates the ranking of an entry.

In essence, voters only see the one good knee on an entry, not the other stump leg.

 

[Kyrbi0]

Still no time to fully contribute, but I must comment to say this: @Naze is performing the single-most useful & necessary function of marshalling this conversation & requiring specific responses. In turn, I should also respond (though I think there is more going on than just those 2): #1 I agree with mostly, #2 I mostly disagree with, unless certain concessions/changes to how Polls work (e.g. voting styles, expectations of Voters, etc) are made.

That is in part due to my second reason for posting, and that is to say that, on the whole, I completely agree with Direfury, for the reasons he's stating (and with some of his propositions).

If I may be so bold, I will reiterate what I believe to be a third topic (or rather zeroth):

[0th Topic] : What is the point of any given Contest? Is it to have fun, to increase the number of resources, to engender community, or as 'tests of skill' to determine who is best at something? Or some combination?

I think the answer to that will have a direct impact on how we structure these conversations.​

 

[Direfury]

Contests are, in general, whatever the hell the entrants want them to be. Personally, I always saw them as a way to prove myself. Make up for the lack of acknowledgement I tend to get.

Always bugged me how few comments my uploads got, compared to most. Even when I'd do something completely new to the community.

 

[Sin'dorei300]

I agree with 2 or more judges, but i disagree judges to be chosen from among those who have little to no experience in the art field associated with the contest.
Also, because it's related to the contests as well, i suggest to adopt the same percentages ratio judges-poll for all the contests(regardless of the sections), e.g. 70%(judges)-30%(poll) for all. Additionally, to decide for all the contests - whether polls will be multi or simple choice.
It doesn't seem fair to me that some contests have multichoice polls and other one choice polls.
Until recently, we were not allowed to have multichoice polls, now i see that it has remained at everyone's will.
It would be nice to have those things(even the calculation formulas) stated somewhere in the Arena - if not as a rule, at least informative(it would be better and more orderly, not letting room for interpretation and other stuff).

*Maybe some people prefer to vote in a poll rather than post feedback, so you might want to add a poll with all the suggested options(including the current one as well).

 

[Direfury]

I doubt the idea was to just pick any two goons at random. It was more, approaching someone like Crazy Russian or whatever his name is, to judge an icon contest.

 

[Naze]

[0th Topic] : What is the point of any given Contest? Is it to have fun, to increase the number of resources, to engender community, or as 'tests of skill' to determine who is best at something? Or some combination?

The most relevant factor imo is that we get a lot more interaction with the community by creating resources for contests, as Direfury mentioned. We participants are more prone to comment on eachother's stuff since we're checking the same thread frequently. Not to mention the judging that is an awesome thing. Also there is the competition, etc.

I agree with 2 or more judges, but i disagree judges to be chosen from among those who have little to no experience in the art field associated with the contest.

I doubt the idea was to just pick any two goons at random. It was more, approaching someone like Crazy Russian or whatever his name is, to judge an icon contest.

Just making things clearer: the idea is replacing 1 absent expert judge (who decides 33% of the final score) with 2 non-expert users, with half of the decision factor (16.5%). Preferably if they have at least a minimum of little experience in the art field, of course. CRAZYRUSSIAN is actually an experienced icon maker. This seems to be just a plan B though.

*Maybe some people prefer to vote in a poll rather than post feedback, so you might want to add a poll with all the suggested options(including the current one as well).

Sure, I've also come to think that that's really the best idea. I'll skim through this thread and see if I can organize a poll later, so we can discuss this in a more organized way.

__________________________________________________________________

This is the time for you guys to voice the things you guys want to see changed in arena judging (if you haven't voiced already). Post them here and I may add them to the poll thread for them to be voted later. Right now we have:

[0th Topic] What is the point of any given Contest?
(ofc this isn't going to the poll, as but the question will be kept for the discussion)

Is it to have fun, to increase the number of resources, to engender community, or as 'tests of skill' to determine who is best at something? Or some combination?​

[1st topic] Obligatory 2 judges. Amendment:

In case an expert judge isn't available, 2 non-expert users will be picked in place of the absent expert judge, each of them with half of the judge decision power.​

[2nd topic] Changing final score calculation:

2 Judges: 66%; public poll: 33%
​

[3rd topic] Same percentages and poll options for all contests:

Same percentage of judge/public poll ratio in final score;
Same rules for applying multiple-choice or single-choice polls in all contests.​

 

[Devine]

0th topic: Add to the purpose of contest: to learn from others (just how I see it).
1st topic: I am all for both minimum 2 judges and 1 judge + 2 backup judges, the way naze described it.
2nd topic: I have no problem with 75%-25% if 2 judges are appointed. However, in case we can't rally enough people behind the 1st topic, I'd like to have 50% judge, 50% poll in the case of only 1 judge.
3rd topic: No opinion. I think it's very important to be able to make exceptions to these same rules and percentages when all contestants feel it's appropriate. I would make them more like guidelines than actual rules.

Spoiler: Guidelines

 

[Direfury]

Even without the presence of expertly skilled judges, I'm still against the idea of giving public polls too much influence. Not without some way of making sure some degenerates don't unfairly influence them. At the very least, you can put your faith in a judge. You know they thoroughly examined the entries.

If anything, I say public polling is best used as a tie-breaker. Look at the previous mini-modelling contest. The poll was weighted in favour of the cutest entry, almost causing a tie. I doubt many of the voters actually checked the models out, beyond maybe giving them a cursory glance. So what happens next time? Does someone win an otherwise close contest, because the poll was full of filipinos voting for the only filipino entrant, because they're from the same place? Or because one of the entrants has more friends?

Before it's brought up, I'm not saying that the aforementioned poll skewed the results. My argument is simply that it has the potential to do so. Contests are, by their very nature, based around technique, skill and merit. They need to stay that way.