Contest Judging Needs A Change.

[StormKnight]

Hi all, I would like to bring up a big discussion regarding how contests are judged and the impact of poor management and judges on contests.

I realize i might not be the best person to point this out, but i can't let this fill inside me without speaking.

Recently there was a music contest that its final results, resulted in one of the best and oldest content creators in the server to leave (one of my favorites). @JetFangInferno. now idk about you guys but i'm not going to sit here watching veteran content authors leave because of some serious problems.


Its time to insert my suggestions and point out suggestions of some of my friends.

First, i'll link this: Improving Our Judgement

There is few things i like to point out without discussing massive overhauls of the system first.
Having only 1 Judge for a contest is very bad, bias creeps in and incompetency shows itself. it needs to be a unified work of atleast 2-3 people. so please make a clarified stage where you NEED judges ready to rumble BEFORE the contest starts, it is a problem of management to point out 1 Judge and throw the fate of how it emotionally and objectively impacts the participants into the grinder with it.
This seems to be one of the major things to point out that almost everyone agreed with me when i consulted with them.

Now the first suggestion starts with me is this:

""
If there is a shortage of judges then just don't make a contest. my suggestion can be to have a smaller set of an event that is a level below contests, and is used when people just want to get togather and brainstorm around making good content.

The judges there can be the participants themselves. i know this sounds scary but honestly if you and me were in an event like this and you make a melee faction and you do things better then me i'll be glad to point it out to you.

Also this unique event i'm suggesting that has no name yet doesn't really need a ranked best to worst judging, the fact that your work will be reviewed by other participants who are competent and skilled enough to part take in such an event is a reward itself.
""

What i will add to this now is that participants could vote for anyone in the contest besides themselves, i'll also point out that if you think this was is too dangerous i would like to disagree, Hive was always based on good human will, and the existing system isn't an exception. its only about the argument of refining where set good human will will thrive better.

Second suggestion i will be pulling from @Khyrberos echoing what he and his wife were saying in the link.

""
Basically this: Completely Do Away with the Judging system we now have, vis-a-vis "assigning a certain number of points out of a total to each entry individually, for each Criteria"

In it's place:

Institute a new system for Judges: for each category of Contest Criteria (e.g. Theme, Role, Balance, etc), the Judge simply lines up each entry from Best to Worst in that Criteria. Points will then be assigned automatically based on the rankings.

(e.g. "I the Judge have looked at Entries A, B, and C, and in terms of Balance I think B is the best, while C is better than A, so:

B: 3
C: 2
A: 1")

It's essentially Ranked Choice Voting, but for the Judges. I won't get into it now but it solves a number of issues & eases things for Judges, which I think we can all agree would be incredibly useful.
""

Would also like to insert a hybrid idea that can be attached to either the current system or Khyrberos's system or it should be possible to attach it to any system.

Why can't we use Medal Rewards and granting them to people not in a top 3 spot basis, but in a general basis. we can also have a lesser Sigil of participation.

So basically if 2 peoples work is ultimately extremely good and deserving of a medal, we need to realize that we can never truly create perfect judging so we should have a more forgiving system. not too much, but just... right.

8 people participate in a contest.

2 Earn the medals of Gold.
2 Earn the medals of Silver.
2 Earn the medals of Bronze.
2 Earn Sigils of participation.

Or some combination of that based on how people performed., you get the idea. i really think there is some proper suggestions here to draw from.

So a quick recap.

1 - Don't make contests with 1 judges, and reduce them into a lesser event incase of lack of Judges.
2 - Contest Participants are the judges since they have been talented and competent enough to take part in the first place.
3 - Have a basic A B C via vs via system.
4 - Remove the hard coded mindset of only 1 person taking first/second/third spot and add a small sigil of participation.

I also think there needs to be a standard example you need to achieve before participation.
I can give this example with an Icon contest, say what you have to make has to be on par or above 10 of the best looking ingame icons.

And i really think whenever the time comes Reforged and Classic should have split judging. there are things that look good or are optimal within one mode of the game hence they should only be compared to their base counterparts.

Thank you for reading the thread. i just hope i provided some proper suggestions.

 

[Footman16]

Recently there was a music contest that its final results, resulted in one of the best and oldest content creators in the server to leave

Personally I think this is a childish and ridiculously over-dramatic thing to do over an online contest on the Hive with no monetary value.

Having only 1 Judge for a contest is very bad, bias creeps in and incompetency shows itself

I'd agree with this having multiple judges is definitely ideal to ensure fair ruling however I feel like the second part of this sentence is a little unnecessary considering most judges in contests are relatively competent in their fields.

It's essentially Ranked Choice Voting, but for the Judges.

I don't see an issue with this, in fact I think this is probably a better way to do it.

2 Earn the medals of Gold.
2 Earn the medals of Silver.
2 Earn the medals of Bronze.
2 Earn Sigils of participation.

This imo is a bit much handing out medals for everything. Having a 1st 2nd and 3rd place works for the Olympics and I think it works for our contests here.

say what you have to make has to be on par or above 10 of the best looking ingame icons.

You talk about trying to eliminate subjectiveness and bias and then come up with examples of ridiculous rules like this... like how on earth could you possibly decide on something like this? And setting it as a requirement for a contest would exclude a lot of first time or new members looking to get into the competitions we host here on the Hive. Low quality entries have just as much place as a learning experience.

I can understand people's frustrations with judging and contests. I've entered one and worked briefly on entering another but eventually got side-tracked. In this contest I knew which entry would win before judgement even started and true enough it did because it was flashy but highly impractical imo too much weight was given to the aesthetics rather than the practicality. But there were 3 judges, there was a poll and I learnt a lot in the process of the contest and I never expected to come even within the top 3 but at the end of the day it won and the user that did win put in a lot of effort and so in that regard anyone who puts in a lot of time and effort into their entry has a "right" to win even if you may disagree with the specifics of the judgements themselves.

 

[StormKnight]

You talk about trying to eliminate subjectiveness and bias and then come up with examples of ridiculous rules like this... like how on earth could you possibly decide on something like this? And setting it as a requirement for a contest would exclude a lot of first time or new members looking to get into the competitions we host here on the Hive. Low quality entries have just as much place as a learning experience.

I can understand people's frustrations with judging and contests. I've entered one and worked briefly on entering another but eventually got side-tracked. In this contest I knew which entry would win before judgement even started and true enough it did because it was flashy but highly impractical imo too much weight was given to the aesthetics rather than the practicality. But there were 3 judges, there was a poll and I learnt a lot in the process of the contest and I never expected to come even within the top 3 but at the end of the day it won and the user that did win put in a lot of effort and so in that regard anyone who puts in a lot of time and effort into their entry has a "right" to win even if you may disagree with the specifics of the judgements themselves.

Ok i was simply trying to put some form of a basic standard, if you think its a bad idea then that's ok. you're probably right, i was thinking about a direction and also putting myself in the situation of a participant.
So if i was a participant i would have liked to know what is the basics of the standard i need to achieve before my competitors come out with their WIP's. that was my intention. but i guess its too exposed to bias.

This imo is a bit much handing out medals for everything. Having a 1st 2nd and 3rd place works for the Olympics and I think it works for our contests here.

It really isn't, Hive is not the Olymics, it can use a different system, and also, sometimes it might be easy for example, to decide who deserves a Gold medal, but not really easy on who deserves a silver one while bronze is obvious.

so the results can also be
1 guy gets gold
2 people get silver
1 gets bronze.

I think this is a highly compatible and convenient system that can grant Judges and Participants with the freedom and ability to more easily acomplish their task, and again i am not a fan of handing out medals everywhere, but if a certain situation exists when the skill gap and produced results are soo identical, i would rather the option to be there for such form of reward system then trust the judges for an "objective" review especially when it comes to music that while obviously there is good and bad, there is quite the bit of subjectivity in play.

I'd agree with this having multiple judges is definitely ideal to ensure fair ruling however I feel like the second part of this sentence is a little unnecessary considering most judges in contests are relatively competent in their fields.

I guess that's a good point, but managers then should be more careful on who they pick. sometimes its self evident they approved a less optimal judge.

I don't see an issue with this, in fact I think this is probably a better way to do it.

Eyy @khyrberos will be happy to hear that
honestly i think its a nice way too.

Personally I think this is a childish and ridiculously over-dramatic thing to do over an online contest on the Hive with no monetary value.

Sometimes it is sometimes it is justified. the times that it is justified its obvious whether or not its over dramatic or not. it shows itself. if you cold hardedly stick to this mindset and you're implying that Judges final decision is like "perfect", understand that the same competitive nature that drives people to part take in these contests may then be negatively affected by poor judgement. or good judgement. both is possible is what i'm saying.

 

[deepstrasz]

We're even lucky we have someone once in a while to judge contests, especially music ones. If that is the reason why he left and deleted ALL his resources with it (like someone else did, actually more but one in particular having to do with a contest), then I am afraid I have thought too high of him.
We all have troubles but this exaggeration is uncalled for.

 

[Footman16]

Sometimes it is sometimes it is justified. the times that it is justified its obvious whether or not its over dramatic or not. it shows itself. if you cold hardedly stick to this mindset and you're implying that Judges final decision is like "perfect", understand that the same competitive nature that drives people to part take in these contests may then be negatively affected by poor judgement. or good judgement. both is possible is what i'm saying.

I think you miss my point. I don't believe the judges are "perfect", far from it and we do demand a certain quality to their judgements. As I've pointed out there are times I've disagreed with the judges direction. My point is more about maturity and dealing with when we disagree with judges or whoever. To use an expression from my country I feel like this person has "Thrown the baby out with the bathwater". That is rather than trying to constructively change things, maybe talking to the judge or site administrators and trying to resolve this properly and diplomatically they've thrown a temper tantrum and robbed others of the opportunity to use their resources out of some sense of spite.

 

[deepstrasz]

@Storm Knight to be clear, are you particularly certain that it's because of the most recent music contest that he decided to split?
Consider that you have to know something about music to take JetF's side. The other composers didn't say that much. Bramble more than others. Jet only wrote three words and a dot.
I don't see that appropriate. If you have something to say with arguments do it.

 

[StormKnight]

We're even lucky we have someone once in a while to judge contests, especially music ones. If that is the reason why he left and deleted ALL his resources with it (like someone else did, actually more but one in particular having to do with a contest), then I am afraid I have thought too high of him.
We all have troubles but this exaggeration is uncalled for.

I don't have much else to say to this other then, i agree in the sense that i now understand your point of view not specifically but in a broader aspect.

@Storm Knight to be clear, are you particularly certain that it's because of the most recent music contest that he decided to split?
Consider that you have to know something about music to take JetF's side. The other composers didn't say that much. Bramble more than others. Jet only wrote three words and a dot.
I don't see that appropriate. If you have something to say with arguments do it.

I do know he acted this way because of the music contest but, no i didn't create this thread to continue on a tantrum that exactly why so little of the thread is about it and far more about spitballing ideas to improve the contest judging. but i will be honest i do think if the judgement system is improved the chances of this happening can indeed be reduced. its a 2 side effort, one is to the individual to not lose focus in the moment one is to the situation that is very much controlled and impacted/handled by others that affect the individual.

I'll also answer @Footman16 when i'm saying this, but sometimes people just feel powerless and completely ignored. obviously the right decision is to rather calmly point out their distaste, but i guess sometimes it just gets to you, especially when you feel alone, like nobody is around or nobody seems around at that specific time. but i guess i can say it is in those times that people tend to be extremely critical of how they are treated/judged, though sometimes some Judges could certainly use some better writing skills. and better ways of transforming their thoughts into words because sometimes it just doesn't end up reading good and it adds onto the emotional backlash of the affected people.

 

[Vinz]

If there is a shortage of judges then just don't make a contest.

a clarified stage where you NEED judges ready to rumble BEFORE the contest starts

Having only 1 Judge for a contest is very bad

it is a problem of management to point out 1 Judge

There were two judges, if you check the contest and poll threads. And then the pandemic came and made some people more busy than others.

Why can't we use Medal Rewards and granting them to people not in a top 3 spot basis, but in a general basis.

To the point where everyone has a sigil, and it's devalued, while people still clamor for the top spots and react negatively when they don't get there?

What i will add to this now is that participants could vote for anyone in the contest besides themselves

That's already in place.

not too much, but just... right.

8 people participate in a contest.

2 Earn the medals of Gold.
2 Earn the medals of Silver.
2 Earn the medals of Bronze.
2 Earn Sigils of participation.

Giving out participation trophies is what 'too much' means.

we need to realize that we can never truly create perfect judging so we should have a more forgiving system

With that in mind, perhaps we should also have a more forgiving outlook towards results.

but sometimes people just feel powerless and completely ignored

They must realize that they have a seat at the table - but it is left unused when they don't detail what they have in mind. If a contestant has issues with a judge's review, they must bring it up with the judge in the thread. If the judge sees they made a mistake, they may change their verdicts. Dialog is key. We should all be more comfortable with talking to each other.

That said, in a profile post, JetFangInferno said his resources will be restored, and that he may venture onto the judging side in the future. I look forward to working with him.

 

[Ralle]

Personally I think this is a childish and ridiculously over-dramatic thing to do over an online contest on the Hive with no monetary value.

The no money part just makes it even more clear that one can me emotionally invested and not monetary. You might see it as childish. I see it as a reaction like words but when words have failed.

The hardest part about hosting contests on Hive is finding judges. Last I checked (which is a while ago). Contests started without even a single judge in place. If we are to require 3 judges. We might as well just stop hosting contests.

some of you might be able look beyond your own nose and judge the quality of each other’s work. But as anyone can participate in contests, they will be swarmed with noobs who vote based on who they like the most. It will put a LOT of pressure on the staff to make sure there are no multi accounts.

I haven’t read Khyrberos’ suggestions in a while so I don’t remember them. But from what I see in this thread I’m quite sure it won’t work.

 

[StormKnight]

@Mythic Well i guess then trophy for participation should be ruled out of this idea and the model should only be used for the actual medals.
Participants already get some rep points so its not like they don't get anything. and experience working alongside other talented people.

With that in mind, perhaps we should also have a more forgiving outlook towards results.

Yep totally.

They must realize that they have a seat at the table - but it is left unused when they don't detail what they have in mind. If a contestant has issues with a judge's review, they must bring it up with the judge in the thread. If the judge sees they made a mistake, they may change their verdicts. Dialog is key. We should all be more comfortable with talking to each other.

Yeah that's exactly what i was hoping people to realize. but when they don't i somehow feel i should be the one to remind them that. and sometimes that gets out of hand.


Also, yeah i agree with Ralle on that part, when there is no monetary value, emotional and other self instilled values are left to replace the void. and we have to simply improve how this whole system works from time to time, because Hive being based on good human will has always been one of its biggest strengths.

 

[The_Silent]

Maybe higher judging "awards", to attract more judges, so we don't have to select whomever turns up. Seems like people don't really want to judge. And to be fair, it is a large task that takes time, so that makes sense. So create some larger incentive for them to do it.

"I judged 1/5/10/20 contests" or "Respected Contest Judge" User medals maybe.


Obviously, all of this is merely to increase the pool of judges. The second step is to hold them to a high bar, and not select those who repeatedly makes poor judgings.

 

[deepstrasz]

Seems like people don't really want to judge. And to be fair, it is a large task that takes time, so that makes sense. So create some larger incentive for them to do it.

I think it's not the time but the responsibility and how you will be reacted at. It takes more time to create something as a participant but then you don't have any responsibility than to participate. The judge, however will be stigmatized by some contestants if they don't agree with the criticism.

 

[Shar Dundred]

"I judged 1/5/10/20 contests" or "Respected Contest Judge" User medals maybe.

I see what you want to accomplish, but I am not sure if that measure would help the issue in the long run.
It may instead create a new fold between "old" judges and "new" judges once the system has been going a bit.
Users may be more likely to take results from "new' judges less seriously or even not seriously at all.
That may make it less likely, again, for new judges to come around.

 

[The_Silent]

It IS to create a pool of trusted judges, while still incentivising new judges to join. It would be amazing if we had a pool at all to be fair, so while I agree that it could become a problem, it would be a luxury problem compared to now where some contests have had to wait months in the past to just get one judge.

Then again, the users should judge the judgments on the content and critisism pointed out in the accompanying text, as they are the reasons for the scores, not who made them.

It takes more time to create something as a participant...

That depends entirety on the contest, judging every entry in a contest might very well take longer than the creation of one entry. It depends on the quality of each and the inherent complexity of the contest.

 

[deepstrasz]

It IS to create a pool of trusted judges, while still incentivising new judges to join. It would be amazing if we had a pool at all to be fair, so while I agree that it could become a problem, it would be a luxury problem compared to now where some contests have had to wait months in the past to just get one judge.

Then again, the users should judge the judgments on the content and critisism pointed out in the accompanying text, as they are the reasons for the scores, not who made them.

Music contests have been judged by people who were also contestants, therefore, people with know-how. Your idea would require such people to be active as random or site known users would not be competent to judge certain contest categories. I wouldn't want to see something like a model maker judge an icon contest when that person hasn't made any icons at all. ever.

 

[The_Silent]

No, I want qualified people who are not participating to judge. Like I do all the time. Not everyone qualified participates.in all contests as contestants. I want to lure people in who would not otherwise be bothered because of the workload.

 

[Shar Dundred]

It IS to create a pool of trusted judges, while still incentivising new judges to join. It would be amazing if we had a pool at all to be fair, so while I agree that it could become a problem, it would be a luxury problem compared to now where some contests have had to wait months in the past to just get one judge.

I am not calling your idea bad, I merely want to point out that it could lead to new problems later down the line if it is successful in the first place.

 

[The_Silent]

Yeah, I think most of those issues will be self mitigating, because the pool will probably never get very large. So new judges will have join all the time. Can't be beggars and choosers, etc. But yes, I see your point.

 

[Ralle]

I had an idea if we decided to go with more than one judge. They wouldn't be allowed to talk among themselves and the more their results differ from each other, the higher the contestant poll weighs in the results. But two judges is also hard to get.

I agree if we turn up the judging award it could help.

 

[deepstrasz]

They wouldn't be allowed to talk among themselves and the more their results differ from each other,

That's usually what happens, at least I know it's how I do it and nobody asked or gave me any info about their judging prior to the contest results. Also, you can't really monitor that, can you?

 

[Ralle]

Anyway. Back to the problem at hand. There were two judges, it got reduced to one.

I have conferred with Mythic, who manages the arena. We talked about the following:

All contests must have at least two judges. If at some point this is reduced to one or less and further judges cannot be found, a discussion with contestants must be started on how best to proceed. An alternative solution that makes sense can be found with the contestants. Off the top of my head are these ideas:

- The judge vote weight is reduced.
- The contestants manage to convince someone whom the staff couldn't to help judge.
- The contest goes on in the same framework with a single judge.
- They agree to let the community vote for some weight of the overall sum?

I dunno. I just like the thought that a discussion is started when we lose control over the situation rather than moving on and making decisions on our own .

 

[IcemanBo]

I have conferred with Mythic, who manages the arena.

Perhaps titles might be updated, as @Naze seems Arena Moderator. ( hello Naze )

 

[General Frank]

Why not also add the rules that contest hosts are not allowed to be participants of the contest?
It makes me feel uneasy and seems like gatekeeping from the host. We had that problem in the past and it went haywire.

 

[deepstrasz]

Why not also add the rules that contest hosts are not allowed to be participants of the contest?

Why is that a problem?

 

[General Frank]

Why is that a problem?

Someone could make up the rules and criteria to fit what they can do best and not the others.
Also, I think it is just a bad idea.

 

[StormKnight]

Why is that a problem?

Because if one wants to be host they could only host and participate in contests they think they can could win, and they would do that by premature long term preparing while for everyone else this might be a new surprise to prepare and look forward to.

Maybe it doesn't seem like a major issue but its sort of there i guess xP

Anyway. Back to the problem at hand. There were two judges, it got reduced to one.

I have conferred with Mythic, who manages the arena. We talked about the following:

All contests must have at least two judges. If at some point this is reduced to one or less and further judges cannot be found, a discussion with contestants must be started on how best to proceed. An alternative solution that makes sense can be found with the contestants. Off the top of my head are these ideas:

- The judge vote weight is reduced.
- The contestants manage to convince someone whom the staff couldn't to help judge.
- The contest goes on in the same framework with a single judge.
- They agree to let the community vote for some weight of the overall sum?

I dunno. I just like the thought that a discussion is started when we lose control over the situation rather than moving on and making decisions on our own .

That sounds like a really nice improvements. and i like the idea of allowing contestants to agree on that.

Going to also poke @Kyrbi0 into this conversation i guess, its a little late but his name from a discord server was different and i was under the impression he used same names.

On an end note would also like to say i don't think one my own suggestion wasn't really that bad but it did need a refinement so i'll rewrite it i guess.

Basically, don't treat Gold/Silver/Bronze medals like as if they are hard coded to 1 2 3 positions that only can be taken by one person each.
The result of a contest can be so:

1 Person gets Gold Medal
2 People get Silver Medal
1 Gets Bronze.

Or... literally any combination of that. This idea is mostly meant to combat situations where like someone doesn't fall down from a reward just because they got 5 points less from a 100 point based system. where another participant got 85, and he got 80. i believe in these very specific close call situations, its best to allow the judges to hand out 2 medals of the same rank to the participants. because the closer the participants come to eachother the less i trust the judges and its honestly an extremely difficult job to do also. so it will certainly feel relieving for judges be able to say, yeah you both deserve a Bonze/Silver/Gold medal because ultimately both of your works are very close to eachother.

But leave it as it is and i do see why is it hard to find judges at all, cuz its extremely difficult to get those close call situations sorted out.

I really hope you guys get my idea.

 

[deepstrasz]

Someone could make up the rules and criteria to fit what they can do best and not the others.

Indeed.

Because if one wants to be host they could only host and participate in contests they think they can could win, and they would do that by premature long term preparing while for everyone else this might be a new surprise to prepare and look forward to.

Great. However, that contestant could be trying for a challenge too.
When I vote in a poll for the contest theme, it's not because I am sure I can make a piece of music for that theme but because I either like the theme, it wasn't in any other contest before or I want a challenge, to try something new.

This is what I wanted, arguments.

1 Person gets Gold Medal
2 People get Silver Medal
1 Gets Bronze.

Now, we're at the Olympics?

 

[IcemanBo]

But staff makes rules and criteria, or at least approves them. I don't think it's meant host sets up such things alone.

 

[Kyrbi0]

(Thanks for pinging me, StormKnight)

I'm grateful to people like SK, fighting the good fight to improve matters here on the Hive (vis-a-vis Contests). I only wish I had more time & energy to assist.

I think it's fair to say there's agreement on one thing: the way Contests (& the Judging thereof) has gone in the past is substandard at best, untenable at worst. It's become an increasingly unsustainable situation.

However, something I've noticed just in the short bit of discussion here: already comments are beginning to diverge, to go off on tangents. I would submit, to have a chance at success, we need to really focus in on one particular facet at a time, each in it's own thread (perhaps with a 'master-list' thread with links to all of them). The fact is there are a number of possible solutions, but it's gonna be difficult to reason out the pros & cons if we all start going off on other, related-but-ancillary matters (e.g. improving Polls, the role of Hosts, improving Criteria, etc).

~~~

In order to stay on topic, then, I won't be commenting on the topic SK so kindly linked to, salient though it may be; except to say that perhaps that suggestion, properly implemented, will ease the job of the Judges to such a degree that other 'fixes' may not be necessary.

To wit (though again, these should probably be separate threads):
- We should definitely be strictly defining, for each Contest, whether it's Reforged or Classic. I don't believe we should be mixing them (i.e. each Contest picks one or the other, and all Contestants adhere to that).
- I don't quite understand the suggestion about "multiple medals", but I think I disagree (along the lines that it goes against common understanding & starts to become a meaningless "participation trophy" (which we already have in the form of "participation Rep rewards"))
- Having more Judges is obviously preferred/optimal, but the Reality is it's hard enough to find one, more often than not. We need to design & implement systems that compensate for that deficiency (both before (prevention) & after (triage)). To be quite honest, I think we need to look at even some long-standing customs & Rules and consider reworking things from the ground up.
- Some suggestions/systems are big enough to require their own thread, but a short-list:
- - Higher rewards to Judges (more Rep, maybe some kind of cumulative medal?)
- - Ease the burden of Judging ((sub-)rankings instead of raw points, splitting up Judging 'domains' for the larger/multidisciplinary Contests (e.g. art, code, design Judges for, say, a Hero/Techtree Contest), etc)
- - Create/encourage a "Judging Pool" of on-tap Judges
- - Create/encourage a system of "pseudo-Judges"; respected/publicly-chosen Users that can provide Judge-like reports for the entries
- - Greater utilization of the "Challenge" concept, perhaps modifying it into sort of a "mini-Contest" with few/no Judges required
- - Consider some kind of 'soft-skills' training course/regimen for potential/current Judges, to avoid causing undue distress (threading the fine line between Critique and Criticism)

~~~

Alright, I pushed through a headache til 3am to get this written so I'm stopping there.

 

[StormKnight]

Indeed.

Great. However, that contestant could be trying for a challenge too.
When I vote in a poll for the contest theme, it's not because I am sure I can make a piece of music for that theme but because I either like the theme, it wasn't in any other contest before or I want a challenge, to try something new.

This is what I wanted, arguments.


Now, we're at the Olympics?

I'm not exactly sure if i understand the "Now, we're at the Olympics?" question. but Footman16 earlier in the thread said that the current system in place is actually much like the Olympics while my suggestion is not, and i was thinking that, its good that my suggestion is NOT like the Olympics because Hive is not the Olympics and it should have a different system in place.

@Kyrbi0 i think you should scroll up above your own post, pass Icemanbo and deepstratz and read my post there. after Mythic also critisized my idea of participation sigil i thought its best to keep the idea except remove that participation part.

- - Ease the burden of Judging

Which is an alternative solution to this.

 

[Ralle]

We have implemented the following change:

1. All contests must have at least two judges. They cannot be started before this happens.
2. If a contest loses a judge. A conversation will be started among the contestants on how to proceed with the judging. This means that the contestants can agree to just let a single judge judge as normal or some change will happen. It's up for interpretation and just needs to be approved.

and

3. We are going to give 5-10 (maybe 15) rep per entry judged (to the judge) depending on the type of contest and the amount of work involved.

 

[deepstrasz]

I'm not exactly sure if i understand the "Now, we're at the Olympics?" question.

I don't agree with more medals of the same type. Not only does it kill incentive but usually there are few contestants anyway.

 

[Murlocologist]

If you want more fair scores, then you'll need to remove polls as well, because people tend to randomly vote, vote for friends or simply aren't qualified to cast a vote in the first place. But poll is kept to keep the community involved, so if the activity goes before fair scores, why bother with this?

Bigger disrespect towards contestants is waiting half a year to get results, then having a non-perfect review of entries delivered on time.

 

[deepstrasz]

Bigger disrespect towards contestants is waiting half a year to get results, then having a non-perfect review of entries delivered on time.

Indeed. People take this too seriously. We're not actually competing. It's more of a fun fair.

 

[Kyrbi0]

If you want more fair scores, then you'll need to remove polls as well...

Indeed. People take this too seriously. We're not actually competing. It's more of a fun fair.

And this fundamental difference of opinion about What The Point of Contests Are is why I wish to the Light I had had the time & energy to push more for the fulfillment of this topic.

I fear everything else to be decided hinges on this one Question; a Question we have as of yet failed to account for.

We have implemented the following change:

1. All contests must have at least two judges. They cannot be started before this happens.
2. If a contest loses a judge. A conversation will be started among the contestants on how to proceed with the judging. This means that the contestants can agree to just let a single judge judge as normal or some change will happen. It's up for interpretation and just needs to be approved.

and

3. We are going to give 5-10 (maybe 15) rep per entry judged (to the judge) depending on the type of contest and the amount of work involved.

I'm... glad to see quick action as evidence of concern, but I worry it might be too hasty.

While we already did #2 (nice to get it in writing) & #3 (Rep rewards for Judges have been a thing for years AFAIK), #1 just kinda makes things much harder for people. Like, I have a hard time seeing most Contests getting off the ground with this Rule in place. :<

 

[deepstrasz]

Like, I have a hard time seeing most Contests getting off the ground with this Rule in place. :<

Don't worry, most of the contestants will probably go with one judge not to wait too much on results.

 

[Kyrbi0]

Don't worry, most of the contestants will probably go with one judge not to wait too much on results.

That's apparently no longer an option.

1. All contests must have at least two judges. They cannot be started before this happens.

 

[Chaosy]

2 - Contest Participants are the judges since they have been talented and competent enough to take part in the first place.

I am not invested enough in the subject to care much but this is not a good idea.
Just joining a contest does not make you a competent judge. Nor does it mean that you want to.

There have been contest entries that score 20/100 in total and you want such a person to judge others? this might be fine for maps, but for pretty much all other contests I would not trust this person to do a proper job. It's kinda like letting me be a judge for the icon contests without being able to draw. Yet I could technically join with some paint stick figure entry and become a judge.

Additionally, isn't the judge usually taken from the talent pool anyway? it's not like Ralle comes in to judge contests because they had issues finding a judge.
It's often someone who would like to compete but does not have the time right now.
And judging takes less time compared to making an entry I would say. (not that they have a deadline anyway and take months to submit their judging *cough*)
____________________________________________

I have an issue or two with judging myself but nothing that overlaps with your concerns, as far as I can tell.

 

[Ralle]

Which brings number 3. More incentive for people to judge.

We'll see how it goes. Everything can be changed if it won't work. But I wouldn't mind contests being more rare and special.

 

[deepstrasz]

What about we make this as anonymous as possible?
Basically, the judges won't know the entry author until after they give their judging. Voters will not know until the results.
Someone, a moderator will receive the entries with no author names and will pass them to the judges and/or post them on the contest thread.
It will mean excluding WiPs from the contest thread but that's not that important ultimately.

 

[IcemanBo]

• [Poll] What's the point of a contest?
  
• [[[The Prime Question]]] What is the Point of a Contest?
  
• Regarding Contests and the New Teamwork Ruling

^People enjoyed open communiation / fun in context of contest. Maybe it's too risky to cut it off, by introducing 'silent contests' with making anonymous voting and judgement.

 

[Kyrbi0]

What about we make this as anonymous as possible?
Basically, the judges won't know the entry author until after they give their judging. Voters will not know until the results.
Someone, a moderator will receive the entries with no author names and will pass them to the judges and/or post them on the contest thread.
It will mean excluding WiPs from the contest thread but that's not that important ultimately.

Important to who, exactly? : )

This is a great example of what I was getting at in my "Prime Question: What is the Point of a Contest" thread & Poll, which...

• [Poll] What's the point of a contest?
  
• [[[The Prime Question]]] What is the Point of a Contest?
  
• Regarding Contests and the New Teamwork Ruling

^People enjoyed open communiation / fun in context of contest. Maybe it's too risky to cut it off, by introducing 'silent contests' with making anonymous voting and judgement.

... I appreciate your posting for me, IcemanBo. : )

Basically, the tactic you describe really goes a long way towards leaning a Contest in a more 'Competitive' direction (for all the reasons you pointed out, and more; Judges don't know, Voters don't know & aren't biased, etc etc... Also no WIPs means no one is getting "extra special reviews" so that knocks out the whole "is it Teamwork" debate...).

However, this tactic takes a heavy toll on the "Fun"/"Eudaemonia" sides of Contests. For people who value those more than "Competitiveness", full anonymity is totally backwards & dumb. No WIPs to talk & get excited about, no exchange of valuable feedback to improve as a modder, trading in FUN for SUPER SRS BSNS GUYS anonymous Polls & secret messages... Heck the Contest thread would basically be a ghost-town instead of the (temporarily-)thriving hubs of conversation they currently are.

~~~

Let me be clear: neither of those is objectively more important than the other. If people want do have a "super serious Test of Skill" Contest, then by all means run it (& likely that way (with full anonymity))! But (and this is where IcemanBo's comment comes in), we've already asked that Question & while the results weren't 100% conclusive, it's definitely certain that a lot of people value the community-building/resource-enhancing/fun-having/conversation-building aspects of Contests the way they are currently run.

All along I've just wanted us to get an idea of What The Community, on the Whole, Wants out of their Contests. Whether we/the majority generally wants more Competitiveness, more Fun, or more 'Eudaemonia', it doesn't matter; we just need to figure it out & then Play That Up in our Rules, our Criteria, our Judging Guidelines, our Polls, etc etc etc. It's the fundamental Question that, I believe, determines the stance we take on all other questions regarding Contests & how to best accomplish them.

 

[IcemanBo]

Important to who, exactly? : )

To be fair, the WIP rule has no high technical value. It's idea is to prevent starting before official start, with showing contestants are making progress. WIPs can be an indicator, but can't truly detect cheaters.

If it's in rules or not, posting WIPs automatically welcomes others for communication and feedback, which is a great social aspect. Anonymous contests would not solely cut off the rule, but also exclude the possibility for those social interactions.

 

[deepstrasz]

^People enjoyed open communiation / fun in context of contest. Maybe it's too risky to cut it off, by introducing 'silent contests' with making anonymous voting and judgement.

Anonymous voting may mean that we don't know who the voters are. I'm saying, the contestants should be anonymous so we eliminate the popularity issues.

Important to who, exactly? : )

To those watching the contest thread.

Basically, the tactic you describe really goes a long way towards leaning a Contest in a more 'Competitive' direction (for all the reasons you pointed out, and more; Judges don't know, Voters don't know & aren't biased, etc etc... Also no WIPs means no one is getting "extra special reviews" so that knocks out the whole "is it Teamwork" debate...).

Sure but then we shouldn't cry over contest results anymore, at least not so much.

Only problem is now most of us know the styles of the others.

However, this tactic takes a heavy toll on the "Fun"/"Eudaemonia" sides of Contests. For people who value those more than "Competitiveness", full anonymity is totally backwards & dumb. No WIPs to talk & get excited about, no exchange of valuable feedback to improve as a modder, trading in FUN for SUPER SRS BSNS GUYS anonymous Polls & secret messages... Heck the Contest thread would basically be a ghost-town instead of the (temporarily-)thriving hubs of conversation they currently are.

Not really. WiPs could be posted by the moderator receiving them but just without mentioning who they belong to.

If it's in rules or not, posting WIPs automatically welcomes others for communication and feedback, which is a great social aspect. Anonymous contests would not solely cut off the rule, but also exclude the possibility for those social interactions.

As mentioned above, this is still possible.

Anyways, I don't want this necessarily. I'm just tired of such threads. From my point of view, people should just take criticism and understand that not everyone judges the same way. And even two judges don't guarantee results would be that much different.

 

[IcemanBo]

Using moderator as medium to post WIPs seems a bit unpractical. Also the communication /feedback would be pretty much into the void / one way.
For "anon discussion" I would prefer to have open submissions like it is, with usual risk of popularity at poll.

I'm just tired of such threads.

Kind of, too. We are talking so much, but it's not worth if Hive as site isn't prepared, not interested itself, or not responsive enough to work out little steps. Good ideas are bloated away with mass text and time.

 

[deepstrasz]

Also the communication /feedback would be pretty much into the void / one way.

There will also be the risk of knowing who made what when those users won't comment on their own pieces.

 

[Ether]

I think one should comment first before they can cast a vote. Or cast a review before casting vote.
Like unlocking a thing after dropping a comment.

 

[Footman16]

In my opinion we'll never have a perfect system and if we make it too complicated or change it too much we'll end up making the whole thing less accessible to people.

I dont know if theres a perfect solution but what we have just now works well enough. It still needs some tweaks like the minimum judge limit etc but the main thing is that it has worked and still does for most people.

 

[deepstrasz]

I dont know if theres a perfect solution but what we have just now works well enough. It still needs some tweaks like the minimum judge limit etc but the main thing is that it has worked and still does for most people.

Not to be rude here but it looks like we're usually having these debates for the minority.