Arena Contests Changes

[Naze]

Ideas spawned in the other thread and I arbitrarily narrowed them down to these few topics, as the discussion was too wide and inconclusive. Vote on what you think is a viable change here. Voice your opinion leaving a post below, and you may give other suggestions too. If you aren't satisfied with the current arena, please participate.

What is the point of any given contest?
(Ofc this isn't in the poll, but this is relevant to the discussion)
Is it to have fun, to increase the number of resources, to engender community, or as 'tests of skill' to determine who is best at something? Or some combination?

[1st topic] Obligatory minimum of 2 judges.

"What if there isn't another judge?" We'll discuss this scenario in case this topic is voted enough.
​

[2nd topic] Increase Public Poll decision power to 33%:

Judges: 66%; public poll: 33%.
​

[3rd topic] Same percentages and poll rules for all contests:

Same percentage of judge/public poll ratio in final score;
Same rules for applying multiple-choice or single-choice polls in all contests. Right now we don't have rules for this.
​

[4th topic] Justified Public Poll votes:

Votes in Public Polls will only be considered if the user leaves a comment, encouraging people to actually bother analyze the entries.​

 

[Devine]

Voted for 1, voted against 2 & 3.
I'm fine with 75%-25% (2) and standardizing will lead to restriction (3).

 

[Mister_Haudrauf]

What is the point of any given contest?

Generally it is a way to test your skill. BUT mostly see it either as a way to have fun, while others see it as a way to increase the number of good quality resources. Of course there are always members/people want to engender the community in some way.

So it depends on 'Point of View'.

[1st topic] Obligatory minimum of 2 judges.
"What if there isn't another judge?" We'll discuss this scenario in case this topic is voted enough.

I am highly for that. There should always be 2 judges. Have you ever seen a contest in real life with only one judge?

[2nd topic] Increase Public Poll decision power to 33%:
Judges: 66%; public poll: 33%.

While this sounds good for the public. It has it's downsides. So against that.

[3rd topic] Same percentages and poll rules for all contests:
Same percentage of judge/public poll ratio in final score;
Same rules for applying multiple-choice or single-choice polls in all contests. Right now we dont have rules for this.

Against that. Public votes can be manipulated somehow.

[4th topic] Justified Public Poll votes:
Votes in Public Polls will only be considered if the user leaves a comment, encouraging people to actually bother analyze the entries.

That's actually a good idea. Why not? I am for that.

 

[Naze]

Alright people, thank you for participating! I added the [4th topic] Justified Public Poll votes option, a suggestion by @Direfury that I somehow missed in the other thread.

This recently added option shouldn't be a problem since you should be able to vote again and even change votes if you make up your minds.

 

[Deleted member 219079]

[1st topic] Obligatory minimum of 2 judges.
"What if there isn't another judge?" We'll discuss this scenario in case this topic is voted enough.

It'd be nice to have multiple judges per contest, to see my entry get mutually bashed and not have it feel like personal feud.

[2nd topic] Increase Public Poll decision power to 33%:
Judges: 66%; public poll: 33%.

Or judges 100%.

 

[Devine]

Sounds like #4 will be a hassle and will discourage people from voting in the first place.

 

[Lord_Earthfire]

Well, i would like to see the poll percentages to be even more reduced, i always see in triggering contests that e.g. the better presented entry gets the highest vote. But i could see it at 67/33% when option #4 would be introduced.

Edit: I see option 4 as a way to encourage people to actually download all recources and test them.

 

[Kyrbi0]

It hurts to be so busy at a time like this. I promise I will be back to comment.

 

[deepstrasz]

Obligatory minimum of 2 judges.

No. We don't need to prolong contests for unnecessary periods of time or "force" a contestant to shift.

Increase Public Poll decision power to 33%:

Would be the best of the four for me especially if there's only one judge per contest.

Same percentages and poll rules for all contests:

Total equalization isn't really helping against friend votes.

Justified Public Poll votes:

No way. Sure, it's sassy to see it but making it compulsory would kill a lot of votes.

 

[Murlocologist]

The only real problem with contests is the await time between poll and results, and submitting contests to any of changes listed wouldn't resolve or reduce that.
All these options in poll are just a matter of quality assurance editing that may or may not bring any benefit to contests. If you want to make real improvements to contests, deal with its actual problems.

 

[Devine]

The only real problem with contests is the await time between poll and results, and submitting contests to any of changes listed wouldn't resolve or reduce that.
All these options in poll are just a matter of quality assurance editing that may or may not bring any benefit to contests. If you want to make real improvements to contests, deal with its actual problems.

Quality assurance is important. However, this is why I suggested getting the non-expert judges to fill in the space of a judge who bails. It's not that judging takes several months, it's that you need to find a judge that will commit to doing it, within a timeframe, and who is actually good for it. The more people you ask from the start, the more will actually finish in time. Also, I don't think the real problem is the time thing. It's the elitism on this site that makes it difficult for its members to be judged by someone who isn't pr0. There aren't many pr0s and if the pr0s don't want to finish in time, there is nothing you can do about it. There is plenty of members who have been on this site for years and years, who know exactly what a quality resource is, but haven't ever made one. They are community friendly people, who will try their best to get the judging done in time. I'm sure of it. I don't see why everybody thinks so highly of their own skills they can't let someone with less skill than them judge the quality. You can even check the quality of the judging as a host. If there's bullshit in the judging, ask another member. My point is that you don't have to be so uptight about the skill level of the judge, you have to be uptight about finishing in time, and about a somewhat democratic result.

 

[Daffa]

I am convinced with the first one, but for the rest is still not convincing enough.

 

[GhostWolf]

The only real problem with contests is the await time between poll and results, and submitting contests to any of changes listed wouldn't resolve or reduce that.

Just looking at what people posted in these two threads shows that there is a question about objectivity, and trying to make contests more objective. While what you wrote is partially a cause for this problem, it's not the only one.

 

[Shar Dundred]

I voted for 1 and 3.
I think that 2 judges should be standard, if possible and I also agree with standartising the poll-judge-ratio.

I disagree with 2, I think 25-75 is fine.

I am neutral towards 4. Of course, it SOUNDS good, however I see three major problems with it.
a) It could discourage people to vote at all and also decrease the number of meaningful votes.
b) I agree that people should give as much feedback as possible, but you cannot expect a user like me to make a good review of a custom model, since I lack knowledge in that area.
That means you can hardly add a rule that forces a certain quality of feedback without alienating people.
And THAT means that comments like "I vote this because I like it the most" have to be considered legit to justify a poll vote.
On the one hand, there really isn't much wrong with that post, on the other hand, however, the only feedback it provides is that this particular user liked this entry the most. No reasons given.
c) This also means that moderators will have to go through an entire thread just to see whose poll vote is valid and whose is not. This sounds like a minor issue, and maybe it is, but I can imagine that browsing several pages in a thread with maybe about 10 pages could get tiresome after some contests.

 

[Kyrbi0]

There is one major problem with a polled thread being constructed in this way, @Naze (though praise be unto you for doing it at all): aside from reading the thread, there's no way to see who disagrees with a given question. Unfortunately, the only real solution is to break each question out into it's own thread, with "agree"/"disagree" add Poll options (if not a wider spectrum: i.e. "strongly agree, agree, neutral, disagree, strongly disagree")

c) This also means that moderators will have to go through an entire thread just to see whose poll vote is valid and whose is not. This sounds like a minor issue, and maybe it is, but I can imagine that browsing several pages in a thread with maybe about 10 pages could get tiresome after some contests.

Sure, but Poll threads rarely get that long (especially if they are curated to remove extraneous/off-topic posts).

 

[Shar Dundred]

Sure, but Poll threads rarely get that long (especially if they are curated to remove extraneous/off-topic posts).

Hence I pointed out that this might be a minor issue, however, can you be so sure about it staying like that when posts are made essential to justify a poll vote?
Regardless, my other points stand.

 

[DD_legionTN]

Voted for 1, 2 and 4.

[1st topic] Obligatory minimum of 2 judges.

Minimum of 2 judges could helps into making more fairer judging, which could helps people to be more satisfy toward the final judged scores.

[2nd topic] Increase Public Poll decision power to 33%:
Judges: 66%; public poll: 33%.

[4th topic] Justified Public Poll votes:
Votes in Public Polls will only be considered if the user leaves a comment, encouraging people to actually bother analyze the entries.

Personally, I'm against 2. But, 2 is justified if people are needed to leave a comment for their review, thus I don't see why not increase the vote decision power since judges and voters are both judging contestants entries. Of course, 4 has it downside, as Shar Dundred mentioned,

but you cannot expect a user like me to make a good review of a custom model, since I lack knowledge in that area.

And hence would limits it to skilled voters who know how to review the contestants entries. But could leave the discussion later if this vote is considered.

[3rd topic] Same percentages and poll rules for all contests:
Same percentage of judge/public poll ratio in final score;
Same rules for applying multiple-choice or single-choice polls in all contests.

Not really agree with, different type of contests focus on different aspect, and hence affecting the justified votes.




Here I would propose a 5th choice : multiple votes. Sometimes there can be more than 1 entries with amazing work. Forcing a people to choose between 1 is hard, and I would say why not make people vote the the entries they feel with same quality. Of course, the multiple votes would not be unlimited votes, it would have limit. Limit the votes to 1 ~ 3 depend on numbers of entries would be enough.

 

[Kyrbi0]

Here I would propose a 5th choice : multiple votes. Sometimes there can be more than 1 entries with amazing work. Forcing a people to choose between 1 is hard, and I would say why not make people vote the the entries they feel with same quality. Of course, the multiple votes would not be unlimited votes, it would have limit. Limit the votes to 1 ~ 3 depend on numbers of entries would be enough.

Multi-vote polls is an important element that really hasn't been discussed up till now. I wholly agree; in fact, I'd go so far as to say that if there are more than 3 options, 2+ votes is almost necessary. I'd go even farther & say we should utilize preferential/ranked voting systems.

 

[Freddyk]

I voted for 1 and 3, though this poll doesn't address to more important (from my point of view) aspect of the discussion we had on the other thread: who can be judge?

To clarify my thoughts:

• Judges:
  • rating based on deep analysis of all entries
  • requires some time to be done properly (check all anims, special effects + assess quality of mesh, geosets)
  • requires some knowledge in contest category (modelling/texturing/whatever)
  • provides useful critics for contestants
• Poll:
  • choose your favourite(s) out of all entries
  • requires few time (quick test in-game or in model editor)
  • can be done by anybody
  • useful to avoid elitism and allow the whole community to participate

From this point of view, poll should not count too much and should not be complicated (no need for justification).

On the contrary, we should encourage people to judge properly, and thus accept more judges (for example, we could say that anyone with at least 2 ressources with at least 4/5 rating in contest category, are valid judges).

 

[Kyrbi0]

Good points.

(for example, we could say that anyone with at least 2 ressources with at least 4/5 rating in contest category, are valid judges).

This might work for the artistic/resource-based Contests... But what about Hero/Techtree Contests?

 

[Daffa]

Techtree Contest can be done if the judge is a Mod maker (e.g Kam) or has done multiple altered races in the past (e.g Wa666r). They should sufficiently fits the matter.

As per Hero contests... this one gets messy.

 

[Devine]

Techtree Contest can be done if the judge is a Mod maker (e.g Kam) or has done multiple altered races in the past (e.g Wa666r). They should sufficiently fits the matter.

As per Hero contests... this one gets messy.

Can you elaborate? I have no idea what the situation is with hero contests.

 

[Rheiko]

In my opinion, contests are both for fun and a test of skill. See how far you've improved and how much you can compete with others.

[1st topic] Obligatory minimum of 2 judges.

Agreed with this. 2 or more judges can make the judgement of entries more.. fairer, i think.

[2nd topic] Increase Public Poll decision power to 33%:
Judges: 66%; public poll: 33%.

No, don't. I'm thinking about decreasing it even more instead. 75-25? Voting score already takes too much role whenever the judgement score is tight and I think that is enough. Since I believe many people consider contests as a place to test their skills, winning through vote doesn't make you superior in term of skills comparison. For me, at least. So yea, against this one.

[3rd topic] Same percentages and poll rules for all contests:
Same percentage of judge/public poll ratio in final score;
Same rules for applying multiple-choice or single-choice polls in all contests. Right now we don't have rules for this.

Against this. Every contests has different aspect which one must pay attention to and I think it's best if we let each contest has their own fitting poll rules.

[4th topic] Justified Public Poll votes:
Votes in Public Polls will only be considered if the user leaves a comment, encouraging people to actually bother analyze the entries.

I'm not sure about this one, I guess I'm on neutral side since it has its downsides despite being a good thing to encourage people.

 

[Kyrbi0]

Techtree Contest can be done if the judge is a Mod maker (e.g Kam) or has done multiple altered races in the past (e.g Wa666r). They should sufficiently fits the matter.

As per Hero contests... this one gets messy.

That makes the pool of potential Judges incredibly slim (like, beyond those 2... Who?). For example, I've begun to consider Judging in such Contests (only certain ones, and only if not competing, etc)... But I don't have a single resource to my name (except that stupid mask, lol). While I'm sure many would know me as a *potential* race-maker, I wouldn't fit the criteria.

 

[Hantoo]

1. Obligatory minimum of 2 judges

to avoid any bias, for example the one judge's preferences are more toward a certain race, which might or might not unconsciously give the contestant an advantage, and ya know, personal preferences and stuff could makes things a little unfair.


2. Increase Public Poll decision power to 33%

im against this, purely because the vote will most likely be one sided, usually the 2-3 best ones get votes while left the decent one or average one nearly 0 to super little vote. Increasing the poll will make the "marks" to be very very unbalanced.

3 and 4 i don't really get what are those mean

 

[Daffa]

Can you elaborate? I have no idea what the situation is with hero contests.

There's no simple way to define Hero contest judges by resources. A spell maker doesn't qualify that area easily, despite being the closest to Hero contest. We can however, rely on past contests, but that would be quite a dig.

No, don't. I'm thinking about decreasing it even more instead. 75-25? Voting score already takes too much role whenever the judgement score is tight and I think that is enough. Since I believe many people consider contests as a place to test their skills, winning through vote doesn't make you superior in term of skills comparison. For me, at least. So yea, against this one.

Say one who wins via polls vote
In a more serious note, I agree with not increasing poll power. However, I'm not that much of supportive in reduction either. I go with neutral and follows 70-30 for now. I know I lost by a poll vote in one of my days, so I'm a bit bias with this one

That makes the pool of potential Judges incredibly slim (like, beyond those 2... Who?). For example, I've begun to consider Judging in such Contests (only certain ones, and only if not competing, etc)... But I don't have a single resource to my name (except that stupid mask, lol). While I'm sure many would know me as a *potential* race-maker, I wouldn't fit the criteria.

True that. It's pretty limiting. I can name a few others, but it won't justify much anyway. The number is too thin.

I for example, used to work as part of Map Reviewers social group back in the day (2014 or something). However, none my resources fits the condition of 4/5. I can name a few others as well with such regard. I know the standards of such reviewers are different now, but I mean, come on.

 

[Rheiko]

Say one who wins via polls vote

Though I accepted my victory, I didn't find it quite satisfying.

 

[Freddyk]

Well my proposal was just a basic idea to initiate the debate. the most important is to find more judges.

Another idea: Having participated in at least 2 contests of said category, with a rating of at least 60% for your entries, would qualify you for judging.

Of course the numbers in these proposed rules can be discussed and could even be different depending on category, if necessary...

 

[Daffa]

@Freddyk or have both conditions in an OR block. So we can get more potential judges.

The problem is the skilled DO NOT ALWAYS WANT TO JUDGE. They sometimes want to partake too.

 

[Mister_Haudrauf]

Well my proposal was just a basic idea to initiate the debate. the most important is to find more judges.

Another idea: Having participated in at least 2 contests of said category, with a rating of at least 60% for your entries, would qualify you for judging.

Of course the numbers in these proposed rules can be discussed and could even be different depending on category, if necessary...

While that would be good. There's also the problem that some members (that are really good in the Category) simply don't want to participate in contests, because it's either against their nature to compete against others or they don't have the time and the will for that.


I, for example, don't want to participate in any contests. No matter what. But i like to "watch" the contests, while i make my own stuff.

 

[DD_legionTN]

(for example, we could say that anyone with at least 2 ressources with at least 4/5 rating in contest category, are valid judges).

With this, it does actually prove that the judge is skilled. But on the other hand, doesn't mean those people who have less than 2 resources or no resources submit at all is not skilled. Simply because reviewing standard is changing time by time (spell would mostly be maintained, but map/icon/model might not)

Example like me, even I have not submitted a single resource, but I know coding well (at least I feel like)

Having participated in at least 2 contests of said category, with a rating of at least 60% for your entries, would qualify you for judging.

And some people prefer to be contestants rather then judge.

And also, people who can do good in an aspect doesn't mean he can judge properly, I would say they are different things.


Finding a judge is all depending on the one if one is willing to judge, and can't be forced. No matter how justified in our rules, it just simply depending on one's willingness, if one would have time and interest for judging the contest.

 

[Rheiko]

With this, it does actually prove that the judge is skilled. But on the other hand, doesn't mean those people who have less than 2 resources or no resources submit at all is not skilled.

Indeed. Then maybe all we need from this kind of people is just some proof that they have what it takes to be a judge.

Finding a judge is all depending on the one if one is willing to judge, and can't be forced. No matter how justified in our rules, it just simply depending on one's willingness, if one would have time and interest for judging the contest.

Yes, but still, I think it is really preferred to have a judge who have both criterias. One who has the will to judge and also abide by the rules.

 

[Freddyk]

When a contestant receives a review from a judge, I think it's important that the judge is recognized as a skilled person, whose advice has actually some value because he knows what he is talking about.
I feel completely normal to enter in the community as a maker of ressources, before becoming a judge.
If people can see what the judge is capable of, they will give more attention to their reviews.

 

[Devine]

When a contestant receives a review from a judge, I think it's important that the judge is recognized as a skilled person, whose advice has actually some value because he knows what he is talking about.
I feel completely normal to enter in the community as a maker of ressources, before becoming a judge.
If people can see what the judge is capable of, they will give more attention to their reviews.

I think a lot of modders are capable of giving valuable feedback. I don't think you need the best modder's review to learn something about your art, or about how people see your entry.
Just objective opinions by the kind of people who will use your models/icons/skins should be extremely valuable anyway -> Eg. The consumer.
I don't get why people don't want to be judged by someone with less skill than them, most skilled modders from the past have left this place anyway.

 

[fladdermasken]

Things shouldn't be so convoluted. AFAIC that's part of what killed The Arena off in the first place. A contest now usually runs for at least 6 months, including theme discussion, theme polling, endless extensions, judging etc.

It'd be good to take a step back and host them more like they were hosted back when Pyritie ran it. Wrap it up quickly, and move on to the next one.

Skip theme discussion in its entirety and let the host/moderator just pick a theme and run with it. Too much time is wasted squabbling over the theme, and all good suggestions are usually posted within the first few days anyways.

Let the contest run for no more than a month, with no extensions. If it's wrapped up quickly, you don't need to worry about missing out because you can just sign up for the next one straight after.

Get the poll ready before the contest closes, and post it straight after it does. The poll can run for 2-3 weeks, while the judges review the entries.

Post results. Rinse and repeat.

 

[Murlocologist]

I agree, one contest shouldn't be longer than: 1 or 2 months for contestants to make entries + 1 or 2 weeks poll + maximum of a month await for judge reviews. But repeating problem is the last, judging. Because it happens to take months and months for judges to post their review. And that is the biggest problem of Arena that should be discussed before all the things put in poll. What is the point in having two judges if it will take half a year for them to invest 2 hours into writing review? After contest is delayed for so long, they may just as well cancel the whole thing, it is pointless and ridiculous to have such contests.

 

[Naze]

Yeah, but we can't go to their homes and force the judges to sit down and write their reviews. What do you suggest for this? A solution I can see is that we should be able to find another judge if the assigned one fails to deliver in time.

 

[Murlocologist]

Yeah, but we can't go to their homes and force the judges to sit down and write their reviews. What do you suggest for this? A solution I can see is that we should be able to find another judge if the assigned one fails to deliver in time.

Mind you, I actually created whole thread about that.

If judge shows no work in progress on judging in one month time after the end of poll he should be replaced by different person.

...if one has not got time to deal with judging, than he should not accept to judge...

...if judge realizes he won't make it in one month time, he needs to notify host so he can be replaced by different person...

...if he goes inactive without telling host he will not be able to finish his task he should be replaced by different person and receive -rep...

Overall idea: making rules about judges less arbitrary.

 

[Naze]

Mind you, I actually created whole thread about that.

Nice. I didn't see that back in October.

Well, since this slow judging problem was already discussed in that thread with the community, I don't see why we should discuss it again. Now let's continue that with the staff work (I already mentioned you there).

 

[KILLCIDE]

Our Arena Moderator, @Rufus, should be all over this!

 

[Freddyk]

What if we stop "appointing" judges and instead:
- let anyone who is willing, and who matches criterias, become a judge and submit reviews and rating within 1 month after contest end
- after one month, reviews and ratings received by the host are used to calculate the results

The idea is to have as much judges as possible, so even after one or two people have expressed their will to judge and been validated by the host, that doesn't prevent other people to do the same. At the end, even if all these people don't make it in time, hopefully at least two of them will do...

 

[Naze]

Well this is a good idea, even though some judges have been voluntarily applying to the job, like you described, since always. The difference would be that now we're going to officially endorse this behavior. I don't think this would hurt. Obviously this is no guarantee that we're gonna have enough judges, but it may help. I just don't think this will be enough for us to stop appointing judges, however we can do both.

 

[Devine]

What if we stop "appointing" judges and instead:
- let anyone who is willing, and who matches criterias, become a judge and submit reviews and rating within 1 month after contest end
- after one month, reviews and ratings received by the host are used to calculate the results

The idea is to have as much judges as possible, so even after one or two people have expressed their will to judge and been validated by the host, that doesn't prevent other people to do the same. At the end, even if all these people don't make it in time, hopefully at least two of them will do...

That honestly sounds wonderful.

@Murlocologist I'm not really sure if -repping a failed judge is the way to go, but otherwise I get the gist of it. To be honest, I think we should cover ourselves for a falling out. That's the only way as far as I can see.

 

[Kyrbi0]

What if we stop "appointing" judges and instead:
- let anyone who is willing, and who matches criterias, become a judge and submit reviews and rating within 1 month after contest end
- after one month, reviews and ratings received by the host are used to calculate the results

The idea is to have as much judges as possible, so even after one or two people have expressed their will to judge and been validated by the host, that doesn't prevent other people to do the same. At the end, even if all these people don't make it in time, hopefully at least two of them will do...

That's... Actually surprisingly a really good idea. Quite intriguing. It actually kills 2-3 birds with one stone, in a way. I'll have to speak more in this later, though.

"Matches criteria" is on of the most important parts, though, IMO. I've had some real "fun" experiences with Judges that did not, or even made up their *own* Criteria. :<

 

[fladdermasken]

Or, if it gets drawn way out of proportion and nobody delivers, just drop the judging for that contest and let the poll decide the winner. Better luck next time. It's better than letting it drag on for another six months.

You do with what you have, and the dearth of reliable judges has been an obstacle for as long as this has been talked about. Get things moving instead.

 

[Kyrbi0]

Or, if it gets drawn way out of proportion and nobody delivers, just drop the judging for that contest and let the poll decide the winner. Better luck next time. It's better than letting it drag on for another six months.

You do with what you have, and the dearth of reliable judges has been an obstacle for as long as this has been talked about. Get things moving instead.

Don't really love that, to be honest. But I can understand the reasoning behind it, and if we indeed had Contests running more frequently as a result (& were OK with reusing Contest Themes in general), might not be a huge issue.