1. Vote on the best bounty the oceans have to offer! The poll for the 34th Modeling Contest has begun.
    Dismiss Notice
  2. Awaken what lies in the heart of your swarm. The 17th Techtree Contest has arrived!
    Dismiss Notice
  3. Check out the Staff Job Openings thread.
    Dismiss Notice
Dismiss Notice
Hive 3 Remoosed BETA - NOW LIVE. Go check it out at BETA Hive Workshop! Post your feedback in this new forum BETA Feedback.
Dismiss Notice
60,000 passwords have been reset on July 8, 2019. If you cannot login, read this.

Contest Judging Needs A Change.

Discussion in 'Site Discussion' started by Storm Knight, May 14, 2020.

  1. Ralle

    Ralle

    Owner

    Joined:
    Oct 6, 2004
    Messages:
    7,723
    Resources:
    21
    Tools:
    2
    Maps:
    5
    Tutorials:
    14
    Resources:
    21
    Anyway. Back to the problem at hand. There were two judges, it got reduced to one.

    I have conferred with Mythic, who manages the arena. We talked about the following:

    All contests must have at least two judges. If at some point this is reduced to one or less and further judges cannot be found, a discussion with contestants must be started on how best to proceed. An alternative solution that makes sense can be found with the contestants. Off the top of my head are these ideas:

    - The judge vote weight is reduced.
    - The contestants manage to convince someone whom the staff couldn't to help judge.
    - The contest goes on in the same framework with a single judge.
    - They agree to let the community vote for some weight of the overall sum?

    I dunno. I just like the thought that a discussion is started when we lose control over the situation rather than moving on and making decisions on our own .
     
  2. IcemanBo

    IcemanBo

    Joined:
    Sep 6, 2013
    Messages:
    6,085
    Resources:
    23
    Maps:
    3
    Spells:
    11
    Template:
    1
    Tutorials:
    5
    JASS:
    3
    Resources:
    23
    Perhaps titles might be updated, as @Naze seems Arena Moderator. ( hello Naze ;) )
     
  3. General Frank

    General Frank

    3D Modeling Moderator

    Joined:
    Nov 19, 2005
    Messages:
    4,185
    Resources:
    326
    Models:
    191
    Icons:
    126
    Packs:
    7
    Tutorials:
    2
    Resources:
    326
    Why not also add the rules that contest hosts are not allowed to be participants of the contest?
    It makes me feel uneasy and seems like gatekeeping from the host. We had that problem in the past and it went haywire.
     
  4. deepstrasz

    deepstrasz

    Map Reviewer

    Joined:
    Jun 4, 2009
    Messages:
    15,131
    Resources:
    1
    Maps:
    1
    Resources:
    1
    Why is that a problem?
     
  5. General Frank

    General Frank

    3D Modeling Moderator

    Joined:
    Nov 19, 2005
    Messages:
    4,185
    Resources:
    326
    Models:
    191
    Icons:
    126
    Packs:
    7
    Tutorials:
    2
    Resources:
    326
    Someone could make up the rules and criteria to fit what they can do best and not the others.
    Also, I think it is just a bad idea.
     
  6. Storm Knight

    Storm Knight

    Joined:
    Nov 12, 2016
    Messages:
    552
    Resources:
    0
    Resources:
    0
    Because if one wants to be host they could only host and participate in contests they think they can could win, and they would do that by premature long term preparing while for everyone else this might be a new surprise to prepare and look forward to.

    Maybe it doesn't seem like a major issue but its sort of there i guess xP


    That sounds like a really nice improvements. and i like the idea of allowing contestants to agree on that.

    Going to also poke @Kyrbi0 into this conversation i guess, its a little late but his name from a discord server was different and i was under the impression he used same names.

    On an end note would also like to say i don't think one my own suggestion wasn't really that bad but it did need a refinement so i'll rewrite it i guess.

    Basically, don't treat Gold/Silver/Bronze medals like as if they are hard coded to 1 2 3 positions that only can be taken by one person each.
    The result of a contest can be so:

    1 Person gets Gold Medal
    2 People get Silver Medal
    1 Gets Bronze.

    Or... literally any combination of that. This idea is mostly meant to combat situations where like someone doesn't fall down from a reward just because they got 5 points less from a 100 point based system. where another participant got 85, and he got 80. i believe in these very specific close call situations, its best to allow the judges to hand out 2 medals of the same rank to the participants. because the closer the participants come to eachother the less i trust the judges and its honestly an extremely difficult job to do also. so it will certainly feel relieving for judges be able to say, yeah you both deserve a Bonze/Silver/Gold medal because ultimately both of your works are very close to eachother.

    But leave it as it is and i do see why is it hard to find judges at all, cuz its extremely difficult to get those close call situations sorted out.

    I really hope you guys get my idea.
     
  7. deepstrasz

    deepstrasz

    Map Reviewer

    Joined:
    Jun 4, 2009
    Messages:
    15,131
    Resources:
    1
    Maps:
    1
    Resources:
    1
    Indeed.
    Great. However, that contestant could be trying for a challenge too.
    When I vote in a poll for the contest theme, it's not because I am sure I can make a piece of music for that theme but because I either like the theme, it wasn't in any other contest before or I want a challenge, to try something new.

    This is what I wanted, arguments.

    Now, we're at the Olympics?
     
  8. IcemanBo

    IcemanBo

    Joined:
    Sep 6, 2013
    Messages:
    6,085
    Resources:
    23
    Maps:
    3
    Spells:
    11
    Template:
    1
    Tutorials:
    5
    JASS:
    3
    Resources:
    23
    But staff makes rules and criteria, or at least approves them. I don't think it's meant host sets up such things alone.
     
  9. Kyrbi0

    Kyrbi0

    Joined:
    Jul 29, 2008
    Messages:
    8,538
    Resources:
    2
    Models:
    1
    Maps:
    1
    Resources:
    2
    (Thanks for pinging me, StormKnight)

    I'm grateful to people like SK, fighting the good fight to improve matters here on the Hive (vis-a-vis Contests). I only wish I had more time & energy to assist.

    I think it's fair to say there's agreement on one thing: the way Contests (& the Judging thereof) has gone in the past is substandard at best, untenable at worst. It's become an increasingly unsustainable situation.

    However, something I've noticed just in the short bit of discussion here: already comments are beginning to diverge, to go off on tangents. I would submit, to have a chance at success, we need to really focus in on one particular facet at a time, each in it's own thread (perhaps with a 'master-list' thread with links to all of them). The fact is there are a number of possible solutions, but it's gonna be difficult to reason out the pros & cons if we all start going off on other, related-but-ancillary matters (e.g. improving Polls, the role of Hosts, improving Criteria, etc).

    ~~~

    In order to stay on topic, then, I won't be commenting on the topic SK so kindly linked to, salient though it may be; except to say that perhaps that suggestion, properly implemented, will ease the job of the Judges to such a degree that other 'fixes' may not be necessary.

    To wit (though again, these should probably be separate threads):
    - We should definitely be strictly defining, for each Contest, whether it's Reforged or Classic. I don't believe we should be mixing them (i.e. each Contest picks one or the other, and all Contestants adhere to that).
    - I don't quite understand the suggestion about "multiple medals", but I think I disagree (along the lines that it goes against common understanding & starts to become a meaningless "participation trophy" (which we already have in the form of "participation Rep rewards"))
    - Having more Judges is obviously preferred/optimal, but the Reality is it's hard enough to find one, more often than not. We need to design & implement systems that compensate for that deficiency (both before (prevention) & after (triage)). To be quite honest, I think we need to look at even some long-standing customs & Rules and consider reworking things from the ground up.
    - Some suggestions/systems are big enough to require their own thread, but a short-list:
    - - Higher rewards to Judges (more Rep, maybe some kind of cumulative medal?)
    - - Ease the burden of Judging ((sub-)rankings instead of raw points, splitting up Judging 'domains' for the larger/multidisciplinary Contests (e.g. art, code, design Judges for, say, a Hero/Techtree Contest), etc)
    - - Create/encourage a "Judging Pool" of on-tap Judges
    - - Create/encourage a system of "pseudo-Judges"; respected/publicly-chosen Users that can provide Judge-like reports for the entries
    - - Greater utilization of the "Challenge" concept, perhaps modifying it into sort of a "mini-Contest" with few/no Judges required
    - - Consider some kind of 'soft-skills' training course/regimen for potential/current Judges, to avoid causing undue distress (threading the fine line between Critique and Criticism)

    ~~~

    Alright, I pushed through a headache til 3am to get this written so I'm stopping there.
     
  10. Storm Knight

    Storm Knight

    Joined:
    Nov 12, 2016
    Messages:
    552
    Resources:
    0
    Resources:
    0
    I'm not exactly sure if i understand the "Now, we're at the Olympics?" question. but Footman16 earlier in the thread said that the current system in place is actually much like the Olympics while my suggestion is not, and i was thinking that, its good that my suggestion is NOT like the Olympics because Hive is not the Olympics and it should have a different system in place.

    @Kyrbi0 i think you should scroll up above your own post, pass Icemanbo and deepstratz and read my post there. after Mythic also critisized my idea of participation sigil i thought its best to keep the idea except remove that participation part.

    Which is an alternative solution to this.
     
  11. Ralle

    Ralle

    Owner

    Joined:
    Oct 6, 2004
    Messages:
    7,723
    Resources:
    21
    Tools:
    2
    Maps:
    5
    Tutorials:
    14
    Resources:
    21
    We have implemented the following change:

    1. All contests must have at least two judges. They cannot be started before this happens.
    2. If a contest loses a judge. A conversation will be started among the contestants on how to proceed with the judging. This means that the contestants can agree to just let a single judge judge as normal or some change will happen. It's up for interpretation and just needs to be approved.

    and

    3. We are going to give 5-10 (maybe 15) rep per entry judged (to the judge) depending on the type of contest and the amount of work involved.
     
  12. deepstrasz

    deepstrasz

    Map Reviewer

    Joined:
    Jun 4, 2009
    Messages:
    15,131
    Resources:
    1
    Maps:
    1
    Resources:
    1
    I don't agree with more medals of the same type. Not only does it kill incentive but usually there are few contestants anyway.
     
  13. Murlocologist

    Murlocologist

    Joined:
    Mar 22, 2016
    Messages:
    747
    Resources:
    227
    Models:
    16
    Icons:
    211
    Resources:
    227
    If you want more fair scores, then you'll need to remove polls as well, because people tend to randomly vote, vote for friends or simply aren't qualified to cast a vote in the first place. But poll is kept to keep the community involved, so if the activity goes before fair scores, why bother with this?

    Bigger disrespect towards contestants is waiting half a year to get results, then having a non-perfect review of entries delivered on time.
     
  14. deepstrasz

    deepstrasz

    Map Reviewer

    Joined:
    Jun 4, 2009
    Messages:
    15,131
    Resources:
    1
    Maps:
    1
    Resources:
    1
    Indeed. People take this too seriously. We're not actually competing. It's more of a fun fair.
     
  15. Kyrbi0

    Kyrbi0

    Joined:
    Jul 29, 2008
    Messages:
    8,538
    Resources:
    2
    Models:
    1
    Maps:
    1
    Resources:
    2
    And this fundamental difference of opinion about What The Point of Contests Are is why I wish to the Light I had had the time & energy to push more for the fulfillment of this topic.

    I fear everything else to be decided hinges on this one Question; a Question we have as of yet failed to account for.


    I'm... glad to see quick action as evidence of concern, but I worry it might be too hasty.

    While we already did #2 (nice to get it in writing) & #3 (Rep rewards for Judges have been a thing for years AFAIK), #1 just kinda makes things much harder for people. Like, I have a hard time seeing most Contests getting off the ground with this Rule in place. :<
     
  16. deepstrasz

    deepstrasz

    Map Reviewer

    Joined:
    Jun 4, 2009
    Messages:
    15,131
    Resources:
    1
    Maps:
    1
    Resources:
    1
    Don't worry, most of the contestants will probably go with one judge not to wait too much on results.
     
  17. Kyrbi0

    Kyrbi0

    Joined:
    Jul 29, 2008
    Messages:
    8,538
    Resources:
    2
    Models:
    1
    Maps:
    1
    Resources:
    2
    That's apparently no longer an option.
     
  18. Chaosy

    Chaosy

    Tutorial Reviewer

    Joined:
    Jun 9, 2011
    Messages:
    11,087
    Resources:
    18
    Icons:
    1
    Maps:
    1
    Spells:
    10
    Tutorials:
    6
    Resources:
    18
    I am not invested enough in the subject to care much but this is not a good idea.
    Just joining a contest does not make you a competent judge. Nor does it mean that you want to.

    There have been contest entries that score 20/100 in total and you want such a person to judge others? this might be fine for maps, but for pretty much all other contests I would not trust this person to do a proper job. It's kinda like letting me be a judge for the icon contests without being able to draw. Yet I could technically join with some paint stick figure entry and become a judge.

    Additionally, isn't the judge usually taken from the talent pool anyway? it's not like Ralle comes in to judge contests because they had issues finding a judge.
    It's often someone who would like to compete but does not have the time right now.
    And judging takes less time compared to making an entry I would say. (not that they have a deadline anyway and take months to submit their judging *cough*)
    ____________________________________________

    I have an issue or two with judging myself but nothing that overlaps with your concerns, as far as I can tell.
     
    Last edited: May 20, 2020
  19. Ralle

    Ralle

    Owner

    Joined:
    Oct 6, 2004
    Messages:
    7,723
    Resources:
    21
    Tools:
    2
    Maps:
    5
    Tutorials:
    14
    Resources:
    21
    Which brings number 3. More incentive for people to judge.

    We'll see how it goes. Everything can be changed if it won't work. But I wouldn't mind contests being more rare and special.
     
  20. deepstrasz

    deepstrasz

    Map Reviewer

    Joined:
    Jun 4, 2009
    Messages:
    15,131
    Resources:
    1
    Maps:
    1
    Resources:
    1
    What about we make this as anonymous as possible?
    Basically, the judges won't know the entry author until after they give their judging. Voters will not know until the results.
    Someone, a moderator will receive the entries with no author names and will pass them to the judges and/or post them on the contest thread.
    It will mean excluding WiPs from the contest thread but that's not that important ultimately.