1. The long-awaited results for Concept Art Contest #11 have finally been released!
    Dismiss Notice
  2. Join Texturing Contest #30 now in a legendary battle of mythological creatures!
    Dismiss Notice
  3. The Aftermath has been revealed for the 19th Terraining Contest! Be sure to check out the Results and see what came out of it.
    Dismiss Notice
  4. Hivers united and created a bunch of 2v2 melee maps. Vote for the best in our Melee Mapping Contest #4 - Poll!
    Dismiss Notice
  5. Check out the Staff job openings thread.
    Dismiss Notice

Removing Review Ratings

Discussion in 'Latest Updates and News' started by Ralle, Oct 7, 2016.

  1. Ralle

    Ralle

    Owner

    Joined:
    Oct 6, 2004
    Messages:
    11,150
    Resources:
    22
    Tools:
    3
    Maps:
    5
    Tutorials:
    14
    Resources:
    22
    We've decided to remove the ratings moderators and reviewers give to resources. This means that, no matter the person rating a resource, their votes will count equally. There are many reasons behind this, and a few are drawn attention to below.

    1. We cannot be consistent over time.

    The Hive has had a lot of moderators since we first opened shop to where we are now. It is simply not possible that the staffers aboard at the moment are giving ratings consistent to those that were aboard nearly ten years ago. The current system is flawed in this respect, but by getting rid of staff ratings we leave the decision about a resources quality to the user.

    2. The opinion of a moderator/reviewer should not matter more than the users' when the users are the ones primarily going to use the resource.

    Moderators and reviewers have many opinions on how they like to see resources turn out. This is all great. Feedback is wonderful. However, this site is for people to submit resources so other users can benefit from them. Therefore, the user matters just as much as a moderator. By having a moderator or reviewer aboard only to review or reject an item, we are roping in the users to help motivate and shape the product of the author's work.

    3. Motivation.

    Many users strive for a good approval from the staff, and this is cool - but a resource lasts longer than the approval. By making ratings general instead of specific, we hope to see authors continue to be interested in developing their work on THW.

    A staffer will still decide where the resource goes, whether to approve, delete, or put it in the low quality section. They still post a review, but ultimately the only thing that differs from a normal user is where the resource ends up.
     
  2. A Void

    A Void

    Joined:
    Mar 29, 2011
    Messages:
    2,481
    Resources:
    10
    Models:
    2
    Spells:
    1
    Tutorials:
    7
    Resources:
    10
    Nice! This should have been here right from the start. Old review and rating system had flaws where user reviews didn't line correctly with moderator ones which resulted in biased ratings on the resource.
     
  3. deepstrasz

    deepstrasz

    Map Reviewer

    Joined:
    Jun 4, 2009
    Messages:
    8,897
    Resources:
    1
    Maps:
    1
    Resources:
    1
    Good. It was confusing especially when one moderator approved a resource and might have or not rated it.
     
  4. Kyrbi0

    Kyrbi0

    Joined:
    Jul 29, 2008
    Messages:
    7,672
    Resources:
    1
    Models:
    1
    Resources:
    1
    This is a brilliant idea, and I'm quite glad to see it happen. I might even start rating resources now.

    This basically puts the power of model ratings into the hands of the most concerned party, while leaving with the Stewards of the Site the ultimate "keep/discard" decision; it always struck me as weird/confusing, the dichotomy between moderator & user reviews.

    It also solves the long-standing issues of "moderator review discrepancies" and "moderator preference"; especially the latter, where approval often hinged upon whether the mod liked it (not whether it was structurally sound & fulfilled the requirements of the section).

    ~~~

    Question: what was my question?

    Second question: Might be slightly off-topic, but can I say again that the name "Substandard/Too Simple" is just shy of offensive, in my opinion... There's got to be a better way to differentiate between models than that. I mean, we could go for "1st Class" and "2nd Class", or "Grade A" and "Grade B" (like eggs). Or heck, reverse it and turn the "Substandard" section into just the plain old "Model" section, while calling the currently "Model" secoin the "Superior Model" section or "High-Quality Model" or "Top Quality Model" or "Choice Model" or "Prime Model" or "Fine Model" or any number of synonyms. Basically lauding those who do better than average rather than shaming those who do worse than average.

    Heck, the "Average Model Section" even sounds better. Still not ideal.
     
  5. Chaosy

    Chaosy

    Joined:
    Jun 9, 2011
    Messages:
    10,529
    Resources:
    17
    Maps:
    1
    Spells:
    10
    Tutorials:
    6
    Resources:
    17
    And there goes my only motivation to create stuff for the spell section.

    Getting high ratings from a mod actually meant something.
     
  6. Kyrbi0

    Kyrbi0

    Joined:
    Jul 29, 2008
    Messages:
    7,672
    Resources:
    1
    Models:
    1
    Resources:
    1
    You can still garner increased satisfaction from a good moderator review more than a good user review. And now you'll be saved from a personally biased moderator giving you a bad review and/or refusing to approve your spell, thus sinking any chances of being seen by anyone.
     
  7. Chaosy

    Chaosy

    Joined:
    Jun 9, 2011
    Messages:
    10,529
    Resources:
    17
    Maps:
    1
    Spells:
    10
    Tutorials:
    6
    Resources:
    17
    Not really the same thing anymore, it's no longer special.

    I do like the part about mods not being able to give me shitty rating because reasons, but at the expense of mods being useless? (from my motivation standpoint xd)
    Meh.

    Also, this change will make searching harder.
    Because the user ratings are not accurate. They are throwing 5/5's at their friends just because. However, mod reviews could be trusted most of the time. If I filter icons by 5/5 there will be good stuff. Filter by user ratings and you get 3/5 worthy stuff.
     
  8. deepstrasz

    deepstrasz

    Map Reviewer

    Joined:
    Jun 4, 2009
    Messages:
    8,897
    Resources:
    1
    Maps:
    1
    Resources:
    1
    No. Please. It's exactly like using the anatomic word for a sexual organ or a slang word. It's the same thing. It's like calling a janitor a cleaning department agent or something. It's all stupidly psychological.
    However, I agree it shouldn't have two names but one. Substandard sounds fine.
    A/one mod is not enough. You'll need more to at least make it feel like a jury.
     
  9. Chaosy

    Chaosy

    Joined:
    Jun 9, 2011
    Messages:
    10,529
    Resources:
    17
    Maps:
    1
    Spells:
    10
    Tutorials:
    6
    Resources:
    17
    Depends on the section.
    For spells it's more or less objective.
    The mod only needs to be skilled enough in the field, it's very hard to give reasons to give a person a low rating because you hate that one.

    Either you have leaks or you don't.
    Either you have good configuration or you don't.

    It's rarely inaccurate in my experience so far.

    I can see it being more of an issue in other sections where I have less experience.
     
  10. Kyrbi0

    Kyrbi0

    Joined:
    Jul 29, 2008
    Messages:
    7,672
    Resources:
    1
    Models:
    1
    Resources:
    1
    Yeah, except it isn't any different. The "specialness" is entirely subjective and all in your head. So you determine whether it's special or not.

    Example: I can get a 5/5 rating for a resource from anyone, and sure it feels good. But a 5/5 from a friend of mine, or from a modder whose opinion I respect/value, will intrinsically mean a lot more/be 'special', to me. Doesn't matter what anyone else says/does (or the fact that in terms of the site's resource review calculations, it's all the same), it's still special... To me.

    They aren't useless. They are exactly as useful as everyone else.

    Hm. I've never really searched by rating. I agree that people will be people; tending to be subjective. Just like Polls.

    So you aren't affected by that kinda stuff. Good for you.

    I'm not talking about using a euphemism or tumblr-special-snowflake garbage. It's just that there are loads of models in the Substandard/Too Simple section that are truly not "substandard", nor "too simple". They *are* the standard, if anything. It's just kind of like a jerk-move, in my opinion.

    Yeah, the artistic resource sections (skins, icons, models especially) is pretty much the exact opposite. Lots of subjectivity.
     
  11. WhiteFang

    WhiteFang

    Joined:
    Jul 6, 2014
    Messages:
    3,249
    Resources:
    0
    Resources:
    0
    Overall,I do agree with this move.

    I see what you are saying,and understand.However,if we add a 'superior model section' we will still have the same issue,just different.Those who made 'regular' models will still look bad and will envy those who have 'superior' models.

    QFT.

    At the end of the day,regardless if it's mods or user ratings,there will always be bias

    Another thing is inexperience.Correct me if I'm wrong,but on the new hive you don't even have to comment on a resource to rate.Any random person can come and deal out 1/5s and 5/5s.Also,a inexperienced user may come around and look at a,let's say icon, and he sees nothing wrong with it.He then simply gives it a 5/5.Just because in his inexperience he could not see anything wrong
     
    Last edited: Oct 8, 2016
  12. Quilnez

    Quilnez

    Joined:
    Oct 12, 2011
    Messages:
    3,195
    Resources:
    37
    Icons:
    2
    Tools:
    1
    Maps:
    7
    Spells:
    21
    Tutorials:
    2
    JASS:
    4
    Resources:
    37
    I was thinking to add my ratings stats to my portfolio, but nevermind. It's no longer reliable. :(
     
    Last edited: Oct 8, 2016
  13. Ralle

    Ralle

    Owner

    Joined:
    Oct 6, 2004
    Messages:
    11,150
    Resources:
    22
    Tools:
    3
    Maps:
    5
    Tutorials:
    14
    Resources:
    22
    Chaosy, a moderator is just a user who has janitorial duties and happens to be skilled in his field. Getting positive feedback from a skilled person should be all that matters if feedback is your motivation.
     
  14. Chaosy

    Chaosy

    Joined:
    Jun 9, 2011
    Messages:
    10,529
    Resources:
    17
    Maps:
    1
    Spells:
    10
    Tutorials:
    6
    Resources:
    17
    And the president of USA is just a man. Yet I would be way more happy if I got a compliment from the president, rather than my sister.

    So I don't think it's that weird that I don't value the feedback of everyone equally.

    It also goes beyond just words.
    If you win, say a football competition you get a medal. Physical proof of what you achieved.
    This used to be the case with mod ratings, but with them gone - the feeling of accomplishment is gone.
     
  15. Kyrbi0

    Kyrbi0

    Joined:
    Jul 29, 2008
    Messages:
    7,672
    Resources:
    1
    Models:
    1
    Resources:
    1
    I disagree. Making a model that is graded as "regular" is not so painful as being graded "substandard/too simple". Literally "not up to the standards".

    And even if that's the case, that's why I first suggested "Class" or "Grade". Grade A/B is fine. First/Second Class is fine.

    Precisely. And that intrinsic value you ascribe to certain people cannot be taken away, regardless of what the site does. So you're good there.
     
  16. Chaosy

    Chaosy

    Joined:
    Jun 9, 2011
    Messages:
    10,529
    Resources:
    17
    Maps:
    1
    Spells:
    10
    Tutorials:
    6
    Resources:
    17
    Read the the edited part of the post.
    It's not the same.
     
  17. BlueSaint

    BlueSaint

    Joined:
    Jun 18, 2012
    Messages:
    2,726
    Resources:
    3
    Tools:
    1
    Spells:
    2
    Resources:
    3
    I don't like this "blending in with the herd" thing. Moderator should be high authority and carry the responsibilities of his own field. Rather step down as a mod than blame the system (you said "we've decided" so mods have discussed this).

    Users should be expected to be immature, giving ratings of 5/5, 4/5 or 1/5, or nothing at all, without further evaluation. Also prone to bias, e.g. model by -Grendel- gets them hard before even seeing whether the asset is functional.

    I'd cut down from the other end: dump down user ratings (e.g. like/dislike) and make the rating spectrum wider for mods, or amplify it in another way.
     
  18. Chaosy

    Chaosy

    Joined:
    Jun 9, 2011
    Messages:
    10,529
    Resources:
    17
    Maps:
    1
    Spells:
    10
    Tutorials:
    6
    Resources:
    17
    Aye. Too biased.
     
  19. SpasMaster

    SpasMaster

    Joined:
    Jan 29, 2010
    Messages:
    1,855
    Resources:
    4
    Icons:
    2
    Maps:
    2
    Resources:
    4
    So, ummm. I'll share my opinion on this topic.
    It will be primarily focused on Map Moderation, since well... that's what I do.

    Overall I can see reasoning behind this change. A moderator might decide that a map is not "good", while the community loves it. A good example could be very popular maps like DotA, Elemental TD, Castle Fight, Enfo's Team Survival, etc - those maps would possibly get average ratings here, whilst adored by the community.

    However, I personally always saw Moderator Ratings as a rating given to a map based on the THW requirements/standards. I remember few years back seeing a topic where DotA was discussed (the wc3 mod) and someone said that it probably wouldn't get more than 4/5 on Hive. Which made sense, if the Moderator Rating symbolised what I pesonally saw it as - the rating of the map based on THW criteria. DotA lacks in several fields that are highly valued in the Hiveworkshop modding community. Does that make it a bad map? Hell, no. Yet, the Moderator Rating would have judged the map based on "modding" criteria. Not "popularity". A map like Gaias or TKoK, or even that 'Diablo 3 into Warcraft 3' mod have a much greater modding value in them compared to DotA - which lacks in those fields compared to the quoted maps.
    There was a nice distinction between User Rating which symbolised the rating of the players and the Moderator Rating which symbolised the "modding" rating.
    At least, that's how I've always seen it.

    All that ^ is my opining on the topic from a general point of view.

    Now, let me get all personal. My personal issues shouldn't impact how the entire moderation system functions, yet I'd like to share them.
    Rambling

    I joined in 2010 when getting your map approved took months and you had to have invested at least some time in your resource for it to be approved. Maps were put on hold and rejected all over the place, due to simple terrains, code leaks, these sort of things. Even map descriptions! Many people were not familiar with the site rules and requirements. Guess what, though. After waiting for months, a moderator came, slapped you in the face by telling you that you need to improve on various fields: terrain, coding, description, custom creatures/spells/heroes. Then he told you that he would pay you a visit soonTM. From my experience and point of view this was incredibly motivational. There was a 'THW judge' who knew and imposed the THW rules and requirements and you had to work on your resource for it to be approved. Oh, yes - ratings didn't come until much later. But I guess WC3 is a bit less lively nowadays and all of that sounds too harsh - I get it. However, as someone who has followed all those rules strictly and as someone who has worked on insane amount of features for years, I feel somewhat bad regarding this change. Let me explain: I have recieved a Director's Cut Rating for Sunken City in 2013 after @-Kobas- gave me big lists of things I must work on and improve on to get the rating. And I did work hard. And I did clear every single note from his list. And I got the rating. And now it's gone.

    I can see why.. I can see the reasons. Yet I feel like I've lost something I worked hard for to get. Don't get me wrong, I don't want a freaking medal or something. It's just that I can't shake the feeling of losing something.
     
  20. Shar Dundred

    Shar Dundred

    Joined:
    May 6, 2009
    Messages:
    3,323
    Resources:
    10
    Packs:
    1
    Maps:
    8
    Tutorials:
    1
    Resources:
    10