Absolving Presumed Bias in Hive Map Reviews

[Daffa]

Greetings,

Brief introduction:
This contest has been opened to absolve presumed biased opinions on hive reviews accused of judging a map based on the author not their work.

Expanded introduction:
There have been some discussions within private talks with regards to moderation of the map section. This problem has escalated in the map department and there has been assumptions that map moderators have been unfair in their ruling and judgement. Now, in an attempt to rectify whether map moderators have been unfair or if map authors failed to see fault in their work, a proposal to ensure the fairness and good practice of the Hive Workshop needs be undertaken.

The rules would be as follow :

1. All maps must be void of author names, so as to eliminate favouritism
2. Control maps will be issued to establish a baseline score out of 5, and to obscure the maps, to be judged from, biased views.
3. No author is to comment on the contest, in any shape means or form, so that they can not directly or in directly control the judges scores
4. After a winner has been established their identity and the identity of the other contestant will remain undisclosed, we are not here to hurt anyone reputation.
5. No member of staff is allowed to participate in rating maps, this is a decision to be made by the public's view over what makes a good map.
6. Only members registered at least for 6 months are allowed to participate, preventing scoring manipulation by opening multiple accounts.

To ensure the fairness of the discussion, there will be mediators appointed to the matter. These mediators will be neutral and meant to ensure the stability of the procedure.

hopefully, this contest will establish a greater understanding between hivers, the moderators and reviewing staff. Map reviews should not be based on the reputation of the author but the entertainment value and enjoyment of the game.

EDITS :
[23/07/2018 16:00 GMT+7] Changed the participation rule
[22/07/2018 23:59 GMT+7] Changed the 1 month member rule

 

[Ralle]

So much of contests is to show WIPs of what is done and commenting on each other. We've already established this as a community. What you're proposing sounds more like an exam situation than a for-fun contest.

 

[Maxwell]

I don't find a lot of sense in this one since map authors will be anonime:

No member of staff is allowed to participate in rating maps, this is a decision to be made by the public's view over what makes a good map.

I do not know the backstory of this rule, but I believe it was created to stop Map Reviewers from commenting.

 

[Ralle]

I am still unsure about this whole thread. Are we debating how to change map contests in the future or do you want to make a new contest?

 

[Shar Dundred]

Both?

 

[Ralle]

But if it's a contest then it should be in contest submission. It should have a theme, it should be so much more. That's why it's confusing to me.

 

[Shar Dundred]

I understand this as a suggestion as in how to handle future contests and an attempt to make
it "official".

 

[Maxwell]

Not really. That was supposed to be an anonime map contest initiated by two Users, that would show "how much biased Map Reviewers are".

 

[Shar Dundred]

Well... In that case...
@deepstrasz, I summon thee to discuss this.
I really got nothing to do with Map Section anymore, so I'll leave this to you guys.

 

[deepstrasz]

No member of staff is allowed to participate in rating maps, this is a decision to be made by the public's view over what makes a good map.

Basically, what @TheLordOfChaos201 might have meant is to remove maps being approved by mods/revs based solely on their decision and only be approved by popular demand.
But he called it a case control study.

 

[Deleted member 247165]

Would have been better if the contest had a pool with a bunch of maps and a judge consisting of 4-5 random users on Hive (not selected ones) so that the maps could receive objective feedback.

 

[MasterBlaster]

So... you want to start a contest where people will rate maps without knowing their creators' identity to see whether the results deviate from the way people are rating maps in general? Because that's how I understand this thread.

---

Let's start with the elephant in the room - to me it's obvious that people will be more lenient and positive in reviewing maps made by users they like or critical and negative in regards to the maps made by people they aren't too fond of. That's just human nature. And unless someone pulls some serious sh*t out of his a*s to criticize or praise something because he likes/dislikes the author, I don't really mind reviews being a little biased based on personal likings.

And let's face it, reviews are subjective by nature, so even if we removed this aspect of the process, reviews wouldn't suddenly become objective anyway - hell, someone could review the same map differently based on what kind of a day he has, so yeah...

But... should we even strive for objectivity? To me making W3 maps is a hobby and this whole site is built around people who share a certain passion and want to have fun together. Piling up rules and restrictions will only make this feel more like a chore instead of a fun online activity... And for what? For a contest that doesn't mean anything outsite of granting the winner a tiny bit of bragging rights?

Personally, I think it's just not worth it and I'd rather see contests remain a for-fun activity where it doesn't really matter who wins, but that all parties involved have fun from creating something and showing it off to the community.

---

As for reviews, the way I see it that the only thing that a reviewer can do is not approve the map, other than that - his rating is basically the same as anyone else's, to a point where if you write a detailed review it's even likely to be highlighted in the review section, despite you not being a reviewer.

And if this is a problem, i.e. if some users feel that their maps weren't approved due to some sort of a personal bias from the reviewer, I think it would be better to just have them ask another reviewer or a staff member to reevaluate the map taking into account what normal users think about it than to impose some restrictive rules for the whole of the map section.

Oh, and P.S. And remember that there will always be as*hats who won't accept even the most rational and well-meaning critique no matter what, thus they will feel hurt by even the most honest and fair reviews if they aren't 100% praise of their work. This cannot be helped.

 

[Daffa]

So much of contests is to show WIPs of what is done and commenting on each other. We've already established this as a community. What you're proposing sounds more like an exam situation than a for-fun contest.

More of a case control study, it's just formally being called 'a contest'.

I am still unsure about this whole thread. Are we debating how to change map contests in the future or do you want to make a new contest?

But if it's a contest then it should be in contest submission. It should have a theme, it should be so much more. That's why it's confusing to me.

A case control study to proof whether reviewer are bias or map makers failing to see their fault

 

[tulee]

So what if it was found that there was bias? The value in doing this kind of "experiment" I am guessing is because you would then want to make the names anonymous for all maps submitted on hive if bias is shown?

Many of the people who post their maps on hive workshop presumably do so not only because they want to share their maps with the community, but also because they want the community to know they (the authors) made the maps.

So assuming we are not going to make all maps anonymous anyways, where is the value of even conducting this kind of experiment?
If you are doing this just to show people who have been arguing there is bias that they are wrong, then you are doing this experiment in bad faith, since hive workshop will probably not be changing the submission rules anyways

 

[TheLordOfChaos201]

this is a very good contest. I've noticed in the past that moderators maps get approved almost immediately after they are uploaded, but for the average user they can wait for months before getting a poor review and a rejection or waiting for improvements. a lot of the times though these mod maps look good the game play is impractical, boring or broken. I remember reviewing a map similar to this where I was attacked for pointing out all the flaws. the problem with this is that the general publics comments were being purposely oblivious of the flaws because they wanted a good review in turn for their own pending maps

I'm fine with the author of the work being revealed after the map is approvedpved but not before.

for instance if @Ralle were trying to get a map approved. how many people would look at his name and title and give an honest review? however if his submission is totally anonymous the reviews will more likely be honest without any alternative intention or trying to earn a favor

well that's what I think anyway from my long stay here

 

[tulee]

this is a very good contest. I've noticed in the past that moderators maps get approved almost immediately after they are uploaded, but for the average user they can wait for months before getting a poor review and a rejection or waiting for improvements. a lot of the times though these mod maps look good the game play is impractical, boring or broken. I remember reviewing a map similar to this where I was attacked for pointing out all the flaws. the problem with this is that the general publics comments were being purposely oblivious of the flaws because they wanted a good review in turn for their own pending maps

I'm fine with the author of the work being revealed after the map is approvedpved but not before.

for instance if @Ralle were trying to get a map approved. how many people would look at his name and title and give an honest review? however if his submission is totally anonymous the reviews will more likely be honest without any alternative intention or trying to earn a favor

well that's what I think anyway from my long stay here

It doesn't matter that the names are revealed after approval. You are forcing submissions to be anonymous until they are approved. That is something only the map makers can decide.

"You are have noticed in the past"... is not a valid criticism, that is your subjective opinion.

This sounds more like people purposely causing trouble just because they enjoy seeing drama.

 

[deepstrasz]

for instance if @Ralle were trying to get a map approved. how many people would look at his name and title and give an honest review?

I for one, will, not would, WILL. Now, please, off with this conspiracy. It's a hobbyist site, not a job fight for who's to get more money.

 

[TheLordOfChaos201]

Ok?!

Why the sudden hostility. They aren't accusing anyone of anything that isn't already part of human nature. This test simply removes that element that might result in human error/favouritism. What we want to accomplish is to remove the possibility of corruption all together, I think if anything that is the goal of this contest. What difference would it make to have the authors name revealed only after the map is approved, as apposed to before. Considering your argument that many map makers just want the recognition, they won't be vying for attention for a rejected map would they?

As it stands the only think that revealing the authors name does, is that it allows you to make prejudice assumptions about the map before playing it.

In schools when they have a written exam, do the modulators look at the name of the student before grading their work? No, that would be unethical, instead they mark the exam with out any preconceptions as to whether it will be a high or low scoring student/paper.

Now the same should apply to maps. We aren't rating the map according to the name, or at least we shouldn't be, as the hiveworkshop we should strive to maintain excellent warcraft content not merely good members.

 

[tulee]

Ok?!

Why the sudden hostility. They aren't accusing anyone of anything that isn't already part of human nature. This test simply removes that element that might result in human error/favouritism. What we want to accomplish is to remove the possibility of corruption all together, I think if anything that is the goal of this contest. What difference would it make to have the authors name revealed only after the map is approved, as apposed to before. Considering your argument that many map makers just want the recognition, they won't be vying for attention for a rejected map would they?

As it stands the only think that revealing the authors name does, is that it allows you to make prejudice resumptions about the map before playing it.

In schools when they have a written exam, do the modulators look at the name of the student before grading their work? No, that would be unethical, instead they mark the exam with out any preconceptions as to whether it will be a high or low scoring student/paper.

Now the same should apply to maps. We aren't rating the map according to the name, or at least we should be, as the hiveworkshop we should strive to maintain excelent warcraft content not merely good members.

I have been reading your posts. How can you pretend to be making a proposal for the good of hiveworkshop? Everything you have said in the past comes across as trolling. The question you should be asking is why you do not have better things to do than cause trouble on a warcraft 3 fansite.

Maybe lose the "blame everyone else for my problems" attitude, and then maybe your suggestions will be taken seriously.

 

[deepstrasz]

In schools when they have a written exam, do the modulators look at the name of the student before grading their work?

No but they have the name on the paper so they know who to give it back to.

Now the same should apply to maps. We aren't rating the map according to the name, or at least we should be, as the hiveworkshop we should strive to maintain excelent warcraft content not merely good members.

Good luck with that. You'll have to convince @Ralle to create a system where the username of the approved/rejected/substandarded resource would appear only after such a decision was taken. From my PoV, this won't change quite anything. I don't care whose map it is as long as it's fit for approval or not. Check my "portfolio".

 

[Ralle]

And the topic is changing to the maps section and not this contest.

Let's keep things separate. If you want to change resource reviewing (Not just maps) create a new thread. If you have a problem with a staff member being biased, post in Staff Contact.

 

[TheLordOfChaos201]

This contest will serve as a platform to undertake nameless reviews, where by the author's name is void from there map. There will be no further functions or changes needed of the hive workshop as they currently have all they need to implement this change.

This project was opened in site discussions so that it could be discussed before being made, what might be the best means of going about this project?

That sort of thing

Right now we have a well grounded idea of how this project should operate.

1. Maps submitted with names void(in pm)
2. Public voting not the reviewing staff(to determine what is fun to play not a match to review conditions)
3. Control maps implanted in contest to create a baseline score(further prevents bias reviews and determines what a substandard score is)
4. All participants must be a hive veteran of 6 months or more
5. Mediators must be appointed to keep the peace(may not be necessary, but more necessary for this discussion considering the opposing views)

 

[The_Silent]

When it comes to map moderation, I fear that the practical implications of such a change would be that map moderation would be slowed to a crawl.

I don't know about ratings, but if a user had a history of good, unbugged, etc. map content, isn't it fair to trust them?

Practically speaking, maps is just such huge and detailed objects, that some trust has to be put into users for moderation not to take around 10 hours of work per map I fear.

Regarding contests:
During the contest, it would be impossible as people have to upload wips, etc. But sure, remove the names in the contest polls. Can't see how it could harm, voters is supposed to atleast click on the submission post anyway, and this might encourage that.

 

[TheLordOfChaos201]

When it comes to map moderation, I fear that the practical implications of such a change would be that map moderation would be slowed to a crawl.

I don't know about ratings, but if a user had a history of good, unbugged, etc. map content, isn't it fair to trust them?

Practically speaking, maps is just such huge and detailed objects, that some trust has to be put into users for moderation not to take around 10 hours of work per map I fear.

Personally I can not comment on how long it takes to review a map. Though I have reviewed maps in the past it has never been from the capacity of a moderator, or by their standard. Often I play a game only until I realise it isn't to my liking or I have stumbled onto a bug I really can't ignore, and yet still I sometimes find that these same maps have already been approved. A lot of the time it is something small like models too big obscuring my view, or broken mechanism and or bobbing of the camera view. Now in my mind I would reject these maps but there is nothing in the moderators hand book that says I should?

Correct me if I am wrong...

or perhaps provide a list of what is deemed the criteria for a map to be approved????

You see this is what I feel is part of this contests objective, what does the public see as a good map? What would they reject based on their enjoyment and disapproval?

 

[deepstrasz]

You see this is what I feel is part of this contests objective, what does the public see as a good map? What would they reject based on their enjoyment and disapproval?

Because we're looking at it from a technical level too, not only what ice cream flavour the majority likes shoving in their mouths.

 

[TheLordOfChaos201]

Because we're looking at it from a technical level too, not only what ice cream flavour the majority likes shoving in their mouths.

give the public what they want

the essence of this debate

not to throw mud at anyone... but Mr name-ommited, a moderator on the hive, has an approved map which is impossible to play in my opinion. the walls are so high in the maze map that at times you can't use your cursor to click in the direction you want to go. the ground is uneven and as a result you get a lot of jumpy jerky actions from the camera which makes one feel sea sick. I suggested they added in keyboard directionality movement options but they refused, even after I modified their map and added them myself, saying 'not being able to go in the direction you want to is part of the game play. that people like their map so they arent going to make any changes'

now it was obvious to me why this map was approved, but if it wasn't approved do you think their attitude would be the same?
biased and unfair approvals hurt the hive no matter how you cast the light on it

 

[Chaosy]

I wont give any specific details for which contests, but I am guilty of voting for people that I know by name in contests.
Not because I am best buddies with them but rather to save time.

If there are 10 entries I choose to check 2-3 of them. So I pick a few users I know have experience that is relevant to the field, for spells I'd check out Tank-commander for example.
I do not even look at the other submissions because I am a lazy bastard.

Now, I imagine that I am not the only one that does it but I cannot prove it.
So I think that some bias is definitely at play in the public poll

I have a harder time to think it happens among judges however as they are forced to write down the reasons to validate their rating.
Yes, if you hate a person you can remove a point or two without it seeming out of place. But you cannot make a 5/5 submission seem like a 3/5 submission, at least I do not believe that until I see it.

 

[deepstrasz]

give the public what they want

the essence of this debate

EpicWar for that.

do not even look at the other submissions because I am a lazy bastard.

That's your problem, entirely.

 

[Chaosy]

And you believe I am the only lazy bastard in all of humanity?
If I do it, I am fairly confident that others to do to, not everyone, but some.

Though they might not be brave (or stupid) enough to admit it

 

[TheLordOfChaos201]

And you believe I am the only lazy bastard in all of humanity?
If I do it, I am fairly confident that others to do to, not everyone, but some.

Though they might not be brave (or stupid) enough to admit it

Well you were certainly brave, I commend you and respect what you have done for this debate

 

[Chaosy]

Thanks.

The other side of the debate however:

If it is a large contests with 20 entries, and I need to go through every single one..
Quite honestly, I will not bother to vote.
Once again, perhaps I am the only one, who knows. But it is a possibility to consider.

 

[TheLordOfChaos201]

Thanks.

The other side of the debate however:

If it is a large contests with 20 entries, and I need to go through every single one..
Quite honestly, I will not bother to vote.
Once again, perhaps I am the only one, who knows. But it is a possibility to consider.

pictures tell a thousand words

if we use many pictures of every map something is bound to catch your eye and interest

you will become excited about reviews again, instead of just devoting your self to a name

 

[Chaosy]

That is true.
That'd help a lot, somehow I doubt it will happen though as it'd mean slightly more effort than just linking the contest entry post like we do currently.
And even then, not everyone posts a GIF of their spell entry (a default picture does not really capture a spell, even more so if it is complex)

 

[MasterBlaster]

Then people would vote based on the images (i.e. I've liked this gif the most so I will vote on this). The reality is that if someone doesn't want to put in the effort to make sure his vote is well cast, there's nothing you can do to force him.

 

[Ralle]

God damn it. And you took it off topic again. I should have moved this thread to the arena before it turned into this.

If you want to talk about moderation/reviewing, don't limit it to maps. Make it broad enough to fit all resources because in this specific case, it doesn't matter which section you are on about. Bias can be found everywhere.

If a review is biased (@TheLordOfChaos201 thanks for bringing exact examples instead of posting in Staff Contact by the way) post in Staff Contact and let's take a look at it all together.

Resources will never show up without author names or have author names hidden from mods/reviewers, why? Because some users are extra sensitive and need to get their reviews with a spoon full of sugar and unicorn tears so they don't get angry and rage quit. Removing names will make it impossible to know how to behave with specific people... Also, mods/revs start to learn the style of the users and can help them improve. Without knowing who they are each time will remove this and make things worse in most ways.

The best way to combat bias is to actually combat the instances of bias, not change the system.

 

[Chaosy]

Spell contests are a bit special as the code is equally important as the visuals. But for most contests a good screenshot should give pretty solid ground, no?

@Ralle
Damn, I was pretty sure contests were the actual problem and not the map section xD

 

[TheLordOfChaos201]

Spell contests are a bit special as the code is equally important as the visuals. But for most contests a good screenshot should give pretty solid ground, no?

@Ralle
Damn, I was pretty sure contests were the actual problem and not the map section xD

I could personally go to every mods map and point out the flaws but they aren't likely to take my view seriously. I'll likely be banned for trolling

talking to staff will not solve this problem

 

[Chaosy]

You start to sound like @ShadowShaman.mn lol

 

[TheLordOfChaos201]

You start to sound like @ShadowShaman.mn lol

this isn't about pointing the blame at staff. the second we do that the instant this whole thing will go no where

this is about creating a separate system to at least give it a shot.

 

[Shar Dundred]

could personally go to every mods map and point out the flaws

this isn't about pointing the blame at staff.

.

talking to staff will not solve this problem

I'll likely be banned for trolling

 

[deepstrasz]

Voting for a map solely on screenshots (in contests) is not only hypocritical... but indeed biased. You should refrain from voting at all. This works for concept art and icon contests because, obviously why.

 

[Chaosy]

You can judge certain things.
Theme, creativity, visuals and such.

But I get your point, it does not work for maps.

 

[TheLordOfChaos201]

Voting for a map solely on screenshots (in contests) is not only hypocritical... but indeed biased. You should refrain from voting at all. This works for concept art and icon contests because, obviously why.

I did not say they would vote on the image alone

merely that if you feel unmotivated to plausibly each map you would likely find what you deem interesting in the image and then become excited about playing that map in particular and then reviewing it

 

[deepstrasz]

merely that if you feel unmotivated to plausibly each map you would likely find what you deem interesting in the image and then become excited about playing that map in particular and then reviewing it

From what I remember, contestants are required to upload pictures of their maps alongside WiPs.

 

[TheLordOfChaos201]

From what I remember, contestants are required to upload pictures of their maps alongside WiPs.

that is good. you see very little is required to set up this contest

so really what are you afraid of?

 

[deepstrasz]

so really what are you afraid of?

Wasting more time than I already am.

 

[pick-a-chew]

The only way to judge maps without bias is to have external reviewers (which is not practical as there probably aren't any). The entire map making process must be peer-reviewed by the Hive's own members to ensure the map is the author's work, hence why contests ask for WiPs.

The proposed trial contest rule-set is not only flawed but it is not viable with the Hive's standards and rules.

 

[Chaosy]

I personally do not really believe there is a bias that is worth noting in the map section.

Getting a map into the approved section is laughably easy. A mod might make it slightly more difficult, but nothing that cannot be fixed with a day's effort.
It was way more biased when a mod gave ratings for maps. A mod can indeed give a 4/5 map 5/5 since it is ultimately subjective.

 

[TheLordOfChaos201]

Agh well I give up then, all I'm hearing is no no no no no, so I guess that means the answer is no

BUT!

I will start reviewing mod maps and giving my personal opinion why they should be rejected

@Daffa the Mage
It all rests on your shoulders now bud

 

[Paillan]

I personally do not really believe there is a bias that is worth noting in the map section.

Getting a map into the approved section is laughably easy. A mod might make it slightly more difficult, but nothing that cannot be fixed with a day's effort.
It was way more biased when a mod gave ratings for maps. A mod can indeed give a 4/5 map 5/5 since it is ultimately subjective.

And that's why I never give scores on my reviews.