• Listen to a special audio message from Bill Roper to the Hive Workshop community (Bill is a former Vice President of Blizzard Entertainment, Producer, Designer, Musician, Voice Actor) 🔗Click here to hear his message!
  • Read Evilhog's interview with Gregory Alper, the original composer of the music for WarCraft: Orcs & Humans 🔗Click here to read the full interview.

Absolving Presumed Bias in Hive Map Reviews

Status
Not open for further replies.

Shar Dundred

Hosted Project: LoA
Level 76
Joined
May 6, 2009
Messages
6,089
Well you are basically just a garbage collector now.
Trashcan or not the trashcan, that is the question.

#BringBackModRatings
I am trying to figure out if you were joking with that statement or not.
I thought you supported the idea that mods/reviewers should only approve/reject and therefore
make the progress of reviewing faster and only judge whether a resource is approvable or not.
 

Chaosy

Tutorial Reviewer
Level 41
Joined
Jun 9, 2011
Messages
13,239
Pros and cons.

The old system was not perfect, but I was really happy when I got my first 4/5 mod rating on one of my spells (after 5+ 3/5 ratings)
It was a huge motivating factor for me, it was also a good metric to filter submissions by as user ratings were a bit unreliable in my opinion.

I had a complaint about how the old rating system worked in a certain fringe scenario involving a map I made but otherwise my experience was good.

I am a bit nostalgic perhaps, but I miss it from time to time.

That said, it was mostly a joke since I understand that the staff was bombarded with "MY MAP WAS UNFAIRLY REVIEWED!"

TLDR: high highs, low lows.

edit: I never got to have a 5/5 rating before the mod rating was removed *triggered*
 
Last edited:
Level 19
Joined
Jul 2, 2011
Messages
2,161
What if I upload a few control maps on my profile and then some maps I made and then some already approved maps

with the authors void

wait a year for them all to be reviewed

then collect my results and decide for myself if the hive is bias based on the scores they give these maps?
 

deepstrasz

Map Reviewer
Level 75
Joined
Jun 4, 2009
Messages
20,236
then collect my results and decide for myself if the hive is bias based on the scores they give these maps?
If you're referring to reviewers, it's user dependent, always will be, even with the normal users. A system where objectiveness isn't the main basis, because we're talking about art, is sure to fail in terms of scientific results.
 
Since I've been trying to keep myself neutral since I WAS suppose to be a neutral party aligned to help this matter, I haven't shed my personal view to the matter. I've been mostly just watch how things unfold. Now, if people excuse me so I can be the not be so neutral guy.

PROS :
+ Maps more inclined to Public Demand
+ Less Staff Control needed

CONS :
- Quality Degradation, I for one don't trust the masses easily and with the suggested methods...
- Systematic Manipulation, if we rely on either rating or downloads, it's easy to manipulate them to gain approval

I've compared Hive restrictions with WC3C's (which is quite known for it's strictness) for a bit, and I know it's laughable how Hive's standard is in a sense. I can say it's above most of smaller sites out there and we have a decent quality control already. I personally agree if maps has to be viewed from two perspective : enjoyable and technical. This in turn ensure we have decent quality maps in database and avoid junkies like some trash campaign I found a few weeks prior.

In short, while this idea helps giving public what they wanted more, the consequences can be dire to the map database. It might even remove the worth of having a map approved at hive, which is a merit on it's own. This is sort of dangerous in the long run.


Now with that out of the system...

I'm currently thinking of a simpler approach to this matter rather than making some sort of complicated test that might be rigged by either side of the play (which is very possible). However, it will take a while before I can even actually come up with something useful for the matter.
 
Level 19
Joined
Jul 2, 2011
Messages
2,161
Since I've been trying to keep myself neutral since I WAS suppose to be a neutral party aligned to help this matter, I haven't shed my personal view to the matter. I've been mostly just watch how things unfold. Now, if people excuse me so I can be the not be so neutral guy.

PROS :
+ Maps more inclined to Public Demand
+ Less Staff Control needed

CONS :
- Quality Degradation, I for one don't trust the masses easily and with the suggested methods...
- Systematic Manipulation, if we rely on either rating or downloads, it's easy to manipulate them to gain approval

I've compared Hive restrictions with WC3C's (which is quite known for it's strictness) for a bit, and I know it's laughable how Hive's standard is in a sense. I can say it's above most of smaller sites out there and we have a decent quality control already. I personally agree if maps has to be viewed from two perspective : enjoyable and technical. This in turn ensure we have decent quality maps in database and avoid junkies like some trash campaign I found a few weeks prior.

In short, while this idea helps giving public what they wanted more, the consequences can be dire to the map database. It might even remove the worth of having a map approved at hive, which is a merit on it's own. This is sort of dangerous in the long run.


Now with that out of the system...

I'm currently thinking of a simpler approach to this matter rather than making some sort of complicated test that might be rigged by either side of the play (which is very possible). However, it will take a while before I can even actually come up with something useful for the matter.
why not discuss it here

maybe we can help
 
I'm honest at that point, i thought i was at the off-topic section for a moment and needed to double check.

WALL OF TEXT INCOMING!


I see that it was for contest, where some of these things are debateable (why the claims some do are futile is something i will come back later to), but lets get the map section out of the way:

Demanding that projects should be send in without the authors name is nothing more than a slap in the face for every recource creator. (i will try to keep the ranting at a minimum)

How do you thing authors build up a community for their maps? By the quality of it (Quality and objectivity and the mutual exclusion of each other is something i will come back later)? Things like Custom hero survival, Vampirism and all it's clones should tell us a different story. They are playable, indeed, but far away from polished, demanding or bugfree. Still people play the heck out of them.

Because of the "franchising" of the map. And this is due to dedicated work of the authors, editors or the community. I don't connect the names of vexorian, gex, shar dundred or crazyrussian, rufus, Aeroblyctos, DSG or bribe (Names are pulled out of my ass, there is no ranking as equal or something like that) with high quality recources for no reason. My bias is not based around arbitrary conditions like skin colour or gender (These being arbitrary takes them anything to base differentiation on). The whole work of them is not seperated from their newest recources, they stand in context. I not only assume that they meet certain conditions, i expect it. And neglecting this work would be nothing short of disrespect.
This doesn't mean that we just should approve them without quality management systems (I mean, a system needs to work, still, that is objectively achieveable). But it means that we shouldn't try to actively damage the progress of people working on their profile/name/whatever. This is not how you build up communities or a portfolio. Thats how you drive people to other sites.

What this means is that any bias coming out of the former work of a synonym is not arbitrary, it is reasoned. Basing assumptions on that can be reasoned upon (although, i give you that, they can fail).
________________________________________

Now, lets take that a stage further, shall we?

The definition of quality itself is subjective.

"Degree to which a set of inherent characteristics fulfills requirements." (ISO Norm 9000)

Requirements are inherently subjective, because they encompass needs and expectations, which they customer doesn't even need to be concious about.

What does this mean?

There are some signs of quality we can objectify. This means: they are still subjective, but we have arbitrary chosen certain measureable, objective parametres to poll them upon. Like TheLordOfChaos201 hinted on a map which got very clunky control and camera, which makes it for they personally non-approveable. What it means is that they have valued the camera as to such a degree detrimentically to the gameplay that it should be sub-standard, while someone else found it meeting the minimum requirements to be approved (This is, by the way, why it can be argued that the abolishment of moderated reviews could overall lead to a decline in quality on the hive. Less demands let more stuff through. But i don't want to argue on that, i am happy that the rule quickening the review process)

It's the same with game design. So many designers have tried to describe what is good game design or what is "fun". And most of them simply fail. More less than others. I read enough blogs of bullshittery (Sadly, i dont have bookmarks of a list anymore, damnit) to know what i am talking about (Mark rosewater tried with his known jonny/spike/timmy apporach to make a systematic take on that, but it is still only an abstractation from reality).

What does that mean? Talking about keeping contests objective and even more so recource submissions is futile. I cannot take someone seriously who says me that icon or skin reviews are objective (Btw, the mods make a great job in that section, much love <3).

But what they are is objectified. The mods keep themself on certain criteria they analyse their work upon. And they can be reviewed themself on these analysis. That is part of quality management. What this means is if you think a review was negatively based towards you because of bias, you can look at the review, see if you there is a dissonance between your view on your work and the analysis, and get a second mod to review your recource again. Thats why it should be normal to have two judges on a contest (of course this is not always possible since really active and knowledgeable member, which are dedicated in the community and have made themself a name, have gotten rare). When two or even three of these reviews fail to meet your expectations, you should ask yourself if your own perspective on your project is flawed or if you just simply don't want to fullfill the criterias. And in that case, you should move on (looking at someone like shadowshaman.mn and other).
___________________________________

What does that mean for contests?

We cannot really stop bias. Even WIP or pictures of the class or just the use of many custom models could lead people to vote for someone they randomly picked (like often said, many ppl dont open more than 3 entries). So we instead of making futile attemps to unify them, we should think of the weighting of our quality analysis tools.

Judges, in comparison to the poll, need to measure parametres objectified. The grade in which they can analyse the entries is far more reasonable than of the poll. On the other hand, we could only really increase the quality control by more people which review the judges or the reviews and which are dislogded from the community. This is not possible for a small community like use, except people would get their ass up and review more maps with reason.
Poll votes are measures without any mean of quality assurance. So these need to be assumed to be completely biased. But, and this is important, they can be taken as quality parametres (even more so, one of the most important parametres in industries).
But this is not why we want to keep them. The community is important for the hive, so we have them in the contests because of ideologic reasons. This means that even if they are biased, we take them because in the end our whole work here revolves around the community. Our doing here doesn't exist in a vacuum.
 
Level 45
Joined
Feb 27, 2007
Messages
5,578
Demanding that projects should be send in without the authors name is nothing more than a slap in the face for every recource creator.
My understanding is that entries are anonymous simply for the poll and the duration of the contest. After which any members who wish to confirm their participation in a map are free to do so and the creator of the winner is officially announced. This type of 'blind audition' is EXTREMELY common in the music world (you enter from where judges can't see you and you play in front of a screen so they can't tell anything about you) and it works fairly well for those purposes. If you want recognition for your work, you may have it after the polls close. If I'm wrong about this, you can ignore me here because I haven't read this whole thread.

Poll votes are measures without any mean of quality assurance. So these need to be assumed to be completely biased.
A lot of science is done on data collected without much QA. If you are trying to eliminate a random bias among the reviewers (not everyone has the same kind of bias, for/against the same people, with the same strength), having a large enough sample size is exactly what you should do. With enough voters (how much is 'enough' here? I am not sure, but some statistics could answer that) all their individual biases cancel each other out and you actually get a representative distribution. Someone with a more solid statistics vocabulary might be able to explain better why that is, but I know from physics that it's the same thing as reducing random uncertainty in measurement.

Now, if a significant portion of the community has a specific bias for/against some members/projects that is NOT random, it's a systematic bias (systematic uncertainty in physics) that does affect your results. This bias can only be overcome/'removed from your data' if you know what the actual specific bias people have is and have a way to quantify it. I don't really see that happening in this case, but I also see the bias in these polls to be much more likely to be random bias than a systematic bias.
 

Rui

Rui

Level 41
Joined
Jan 7, 2005
Messages
7,551
Damnit, you guys. Always opening up the interesting discussions while I am away. :pnewt:


@Ralle maybe @Daffa the Mage can help clarify. This thread seems related to map section moderation. Not the Arena. @Daffa the Mage introduces with “this contest has been opened (...)”, so I think “contest” is being used loosely, and refers to this thread.

As a matter of curiosity, I am slightly guilty of this in what concerns @CMarket, but in opposite polarity. I kept giving him 4/5 when his cinematics in general might've been worth 5/5. :pmeh: I was somewhat his friend and loved his cinematics, so I put up harsher thresholds when reviewing his work.

Where are all the links showing examples of such behavior?
Was amidst my first thoughts. How many instances are we talking about? 10? 100? Occasional occurrences, or systemic calamity? This is relevant only because no system is going to be perfect, so there will always be some situations of unfairness.
 
My understanding is that entries are anonymous simply for the poll and the duration of the contest. After which any members who wish to confirm their participation in a map are free to do so and the creator of the winner is officially announced

For contests, this would be a reasonable approach. We can send the mods the wips, so the peer reviewing process is still there. I was talking about map submissions because people took that suggestion in the heat of the discussion also into the map section, too. Where in my opinion it doesnt make any sense.


A lot of science is done on data collected without much QA. If you are trying to eliminate a random bias among the reviewers (not everyone has the same kind of bias, for/against the same people, with the same strength), having a large enough sample size is exactly what you should do.

While this is definetly the case, i would like to question what we measure with a community poll. A huge part of what we measure is the presentation of the map, mostly even outside of the map, like youtube videos, and how it fares against an average consumer taste. In fact, we measure to a huge part attributes that some people don't think belong qualify the map itself, but its PR. This is why we had that many discussions in the past about that topic.
It is not that the measured value is biased, the problem is that it is in the eyes of some people the poll is an unqualified variable to measure quality of the maps properties alone.

Its like measuring the surface of an object when you want to describe gravity. It may have some impact on your result (due to hydrodynamic effects and resistance and thus a deacceleration of the object in air), even a mayor one, but it encompasses much more effects than what you want to describe, which makes using it doubtfull and only obsfucate your result. And a hogher sample size won't change somethinv there because your model is flawed itself.
 
Level 12
Joined
Mar 24, 2011
Messages
1,082
I'll put that in front, I have no idea what is happening in map contests so I am speaking "resources > maps" only

tbh, content creators have to be a bit more objective towards their own creations, stop picking up the stick any time anybody says something that doesn't please them or doesn't put their work on a pedestal and take some bloody constructive criticism.

So far I have not seen a map review that says "Bad/Rejected/Fix/Etc." that doesn't actually deserve it.
"My map is good, my friends who play it with me say it is good"
Well, you have green icons, that spell does nothing, that other spell bugs the unit, balance is off the charts, the terrain is made out of a flat dirt with a few trees, can't see a single doodad or any flavor and where is the bloody music coming from ?

I know that is an extreme case, but it is mostly the case of the loud people who are unhappy with their own incompetence.

In the case that the map is actually any good, I am guessing that a map reviewer has given 3 or 4 and the creator thinks that it deserves a 5 ?
Well, grow up and suck it up.

I haven't seen any of those though... usually it is a non-map-reviewer who gives a crap rating like 1 or 2 with the pretext "I don't understand this" "Too hard" "Game no like" without any objective reviewing.

So... happy reviewing?

-Ned
 
Level 19
Joined
Jul 2, 2011
Messages
2,161
new point of order

now this discussion is about maps being reviewed and to a larger extent, why some are rejected and others approved.

the question I pose to you is, what maps some maps worth approving and others not?

I need a default comparison list so I can check automatically off what qualifies a map

I'm designing a program^-^

I actually already have many parts of the programming but I stopped working on it because the staff kept telling what I'm doing is impossible and I'm just trolling them..... :(

bunch of meanies
 
Level 19
Joined
Jul 2, 2011
Messages
2,161
thanks... unfortunately those are very hard criteria to add to a computer generated map summary

right now program can just tell the following

percentage of flat terrain
number of doodles
number of custom united, spells and upgrades.
number of spelling errors

though I guess coupled ralle's trigger error checker... it isn't so bad :?

none of which qualify or disqualify a map
 
For a programm it is inherently impossible to measure if a game is following the explicetly rules it should follow, without being given the in depht rules by the programmer (meaning the one who programmed it need to have reviewed the map to the core already. And then it gets quite redundand. Even iteration-based bots need to know which results they shall judge as positive and which one as negative.

So a bot cannot test if a map is playable without having someone else to go in and check hinself.

If you manage to do that, though, please think about writing some papers about that and prepare to get sone prizes.
 
Level 22
Joined
Sep 24, 2005
Messages
4,821
@TheLordOfChaos201
That was why they initially said it was impossible.

"4. Quality Standard
- Fully functioning gameplay"

How can you automate that? I'm not saying you can't, but the staff would be more interested on a working prototype than promises. You made a lot too.
 
To be honest, no computer generated materials will exactly determine is a map approvable or not. This also applicable to all sections, with no exceptions.

If we count based on specific criteria to auto approve a map, a clever mapmaker will be able to manipulate their map to fit the needs of the filter, even if the map is not acceptable. There are also a lot of ways a map can be done. Take the terrain : it can be intentionally average to help gameplay or it's auto generated which wilk show empty map terrain from the terrain file. This extends to triggers and other aspects as well. A map can be approved if the leak is minimal and will not cause any possible lag even on low end engines. We can also talk about import usage and other stuffs, which varies between each map.

In short, you cannot decide a map with a set of exact standard if it's approvable or not. This is why map reviewer and moderator are needed to handle the case, and they aren't paid for it.

Another one would be model. You cannot decide if a high poly model is not approvable instantly. It's also impossible to set a model to approves just because it's usable.

In icon section, it comes to art and visual check of the icon. You cannot easily set an icon as approved or rejected just because it's lacks something.

In spell section, spellmakers have different taste in coding. For example, I favor Unit Indexer or Hashtables while another spellmaker would prefer Dynamic Indexing. Some would go with Linked List and so on. The triggers also varies between users as well. This in turn makes it difficult to set an exact rule that can be passed automatically.
 
Level 19
Joined
Jul 2, 2011
Messages
2,161
@TheLordOfChaos201
That was why they initially said it was impossible.

"4. Quality Standard
- Fully functioning gameplay"

How can you automate that? I'm not saying you can't, but the staff would be more interested on a working prototype than promises. You made a lot too.
you could still use it to automatically disqualify maps with given reasons

though I question the wisdom in rejecting a map for spelling errors or a lack of custom models, units and spells
 
Level 22
Joined
Sep 24, 2005
Messages
4,821
Indeed, it can be used. If the tool existed and the tool is accessible to the people who needs them. Right now both, I assume, aren't.

EDIT:

"though I question the wisdom in rejecting a map for spelling errors or a lack of custom models, units and spells "
Is this true? I'd like to see some examples of it. Thanks.
 
Level 19
Joined
Jul 2, 2011
Messages
2,161
Indeed, it can be used. If the tool existed and the tool is accessible to the people who needs them. Right now both, I assume, aren't.
it works... it is just a little bulky and complicated at the moment

I need to find a way of looking inside a map without an mqs viewers.

an automatic way... otherwise map reviewers will spend more time trying to get this program to run than reviewing programs.

it would be neat if you could just point the program to the of directory of a map and bamboo instant review

right now it is looking like you need to manually extract 5 files from maps before you get that report
 
Level 19
Joined
Jul 2, 2011
Messages
2,161
Since the maain topic has been dropped, I suggest you make a different thread for the tool instead.
you are dropping the topic! !!?

I thought it was on going. you said you had a grand master plan you needed to think on and weren't willing to share! !??

nah it is ok I think I have just about everything I can do for the automated program. it just needs to be more user friendly
 
Level 12
Joined
Mar 24, 2011
Messages
1,082
Yeah, players are having a hard time coding a good AI but yeah.
You mean "any" ? :)

@Paillan It would be a waste of resources kind of... Let's run an advanced AI on a quantum computer to evaluate war3 maps... Could do for training, e.g. Deep Mind tried chess just as a test, then I could only imagine it is now being used to solve more important problems. No time for games for you advanced piece of machinery.

-Ned
 
Level 19
Joined
Jul 2, 2011
Messages
2,161
You mean "any" ? :)

@Paillan It would be a waste of resources kind of... Let's run an advanced AI on a quantum computer to evaluate war3 maps... Could do for training, e.g. Deep Mind tried chess just as a test, then I could only imagine it is now being used to solve more important problems. No time for games for you advanced piece of machinery.

-Ned
Right now the program will just make a very good map rejection software

It can detect swear words, nude words, and spelling errors.

All of which can get a map rejected when in excess.

Which would at least cut down on the maps waiting for pending and prevent 'time waster' maps
 
Level 20
Joined
Feb 23, 2014
Messages
1,265
Are swear words even that big of an issue? @deepstrasz

I'd say that a far bigger issue is checking whether the map actually works - you know, if you can play it from start to finish, whether all objectives/quests can be completed, etc. Another thing would be to check if all the custom assets are working, for example - if there aren't any disappearing models or green icon boxes, etc.

There's also checking for bugs, seeing if the terrain is at least half-decent (for example, if it isn't just a flat space with one texture and little to no doodads) or if the map itself is even remotely fun or balanced...

I can't imagine a tool that would be able to test for all those things and come back with results that are reliable enough to be believed without the mod playing through the map.
 
Level 19
Joined
Jul 2, 2011
Messages
2,161
Are swear words even that big of an issue? @deepstrasz

I'd say that a far bigger issue is checking whether the map actually works - you know, if you can play it from start to finish, whether all objectives/quests can be completed, etc. Another thing would be to check if all the custom assets are working, for example - if there aren't any disappearing models or green icon boxes, etc.

There's also checking for bugs, seeing if the terrain is at least half-decent (for example, if it isn't just a flat space with one texture and little to no doodads) or if the map itself is even remotely fun or balanced...

I can't imagine a tool that would be able to test for all those things and come back with results that are reliable enough to be believed without the mod playing through the map.
I'm not trying to make a fully automated system, merely I'm trying to add something that could reduce the load on mods and allow more qualified maps to be reviewed first.

With @Ralle 's trigger testing software and my spell check swear word and porn check, maps can be run through both and if they are cleared then they put into the immediate review list where as other maps can be put on pending until trigs and spelling is fixed
 
Level 19
Joined
Jul 2, 2011
Messages
2,161
I'm asking for anyone with a computer.

I need help on this program, but it seems no one is available for testing.

Here is what a generated report looks like so far:

Total Objects: 431
Number of Custom:
Items: 166
Abilities: 173
Units: 57
Doodads: 1
Destructibles: 0
Buffs: 19
Upgrades: 0


List of swear words detected: 2
ass
bite


Flat land: 87.0%
Number of doodads loaded with map: 297
Number of units/buildings loaded with map: 197
Imported files used: 32 of 32 files used
Estemated In Game loading time: 44.46 seconds


Please I really need help checking to see if this program works, the mods say it doesn't work but it works fine on my computer. I need someone else to download it so I can verify if this is a problem.
 
Level 12
Joined
Mar 24, 2011
Messages
1,082
Right now the program will just make a very good map rejection software

It can detect swear words, nude words, and spelling errors.

All of which can get a map rejected when in excess.

Which would at least cut down on the maps waiting for pending and prevent 'time waster' maps
Would it reject my map if I have a lot of assassins ?

PS: Nude words ? Better cloth them or they gonna catch a cold ;)
PS2: How is "bite" a swear word ????
PS3: I would highly suggest to move any further discussion about your tool to your thread.

-Ned
 
Level 19
Joined
Jul 2, 2011
Messages
2,161
Would it reject my map if I have a lot of assassins ?

PS: Nude words ? Better cloth them or they gonna catch a cold ;)
PS2: How is "bite" a swear word ????
PS3: I would highly suggest to move any further discussion about your tool to your thread.

-Ned
Bite is a swear word when in connection to ass :p

I do actually have a thread open on the topic, but sadly people are ignoring me there:(

Oh well if the hive isn't going to try... I can't either :eek:
 
Status
Not open for further replies.
Top