Resource Ratings (Spells used as example)

[Nestharus]

The rating system for Spells should be changed I think : ).

As it stands, it's extremely difficult for people to find high quality resources. This is because resources are rated by moderators based on a coolness factor, not on how well it was designed + coded.

A resource should get a 5/5 if it was coded perfectly, meaning excellent design, excellent API, excellent ease of use, no leaks, and so on : D. It should get 6/5 if it goes above and beyond, by perhaps including in-depth documentation like interactive tutorials, tons and tons of examples, and so on (like save/load with snippets does).

I just don't agree with how things are being currently rated. There are 5/5 resources that should be rated 2 or 3/5 and there are lots and lots of 3/5 resources that should be 5/5. A coolness factor approach is too damn subjective.

Does this mean that if File I/O was perfect that it should be 6/5? Hell no, because it doesn't have the in-depth examples and tutorials that Save/Load With Snippets has. At most, it'd be a 5/5, and in it's current state, a 2/5. It may be cool and so on, but a resource shouldn't be graded on coolness factor or complexity, it should be graded purely on the quality of the resource's code.

This means that moderators will be able to approve anything and everything as well, so long as they work : ), even if they have memory leaks. 0/5 = doesn't even work, 1/5 = works, but crap, etc, lol. We can start grading resources by quality and actually make the rating system useful to users : D.

 

[HammerFist132]

I rly like ur thinking, but I have found jundging of Magtheridon96, rly good, I had posted few spell and he rejected them, there was no leaks, but for excample I had too many If then else, but yes for Excample I definitely think to Spell(better system)Full screen inventory should be DC and not 5/5 while I dont think to thank chopter with bombs should be dc.

 

[Nestharus]

I don't think that the Full Screen Inventory should be DC... it doesn't have in-depth examples and it doesn't have the code to support it. If anything, I'd rate it 4/5.

Save/Load with Snippets is coupled with a 100 Lesson interactive tutorial and a massive map loaded with examples. That = diractor's cut quality. It's either that, or I went way above and beyond even DC there, lol...

edit
If it was laid out a bit better, it could be 5/5, but it's nowhere near 6/5 atm.

The inventory isn't dynamic enough: you can't create your own styles easily. It's pretty a poorly detailed example of his own inventory that you are expected to use. There are certainly lots of tools for creating inventories, but they all lead to a very static approach to inventories. I think that a 4/5 rating is very appropriate for the vjass full screen inventory in its current state : ).

edit
resource I was talking about

http://www.hiveworkshop.com/forums/spells-569/save-load-snippets-v2-1-0-5-a-202714/?prev=mmr=6

Notice all of the tools, examples, and tutorials it comes with. It also links to an interactive tutorial (written to support it of course) that teaches you how the various snippets operate + how to do really advanced stuff.

Of everything include above, the only thing I did not write is Table

I'm only linking this as reference to what I think merits Director's Cut. It needs to have excellent design, excellent API, ease of use, excellent code, and in-depth documentation + examples in mass.

Here are what I think the ratings should be for
0: does not work
1: works, but many design issues and code issues + possible leaks
2: leakless, but design issues and code issues, poor data structure choices
3: design issues, not as generic as it could be, possibly has extra things that shouldn't be in there
4: not as generic as it could be, too static of a layout. Pretty good modular design.
5: extremely modular, excellent data structure usage, excellent code, no leaks, excellent API, ease of use, excellent design, one example, documentation covering API and possible small code snippet examples of specific uses (this is what I typically aim for, the 6/5 is too much work >.<)
6: in-depth examples and tutorials on many possible uses, lots of resource examples using it. For a spell, perhaps 3-4 fully functioning heroes using that spell with other spells that work in conjuction with it and a themed map to showcase the spell, which includes a fully playable mini-game using the spell. For systems, interactive tutorials, resources using those systems, maps using those systes, regular tutorials, examples of resources using those systems, examples of using those systems, and possibly even a game utilizing the system (for save/load with snippets, I worked at getting it into a variety of maps to showcase it, but I never did link to those maps).

As is seen by my above descriptions, 6/5 requires an insane amount of time and work. It really is about going above and beyond. 5/5 is just doing what you need to do so that most people will be able to use it. 6/5 is doing the 5/5 with style. This is why I typically aim for 5/5 when I make resources : P. The only exception to this has been my save/load stuff, ; ). Actually, my damage stuff has the potential to go to 6/5 too if I decide to do all of the tuts and so on.

 

[Ralle]

So lets imagine if I built a spell that was extremely boring and useless, but it had the best API you have ever seen and has no leaks at all and a binder full of documentation. It would be DC'ed but nobody would use it as it was so boring. A spell like this should get a 2 or something like that.
Coolness definitely factors in. A really cool spell with poor coding should be able to get you a 3. To get it higher, it would need better code, so it goes both ways.

If we made a weight scale, I would say:
60% coolness
40% coding

So you can rate the resource in both categories and summarize the weighted rating.

 

[Nestharus]

Coolness is purely subjective though, and it's extremely unfair to the author. It also makes the ratings rather useless since they all become very subjective.

If I'm a person and I'm looking for a well coded system, I want the 5 to return well coded systems, not systems that the moderators thought were cool.

If I want high quality spells that I know are guaranteed to work and work well (meaning well coded), I should be able to input a 5 and get those. The coolness factor should come from categories, or perhaps it should be an altogether different rating. You can't really combine them both, otherwise you kill the usefulness of the rating.

The same goes for maps.

Coolness can't really be quantified or rated because again, it's subjective. The only thing that can be rated and rated fairly is quality: the craftsmanship.

Rating based on coolness is like giving art ratings, which you just can't do : \. You can rate art on technique, but you can't rate it on how awesome it is.

For me, right now, ratings in every single category are completely useless. When I used to browse for models, I would have to browse over 10000s of 1000s of models to search for models that were of high quality and guaranteed to work (essentially every single model on the site as some of the categories weren't very useful either). Why did I have to do this? Because ratings didn't give me those results, they gave me the models that the moderators thought were cool. I needed models for very specific purposes, cool or not. The same goes for spells. I don't care whether a spell is cool or not, I care whether it fills the need I have for it, which is where search comes into play. The only thing I'd care about in ratings would be quality, not coolness, because cool or not, I'm going to want something specific for the map. This is why I say that the coolness factor in a rating is completely useless. Even if they were split up into 2 ratings, it'd still be useless, because map makers don't care about coolness, they care about whether they need it for their map or not. They should be able to filter out quality resources =). I think that if you did this, most mappers would always look for resources that are initially 4+ and then expand to 3+ if they couldn't find what they were looking for. I'd personally do 5+ and then code it if I didn't find what I was looking for : o. I doubt any mapper would use anything <= 2.

See how the ratings suddenly become extremely useful when doing ratings based on quality and not coolness (0 weight involved)?

 

[PurplePoot]

I think it depends on what you mean by "coolness"; its subjectivity is certainly an issue, though.

Nobody (or few people) cares about some really elegant code that doesn't do anything interesting, but similarly a resource which is flashy but messes up your map due to its code being awful is equally problematic. I definitely agree that the art section moderation does not seem based on actual practical ingame use, but I think that's an issue that goes much farther than just the ratings system.

 

[Anachron]

I can partly agree with this.

While my Full Screen Inventory which has video tutorials how to work with it and documentation and a lot of features, is well coded and with comments it gets a 5/5.
But ROFLCOPTER gets a 6/5, and I have never seen that spell in any map, while I know a few maps that use my FSI.

I don't say I should get a DC, but I don't see how this is less than a spell.
So yeah, coolness is overrated, imho.

 

[Ralle]

[...] Nobody (or few people) cares about some really elegant code that doesn't do anything interesting, but similarly a resource which is flashy but messes up your map due to its code being awful is equally problematic. [...]

Yep.

 

[Nestharus]

While my Full Screen Inventory which has video tutorials how to work with it and documentation and a lot of features, is well coded and with comments it gets a 5/5.

Oh it has video tuts and etc, then I change my cast to 5/5 for it, and if a couple of things are changed, 6/5 =). My bad there, didn't see all the tuts ;o.

So flashiness then is more important than how useful it is for a given map?

If I'm a map maker, I don't care about finding spells that fit my hero, I only care about flashiness? That doesn't sound right to me, I really don't think that mappers think that way. I know that I certainly didn't when I did mapping. I just coded my own spells to fit my hero actually ;o.

 

[Magtheridon96]

Okay. Here's the thing.

Before, I often saw TRD spamming 5/5s and 4/5s everywhere, as if they were valueless.
I was at first striving to give this 5/5 it's rightful value, but I have to admit that I have made a mistake by taking it too far at one point and giving 4/5s to resources that deserve a 5/5.

If you are not satisfied with your rating, please, do tell me, and I will change it right away after re-evaluating your resource, even if I have to do it for all the other 400-ish resources I've evaluated so far.

Coding is what makes up the 4 out of the 6 points.
1 point goes to flashiness (It's a small factor, but there is a difference between an amazing spell with all these cool buffs, projectile physics and cleverly used effects, and a dull one that gives a buff to a targeted unit.)

The 6/5 is for resources that got a 5/5, but are above and beyond that.

 

[Nestharus]

Ok, how about this. We use a point system as Dr Super Good suggested in the chatroom with a checklist of pass/fail. These points add up and are then translated into ratings.

The scale should be as follows ; D

1: support
4: quality
1: innovation

For the resource to be approved, it should have a minimum # of points in support and quality

Example: minimum .3 support and 2.7 quality to pass

The checklist could be as follows-
Tutorials (.05 points for each one up to 4 (sections)) = .2 (optional)
Examples (.05 points for each one up to 10) = .5 (optional)
API Docs (.3) = .3 (required)

.2 + .5 + .3 = 1

For example, Save/Load Snippets has an interactive tutorial that includes I think around 75 lessons (if I wanna be accurate). If we split it up into sections they cover, it'd probably be 30ish, meaning that that alone hits 30 on the tutorial thing.
The tutorial has many examples and the map also includes examples. I'd ballpark the number at around 20, which hits the examples.
API Docs are of course included, which hits the .3 requirement.

We can go on with the Quality and Innovation checklists as well =).

 

[Dr Super Good]

Get rid of silly continuous scales. They are too easy to be biased, difficult to generate and often feel unfair to the resource creator. They are also difficult to maintain if a resource is updated.

Half the problem with resource moderation is trying to produce a fair ratting for a resource and moderators often get complained at if they do not rate what the author wanted (mostly a 5/5).

My idea is to redefine the entire moderation process into a simple checklist that also acts as a ratting to help potential users decided on the quality of a system.

An example of such a template. Obviously a table would make it look better but I have no clue how to make one in the forums.

Each system or spell in a map is given a separate entry containing...
Is in a commonly used language: JASS
Is easy to modify: NO
Is well documented: NO
Is self-contained: YES
Does not contain preventable leaks: YES
Manages game resources efficiently: YES
Does not noticeably show bugs: YES

The actual test map is given a rating containing...
Complies to all Hive Workshop site rules: YES
Map allows sufficient testing: NO

Where the resource is sent is dependent on the results of the test. For example if a map contains spells which mostly have all green results then it would be sent to a high quality section. If a map contains spells that fail most of the tests (red results) then it is sent to a sort of dumping ground. Both these sections are publicly available except the high quality section emphasize resources more (such as appearing on main page, having description next to image in list etc). If a map fails the general site rules test then it is removed from public access for potentially legal reasons.

On top of this individuals can give their own opinion on a resource. This comes in the form of recommendations (thumbs up) and deterrents (thumbs down). When an opinion is given the user needs to specify a brief reason why and it ties that opinion to the revision data (last update). People can publicly view the list of opinions for a resource seeing who gave them, their reason and for what revision (update date). Users can change their opinion at any time.

Opinions will not create a rating. This is because a poor resource may improve over time so lots of negative opinions might leave a permanent bad mark as the users never update them. Instead they will be given in the form of the last 100 opinions showing how many were positive and how many were negative. Obviously if a user is the creator he cannot state his opinion due to extreme bias.

Users who are skilled in their field or who have a high position will be given special opinions. If more than 1 such person gives a positive opinion for a resource then the resource gets a recommended status (and appropriate eye candy).

The entire idea is based around peer assessment with moderators only having to do simple and easy to fill in tests. These tests may even be hard-coded into the moderation system meaning all he has to do is tick boxes. The actual quality of a resource would be decided by users. This saves moderators a lot of effort and will produce more accurate results.

It would also mean you could do things like allowing moderators to moderate their own resources. Dedicated pages could be made for each test clearly stating what is needed to pass or fail them with examples making the results absolute.

In the end the users of a resource do not care if it gets a number like 96.5/100 or 4.66/5 and things. They care if people find it good and if it is of a reasonable quality.

 

[Pharaoh_]

You are basically suggesting a reinvention of the wheel, based on your own resources, which is acceptable, to an extent, given that we can get them as examples of what your theory stands for, but unfortunately, it's not enough to make the decision, which will basically turn the tides of how things were being processed so far.

When I was moderating, I was mostly aiming for the originality of the spell and how well the coding was executed. Most of all, I valued the way the spell's effect would come into play. Yes, there is a spell that damages, but there are tons of ways to damage a unit, such as custom damage, true damage (remove life instead of calculating defenses), physical/magical damage, damage per second, damage per X seconds, damage upon movement, damage upon unit's actions, etc. Therefore, a combination of a nice and clean code, configurable, with well searched models to support the visual aspect and an original way of materializing the concept was what made me increase the rating of the resource.

I agree that there should be a more objective way of rating our resources, but let's face it, each moderator has a different background, coding-wise, gaming-wise, etc., so we can't expect absolutely equal results among them.

 

[Nestharus]

So.. perhaps different rating systems based on the resource type?

System, Spell, etc?

I think that the stuff you outlined is innovation really ; ). So perhaps Spells should place greater emphasis on innovation than systems should. Or maybe innovation should be higher, but the criteria for innovation should change depending on the resource type. Or perhaps, we should try to come up with a general idea of innovation regardless of the resource type, hm...

And as for Dr Super Good's pass/fail, I think that we can combine it with ratings by assigning points to each field ; ), and then requiring that certain fields bet met ;o, and those minimum fields would add up to a rating of 3/5.

 

[Dr Super Good]

You seem to be missing the point of how time consuming and difficult ratting is...

 

[Nestharus]

You seem to be missing the point of how time consuming and difficult ratting is...

I'm not. There's a reason that I never want to be a moderator and hate doing reviews : P.

Btw, I don't see how ratings would be difficult. On your rubric, we'd just assign points, then the moderator just adds up all the points and rounds down to the nearest whole and they're done : \. Where is the difficult here? The adding? : P

 

[Dr Super Good]

we'd just assign points

Which is the difficulty. A boolean value can be made much more absolute and less variable so comming to a choice is much easier.

The rating system was half the reason I stopped moderating resources. As it was 1/3 would argue with me over the ratting I gave.

 

[rulerofiron99]

I think your binary rubric is a great idea, but only as a supplement to the current system.

In the same way that authors can specify whether or not a spell/system follows the JESP standard, we should add all of those points in that list.

The spell's numeric rating then is more to do with:
- aesthetics
- innovation
- usefulness
- coolness

 

[Dr Super Good]

- innovation

This is a stupid field. Who cares if something is new as long as it is done well.

 

[rulerofiron99]

This is a stupid field. Who cares if something is new as long as it is done well.

Without innovation, we'd all be playing Warcraft 3, except that there would only be a Paladin and Footmen.

Innovation is an important part of new, interesting content. At the end of the day, we play and mod a game. A 5% reduction in CPU usage doesn't make the game any more fun, and in fact with such an old game that 5% reduction is insignificant with today's powerful PCs. Innovation and ideas, however, do impact the gameplay experience.

To put your words in an example, it's like making a Footman model with 20 more poly's, instead of making a Footman with an animation that makes him throw his shield.

 

[Nestharus]

Without innovation, we'd all be playing Warcraft 3, except that there would only be a Paladin and Footmen.

Innovation is an important part of new, interesting content. At the end of the day, we play and mod a game. A 5% reduction in CPU usage doesn't make the game any more fun, and in fact with such an old game that 5% reduction is insignificant with today's powerful PCs. Innovation and ideas, however, do impact the gameplay experience.

To put your words in an example, it's like making a Footman model with 20 more poly's, instead of making a Footman with an animation that makes him throw his shield.

Imma use BigInt as an example. I had complaints that BigInt wasn't fast enough, so I had to redo a lot of it to make it as absolutely fast as possible. Nobody has complained about the latest version BigInt thus far : ).

Why were people complaining? It'd freeze the game for a moment.

The first design of BigInt was as optimized as possible, and this still wasn't good enough. I had to come up with new creative designs to reduce operations.

 

[rulerofiron99]

Imma use BigInt as an example. I had complaints that BigInt wasn't fast enough, so I had to redo a lot of it to make it as absolutely fast as possible. Nobody has complained about the latest version BigInt thus far : ).

Why were people complaining? It'd freeze the game for a moment.

The first design of BigInt was as optimized as possible, and this still wasn't good enough. I had to come up with new creative designs to reduce operations.

Yes, it's definitely important that things work properly - something freezing for a moment is definitely significant.

My post was arguing against DSG's notion that innovation isn't important.

 

[Nestharus]

My post was arguing against DSG's notion that innovation isn't important.

Well, it isn't.. lol

what's important is performance and functionality. If your thing is more performant and more functional, innovation or not, your thing wins. It's just that innovation typically goes hand in hand with performance and funcionality =).

 

[rulerofiron99]

A better performing damage assigning engine is still just a damage assigning engine.

A 3-dimensional knockback system, even if coded "badly" (but playable), adds to a map's value.

I am not arguing as innovation being more important than performance and functionality - if you read my first post on this page, I agree with the suggested system of classifying a resource by certain standards, however that the numeric score should still be included for the "coolness" factor.

Both can coexist, I am in no way saying that we should ignore functionality and performance. What I am saying is that both are important, and separate from each other.

If innovation isn't important to you, why are you even on a modding website? Innovation is an important part of modding, a central driving factor behind our efforts.

 

[Nestharus]

Coming up with something new is good, but giving something new extra points just because it's new isn't good. That's like giving extra brownie points to someone because they were the first to do it. What if the resource was suck and someone else came along and remade it but 1000x better? omg, now they get a lower rating because their resource wasn't *innovative* even though it's actually higher quality and better, so people end up using the first resource even though it's worse just because it was the first one of its kind.

Is that fair? Not at all.
Does that help the users? Not even.

We've been striving to kill this sort of attitude in the JASS section, and that attitude was in fact killed. Even if you are innovative, if a new resource comes out that does the same job but better, yours is going to get summarily rejected (even if it was already approved).

We did this with save/load in the Spells section too. If you look for save/load systems, only Save/Load with Snippets will be there because it deprecated every other save/load system.

As for a coolness factor, this too is really unfair. You may create a system or something that isn't flashy but extremely useful. Is unit indexing cool? Not at all. If it's perfect and has absolutely great support, why shouldn't it get a 6/5? Just because it's not flashy or cool? That's hardly fair. That's like putting a cap on certain resoruce types just because you don't think that they are cool. That's like saying, we only support flashy spells, everything else can suck it and get low ratings, even if more work was put into them than a flashy spell.

 

[coolyoshi1]

If we made a weight scale, I would say:
60% coolness
40% coding

^^^^

 

[SA Dashie]

Yeh I'm all in with Nestharus in this. If a spell is cool then let the user decide. it's more of a personal based opinion anyway, as from quality in coding and usefulness etc can't be argued from as easily and makes more sense when a moderator rates a recourse in this kind of way.

 

[Anachron]

It's funny, because my CustomInventory once was Directors Cut (by TRD), because of being something unique.

Then hvo evaluated it and put it to 5/5, in respect to TRD.

I am totally fine with it, but then the Roflcopter spell does not deserve it either.
The CI has thousands of bug fixes and it just works. It has a commented code and I wrote a little document about the libraries.

So, hvo didn't think it was worth a DC (but had no point to complain about whatsoever) and TRD said it is perfectly coded and also super cool.

And yet it's a 5/5.

 

[Nestharus]

Anachron, the problem is its design. You kind of combine a demonstration of the custom inventory system by defining your own inventory with a set of back code. People end up using your defined custom inventory rather than making their own.

Furthermore, it's set up like a huge, huge, huge, huge, huge pack, which makes things harder to find. The naming is also wth. Yea, you have CI for CustomInventory, but naming stuff like that implies coupling. Each resource should be independent but be able to be used together to create something awesome -> a custom inventory.

This is why I'd really rate it 4/5, but I'm a frigging nazi when it comes to this stuff, so I'm much harsher than others are : P.

If you fix the stuff I mentioned and treat your own custom inventory as a demonstration, walking through to teach people how to use the resources together, then I'd gladly support it as a 6/5 =D.

It may have good support, but it doesn't have good design.

 

[Almia]

I agree w/ you,seems to be good.

If you saw my spell Split Earth and compared it to Jazztatic's Voodoo Drain.

Voodoo Drain got 4/5 but mine got 3/5 w/c i have much more longer code than Voodoo Drain

 

[busterkomo]

So, hvo didn't think it was worth a DC (but had no point to complain about whatsoever) and TRD said it is perfectly coded and also super cool.

And yet it's a 5/5.

I don't remember any of this, but I'm not really the type of person to do something without justification.

 

[Anachron]

I don't remember any of this, but I'm not really the type of person to do something without justification.

To quote yourself from my resource thread:

18:27, 18th Feb 2010

The_Reborn_Devil:

Ohsh-
That describes pretty much what I'm thinking.
My eyes were sore even before I was done reviewing half of the code.
The code looks really good and I couldn't find just one little thing that would be enough to reject this...
This is the best inventory system I've seen and I've seen maaaany.


Edit:
Since you updated I had to review it again. It looks good, but I did notice one thing >

1. You don't need those null filters like in CIEventHandler
That bug has been fixed.

Other than that the code looks good and it's very easy to use.
Would be more awesome if the system had fewer lines though as this system is frikken HUGE. It hurts to review it.


Status: Approved
Rating: Director's Cut! [6/5] Plz?

hvo-busterkomo: After evaluating this system I do not think it meets the standards to be a Director's Cut. However, in respect to The_Reborn_Devil, I have left it as Highly Recommended.

So, you didn't give any feedback, just that you don't think it deserves DC.

 

[Pyramidhe@d]

HEY LOOK. A BELL GRAPH. ignore the comment about the IQ thingy. its not a lame quip about everyone's IQ around here. i just couldnt find a better clearer image.

when you take a test, you get grades like Participation, Credit, Distinction, High Distinction.

just like Useful, Recommended, Highly Recommended, Director's Cut.

so to go from Useful to Recommended, you do a considerable more amount of effort. and to go from Recommended to Highly Recommended, you do an exponentially more amount of effort. and so on and so forth.

so doing one thing better than another piece of resource doesnt mean it gets a higher rating. you must do an exponentially better job. so you really cant have a tickbox list of things to do to get a director's cut. there would be a list of criterias, but those list probably are not the definitive nor the only basis.

After evaluating this system I do not think it meets the standards to be a Director's Cut.

well i had this situation before, talking about art as an example, where an artwork is good. there is no doubt. but it doesnt reach that level of excellence that warrants it a director's cut. i guess judging between good and then better at that level will unfortunately fall in to subjective views. it is mostly just instinctive. its the difference between "damn, that's good" to "holy fucking shit...that is...awesome". it's not really concrete. but its just simply put as the "wow factor". ya. that's cliche but its the only usable explanation.

innovation
This is a stupid field. Who cares if something is new as long as it is done well.

umm. ya. exactly. the exact opposite of that comment. being innovative will be that "wow factor" in getting a exponentially better rating. because you can draw or code if you were taught. but you cant be taught to do something innovative.

what i mean by innovative, lets have coding for example. Person A could Code something to perform a task. Person B codes something to perform the same task but is done in a completely different way from the usual way. Person B then expands on his own coding that will allow a greater efficiency when the coding is faced with a more complex system. it could just compile more efficiently or it could have a process where it might just completely skip a pre-existing redundancy and save on processing. that will be being innovative.

a simpler example would be.

writing 8x5 instead of 8+8+8+8+8.

if there is a new way to do the same thing but better in some way, why wouldnt you care about it?

 

[Dr Super Good]

There is a difference between redundancy and being innovative. Being innovative in such an old game is nearly impossible. Making something that is redundant is possible and pointless to do.