Advertisement

Choose Your Background

  • Noah Bradley: A place to call Home

  • Noah Bradley: Their Mournful Tones



A different fundamental approach to the rating system

#1
I'm a teacher, and one frustration I have with the rating system is that any given piece of writing will, and should, receive different levels of assessment based on how it is being judged.

The exact same paper could get an A in an 8th grade class, and would get an F in university. To give one overall grade depends on the perspective that the reviewer has in judging it, on what "stage" they see the author as presenting it. Whether the author means it as a serious entry into the finest works of humanity, or as a fun romp, or as a learning experience makes a big difference.

So, I think it'd be useful to include a different sort of master category for stories - the stage.

Example stages could be:

World Stage: The author feels that this work ought to be judged according to the standard of the best humanity has to offer. It should be judged alongside Douglas Adams, Isaac Asimov, and modern major works, like Harry Potter.

Standard Publication: The author feels that this work is on par with normal published works, and feel it should be judged accordingly.

Just For Fun: The author's writing is whimsical, or at some level, not pursuing perfection, and as such, it should not be taken seriously. Therefore, it should be judged by how fun it is to read, rather than any serious standards.

Learner's Lounge: The author feels that this work ought to be judged with the understanding that the author is learning or otherwise presenting a notably imperfect work, but it nonetheless has value.


If an author marks his story as World Stage, he's inviting heavy criticism and harsh reviews. Getting 3 stars average is a major achievement. If marked as Standard Publication, he's inviting serious criticism, but 

Many stories that get a 5 star rating under the current system ought to get 3 stars or less with such a standard. 

Take Arcane Emperor, for instance - it's a fun story, to be sure, and a lot of people like it. By the standards of "just for fun," it definitely deserves 4-5 stars. By "Standard Publication," it should get 3-4 stars, tops, because of poor technical issues, like pervasive grammar issues. By "World Stage," it should get 2 stars, tops, and that'd be wildly generous. I've spoken to the author about technical writing skills, and the author spoke of being unconcerned about the technical errors, since they weren't bad enough to entirely wreck the story. That implies that the author feels the work should be judged by "Just For Fun" standards.

The basic idea of allowing works to be evaluated, not by a global standard, which invites excessive flattery (shown by the insanely high ratings that stories get, on average), but rather, by different standards according to what the author is going for, can be implemented in a variety of ways.

It could even be part of the review process exclusively, not letting the author decide. Reviewers could pick which "stage" they feel the story is best judged by, and judge it accordingly.

However it is implemented, it would be an enormous commitment and overhaul, to be sure. That said, I feel the idea would address the key, well established flaw with RRL, which is the near uselessness of the reliability of the ratings.
Writing Sorcery in Boston

- RubyCona
Reply
#2
It's a nice thought, but the problem with your premise is the fact that you expect a different pool of readers for each of your stages when in reality every author in RRL is sharing a unified reader base.

It doesn't matter if as an author I want people to judge my work just for the fun quality of my story instead of hoping to be the next literary gem destined to win the Nobel prize one day, I will still be judged by the casual readers, the trolls, the grammar nazis, the serious reviewers and all the other tropes out there.

It is my experience that 90% of the people who review the work of others because they have a strong knee-jerk reaction to one story, and not to actually help the author.

Book 1, author pours heart and soul in the story because he is hoping to become a serious author in the future:
Casual reviewer: Meh, no harem.  take my 0.5-star review!

Book 2, an author is just bored and think it will be hilarious to make a parody of other works in RRL.
Casual reviewer: still no harem, 0.5 stars to you as well!
Check my fictions
"Invincible"
"Gamer of the Dead"
And the newest  "Slam No Basuke"
Reply
#3
It's not that I expect a different pool of readers. Any one reader can judge the same work by different standards. I have students and colleagues do the same thing all the time.

It's that I'm hoping that reviews, etc, can be attached to a certain judgement standard.

Also worth noting that both of the example reviews you mentioned are invalid, according to current RRL rules, and ought to be reported for removal. Those are not proper ratings.

By the current system, by having practically all remotely half decent stories have nearly 5 star ratings, then literally anything other than 5 stars is hurting their stats. My story is 4.7 stars - it takes two five star ratings just to recover from a single 4 star rating, let alone 3s or lower.

Which means that authors feel like they're being punished by 4 star ratings, when, with a remotely sane review system, 4 stars should be honestly a good review.

With something like a "World Class Stage," inviting reviews to be actually serious, then it becomes acceptable to give a 3-star rating to a story you like, and have that be a good thing. Rather than needing to give 5 stars just to avoid hurting their score, you can give a more nuanced appraisal.

Regardless of whether my suggestion is the best solution, the fact remains that the current system is extremely flawed, far beyond simply RRL. RRL is known for this, in particular, but it's a problem at large.
Writing Sorcery in Boston

- RubyCona
Reply
#4
I don't really get what problem you're trying to solve here. To get more accurate reviews, I guess? But why would dividing the reviews up into more categories address the underlying issue - the unreliability of the reviews in the first place. The numbers would be just as skewed in the new system.

In the first place I think that our current star system already has the most important - unspoken - qualifier. That is to say, if you see a story with five stars you have to qualify that statement with "it is a story with five stars... on RRL". That is the category we are all laboring within to begin with and probably the only category the userbase is qualified to weigh in on. I mean how many people come to this site with both the time, ability, and inclination to weigh the new fiction they started reading on a "world class stage"? I somehow doubt there are a bunch of Masters and PhDs with degrees in technical editing and literary theory running around.

The reviews are skewed because they come from the specific userbase of this website and are reflective of our niche communities' tastes and interests, which isn't really a bad thing I don't think - just a reality everyone should be aware of.
Read my story if you want it's called Once More
Reply
#5
(28-10-17, 12:08 AM)rubycona Wrote: By the current system, by having practically all remotely half decent stories have nearly 5 star ratings, then literally anything other than 5 stars is hurting their stats. My story is 4.7 stars - it takes two five star ratings just to recover from a single 4 star rating, let alone 3s or lower.

Which means that authors feel like they're being punished by 4 star ratings, when, with a remotely sane review system, 4 stars should be honestly a good review.

Dude. One of my stories have 4 0.5 star ratings. For some reason it also has 4 5.0 star ratings (and 1 4.0 rating) to swing it to 2.94. And you know what? The exact number doesn't matter to me. What matters is whether more people like the story or not (5 versus 4). Is 2.94 a terrible rating? Sure. But it doesn't tell the whole story (I just have to use that word...). But rather than the value itself, I look at how many people rated it 0.5 as opposed to those who rate it highly. If there's more people in one side than the other, then I can tell whether it's worth continuing the story regardless of whether it's a 2.75 or 3. Recover? What do you mean, "recover"? It sounds like you want to maintain your story at a 5.0 rating to justify that it's a good story. I mean, I don't think that's the case, but honestly, let's not get obsessed with ratings.

I used to do that in the past, but after a while I realized it's better to measure myself by seeing how many readers like the story as opposed to being obsessed with the actual value.

Not to mention, your idea is hardly going to solve the problem. Almost everyone, including the authors of Arcane Emperor, etc. will more than likely put their work in Just for fun, and the guys whose English is their second or third language will just stay in Learners' Lounge. And most likely it'll just go back to the whole 5.0 ratings in the just for fun category (where the vast majority of the works will be). It changes nothing, except with a couple of World Stage categories that nobody will use. Because let's be honest - if they think their works are master class or publication standard, they wouldn't be posting here in a writing website. They would be submitting them to publication companies or self-publishing on Amazon.

Honestly, authors feel like they get punished by 4.0 ratings? Then what about those of us who have ratings of 0.5, 1.0, 1.5 or 2.0? Those of us with ratings of 3.0 and less? Should we pack up and quit writing for good?
Reply
#6
I agree about the knee-jerk reaction reviews. Holy crap. I just changed the ending to my novel to better fulfill the needs of my next book. Two people gave me .5 stars cause they hated the change. The one thing I wish for, and I know its never going to happen. I'd like to see who gave me those reviews, so I can at least ask them why.

On the note about review pool. The system we have isn't perfect, far from it, but it does even out. What we do have is a slice of the real people. There are smart, consciences readers who can articulate their reasons for liking a novel. Then we have the morons and the lazy. Just like being Amazon, the number of stars you get averages out. I have two (.5 star) reviews, and two (1 star) reviews now. On the flipside, I have like 37 (five star) reviews. I know my novel isn't perfect.

On Newgrounds they used to have a weighted system. Users who gave good reviews, or were active were treated as being super reviewers. My buddy Chris used to do this. His vote would be worth like fifty or a hundred reviews. I remember this because sometimes he'd find some awful piece of drivel. He'd be like, "be banished the void." and slam negative review button.

I just went to look at their system. (Haven't been on Newgrounds in years! Wow, it's changed.) Copied their stuff here.



Voting Power -
Voting Power is based on a user's Level and Blam / Protection points. Take a look at the respective sections to see how voting power is affected by each.



Level
A level is a number (between 1 through 60, inclusive) along with an icon of a fist, glove, or weapon for a user based on that user's experience. Levels serve no purpose except to increase the voting power in the Flash Portal and to indicate status. For this reason a higher level often commands a certain level of respect.



Experience Points
Much like a traditional role playing game, registered users may acquire experience points and gain levels resulting in higher status. This is done by voting on five games and/or movies per day, which results in a deposit of ten experience points. You can only log ten points per day, there are no exceptions.

Experience points add weight to your votes. Users with an abundance of points will have the strength of multiple voters and can hurt the movies they hate or help the movies they love. Remember, you must be logged in for your voting power to be augmented!

Experience points also unlock secrets on your account page.

When the system was introduced in 1999, users could vote on each unique submission once daily. This meant you could return to Newgrounds on a daily basis and upvote the content you liked, or downvote the content you disliked. This resulted in a habit of "zero bombing" that was frowned upon and resulted in many discussions about the voting system.

In 2015 the voting system was changed to allow only one vote per unique submission.



Blams / Saves
Blam and Protection points are earned by voting on a movie or game that is "under judgment." If a user votes a 0-1 on a movie and it is subsequently deleted (see deletion qualifications below), the user receives one Blam point. Conversely, when a user votes 2-5 on a movie that successfully passes the judgment phase then that user receives a protection point.

Movies are deleted if they fall below the listed score at the listed number of votes:
50 votes: 1.0
70 votes: 1.5
90 votes: 1.75
Blam and Protection points augment a user's voting power (which otherwise would be strictly based upon experience). The increase is a percentage bonus based on the user's level. You also get a fun little badge to show how 1337 you are.
Reply
#7
You know, while we're on this topic, and because we all know the rating is highly subjective, we can ditch the numerical value system altogether (keep it for the reviews). So instead of rating from a scale of 0.5 to 5.0, readers can dislike it like it in the same manner as Facebook or something. So you have Hate, Dislike, Neutral, Like, Love. Kind of equivalent to the old rating of 1, 2, 3, 4 and 5, but it more clearly emphasizes how subjective the rating is rather than make it look like a grade. Furthermore, it will not be as discouraging to new writers who aren't that good in the language because if you like the story even though the grammar or whatever deserves a low rating, you can still Like the story or something.

Of course, I don't know what's worse - to see half of your readership giving you 0.5 ratings, or tell you directly that they Hate the story, so this idea might not be practical at all.
Reply
#8
The problem with this system is that the vast majority of reviewers either don't care about or don't understand the finer points of literary craftsmanship. A reader might say "This chapter needs to be fixed author." and then say something about showing instead of telling, whereas their actual issue isn't the issue they named but rather the fact that there wasn't significant plot progression in a chapter.

I encounter these kinds of reviews more often than not, and that means you need to analyze and reinterpret a reader review to make sense of it.

Another thing that I should mention is that this type of review stratification already exists. I, as somebody who has written multiple stories placed quite highly on various independent accounts (so no readers checking out my works strictly because they enjoyed a different story of mine, they're all independent), have noticed that once you hit about 100 reviews or around the 500-800 follower mark your reviews will start becoming much harsher and more critical. This is because your story is stacking up in a different crowd among different stories. It's no longer a matter of you simply being on the map, but rather a matter of what your rank is among other good fictions.

Ultimately however, the ranking system is highly subject to taste. Invariably, higher ranked fictions will be stories that people on RoyalRoadl simply like more. There's little that can be done to change than, since any new implemented system is bound to follow suit.

If you want a different kind of feedback, your most reliable option is to simply publish your story for review in different communities. If you want feedback in a 'learners lounge' category, post your story something like Reddit Writing prompts or the spacebattles creative writing forum. Or here on RoyalRoadl, though most high-visibility stories on this website fall more under your just-for-fun category. Fictionpress and Wattpad tend to cater to this and the semi-serious crowd.

If you want to see if your stories are competitive in a live market then your best bet is to start self-publishing. There's a mountain of quality difference between what I publish on sites like this one as practice work, and what I actually self-publish over at amazon.  And because they're published works they're held to a higher standard.

If you plan on moving to an even higher playing field than that so you can start contending with the big boys, your best option is to find a traditional publisher. If you can't get one, your story probably isn't good enough.

Finally 'World-class' literary works can't really be fully evaluated for years. What makes or breaks them is the readership. Harry Potter did not get very flattering reviews from other writers when it was first introduced to the masses. I personally find 'A Song of Ice and Fire' to be somewhat poorly planned. More exaggerated examples of this phenomenon are the Twilight and 50 Shades of Gray books. Universally mocked among literary communities but still quite successful. It's my opinion that this isn't really a category you can claim to aim for. You just write your best and maybe your fans will decide to put you here.

Ultimately that's my advice to any aspiring writer. Don't write to be the next Hemmingway or Shakespeare. Write the best story you can, focusing your efforts on what you consider to be the most important. Leave the rest up to the readers.
Reply





Users browsing this thread: 1 Guest(s)