Star Scale Scores Modified

Org Profile · Post by **Phade** » Sun Jul 13, 2003 9:08 pm

Hey All,

I've written a script that detects if a member is giving about the same score each time they rank a local download with the star scale and then ignores that member. What this means is that if someone keeps giving almost all 1's or 5's or 3's or whatever, all of that members scores are ignored since they obviously aren't taking the scoring seriously. There are some members who have score hundreds of vids and given them all the same score: some all 5's, some all 1's, some all 3's. Of the 3000 or so members giving star scores, only about 160 are affected by this rule.

Hopefully this will make the star scale more reflective of people who actually put thought into their scoring.

Phade.

Deaths_ally · Post by **Deaths_ally** » Sun Jul 13, 2003 11:18 pm

interesting Idea.
now wouldn't it be nice to do the same to the opinions [/way off track thought]

I like that idea phade. not that i'm really fussed.. I'm going to try and avoid local downloads until I donate.

EarthCurrent · Post by **EarthCurrent** » Mon Jul 14, 2003 12:07 am

So if I did nothing but download average videos all deserving 3s my scores are ignored?

hmmm....

TaranT · Post by **TaranT** » Mon Jul 14, 2003 12:14 am

So that's why all my ratings dropped.

Just out of curiosity, does this algorithm account for number of ratings that a person has given? Someone who has given only two ratings, both 5-stars, could be doing the right thing, as opposed to someone who has given 392 ratings, all 5-stars.

dwchang · Post by **dwchang** » Mon Jul 14, 2003 12:22 am

TaranT wrote:So that's why all my ratings dropped.

Just out of curiosity, does this algorithm account for number of ratings that a person has given? Someone who has given only two ratings, both 5-stars, could be doing the right thing, as opposed to someone who has given 392 ratings, all 5-stars.

Whoa good point. I also think it should take into account averages given AND # of ratings...like someone who has a rating below 1.5 or something and 50+ ratings is probably doing something malicious. Then again, I guess it is possible to download that many bad videos.

Org Profile · Post by **Phade** » Mon Jul 14, 2003 9:15 am

Hey,

The formula goes something like: You must have a ratings count greater than a certain number and a standard deviation less than a certain small number. This way, if you are just starting out and download videos starting at the top of the star scale list, your score will stay until you reach a certain number. By the time you reach the count criteria, your standard deviation should be greater than the small number unless you are just handing out all 5's. The check is updated frequently.

I hope that clears it up some.

Phade.

turboneko · Post by **turboneko** » Mon Jul 14, 2003 11:09 am

Phade wrote:There are some members who have score hundreds of vids and given them all the same score: some all 5's, some all 1's, some all 3's. Of the 3000 or so members giving star scores, only about 160 are affected by this rule.

Hopefully this will make the star scale more reflective of people who actually put thought into their scoring.

dwchang wrote:Whoa good point. I also think it should take into account averages given AND # of ratings...like someone who has a rating below 1.5 or something and 50+ ratings is probably doing something malicious. Then again, I guess it is possible to download that many bad videos.

Hmmm... well, we are in those 160 that received the nice email from Phade

I have to admit that having a star rate average around 1.50 should be enough to be catalogued as a "video basher", but let me explain the reasons behind this low scores.

First of all, I am very very critic: can't do much about that. This goes back to the times when I was giving out free reviews and hasn't really changed: when 5 is the average (2.5 in terms of stars), I can't see why I should rate 7 or 8 something that's just average.

This doesn't mean that I am not able to rate high: when the video really deserves it I am more than happy to give high marks (or stars). But honestly, of the nearly 20,000 vids that are registered on this site, how many are really enjoyable?

I'd say that we can divide them in the following percentages:

20% are really crappy
50% are just ordinary, average videos that I'll watch just once
15% are nice vids that I'll replay more than once
5% are the really good ones

Now, what makes a good video for me? First of all a good, solid concept. I am really tired of videos made just for the sake of it, without any reason and any planning behind the music/anime combo. I understand that it's not possible to make a masterpiece out of every music video (I've made quite a few senseless videos myself), but a bit of careful planning it's needed if you want to deliver something that is interesting.

Another important point that is seriously understimated is the lenght of the song. If you want to make an 8:00 min video be my guest, but be EXTRA sure that you are going to make it enjoyable. The average attention span drops after 2:50, that's unavoidable.

Now, back to the star scale. I've downloaded quite a few videos that were popping up in the first places with high scores (like 4.80) and only 5 or 6 ratings: most of them honestly sucked (here are the first 1's).

Second, I have a taste for bad videos/movies/stuff, so sometimes I download them just for fun (here they go the other 1's).

Third, the scale doesn't give you much space when you have to rate a video. If there is a video that completely sucked, but had at least one or two nice shots, I'll still rate it 1 because I feel that 2 is too much for just a couple of nice edits. Maybe allowing half scores would help and will make scoring a little more precise.

Fourth, the carrot has given to everybody what they always dreamt of: cheap cheap hosting. This gave to a lot of people the great opportunity of finally sharing their work... but also opened the door to tons of crappy videos. Let's face it: almost everybody can make a music video with little or no knowledge if he/she really wants. Hell, how many times I've put another song on an existing video and found out that it works better than the original?!?

In the overall, I admit that I may have been a little too harsh in some cases... and with the stars it's even worst because there is no comment associated with the rating (hey, what about the option of adding a two liner to your star rate?

): in writing opinions I had the opportunity of fully explaining my scores.

Well, I hope that this explains a little bit my position: I do not give out straight ones just for the fun of it (hopefully this came out from my post

). If anybody has pointers, comments and/or rants, I am open to any discussion.

- Marco

Org Profile · Post by **Phade** » Mon Jul 14, 2003 11:38 am

Hey turboneko,

Actually, your scores are still in because your standard deviation is above the cutoff limit. I wrote you a note because your count is high and average is quite low. Of the ~3000 members leaving a star rating, your average is the 22nd lowest, counting the 7 people who gave just one score of 1 (counting the people who have given 5 or more scores, you are 13th from the lowest). There are some people who have given ~50 scores with an average of less than 1.2. There is also a member who has given 200+ scores, all of them 5. It's those people who are being ignored.

Phade.

dwchang · Post by **dwchang** » Mon Jul 14, 2003 1:11 pm

Phade wrote:Hey turboneko,

Actually, your scores are still in because your standard deviation is above the cutoff limit. I wrote you a note because your count is high and average is quite low. Of the ~3000 members leaving a star rating, your average is the 22nd lowest, counting the 7 people who gave just one score of 1 (counting the people who have given 5 or more scores, you are 13th from the lowest). There are some people who have given ~50 scores with an average of less than 1.2. There is also a member who has given 200+ scores, all of them 5. It's those people who are being ignored.

Phade.

It might also be nice to have an automated system which mails people to tell them their scores are being dropped (useless?). It might lead them to remedy the situation.

I also think a comment or two would be nice (~255 characters ). That also goes along with my suggestion for non-anonymous

. At the same time, it *somewhat* defeats the purpose of a system (very fast), by adding 20 seconds to the time it takes to do star-ratings...which could discourage *slightly*.

turboneko · Post by **turboneko** » Mon Jul 14, 2003 1:32 pm

Thanks for the reply Phade... it's nice to know that I'm still in

About the star system... I agree with dwchang and actually it could be modified in this way:

1) When you star-rate a video, you are not required to submit any comment

2) If you do not submit a comment, your rating will be anonymous

3) If you do submit a comment, there should be a checkbox that allows you to choose between anonymous/non anonymous

4) It should be possible to see for each video all the non-anonymous star "comments" that people left: making this stuff public may help in understanding if the video suits the personal taste before downloading it without completely spoil it (as it could happen with opinions).

The checkbox at point 3 is mainly to avoid "revenge" ratings... if stars were not anonymous I would have already collected tons of them

On the other end, if you have the possibility of justifying your low score with a small comment, people would not take it so bad (hopefully

) and it may actually help the feedback system.

What do you guys think of it?