Discussion of the new credit system

Message boards : Number crunching : Discussion of the new credit system

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 . . . 8 · Next

AuthorMessage
Astro
Avatar

Send message
Joined: 2 Oct 05
Posts: 987
Credit: 500,253
RAC: 0
Message 24686 - Posted: 24 Aug 2006, 12:43:32 UTC

Here is the "credit overview" from boincstats as it appeared as of today. It will be interesting to see what happens to it's currently linear ascension. Any subsequent drop will IMO be the result of reduced credit granted to those using optimized Boinc clients and the total loss of users due to the new credit systems implementation. The loss might possibly be offset by the addition of new users coming to rosetta after this change. All of this will be impossible to narrow down, but it will be interesting to see anyhow.

ID: 24686 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Trog Dog
Avatar

Send message
Joined: 25 Nov 05
Posts: 129
Credit: 57,345
RAC: 0
Message 24687 - Posted: 24 Aug 2006, 12:49:06 UTC - in response to Message 24675.  



Looking at your post, there might be a very easy solution : people get the running average, BEFORE their claim is taken into account. This would make it useless to overclaim, in fact it would NOT give you more credits, but it WOULD give more credits to those that report WU's after yours. How's that for discouraging optimized clients? You would only benefit from overclaiming if you were actually the first to report a certain WU (what are the odds?). In all other cases, overclaiming would give more credits to everyone behind you - but not yourself, thus you would not get higher in the ranking - the opposite would be true.


Example : first guy reports, claims and gets 100 credits. You report, claiming 300, getting 100, setting the running average to 200. Third guy reports, claims 100, and gets 200, running average goes down to 166, etc....



G'day Los Alcohilocos~Sloom

If you overclaim you won't get the benefit on your first result, but you will on every subsequent result from the same type of wu. Sounds like a great incentive to me - not only can I doctor my subsequent credits but also those of all my teammates.

The fact that everybody else potentially benefits is not really a disincentive, if you and your teammates get your results before everyone else.

Potentially this means as a team you all set your runtime to the minimum, and your credit claims to the maximum.
ID: 24687 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
R.L. Casey

Send message
Joined: 7 Jun 06
Posts: 91
Credit: 2,728,885
RAC: 0
Message 24688 - Posted: 24 Aug 2006, 12:49:21 UTC - in response to Message 24686.  

Here is the "credit overview" from boincstats as it appeared as of today. It will be interesting to see what happens to it's currently linear ascension. Any subsequent drop will IMO be the result of reduced credit granted to those using optimized Boinc clients and the total loss of users due to the new credit systems implementation. The loss might possibly be offset by the addition of new users coming to rosetta after this change. All of this will be impossible to narrow down, but it will be interesting to see anyhow.



Many new users and hosts have arrived over the past day... a good sign!

Users (last day ) : 78,055 (+377)
Hosts (last day ) : 172,223 (+709)
ID: 24688 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
R.L. Casey

Send message
Joined: 7 Jun 06
Posts: 91
Credit: 2,728,885
RAC: 0
Message 24691 - Posted: 24 Aug 2006, 12:59:44 UTC - in response to Message 24682.  
Last modified: 24 Aug 2006, 13:17:41 UTC

...I fully support the suggestion to wait until the first 10.000 results / 100.000 models are returned to start already with a very good average. I would even go further and stop changing the credit/model-ratio after a proper ratio has been establised. This would lead to the same credit/model-ratio applied to all participants.


Hope they have a good look at this suggestion.

The fact that all get the same credit/model sounds good to me.

Anders n


(R.L. Casey 11:42 UTC):
I just received "Pending Credit" for a WU just reported. Hmmm, perhaps the suggestion already has been implemented to wait for a stable average.


I've checked a number of results, and they show that the switch to granting "Pending Credit" happened today between 11:23 UTC and 11:27 UTC.


This fact has now been confirmed (edit: REPORTED from this post) in the 'The new credit system explained' thread.
ID: 24691 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
DigiK-oz

Send message
Joined: 8 Nov 05
Posts: 13
Credit: 333,730
RAC: 0
Message 24692 - Posted: 24 Aug 2006, 13:05:03 UTC - in response to Message 24687.  
Last modified: 24 Aug 2006, 13:06:37 UTC



Looking at your post, there might be a very easy solution : people get the running average, BEFORE their claim is taken into account. This would make it useless to overclaim, in fact it would NOT give you more credits, but it WOULD give more credits to those that report WU's after yours. How's that for discouraging optimized clients? You would only benefit from overclaiming if you were actually the first to report a certain WU (what are the odds?). In all other cases, overclaiming would give more credits to everyone behind you - but not yourself, thus you would not get higher in the ranking - the opposite would be true.


Example : first guy reports, claims and gets 100 credits. You report, claiming 300, getting 100, setting the running average to 200. Third guy reports, claims 100, and gets 200, running average goes down to 166, etc....



G'day Los Alcohilocos~Sloom

If you overclaim you won't get the benefit on your first result, but you will on every subsequent result from the same type of wu. Sounds like a great incentive to me - not only can I doctor my subsequent credits but also those of all my teammates.

The fact that everybody else potentially benefits is not really a disincentive, if you and your teammates get your results before everyone else.

Potentially this means as a team you all set your runtime to the minimum, and your credit claims to the maximum.



Maybe, maybe not. I would have to do some longterm calculations on this, but point is that you will probably not gain much yourself. By the time you get the next WU from the same batch in, the effect of your previous claim will have diminished. Especially if the WU has had a lot of results reported already. Chances that some of your teammates gain anything are slim, too. And even if you do gain anything, fact is that other members/teams will have a higher chance of benefitting from you, unless you're in a team that is over half the active participants. So, you might be winning in actual points, but other teams will be winning more. So you're actually helping your opponents more than you help yourself.

Having said that, there's a loophole in that when someone reports their work in large batches (like 10 WU's). In that case, 9 out of the 10 would benefit (assuming that no ther WU's are sent by other members inbetween yours). Still, much more than 9 following your contribution would benefit....

Besides, your reasoning goes for the current system as well - and more so, because you gain immediately no matter what the effect on your next or teammate's WU.
ID: 24692 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Trog Dog
Avatar

Send message
Joined: 25 Nov 05
Posts: 129
Credit: 57,345
RAC: 0
Message 24693 - Posted: 24 Aug 2006, 13:05:47 UTC - in response to Message 24679.  

Trog Dog, for someone to do anything, there has to be a motivation to do so. If someone hasn't already switched to using an optimized boinc core client, when you actually got whatever you claimed, then I don't think there's much motivation to switch to one now. 90% + of the active attached hosts are using standard boinc clients. Remember only the first few results returned of a new wu could be granted substantially higher claims, even when averaging. I doubt there's much incentive to switch to one now.

Yes, this might be a loop hole, but it's a SMALL one that might be exploited. Should it be addressed/considered? Yes I don't think it's much to get worried over though.

In Ralph, we discussed "cherry picking", however, from my own results I see no way that I could pick and choose which gave more. I think by the time enough results come in that "cherry Picking" might be possible, they'd be issuing a different wu and it wouldn't matter.



G'day mmciastro

There's plenty of psych/legal studies that have been carried out that correlate the propensity to "infringe" to the liklihood of getting caught. Typically increased penalties for speeding won't cause drivers to slow down, but an increased police/highway patrol presence , speed cameras will(ie chances of being caught will). So let's move from the abstract.

Suddenly under this new credit system a user (of an optimised client) can claim "but it wasn't benefiting me" - so less stigma. If everybody is getting more credits, how do you identify the users claiming more - check every wu returned? Only the project can do that.

Something else that just came to mind - intraproject it won't necessarily matter - interproject it will. I want to get my team to number one in BOINC combined statistics - solution have the team attach to Rosetta with bogus benchmarks.

Wait and see - I hope I'm wrong, but I've seen too many arguments over credits and "what MY boxes are ENTITLED to", to view things any differently.
ID: 24693 · Rating: -1 · rate: Rate + / Rate - Report as offensive    Reply Quote
Mod.DE
Volunteer moderator

Send message
Joined: 23 Aug 06
Posts: 78
Credit: 0
RAC: 0
Message 24694 - Posted: 24 Aug 2006, 13:09:21 UTC - in response to Message 24691.  

This fact has now been confirmed in the 'The new credit system explained' thread.

Actually I put this in after I read your post. ;-)
I am a forum moderator! Am I?
ID: 24694 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
R.L. Casey

Send message
Joined: 7 Jun 06
Posts: 91
Credit: 2,728,885
RAC: 0
Message 24695 - Posted: 24 Aug 2006, 13:22:18 UTC - in response to Message 24694.  

This fact has now been confirmed in the 'The new credit system explained' thread.

Actually I put this in after I read your post. ;-)


OK :-) I tweaked the prior post to say it was 'reported'... ;-)
ID: 24695 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Trog Dog
Avatar

Send message
Joined: 25 Nov 05
Posts: 129
Credit: 57,345
RAC: 0
Message 24697 - Posted: 24 Aug 2006, 13:29:20 UTC

https://boinc.bakerlab.org/rosetta/forum_thread.php?id=2197#24664

I'm making a point of this because I think it should be considered as a possibility.

I don't like tilting at windmills, but for the integrity of this project the cynical viewpoint must be considered.
ID: 24697 · Rating: -1 · rate: Rate + / Rate - Report as offensive    Reply Quote
FluffyChicken
Avatar

Send message
Joined: 1 Nov 05
Posts: 1260
Credit: 369,635
RAC: 0
Message 24699 - Posted: 24 Aug 2006, 13:44:40 UTC - in response to Message 24688.  

Here is the "credit overview" from boincstats as it appeared as of today. It will be interesting to see what happens to it's currently linear ascension. Any subsequent drop will IMO be the result of reduced credit granted to those using optimized Boinc clients and the total loss of users due to the new credit systems implementation. The loss might possibly be offset by the addition of new users coming to rosetta after this change. All of this will be impossible to narrow down, but it will be interesting to see anyhow.



Many new users and hosts have arrived over the past day... a good sign!

Users (last day ) : 78,055 (+377)
Hosts (last day ) : 172,223 (+709)



Though there's been a drop of active users (and about 10% drop of active hosts)

Team mauisun.org
ID: 24699 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Trog Dog
Avatar

Send message
Joined: 25 Nov 05
Posts: 129
Credit: 57,345
RAC: 0
Message 24701 - Posted: 24 Aug 2006, 13:49:27 UTC - in response to Message 24692.  




Maybe, maybe not. I would have to do some longterm calculations on this, but point is that you will probably not gain much yourself. By the time you get the next WU from the same batch in, the effect of your previous claim will have diminished. Especially if the WU has had a lot of results reported already. Chances that some of your teammates gain anything are slim, too. And even if you do gain anything, fact is that other members/teams will have a higher chance of benefitting from you, unless you're in a team that is over half the active participants. So, you might be winning in actual points, but other teams will be winning more. So you're actually helping your opponents more than you help yourself.

Having said that, there's a loophole in that when someone reports their work in large batches (like 10 WU's). In that case, 9 out of the 10 would benefit (assuming that no ther WU's are sent by other members inbetween yours). Still, much more than 9 following your contribution would benefit....

Besides, your reasoning goes for the current system as well - and more so, because you gain immediately no matter what the effect on your next or teammate's WU.


G'day again LosAlcoholicos~Sloom

"Maybe, maybe not" - that uncertaintity would be enough for some individuals and/or teams to do what I'm hypothesising. That afterall, has been amongst the reasons given over the last weeks/months as to why optimised clients are used - "everybody else is using them" "team ?? are using them so why shouldn't we" "os ?? users are using them so why shouldn't we" "my cpu is disadvantaged compared to your cpu, so I'm using them".

"by the time you get your next wu from the same batch in" - what about mutiple hosts - same user multiple hosts? Never see the same wu type across multiple hosts?

For every cynical & devious idea I'm thinking up, I bet that there are ten more out there in the wild. Prove me wrong - and I say that without cynicism, because I hope that I am wrong. Only time will tell.
ID: 24701 · Rating: -1 · rate: Rate + / Rate - Report as offensive    Reply Quote
R.L. Casey

Send message
Joined: 7 Jun 06
Posts: 91
Credit: 2,728,885
RAC: 0
Message 24703 - Posted: 24 Aug 2006, 14:03:01 UTC - in response to Message 24699.  

Here is the "credit overview" from boincstats as it appeared as of today. It will be interesting to see what happens to it's currently linear ascension. Any subsequent drop will IMO be the result of reduced credit granted to those using optimized Boinc clients and the total loss of users due to the new credit systems implementation. The loss might possibly be offset by the addition of new users coming to rosetta after this change. All of this will be impossible to narrow down, but it will be interesting to see anyhow.



Many new users and hosts have arrived over the past day... a good sign!

Users (last day ) : 78,055 (+377)
Hosts (last day ) : 172,223 (+709)



Though there's been a drop of active users (and about 10% drop of active hosts)

Yes, that's true, and I don't have the benefit of long-term stats; however, I did check Einstein@home & SETI@home, and they, too, display a similar decrease in user and host activity. All three had apparent minima in mid-August and now activity seems to be on the rise. It may be an effect due to summertime vacations and travel--or maybe too hot for some to run the crunchers?? :-)
ID: 24703 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
DigiK-oz

Send message
Joined: 8 Nov 05
Posts: 13
Credit: 333,730
RAC: 0
Message 24704 - Posted: 24 Aug 2006, 14:13:27 UTC - in response to Message 24701.  




Maybe, maybe not. I would have to do some longterm calculations on this, but point is that you will probably not gain much yourself. By the time you get the next WU from the same batch in, the effect of your previous claim will have diminished. Especially if the WU has had a lot of results reported already. Chances that some of your teammates gain anything are slim, too. And even if you do gain anything, fact is that other members/teams will have a higher chance of benefitting from you, unless you're in a team that is over half the active participants. So, you might be winning in actual points, but other teams will be winning more. So you're actually helping your opponents more than you help yourself.

Having said that, there's a loophole in that when someone reports their work in large batches (like 10 WU's). In that case, 9 out of the 10 would benefit (assuming that no ther WU's are sent by other members inbetween yours). Still, much more than 9 following your contribution would benefit....

Besides, your reasoning goes for the current system as well - and more so, because you gain immediately no matter what the effect on your next or teammate's WU.


G'day again LosAlcoholicos~Sloom

"Maybe, maybe not" - that uncertaintity would be enough for some individuals and/or teams to do what I'm hypothesising. That afterall, has been amongst the reasons given over the last weeks/months as to why optimised clients are used - "everybody else is using them" "team ?? are using them so why shouldn't we" "os ?? users are using them so why shouldn't we" "my cpu is disadvantaged compared to your cpu, so I'm using them".

"by the time you get your next wu from the same batch in" - what about mutiple hosts - same user multiple hosts? Never see the same wu type across multiple hosts?

For every cynical & devious idea I'm thinking up, I bet that there are ten more out there in the wild. Prove me wrong - and I say that without cynicism, because I hope that I am wrong. Only time will tell.



I will not prove you wrong, or try to counter your devious ideas. The suggestion I made just takes one extra reason OUT of overclaiming. It doesn't prevent every single cheat you can come up with.

In the current system, overclaiming gives you extra credit, and MIGHT make a difference on your next WU or your teammate's. But YOU are the main beneficiary.

In my suggestion, the first reason is gone. What remains is you MIGHT get more credits on your next WU or your teammate's.

So, I still think that it could be implemented, and probably very easy :

Cuurent system : Calculate a new rolling average, then assign average points

My idea : assign averige points, then calculate new rolling average.
ID: 24704 · Rating: -1 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Trog Dog
Avatar

Send message
Joined: 25 Nov 05
Posts: 129
Credit: 57,345
RAC: 0
Message 24705 - Posted: 24 Aug 2006, 14:31:19 UTC - in response to Message 24704.  




I will not prove you wrong, or try to counter your devious ideas. The suggestion I made just takes one extra reason OUT of overclaiming. It doesn't prevent every single cheat you can come up with.

In the current system, overclaiming gives you extra credit, and MIGHT make a difference on your next WU or your teammate's. But YOU are the main beneficiary.

In my suggestion, the first reason is gone. What remains is you MIGHT get more credits on your next WU or your teammate's.

So, I still think that it could be implemented, and probably very easy :

Cuurent system : Calculate a new rolling average, then assign average points

My idea : assign averige points, then calculate new rolling average.


Thanks for your ideas Sloom. We'll soon see what happens ;) I hope that I am proved wrong.

ID: 24705 · Rating: -1 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Hoelder1in
Avatar

Send message
Joined: 30 Sep 05
Posts: 169
Credit: 3,915,947
RAC: 0
Message 24707 - Posted: 24 Aug 2006, 14:33:52 UTC - in response to Message 24699.  
Last modified: 24 Aug 2006, 14:38:14 UTC

Many new users and hosts have arrived over the past day... a good sign!

Users (last day ) : 78,055 (+377)
Hosts (last day ) : 172,223 (+709)
Though there's been a drop of active users (and about 10% drop of active hosts)
Yes, there has been quite a dramatic drop in active users and hosts over the last two months - not just in Rosetta but in all of BOINC. But since yesterday active hosts and users seem to be on the rise again in Rosetta while the numbers for all of BOINC are still low. It seems people are voting with their feet (or rather their mouse and keyboard) for the new credit system.
Team betterhumans.com - discuss and celebrate the future - hoelder1in.org
ID: 24707 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Paydirt
Avatar

Send message
Joined: 10 Aug 06
Posts: 127
Credit: 960,607
RAC: 0
Message 24720 - Posted: 24 Aug 2006, 15:12:47 UTC

I'm not sure what math is being used to assign credits... If they are going to wait for ## or ### results before they start granting credit, then I think it makes the most sense to go with a median, or a mean/average with the outlying tails thrown out. So if you had...

20 25 30 30 30 32 34 35 100

The median would be 30, the mean with the lowest and highest observation thrown out would be 31. Straight mean is 37.


If you go with a straight mean, then you have to monitor the misreporting results, and you would have to adjust the granted results for all reporting computers. It would be a hassle to administer. If you could run a median or a "mean/average minus outliers", then this problem goes away.

-Bradford
ID: 24720 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Keith Akins

Send message
Joined: 22 Oct 05
Posts: 176
Credit: 71,779
RAC: 0
Message 24724 - Posted: 24 Aug 2006, 15:27:59 UTC

Let's see what happens. Will be interesting to see how long credit validation takes. Other projects that take three or more WU's to validate results takes up to three days for some to grant credit. Here we're talking 100's to 1000's.

This is shaping up to be a very interesting experiment.
ID: 24724 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Saenger
Avatar

Send message
Joined: 19 Sep 05
Posts: 271
Credit: 824,883
RAC: 0
Message 24725 - Posted: 24 Aug 2006, 15:31:02 UTC - in response to Message 24724.  

Let's see what happens. Will be interesting to see how long credit validation takes. Other projects that take three or more WU's to validate results takes up to three days for some to grant credit. Here we're talking 100's to 1000's.

This is shaping up to be a very interesting experiment.

But that's 3 per single WU, here it's 1000 (is it btw?) per type of WU.
Let's wait and see (Abwarten und Tee trinken);)
ID: 24725 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
FluffyChicken
Avatar

Send message
Joined: 1 Nov 05
Posts: 1260
Credit: 369,635
RAC: 0
Message 24728 - Posted: 24 Aug 2006, 15:37:40 UTC - in response to Message 24707.  

Many new users and hosts have arrived over the past day... a good sign!

Users (last day ) : 78,055 (+377)
Hosts (last day ) : 172,223 (+709)
Though there's been a drop of active users (and about 10% drop of active hosts)
Yes, there has been quite a dramatic drop in active users and hosts over the last two months - not just in Rosetta but in all of BOINC. But since yesterday active hosts and users seem to be on the rise again in Rosetta while the numbers for all of BOINC are still low. It seems people are voting with their feet (or rather their mouse and keyboard) for the new credit system.



I had written a longer one before with graphs, but my connection dropped.

The problem with the graphs is we will not see the effect until a month or so. Since it is a graph of granted in last 30 days. So what you are really seeing is the people that stopped a month ago.

Team mauisun.org
ID: 24728 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Hoelder1in
Avatar

Send message
Joined: 30 Sep 05
Posts: 169
Credit: 3,915,947
RAC: 0
Message 24734 - Posted: 24 Aug 2006, 15:53:14 UTC - in response to Message 24720.  
Last modified: 24 Aug 2006, 15:55:59 UTC

The median would be 30, the mean with the lowest and highest observation thrown out would be 31. Straight mean is 37.
I made that same suggestion some time ago and David Kim explained that the only numbers he has available are the cumulative claimed credit and cumulative number of returned models for each batch of WUs, so he can't calculate the median from that. Anything else would require some lengthy messing with the database. And yes, he also mentioned a correction factor to account for outliers that would need to be reviewed occasionally.
Team betterhumans.com - discuss and celebrate the future - hoelder1in.org
ID: 24734 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Previous · 1 · 2 · 3 · 4 · 5 . . . 8 · Next

Message boards : Number crunching : Discussion of the new credit system



©2024 University of Washington
https://www.bakerlab.org