Report work units with unusual amount of credits granted

Message boards : Number crunching : Report work units with unusual amount of credits granted

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Mod.DE
Volunteer moderator

Send message
Joined: 23 Aug 06
Posts: 78
Credit: 0
RAC: 0
Message 24570 - Posted: 24 Aug 2006, 7:49:01 UTC
Last modified: 24 Aug 2006, 7:50:28 UTC

This thread is supposed to be a collecting point of work units, which got an unusual amount of credits granted.

It has a similar function like the "Report Problems with Rosetta Version XXX" thread, where you can report work units which produce an error.

Please don't discuss the credit system as a whole in this thread, just report the work unit, for which you think the granted credits are inappropriate.
I am a forum moderator! Am I?
ID: 24570 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile anders n

Send message
Joined: 19 Sep 05
Posts: 403
Credit: 537,991
RAC: 0
Message 24597 - Posted: 24 Aug 2006, 8:36:53 UTC
Last modified: 24 Aug 2006, 8:37:12 UTC

Any thoughts to when a WU gets to much or to little?

Is it 20-30% up or down or is it the wu-s with 0 or very much?

Anders n
ID: 24597 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Mod.DE
Volunteer moderator

Send message
Joined: 23 Aug 06
Posts: 78
Credit: 0
RAC: 0
Message 24599 - Posted: 24 Aug 2006, 8:42:52 UTC - in response to Message 24597.  

Any thoughts to when a WU gets to much or to little?

Is it 20-30% up or down or is it the wu-s with 0 or very much?

Anders n

That is up to you to decide. A variation of 10% is certainly nothing to report and even 20% seems not way off for me (if there is no pattern). This thread is just meant to collect "suspicious" WU and then we will see if it is a "natural" variation or something to bugfix.

I am a forum moderator! Am I?
ID: 24599 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile anders n

Send message
Joined: 19 Sep 05
Posts: 403
Credit: 537,991
RAC: 0
Message 24601 - Posted: 24 Aug 2006, 8:46:22 UTC
Last modified: 24 Aug 2006, 8:46:29 UTC

Then I think this one got to much credit.

https://boinc.bakerlab.org/rosetta/result.php?resultid=34074163

Anders n
ID: 24601 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
FluffyChicken
Avatar

Send message
Joined: 1 Nov 05
Posts: 1260
Credit: 369,635
RAC: 0
Message 24611 - Posted: 24 Aug 2006, 8:58:12 UTC - in response to Message 24601.  

Then I think this one got to much credit.

https://boinc.bakerlab.org/rosetta/result.php?resultid=34074163

Anders n


That's well within the tollerances of this new system.
That variation was seen over at Raplh. the the code for working it out has changed I wouldn't expect the variation to have altered that much.
All you credits seem to be on average higher.
Team mauisun.org
ID: 24611 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile anders n

Send message
Joined: 19 Sep 05
Posts: 403
Credit: 537,991
RAC: 0
Message 24616 - Posted: 24 Aug 2006, 9:07:07 UTC - in response to Message 24611.  
Last modified: 24 Aug 2006, 9:08:49 UTC

That's well within the tollerances of this new system.
That variation was seen over at Raplh. the the code for working it out has changed I wouldn't expect the variation to have altered that much.
All you credits seem to be on average higher.


Well I don´t think ~55% is within the tollerance.

As for getting higher credit it must be the optimized clients pushing the

average up.

Anders n

Edit Spelling

ID: 24616 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Aaron Finney

Send message
Joined: 8 Oct 05
Posts: 52
Credit: 109,589
RAC: 0
Message 24620 - Posted: 24 Aug 2006, 9:11:09 UTC - in response to Message 24570.  

This thread is supposed to be a collecting point of work units, which got an unusual amount of credits granted.

It has a similar function like the "Report Problems with Rosetta Version XXX" thread, where you can report work units which produce an error.

Please don't discuss the credit system as a whole in this thread, just report the work unit, for which you think the granted credits are inappropriate.


Rosetta shouldn't be your testing ground. You have RALPH for that.

If what you are saying is that the credit system you implemented with NO WARNING, still needs more testing....

I fear for this project.
ID: 24620 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
FluffyChicken
Avatar

Send message
Joined: 1 Nov 05
Posts: 1260
Credit: 369,635
RAC: 0
Message 24623 - Posted: 24 Aug 2006, 9:16:40 UTC - in response to Message 24616.  
Last modified: 24 Aug 2006, 9:18:06 UTC

That's well within the tollerances of this new system.
That variation was seen over at Raplh. the the code for working it out has changed I wouldn't expect the variation to have altered that much.
All you credits seem to be on average higher.


Well I don´t think ~55% is within the tollerance.

As for getting higher credit it must be the optimized clients pushing the

average up.

Anders n

Edit Spelling



Well that's was up to the implementers to decide, not me :-) and we'll have to live with it.
Yep, the optimised clients push the average up, though the linux users with srock bring it down. (though I think overall it'll certainly be up)

The idea they are hoping is you'll also get a -45% granted credit. Though they've never shown the sort of distribution and error from mean results.

Mind they've only just implemented it using Rosetta with a rolling avererage as the scoring methed as opposed to using it from Ralph and a fixed average as the scoring method.
Team mauisun.org
ID: 24623 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile adrianxw
Avatar

Send message
Joined: 18 Sep 05
Posts: 653
Credit: 11,663,494
RAC: 723
Message 24625 - Posted: 24 Aug 2006, 9:25:31 UTC
Last modified: 24 Aug 2006, 9:28:46 UTC

If what you are saying is that the credit system you implemented with NO WARNING, still needs more testing....

The boards here have been discussing the new credit system for months now, and it had been stated repeatedly that the new credit system would not be introduced until CASP finished. CASP finished. I believe ample "warning" was given.

As for testing, a testbed can isolate obvious errors, but unless you run every wu on the testbed before running it on a production service, you will never be able to categorically say it works 100%. Therefore, the project has created a place where you can point out dubious looking results, and have them checked to make sure there isn't a lurking problem. Testing every wu would be futile.

I see this as positive, rather then negative, and to be applauded.

To get back to the topic in hand, I found this one somewhat suprising. It is one of mine crunched with standard clients and cores, as all of mine are. It was before we were being actually credited with the enhanced score however, so although I was "granted" 204, in fact my credit for that day shows I actually received 76. I don't know if that is significant, but it certainly is a big markup.

*** EDIT ***

It's only been there a couple of minutes and already somebody gives me a "minus". Ho hum.
Wave upon wave of demented avengers march cheerfully out of obscurity into the dream.
ID: 24625 · Rating: 5 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile RedQueen

Send message
Joined: 15 Jun 06
Posts: 9
Credit: 163,610
RAC: 0
Message 24638 - Posted: 24 Aug 2006, 9:43:12 UTC - in response to Message 24625.  
Last modified: 24 Aug 2006, 9:43:29 UTC

https://boinc.bakerlab.org/rosetta/result.php?resultid=33631759
Every other work unit I have done averages about 30 credit on that machine, for the same amount of time
ID: 24638 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Mod.DE
Volunteer moderator

Send message
Joined: 23 Aug 06
Posts: 78
Credit: 0
RAC: 0
Message 24640 - Posted: 24 Aug 2006, 9:48:54 UTC - in response to Message 24638.  

https://boinc.bakerlab.org/rosetta/result.php?resultid=33631759
Every other work unit I have done averages about 30 credit on that machine, for the same amount of time


The ...SRAMAN... WU were already identified as not in line and corrected. That was before the new credit system went live. But thanks for reportint it anyway. But in future please only WU granted credit on August, 23rd or later.
I am a forum moderator! Am I?
ID: 24640 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
J D K
Avatar

Send message
Joined: 23 Sep 05
Posts: 168
Credit: 101,266
RAC: 0
Message 24774 - Posted: 24 Aug 2006, 22:10:46 UTC
Last modified: 24 Aug 2006, 22:13:12 UTC

Interesting

Result ID Work unit ID Sent Time reported or deadline explain Server state
explain Outcome explain Client state explain CPU time (sec) claimed credit granted credit

34204684 29727119 24 Aug 2006 8:58:52 UTC 24 Aug 2006 19:02:22 UTC Over Success Done 20,660.52 53.13 68.35
34163312 29687812 24 Aug 2006 2:41:40 UTC 24 Aug 2006 13:16:09 UTC Over Success Done 20,244.34 53.33 74.88
34111383 28656790 23 Aug 2006 18:38:58 UTC 24 Aug 2006 7:07:53 UTC Over Success Done 21,405.50 56.39 132.10
34090014 29620260 23 Aug 2006 15:29:11 UTC 24 Aug 2006 1:08:47 UTC Over Success Done 20,311.30 53.51 61.03

BOINC Wiki

ID: 24774 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
R.L. Casey

Send message
Joined: 7 Jun 06
Posts: 91
Credit: 2,728,885
RAC: 0
Message 24807 - Posted: 25 Aug 2006, 3:49:56 UTC - in response to Message 24775.  
Last modified: 25 Aug 2006, 3:59:13 UTC


(from 'J D K'):
Interesting

Result ID Work unit ID Sent Time reported or deadline explain Server state
explain Outcome explain Client state explain CPU time (sec) claimed credit granted credit

34204684 29727119 24 Aug 2006 8:58:52 UTC 24 Aug 2006 19:02:22 UTC Over Success Done 20,660.52 53.13 68.35
34163312 29687812 24 Aug 2006 2:41:40 UTC 24 Aug 2006 13:16:09 UTC Over Success Done 20,244.34 53.33 74.88
34111383 28656790 23 Aug 2006 18:38:58 UTC 24 Aug 2006 7:07:53 UTC Over Success Done 21,405.50 56.39 132.10
34090014 29620260 23 Aug 2006 15:29:11 UTC 24 Aug 2006 1:08:47 UTC Over Success Done 20,311.30 53.51 61.03
-----------------

(from 'ted3'):
The new system is Lotto. My last 2 results:
CPU-time: 10,061.41 GC: 68.19
CPU-time: 9,951.78 GC: 105.76

So much more points for less work? ...
-----------------


Both 'J D K' and 'ted3' seem to have encountered the same situation, and two factors seem to be unique to these results compared to their other recent results.

1. Most or all of the results shown in comaprison to the apparent "outlying" point values are for CASP Refinement targets for protein "t370", while the two higher values are for protein "t362".
2. In addition--and perhaps more importantly--each of the "t362" WUs were originally sent out on Aug. 16 at around 16:00UTC and 17:06UTC. The first computer getting each of those "t362" targets did not reply, so when the seven-day time deadline was reached on Aug. 23, the targets were sent out to 'J D K' and 'ted3' at that time. As a result, these WUs generated before the credit system switchover was processed after the switchover.

Since other recent results for these two crunchers seem to correlate very well, it seems to me that a simpler explanation for the apparent anomaly in Granted Credit might be either (1) the different protein target (even though the number of residues is about the same), or (2) some glitch in point counting in the new point system when processing a WU that was generated before the new system was in place. In the latter case, the effect would be only temporary.

Perhaps a Project representative will be able to comment.

Serene crunching!

P.S. Thanks much to 'J D K' and 'ted3' for having their computers visible!
ID: 24807 · Rating: 1 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Buffalo Bill
Avatar

Send message
Joined: 25 Mar 06
Posts: 71
Credit: 1,630,458
RAC: 0
Message 24810 - Posted: 25 Aug 2006, 4:16:44 UTC
Last modified: 25 Aug 2006, 4:17:58 UTC

I also got a high credit for a single T362 WU. Most of my credits are higher now as I always used the standard client but this one is quite a bit higher.

https://boinc.bakerlab.org/rosetta/result.php?resultid=34125114

Bill
ID: 24810 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
R.L. Casey

Send message
Joined: 7 Jun 06
Posts: 91
Credit: 2,728,885
RAC: 0
Message 24812 - Posted: 25 Aug 2006, 4:44:00 UTC - in response to Message 24810.  

I also got a high credit for a single T362 WU. Most of my credits are higher now as I always used the standard client but this one is quite a bit higher.

https://boinc.bakerlab.org/rosetta/result.php?resultid=34125114

Bill

Thanks for the info! The WU, #28669731 was also a re-send originally sent out on Aug. 16. Nice to have some more confirmation.
ID: 24812 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Mod.DE
Volunteer moderator

Send message
Joined: 23 Aug 06
Posts: 78
Credit: 0
RAC: 0
Message 24830 - Posted: 25 Aug 2006, 7:24:29 UTC

I moved several posts from the discussion thread over here, since they are dealing with specific WU and may help the project staff in finding bugs.

Yes the average for t362 is much higher than for other WU. This can happen due to initial overclaimed credit. The question is, why does it not correct soon? It still seems that new results of t362 get significantly more credit than other WU. In theory the ratio should get down quite quickly to the average of all other WU.
I am a forum moderator! Am I?
ID: 24830 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
FluffyChicken
Avatar

Send message
Joined: 1 Nov 05
Posts: 1260
Credit: 369,635
RAC: 0
Message 24855 - Posted: 25 Aug 2006, 11:01:15 UTC - in response to Message 24830.  

I moved several posts from the discussion thread over here, since they are dealing with specific WU and may help the project staff in finding bugs.

Yes the average for t362 is much higher than for other WU. This can happen due to initial overclaimed credit. The question is, why does it not correct soon? It still seems that new results of t362 get significantly more credit than other WU. In theory the ratio should get down quite quickly to the average of all other WU.



And that's when cherry picking starts, people dump the 370 and try until they get 362's

Hopefully this is just a blip of the transition over to the newer system. Given used two different system the use Ralph for the credit value (older ones) & the use Rosetta for the credit value (newer ones)

Team mauisun.org
ID: 24855 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
[DPC]Division_Brabant~OldButNotSoWise
Avatar

Send message
Joined: 23 Jan 06
Posts: 42
Credit: 371,797
RAC: 0
Message 24896 - Posted: 25 Aug 2006, 20:22:04 UTC - in response to Message 24855.  
Last modified: 25 Aug 2006, 20:22:43 UTC


And that's when cherry picking starts, people dump the 370 and try until they get 362's

Hopefully this is just a blip of the transition over to the newer system. Given used two different system the use Ralph for the credit value (older ones) & the use Rosetta for the credit value (newer ones)

Spoken about a unfair creditsystem LOL.

I'll put half of my CPU power on D2OL now, the other stay's for another week on rosetta.
If nothing changed with this strange creditsystem, then I'll thank you all for the fun I had in rosetta before this. :)

(no hard feelings)
ID: 24896 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
BennyRop

Send message
Joined: 17 Dec 05
Posts: 555
Credit: 140,800
RAC: 0
Message 24919 - Posted: 26 Aug 2006, 1:50:57 UTC

[DPC]Division_Brabant~OldButNotSoWise: graph the credit/time for each of WUs on a machine (as I've started to do here.) and see if you don't have a fairly smooth graph with just a few WUs that pop up that should be reported here.

For unusual credit reports:


https://boinc.bakerlab.org/rosetta/result.php?resultid=33516118
FRA_t367_CASPR_hom001_6_t367_4_1wolA_IGNORE_THE_REST_1504_1076_84_1
1.5x score.

https://boinc.bakerlab.org/rosetta/result.php?resultid=33628586
t000__BOINC_SRAMAN_periLABRELAX_SAVE_ALL_OUT__1178_6433_0
2.5x score.

https://boinc.bakerlab.org/rosetta/result.php?resultid=33945090
FRA_t322_CASPR_hom001_6_t322_3_1u1zA_IGNORE_THE_REST_202_1079_87_1
1.5x score.

https://boinc.bakerlab.org/rosetta/result.php?resultid=34123801
FRA_t362_CASPR_hom001_6_t362_6_1s5uA_IGNORE_THE_REST_54_1080_96_1
1.5x score.

https://boinc.bakerlab.org/rosetta/result.php?resultid=33458306
FRA_t367_CASPR_hom001_6_t367_4_1o3uA_IGNORE_THE_REST_396_1076_73_1
1.5x score.

https://boinc.bakerlab.org/rosetta/result.php?resultid=33384283
FRA_t367_CASPR_hom001_6_t367_4_1o3uA_IGNORE_THE_REST_1227_1076_48_1
3x score.

https://boinc.bakerlab.org/rosetta/result.php?resultid=33628390
t000__BOINC_SRAMAN_periLABRELAX_SAVE_ALL_OUT__1178_6239_0
3x score.

https://boinc.bakerlab.org/rosetta/result.php?resultid=33233227
t314__CASP7_FOLLOWUP_ABRELAX_SAVE_ALL_OUT_BARCODE_perfectss__1066_252546_0
2/3rds score.

ID: 24919 · Rating: 1 · rate: Rate + / Rate - Report as offensive    Reply Quote
Ananas

Send message
Joined: 1 Jan 06
Posts: 232
Credit: 752,471
RAC: 0
Message 25147 - Posted: 27 Aug 2006, 15:43:55 UTC

@Benny : I think they want to see reports about unusual granted credits, which would point out flaws in the new credits system
ID: 25147 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
1 · 2 · Next

Message boards : Number crunching : Report work units with unusual amount of credits granted



©2024 University of Washington
https://www.bakerlab.org