New Crediting system: questions

Message boards : Number crunching : New Crediting system: questions

To post messages, you must log in.

1 · 2 · 3 · 4 . . . 7 · Next

AuthorMessage
Profile Cureseekers~Kristof

Send message
Joined: 5 Nov 05
Posts: 80
Credit: 689,603
RAC: 0
Message 22511 - Posted: 16 Aug 2006, 5:59:57 UTC
Last modified: 16 Aug 2006, 6:08:33 UTC

I want to start this thread because I (and I guess there are more people with me) have some questions.
Lets use this thread for questions and not for discussion of pros and contras of the crediting system.

On Ralph, one of the project people said:
The new crediting system is pretty simple.

First, we determine how much credit to grant per model for each work unit by running tests on Ralph. We will use the average credit per model from the Ralph tests for production runs on Rosetta@home.
These values will be work unit specific so work units that take longer to generate structures will have higher values.

I have explained this below in a previous post. The only difference is that I decided to keep the old crediting system so users can compare their work with either system. If you take a look at the top participants page you will see two new columns, "Recent average work credit" and "Total work credit". These are the new credit per model based values. I've been using 2 credits per model on Ralph JUST FOR TESTING. Rosetta@home will use work unit specific values determined from the Ralph test runs.
and
Everyone keep in mind that the current standard boinc crediting system will still be used.

Also, minor modifications to the credit/model values will not make that much of a difference in the long run. The important thing to know is that given any credit/model value, users will be on a level playing field. I think we can all agree that this is the major drive/motivation for coming up with a new method. Making sure it closely matches the BOIINC credit values is not as important since we will still use the old system along with the new.

So if I understand well: The 2 crediting systems will be used both together? So you can choose to count on the old or the new system?
Member of Dutch Power Cows
ID: 22511 · Rating: 3 · rate: Rate + / Rate - Report as offensive    Reply Quote
XS_Vietnam_Soldiers

Send message
Joined: 11 Jan 06
Posts: 240
Credit: 2,880,653
RAC: 0
Message 22523 - Posted: 16 Aug 2006, 8:55:07 UTC - in response to Message 22511.  

I want to start this thread because I (and I guess there are more people with me) have some questions.
Lets use this thread for questions and not for discussion of pros and contras of the crediting system.

On Ralph, one of the project people said:
The new crediting system is pretty simple.

First, we determine how much credit to grant per model for each work unit by running tests on Ralph. We will use the average credit per model from the Ralph tests for production runs on Rosetta@home.
These values will be work unit specific so work units that take longer to generate structures will have higher values.

I have explained this below in a previous post. The only difference is that I decided to keep the old crediting system so users can compare their work with either system. If you take a look at the top participants page you will see two new columns, "Recent average work credit" and "Total work credit". These are the new credit per model based values. I've been using 2 credits per model on Ralph JUST FOR TESTING. Rosetta@home will use work unit specific values determined from the Ralph test runs.
and
Everyone keep in mind that the current standard boinc crediting system will still be used.

Also, minor modifications to the credit/model values will not make that much of a difference in the long run. The important thing to know is that given any credit/model value, users will be on a level playing field. I think we can all agree that this is the major drive/motivation for coming up with a new method. Making sure it closely matches the BOIINC credit values is not as important since we will still use the old system along with the new.

So if I understand well: The 2 crediting systems will be used both together? So you can choose to count on the old or the new system?

Hello Kristof!
Yes, my understanding is that there will at least for a time be the 2 systems shown. Smart move, allows for adjustments and to keep a happy family.
Has Bubbles posted any more pics of his shoes? <BG>
I ought to post that story here, these guys might get a kick out of it.
Thanks for your time,
Movieman
ID: 22523 · Rating: -1 · rate: Rate + / Rate - Report as offensive    Reply Quote
BennyRop

Send message
Joined: 17 Dec 05
Posts: 555
Credit: 140,800
RAC: 0
Message 22524 - Posted: 16 Aug 2006, 9:09:00 UTC

since we will still use the old system along with the new.


There were two things that David Kim was talking about that could be described by that quote.

1. They use lab machines (something trusted) on Ralph running the standard boinc client to generate a credit/model for each major WU. The credit/model is thus based on the old credit system. If our Rosetta machines produce models faster than the ones on Ralph, we'll end up with more credits/hour than the machines in Ralph. If our Rosetta machines produce models slower than the machines on Ralph, we'll end up with fewer credits/hour than the machines in Ralph. The new system would have the credit/model based on the old credit system; so both systems would be running at the same time.

2. He mentioned keeping track of the new credit system, plus the claimed score from the old credit system for each of our computers. It should make troubleshooting easier, and comparisons possible.

Mention which of the two cases you're more interested in, and hopefully we can get David (either/or/both) to comment on it..


ID: 22524 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
XS_Vietnam_Soldiers

Send message
Joined: 11 Jan 06
Posts: 240
Credit: 2,880,653
RAC: 0
Message 22526 - Posted: 16 Aug 2006, 9:36:29 UTC - in response to Message 22524.  

since we will still use the old system along with the new.


There were two things that David Kim was talking about that could be described by that quote.

1. They use lab machines (something trusted) on Ralph running the standard boinc client to generate a credit/model for each major WU. The credit/model is thus based on the old credit system. If our Rosetta machines produce models faster than the ones on Ralph, we'll end up with more credits/hour than the machines in Ralph. If our Rosetta machines produce models slower than the machines on Ralph, we'll end up with fewer credits/hour than the machines in Ralph. The new system would have the credit/model based on the old credit system; so both systems would be running at the same time.

2. He mentioned keeping track of the new credit system, plus the claimed score from the old credit system for each of our computers. It should make troubleshooting easier, and comparisons possible.

Mention which of the two cases you're more interested in, and hopefully we can get David (either/or/both) to comment on it..


I have to toss this in just for a little levity:
We don't run machines as slow as what they use in the labs!<BG>
Ok, I would guess that they have some decent equipment and I think I saw that they have a 500 node Linux server? Correct me on that if I'm wrong.
On a machine by machine basis, I know that XS,FreeDC and the Dutch Power Cows have some pertty decent crunchers. I've got high end dual xeons and now a dual Opty to play with but some of these guys make me look like a kid with an abacus.
There are individuals with way beyond $100,000.00 in equipment in their homes.
One of our newer members has 37 PC's running in his house and he isn't the most "Xtreme" of the group. Another runs way over 100 machines for Rosetta, all his own, all from his pocket.
Thanks for your time,
Movieman
ID: 22526 · Rating: -1 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile dcdc

Send message
Joined: 3 Nov 05
Posts: 1829
Credit: 115,712,692
RAC: 55,885
Message 22527 - Posted: 16 Aug 2006, 9:54:46 UTC - in response to Message 22526.  
Last modified: 16 Aug 2006, 9:57:33 UTC


I have to toss this in just for a little levity:
We don't run machines as slow as what they use in the labs!<BG>
Ok, I would guess that they have some decent equipment and I think I saw that they have a 500 node Linux server? Correct me on that if I'm wrong.
On a machine by machine basis, I know that XS,FreeDC and the Dutch Power Cows have some pertty decent crunchers. I've got high end dual xeons and now a dual Opty to play with but some of these guys make me look like a kid with an abacus.
There are individuals with way beyond $100,000.00 in equipment in their homes.
One of our newer members has 37 PC's running in his house and he isn't the most "Xtreme" of the group. Another runs way over 100 machines for Rosetta, all his own, all from his pocket.
Thanks for your time,
Movieman


The new credit system will credit you for the (massive amount of) work you do/have done. The more work the computer does (i.e. the faster the computer) the more credits you get. If you have more CPUs, you get more credits. If you overclock, you get more credits. If you raise the FSB and reduce a memory bottleneck, you'll get more credits.

Looks like a good, fair system.
ID: 22527 · Rating: 1 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Gerry Rough
Avatar

Send message
Joined: 2 Jan 06
Posts: 111
Credit: 1,389,340
RAC: 0
Message 22534 - Posted: 16 Aug 2006, 12:50:16 UTC - in response to Message 22527.  

The new credit system will credit you for the (massive amount of) work you do/have done. The more work the computer does (i.e. the faster the computer) the more credits you get. If you have more CPUs, you get more credits. If you overclock, you get more credits. If you raise the FSB and reduce a memory bottleneck, you'll get more credits.

Looks like a good, fair system.


While I like the new crediting system, what about guys like me who still do some Rosetta on paper and snail mail the results? :0


(Click for detailed stats)
ID: 22534 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Avi

Send message
Joined: 2 Aug 06
Posts: 58
Credit: 95,619
RAC: 0
Message 22535 - Posted: 16 Aug 2006, 12:50:59 UTC - in response to Message 22527.  

From what I understand, there are currently several differnt work units out. Some are on different length proteins, and some are using different methods. The plan is to test each on RALPH and come to an average credit per decoy, for each work unit type (processing and protein mix). The work units that have larger proteins will get more credit.
The point of this is to avoid relying on the FLOPS that BOINC returns, since they can be tweaked.
ID: 22535 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Trog Dog
Avatar

Send message
Joined: 25 Nov 05
Posts: 129
Credit: 57,345
RAC: 0
Message 22537 - Posted: 16 Aug 2006, 13:01:14 UTC - in response to Message 22527.  



The new credit system will credit you for the (massive amount of) work you do/have done. The more work the computer does (i.e. the faster the computer) the more credits you get. If you have more CPUs, you get more credits. If you overclock, you get more credits. If you raise the FSB and reduce a memory bottleneck, you'll get more credits.

Looks like a good, fair system.


And it won't be disputable - higher credits relative to someone else or some other machine means that the machine with the higher credits HAS done more work not just claimed that it has :)
ID: 22537 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
tralala

Send message
Joined: 8 Apr 06
Posts: 376
Credit: 581,806
RAC: 0
Message 22553 - Posted: 16 Aug 2006, 15:09:31 UTC - in response to Message 22537.  

And it won't be disputable - higher credits relative to someone else or some other machine means that the machine with the higher credits HAS done more work not just claimed that it has :)


That's the goal, however right now they are not quite there. Differen WU will give very differen credits/hour and it is not clear how they gonna fix that.

ID: 22553 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Keith Akins

Send message
Joined: 22 Oct 05
Posts: 176
Credit: 71,779
RAC: 0
Message 22558 - Posted: 16 Aug 2006, 15:42:31 UTC

If it's not in an acceptible state, then why move it from RALPH? Seems to me that it needed at least a couple more test runs on RALPH to equal out credits.
ID: 22558 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Feet1st
Avatar

Send message
Joined: 30 Dec 05
Posts: 1755
Credit: 4,690,520
RAC: 0
Message 22559 - Posted: 16 Aug 2006, 15:44:49 UTC - in response to Message 22553.  

And it won't be disputable - higher credits relative to someone else or some other machine means that the machine with the higher credits HAS done more work not just claimed that it has :)


That's the goal, however right now they are not quite there. Differen WU will give very differen credits/hour and it is not clear how they gonna fix that.

Different WUs will give different credits per MODEL. The credit per HOUR should be very consistent on a given box, regardless of which WU or Rosetta approach is being used to study a protein.
Add this signature to your EMail:
Running Microsoft's "System Idle Process" will never help cure cancer, AIDS nor Alzheimer's. But running Rosetta@home just might!
https://boinc.bakerlab.org/rosetta/
ID: 22559 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jose

Send message
Joined: 28 Mar 06
Posts: 820
Credit: 48,297
RAC: 0
Message 22561 - Posted: 16 Aug 2006, 16:03:59 UTC - in response to Message 22558.  
Last modified: 16 Aug 2006, 16:07:46 UTC

If it's not in an acceptible state, then why move it from RALPH? Seems to me that it needed at least a couple more test runs on RALPH to equal out credits.


IMHO:

There is a reason why taking the "new credit system" from RALPH and play it here for comments and tweaking makes sense: the system needs a higher range of computers/os/configurations to see how it works under pressure. RALPH as good as a system for tests that is is, doesnt have a good representation of all the machines/OS present: with the new machines and cpu's now available and soon available the posibilities for credit levels will be endless . (Alas that exciting hardware is not fully represented in RALPH)

I hope people have noticed that the old system is being kept and that will allow for comparison (And yes Back up ) . I just think what would have happened if a RALPH-only -tested credit system collapses due to the complexity of the issues and unfroseen problems that ussually sprout in a complex project like this.. So I appreciate the fact that the old system is there , while a "real" life testing is done to the new.

That said: The only way this testing is going to work, if everyone comes with it with "clean hands" no predetermined agendas, openess to others people's ideas and the care to use the chance to makes sugestions and make them with civility.


The developers and scientists have demonstrated that they pay attention to our needs and our opinions. They are given us a chance seldom given in a DC project.

Let's not blow it. ( Deep inside I know we will not)
ID: 22561 · Rating: 2 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jose

Send message
Joined: 28 Mar 06
Posts: 820
Credit: 48,297
RAC: 0
Message 22562 - Posted: 16 Aug 2006, 16:05:04 UTC - in response to Message 22559.  

And it won't be disputable - higher credits relative to someone else or some other machine means that the machine with the higher credits HAS done more work not just claimed that it has :)


That's the goal, however right now they are not quite there. Differen WU will give very differen credits/hour and it is not clear how they gonna fix that.

Different WUs will give different credits per MODEL. The credit per HOUR should be very consistent on a given box, regardless of which WU or Rosetta approach is being used to study a protein.



So the difference lies in the machines we bring to the project. That is fair for me.
ID: 22562 · Rating: 1 · rate: Rate + / Rate - Report as offensive    Reply Quote
Ethan
Volunteer moderator

Send message
Joined: 22 Aug 05
Posts: 286
Credit: 9,304,700
RAC: 0
Message 22566 - Posted: 16 Aug 2006, 18:17:54 UTC - in response to Message 22562.  
Last modified: 16 Aug 2006, 18:18:40 UTC

Just a message from DK on what their plans currently are:

http://ralph.bakerlab.org/forum_thread.php?id=233#2121

Today's outage will simply add 2 columns to the results section. Everything will continue to run off the old/current credit system. . there will just be additional fields to look at.
ID: 22566 · Rating: 2 · rate: Rate + / Rate - Report as offensive    Reply Quote
Scribe
Avatar

Send message
Joined: 2 Nov 05
Posts: 284
Credit: 157,359
RAC: 0
Message 22591 - Posted: 17 Aug 2006, 5:27:07 UTC
Last modified: 17 Aug 2006, 5:27:25 UTC

Well there are no extra columns on my results and the granted credit is way down on the claimed one!
ID: 22591 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
eplnificant

Send message
Joined: 9 Jun 06
Posts: 6
Credit: 13,121,045
RAC: 0
Message 22593 - Posted: 17 Aug 2006, 5:34:42 UTC

The granted vs. claimed credits is certainly inconsistent with granted being both higher and lower, even on the same PC. I have one that granted 50% more than claimed on one WU, but less than claimed on the other. I'm not even going to try to figure it out; I'll just keep crunching.
ID: 22593 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Avi

Send message
Joined: 2 Aug 06
Posts: 58
Credit: 95,619
RAC: 0
Message 22595 - Posted: 17 Aug 2006, 5:44:26 UTC - in response to Message 22593.  

The granted vs. claimed credits is certainly inconsistent with granted being both higher and lower, even on the same PC. I have one that granted 50% more than claimed on one WU, but less than claimed on the other. I'm not even going to try to figure it out; I'll just keep crunching.

They could be using different algorithms, and your cpu cache is better for one of them.

BTW, could there be someway to see if a CPU performs "abnormally well" using a certain algorithm and prefer sending some to them? i.e. AMD and intel seem to sometime reverse the whet/dhyr results (one is better at one than the other) etc.

My main comptuer returned 3 results after the upgrade and was given almost double credit. I checked from a top team, they got 1/2 to 1/3 of the credit (for 3+ WU's).. prolly optimized clients?
ID: 22595 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Hoelder1in
Avatar

Send message
Joined: 30 Sep 05
Posts: 169
Credit: 3,915,947
RAC: 0
Message 22598 - Posted: 17 Aug 2006, 6:38:55 UTC - in response to Message 22593.  
Last modified: 17 Aug 2006, 6:48:49 UTC

The granted vs. claimed credits is certainly inconsistent with granted being both higher and lower, even on the same PC. I have one that granted 50% more than claimed on one WU, but less than claimed on the other. I'm not even going to try to figure it out; I'll just keep crunching.
I just did a quick test on the Windows users of my team (all using standard client). There were 9 WUs returned since Rosetta came up again and the average of the granted credit is very, very close to the claimed credit (to within 5 % or so). There is some scatter between the granted and claimed credit values: for seven of the 9 WUs the granted credit is within +/-20% of claimed credit. Observation for those who followed the recent activities on Ralph: I think David Kim adjusted the "credit/model correction factor" some more when moving the new system to Rosetta: while in the recent Ralph tests the granted credit on average was still about 10-15% higher than the claimed credit (for Windows/standard client users), the match now seems almost perfect - well as far as I can tell from the 9 WUs I looked at. If you want my opinion, I think David K did a great job so far. :-)
Team betterhumans.com - discuss and celebrate the future - hoelder1in.org
ID: 22598 · Rating: 1 · rate: Rate + / Rate - Report as offensive    Reply Quote
Ananas

Send message
Joined: 1 Jan 06
Posts: 232
Credit: 752,471
RAC: 0
Message 22600 - Posted: 17 Aug 2006, 7:05:01 UTC

I think it looks good, the Linux machines are not under-credited anymore and the time/credits ratio on the same machine for compareable WUs seems to be quite constant.

It looks as if my 5.2.12tx client did increase the credits on some machines even without having calibration enabled :-/ I don't mind if they get adjusted (i.e. reduced) to the new values if that is possible.
ID: 22600 · Rating: 1 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile David E K
Volunteer moderator
Project administrator
Project developer
Project scientist

Send message
Joined: 1 Jul 05
Posts: 1018
Credit: 4,334,829
RAC: 0
Message 22602 - Posted: 17 Aug 2006, 7:22:52 UTC

I'm using the same correction factor but I calculated the credit/model values from R@h results to get things started since tests to determine the values were not carried out on ralph for already existing work units. If I couldn't determine the value because results were not returned yet, the value will get set and then adjust as results come in. We may actually have the credit/model values adjust for all work units as results come in and use ralph tests to serve as a starting point.

Sorry, I didn't have time to add the columns on the php pages. Ralph was unexpectedly (at least to me) shut down for a move to a new location while I was working on the transition, also, it is taking quite a while to extract and add the total_work_credit values to the database.

Please keep in mind that there will be differences when comparing the claimed and granted credits due to the random nature of the predictions. The values should average out though.
ID: 22602 · Rating: 1 · rate: Rate + / Rate - Report as offensive    Reply Quote
1 · 2 · 3 · 4 . . . 7 · Next

Message boards : Number crunching : New Crediting system: questions



©2024 University of Washington
https://www.bakerlab.org