New Crediting system: questions

Message boards : Number crunching : New Crediting system: questions

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 · Next

AuthorMessage
Ethan
Volunteer moderator

Send message
Joined: 22 Aug 05
Posts: 286
Credit: 9,304,700
RAC: 0
Message 23977 - Posted: 21 Aug 2006, 0:16:42 UTC - in response to Message 23976.  

I know the results for any decoy will be an average of the decoys for that entire workunit type. But this is exactly what bothers me. Although the results will even out across the workunit they won't even out on the same machine. Some people will generate less decoys than others for the same workunit in the same amount of time possibly based only on their random number. I have seen this on my own machine after doing the same workunit downloaded multiple times. Users who have the bad luck of getting these long decoys will think it is unfair that someone with the same machine gets more credit for the same workunit for the same amount of time. And if I downloaded a whole 6 workunits at once and they end up being the same workunit and it grants a bit less credit than other workunits I would be forced to either be happy with getting less credit for all of them or to dump all of them. These are the two main issues I want resolved and I don't see that happening with the current system.


Thanks for your input, this is the type of feedback the folks programming the new system appreciate getting. I'm sure they'll do their best to address it as it continues to be developed.
ID: 23977 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
John McLeod VII
Avatar

Send message
Joined: 17 Sep 05
Posts: 108
Credit: 195,137
RAC: 0
Message 23979 - Posted: 21 Aug 2006, 0:30:31 UTC - in response to Message 23976.  

It's known that a given simulation within a WU will vary. However, this variability will be both within a single WU and all the others of the same type. By definition, the credits will be based on the average of these results. Since it's not possible to know in advance how long each decoy will take, it should average out.

The other side of it is one WU type may grant slightly more or less credit than others. They're trying to keep that to a minimum, but as they've posted, they have tools to find people who take advantage of the system with this method.

Once the average credit/decoy is calculated, it's entirely dependant on how fast a given machine is. If Intel cpus can do more science in an hour due to features of its hardware, it will end up getting more credits (and vice-versa).

I know the results for any decoy will be an average of the decoys for that entire workunit type. But this is exactly what bothers me. Although the results will even out across the workunit they won't even out on the same machine. Some people will generate less decoys than others for the same workunit in the same amount of time possibly based only on their random number. I have seen this on my own machine after doing the same workunit downloaded multiple times. Users who have the bad luck of getting these long decoys will think it is unfair that someone with the same machine gets more credit for the same workunit for the same amount of time. And if I downloaded a whole 6 workunits at once and they end up being the same workunit and it grants a bit less credit than other workunits I would be forced to either be happy with getting less credit for all of them or to dump all of them. These are the two main issues I want resolved and I don't see that happening with the current system.

This I did not know.

Is there any interest in FLOPs counting?


BOINC WIKI
ID: 23979 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Hoelder1in
Avatar

Send message
Joined: 30 Sep 05
Posts: 169
Credit: 3,915,947
RAC: 0
Message 23981 - Posted: 21 Aug 2006, 1:03:40 UTC - in response to Message 23976.  
Last modified: 21 Aug 2006, 1:53:15 UTC

These are the two main issues I want resolved ...
I just wanted to point out that on your machine the averaging out of the granted work credit seems to have worked quit well. So far you did 19 WUs for which work credit was granted. For the first 9 of these the sum of the granted work credit is 261.61, for the second set of 9 WUs it is 248.07 - with a difference of just 5%.

Something else to note: It would require a quite detailed and lengthy analysis to establish that one type of WU grants more work credit than another. You would need at least something like 10 WUs of the same type to average out the algorithm-based randomness on the model level. These WUs should perferrable be from the same host or else you would also have to account for the different CPU speeds. Since the WU names don't get exported by XML this analysis would have to be done by extracting the information from the Web pages. I suggest that anyone capable of doing this anaysis and writing a suitable script should go to the stockmarket and try to identify stocks that are doing better than others. ;-)

There is one other complications: If the following statement by David Kim (from a few days ago in this thread) is still valid
We may actually have the credit/model values adjust for all work units as results come in and use ralph tests to serve as a starting point.
than once you did your statistical analysis, the credit/model for a specific WU may likely have been adjusted by new data coming in while you did the analysis - once you have enough data to establish that one WU grants more or less credit than another, than so does the script that calculates the credit/model on the server and the inequality will be corrected.

So bottom line, I very much doubt that we will ever see anyone trying, let alone succeeding, with this trick - in which case there would still be the counter measures that David K mentioned...
Team betterhumans.com - discuss and celebrate the future - hoelder1in.org
ID: 23981 · Rating: 1 · rate: Rate + / Rate - Report as offensive    Reply Quote
BennyRop

Send message
Joined: 17 Dec 05
Posts: 555
Credit: 140,800
RAC: 0
Message 24008 - Posted: 21 Aug 2006, 5:10:27 UTC

date seconds old score new system
Aug 15th: 85,637.94 267.35 267.35
Aug 16th: 84,394.61 263.47 403.61
Aug 17th: 86,390.48 268.61 260.29
Aug 18th: 85,987.86 267.36 276.98
Aug 19th: 85,638.19 266.27 262.05

The 15th is the old system. Starting on the 16th, my first new system report started at 323 points.. and has been adjusted up to 403 now.
The next 3 seem in line.

Given that there were problems with the script used that first day, the results of the 17th, the 18th, and 19th seem to show that the system can average out over time.
(This is on an Athlon 64 3000+ cpu at 2Ghz, 1Gig ram, 24/7 machine that is is only taxed by doing web surfing.)
ID: 24008 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
tralala

Send message
Joined: 8 Apr 06
Posts: 376
Credit: 581,806
RAC: 0
Message 24029 - Posted: 21 Aug 2006, 7:49:36 UTC

There are two different issues:

a) the "natural" variation of decoys in the same WU type due to the randomness of the algorithm
b) different credits/hour-ratios among different WU

With a) we have to live with. However this really should average out over time and if you had just bad luck with one WU which gave you a bad decoys/wu ration you will be lucky with the next one.

b) is a problem with which the project has to deal with and which should be minimized. If not resolved b) would result in cherry picking no matter how difficult it might sound at first glance (human mind is very clever). David Kim mentioned using the RALPH values as a starting point and applying constant correction while the result on Rosetta come in. If this is practical it would counter any attempt in cherry picking and bring all different types of WU to the same value over time. There are probably many other solutions to that but it will require some finetuning. In the end I hope the variation between different WU will be within 10-20% which I think is acceptable.
ID: 24029 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Dave Wilson

Send message
Joined: 8 Jan 06
Posts: 35
Credit: 379,049
RAC: 0
Message 24031 - Posted: 21 Aug 2006, 8:01:48 UTC

Wow, my Power Mac duel 2.5 G5 was getting 1,100 credits per 24 hours now with the new credit system it is getting 220. That is about an 80% reduction in credits. If this keeps up I will make it 100%.

ID: 24031 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile dcdc

Send message
Joined: 3 Nov 05
Posts: 1830
Credit: 119,208,549
RAC: 2,517
Message 24032 - Posted: 21 Aug 2006, 8:14:05 UTC - in response to Message 24031.  

Wow, my Power Mac duel 2.5 G5 was getting 1,100 credits per 24 hours now with the new credit system it is getting 220. That is about an 80% reduction in credits. If this keeps up I will make it 100%.

1,100 credits per day seems very high, even for a 2CPU machine like that. I think an 80% reduction sounds very high too though. There's quite a large variation in the credit awarded under the new system - your previous credits are more consistent. Hopefully that'll improve with testing.
ID: 24032 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Dave Wilson

Send message
Joined: 8 Jan 06
Posts: 35
Credit: 379,049
RAC: 0
Message 24035 - Posted: 21 Aug 2006, 8:45:43 UTC - in response to Message 24032.  

Wow, my Power Mac duel 2.5 G5 was getting 1,100 credits per 24 hours now with the new credit system it is getting 220. That is about an 80% reduction in credits. If this keeps up I will make it 100%.

1,100 credits per day seems very high, even for a 2CPU machine like that. I think an 80% reduction sounds very high too though. There's quite a large variation in the credit awarded under the new system - your previous credits are more consistent. Hopefully that'll improve with testing.


Actually that is about a 16 hour turnaround. This machine has 5.5 gig ram and is almost dedicated full time. Since it has been getting this kind of credits from the beginning I don't think that it is at all out of line. The 80% is of course still accurate.

ID: 24035 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Ananas

Send message
Joined: 1 Jan 06
Posts: 232
Credit: 752,471
RAC: 0
Message 24037 - Posted: 21 Aug 2006, 9:01:20 UTC
Last modified: 21 Aug 2006, 9:10:39 UTC

If your 4.44 is an optimized Client, that would explain the reduction. There has been a 4.43 and a 4.70 but no official 4.44 I think.

With a stock 5.2.13, one PowerMac7,3 Dual I have seen had 1105.87 million intops/sec and 3573.43 million flops/sec
ID: 24037 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
tralala

Send message
Joined: 8 Apr 06
Posts: 376
Credit: 581,806
RAC: 0
Message 24042 - Posted: 21 Aug 2006, 9:38:52 UTC - in response to Message 24031.  

Wow, my Power Mac duel 2.5 G5 was getting 1,100 credits per 24 hours now with the new credit system it is getting 220. That is about an 80% reduction in credits. If this keeps up I will make it 100%.


Dave I understand your frustration however the new credit system awards fair credits based on actual work done (besides some fluctuation in the results which should be reduced). If it happens to be 220 per day that is the actual amount of computing it does (I have and AMD Athlon 64 @2.4 GHZ which gets around 300 per day if fulltime switched on). The rosetta app does not compile as fast for macs as for windows machine I think.
ID: 24042 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jose

Send message
Joined: 28 Mar 06
Posts: 820
Credit: 48,297
RAC: 0
Message 24049 - Posted: 21 Aug 2006, 10:03:50 UTC - in response to Message 24042.  

Wow, my Power Mac duel 2.5 G5 was getting 1,100 credits per 24 hours now with the new credit system it is getting 220. That is about an 80% reduction in credits. If this keeps up I will make it 100%.


Dave I understand your frustration however the new credit system awards fair credits based on actual work done (besides some fluctuation in the results which should be reduced). If it happens to be 220 per day that is the actual amount of computing it does (I have and AMD Athlon 64 @2.4 GHZ which gets around 300 per day if fulltime switched on). The rosetta app does not compile as fast for macs as for windows machine I think.


THat is why there is a gallery of Mac optimizers available and used.
ID: 24049 · Rating: -2 · rate: Rate + / Rate - Report as offensive    Reply Quote
Ethan
Volunteer moderator

Send message
Joined: 22 Aug 05
Posts: 286
Credit: 9,304,700
RAC: 0
Message 24053 - Posted: 21 Aug 2006, 10:32:08 UTC - in response to Message 24049.  

I believe Apple has moved to an Intel platform and claimed a big jump in performance. Unfortunately not all hardware is equal, there will be differences based on how fast your processor is and its architecture.

ID: 24053 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Ananas

Send message
Joined: 1 Jan 06
Posts: 232
Credit: 752,471
RAC: 0
Message 24056 - Posted: 21 Aug 2006, 11:10:36 UTC

The PowerPC does not target the same group of users as the Intel architecture. I doubt that we could satisfy 200+ users on our AIX systems with three CPUs on x86 base. Those IBM machines are not very strong when it comes to float (when we need that for batch runs, we add 3 more CPUs on the fly), but their data throughput is immense.
ID: 24056 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Whl.

Send message
Joined: 29 Dec 05
Posts: 203
Credit: 275,802
RAC: 0
Message 24073 - Posted: 21 Aug 2006, 12:45:49 UTC - in response to Message 23979.  


This I did not know.

Is there any interest in FLOPs counting?

You would have known if you had read more John. ;-)

Other people have made the same good points, but it was all lost in the fuss that broke out, deleted threads, or the threads falling right out of view as nobody is allowed to post in them anymore.

ID: 24073 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Dave Wilson

Send message
Joined: 8 Jan 06
Posts: 35
Credit: 379,049
RAC: 0
Message 24127 - Posted: 21 Aug 2006, 17:29:53 UTC - in response to Message 24042.  

Wow, my Power Mac duel 2.5 G5 was getting 1,100 credits per 24 hours now with the new credit system it is getting 220. That is about an 80% reduction in credits. If this keeps up I will make it 100%.


Dave I understand your frustration however the new credit system awards fair credits based on actual work done (besides some fluctuation in the results which should be reduced). If it happens to be 220 per day that is the actual amount of computing it does (I have and AMD Athlon 64 @2.4 GHZ which gets around 300 per day if fulltime switched on). The rosetta app does not compile as fast for macs as for windows machine I think.

Without getting into a Mac/PC mud slinging war the comparison you mention only proves my point, a single Athlon @ 2.4 GHZ gets 300 while two 64bit PowerPC@2.5 GHZ chips with Altivec enabled clients only gets 220 combined, is more than twice reduced. The Rosetta app has always run fine before but the method for calculating credits is very flawed and I will not believe that these processors are in anyway one third the speed as the athlon.
ID: 24127 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Avi

Send message
Joined: 2 Aug 06
Posts: 58
Credit: 95,619
RAC: 0
Message 24129 - Posted: 21 Aug 2006, 17:41:25 UTC

Its mentioned elsewhere about the optimization (not the boinc client, the core rosetta client) for the architecture its running on. The post said that there is still some "legacy" code, which I would suppose runs slowly. Perhaps with the new credit system, the software developers can see which machines are STILL getting low credit and see if some of the code needs to be updated?
ID: 24129 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Ananas

Send message
Joined: 1 Jan 06
Posts: 232
Credit: 752,471
RAC: 0
Message 24138 - Posted: 21 Aug 2006, 18:15:27 UTC
Last modified: 21 Aug 2006, 18:17:29 UTC

The new system is machine independant, it counts the output, not the claimed power of a machine.

The PowerPC is very weak in floatmath, if the program uses no specific optimisations (vectorizing).

As Rosetta does not have a specific version for the different Mac versions, those big machines are as much screwed as the latest AMD and Intel machines. They all offer more power but the project clients are unable to use it.

A dual Athlon MP 2600+ has beaten a 600k$ AIX machine in SETI Classic by 20% on the same workunits - I could directly compare them as one of those is my machine (no need to tell that it's the smaller of those two) and the other one I could use for a few days for load and performance tests :-)

Everything but basic floatmath is faster on that AIX box than on the PC, but that is what most DC programs need.


A very early demand for BOINC has been to make it report all machine capabilities to the BOINC server, which would allow to return the best client that is available for that system. They are now implementing it I think.

When that has been done, it will pay off for long running projects like Rosetta to compile more than 3 or 4 different basic clients for the basic host infrastructures - and it will pay off to crunch with a cutting edge machine.
ID: 24138 · Rating: 1 · rate: Rate + / Rate - Report as offensive    Reply Quote
Dave Wilson

Send message
Joined: 8 Jan 06
Posts: 35
Credit: 379,049
RAC: 0
Message 24141 - Posted: 21 Aug 2006, 18:25:49 UTC - in response to Message 24138.  



As Rosetta does not have a specific version for the different Mac versions, those big machines are as much screwed as the latest AMD and Intel machines. They all offer more power but the project clients are unable to use it.

It is not the Mac that will be "Screwed" by this I am now switching all my computers to get no new work.
When they are dry, It's been fun till you "Screwed me"

Guessing my final Rosetta contribution at around 380,000.
You don't need or want me anymore.

Dave
ID: 24141 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Ethan
Volunteer moderator

Send message
Joined: 22 Aug 05
Posts: 286
Credit: 9,304,700
RAC: 0
Message 24144 - Posted: 21 Aug 2006, 18:34:20 UTC

The new system is Computer and OS blind. It doesn't check what type of Proc or OS you're running, it just downloads an executable and processes it. If you computer can process it faster than average, your work credits will be higher than average.
ID: 24144 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Ananas

Send message
Joined: 1 Jan 06
Posts: 232
Credit: 752,471
RAC: 0
Message 24152 - Posted: 21 Aug 2006, 18:43:19 UTC

@Dave : I did not want to offend you, I just tried to explain what happened and why. Maybe my choice of words has not been so very good but English isn't my first language :-/
ID: 24152 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 · Next

Message boards : Number crunching : New Crediting system: questions



©2024 University of Washington
https://www.bakerlab.org