Granted Credit taking forever....

Message boards : Number crunching : Granted Credit taking forever....

To post messages, you must log in.

Previous · 1 · 2 · 3

AuthorMessage
Mark Brown

Send message
Joined: 8 Aug 09
Posts: 21
Credit: 602,685
RAC: 0
Message 63345 - Posted: 14 Sep 2009, 21:20:07 UTC - in response to Message 63340.  

The science is way beyond me, but I appreciate those involved that do.

I have been adding computers slowly, 10 so far, in an effort to help with a cause that seems quite interesting and promising.

Although I'm not in it for the points, I enjoy seeing the growth.

I'm going to keep running R@H and have suggested it to several other people. I'm sure the pending jobs wll eventually be processed. I hope my results are helpful.

Sincerely

ID: 63345 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile David E K
Volunteer moderator
Project administrator
Project developer
Project scientist

Send message
Joined: 1 Jul 05
Posts: 1018
Credit: 4,334,829
RAC: 0
Message 63346 - Posted: 14 Sep 2009, 22:02:57 UTC - in response to Message 63343.  

you can ignore the server status page for now. I stopped the non-minirosetta daemons and fired up more assimilators and validators for the minirosetta jobs. 8 assimilators and 4 validators are running on bk1 and bk2. The load on these servers is very high and we're doing what we can with what we have.

The only issue is pending credits. Users will just have to wait a bit longer for their credits to be awarded as our system catches up. The more important issue is that our work unit generators continue to make new work and on that front we're doing fine.



Kind of bouncing back and forth with the various servers these days it seems.
Fighting between work generation and then some unchecked code and now the validators. Being that things supposedly happen in 3's (so to speak) the problems should theoretically be over. (knock on wood, fingers crossed and all that)
Hope to see some stability in the project before the year ends....good luck keeping up with it all. you are doing a good job for one or two people.


The project and servers are stable but they're just slow to catch up right now. I guess people are really used to getting credit right away with our project.


ID: 63346 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
KWSN - Roger the Shrubber

Send message
Joined: 16 Sep 07
Posts: 2
Credit: 9,134,942
RAC: 0
Message 63347 - Posted: 14 Sep 2009, 23:24:46 UTC - in response to Message 63346.  



The project and servers are stable but they're just slow to catch up right now. I guess people are really used to getting credit right away with our project.




Right away? I haven't gotten any credit in 3 days!
ID: 63347 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
LizzieBarry

Send message
Joined: 25 Feb 08
Posts: 76
Credit: 201,862
RAC: 0
Message 63348 - Posted: 14 Sep 2009, 23:34:07 UTC - in response to Message 63342.  

Thanks Yifan. Let's hope so. Though I note the bk1 and bk2 servers aren't running right now. Part of the problem or part of the solution?

It's part of the solution. DEK rearranged the validator servers a bit. They are just temporarily not showing properly on the webpage.

Looking at the credits being awarded now I can see it's working.

Looking forward to a couple of days of high credits now.

I guess people are really used to getting credit right away with our project.

True. At one time they'd be available within 30 seconds, while some other projects routinely take days and more. Thanks for the solution.
ID: 63348 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile David E K
Volunteer moderator
Project administrator
Project developer
Project scientist

Send message
Joined: 1 Jul 05
Posts: 1018
Credit: 4,334,829
RAC: 0
Message 63349 - Posted: 14 Sep 2009, 23:36:38 UTC - in response to Message 63347.  



The project and servers are stable but they're just slow to catch up right now. I guess people are really used to getting credit right away with our project.




Right away? I haven't gotten any credit in 3 days!


yes, I think many users expect to get credit right away. i.e. sooner than 3 days or a week which may be what it takes for our servers to catch up.
ID: 63349 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Gen_X_Accord
Avatar

Send message
Joined: 5 Jun 06
Posts: 154
Credit: 279,018
RAC: 0
Message 63350 - Posted: 14 Sep 2009, 23:38:47 UTC

I've had about half of my "pending" credit get it's granted credit awarded to it, so the servers must be catching up. And yes, this project has spoiled us in the past with how fast credit was granted. Some work units take a day or more on WCG to get validated, I've seen some not be validated even after a week or two over there.
ID: 63350 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Greg_BE
Avatar

Send message
Joined: 30 May 06
Posts: 5659
Credit: 5,691,837
RAC: 1,806
Message 63351 - Posted: 14 Sep 2009, 23:41:38 UTC - in response to Message 63350.  

I've had about half of my "pending" credit get it's granted credit awarded to it, so the servers must be catching up. And yes, this project has spoiled us in the past with how fast credit was granted. Some work units take a day or more on WCG to get validated, I've seen some not be validated even after a week or two over there.


try einstein, there the task has to be "validated" by another user before they grant credit. can take a week or more to have that done.
ID: 63351 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile David E K
Volunteer moderator
Project administrator
Project developer
Project scientist

Send message
Joined: 1 Jul 05
Posts: 1018
Credit: 4,334,829
RAC: 0
Message 63352 - Posted: 15 Sep 2009, 0:24:12 UTC

Just to put a number on things and to get an idea of the impressive amount of work that has to be processed for the batch that is causing the slowdown. There were around 250,000 jobs sent out for this batch and using a close approximation of 5MB of result data per job, that adds up to around 1.2 Terabytes of data that has to be processed, and the data is read twice by our servers, once by the validator and once by the assimilator. That's a lot of data for Yifan and I'm sure he really really appreciates it! If you think it's taking a while to get credit for all this data, imagine the time it will take to analyze it!

I just want to mention that there's been a slowdown in the credit granting and there's some catching up to do in that respect but there does not appear to be much of a slowdown in work and actual throughput. So thank you all for your great work!
ID: 63352 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Gen_X_Accord
Avatar

Send message
Joined: 5 Jun 06
Posts: 154
Credit: 279,018
RAC: 0
Message 63355 - Posted: 15 Sep 2009, 2:47:48 UTC

It looks like some of the servers are down now too, like about half of them. UH-OH!
ID: 63355 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile David E K
Volunteer moderator
Project administrator
Project developer
Project scientist

Send message
Joined: 1 Jul 05
Posts: 1018
Credit: 4,334,829
RAC: 0
Message 63356 - Posted: 15 Sep 2009, 3:27:27 UTC - in response to Message 63355.  

It looks like some of the servers are down now too, like about half of them. UH-OH!

you can ignore that. I have to update that page to reflect some recent changes.
ID: 63356 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Michael H.W. Weber
Avatar

Send message
Joined: 18 Sep 05
Posts: 13
Credit: 6,672,462
RAC: 0
Message 63358 - Posted: 15 Sep 2009, 7:20:59 UTC

Just to say it clearly once more. It is not about the credits why I posted above. It is a new machine I have set up a few days ago and I was wondering whether something is wrong with it or with the Rosetta@home servers.

Michael.
President of Rechenkraft.net e.V.

http://www.rechenkraft.net - The world's first and largest distributed computing association. We make those things possible that supercomputers don't.
ID: 63358 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Michael H.W. Weber
Avatar

Send message
Joined: 18 Sep 05
Posts: 13
Credit: 6,672,462
RAC: 0
Message 63361 - Posted: 15 Sep 2009, 9:36:16 UTC

Could someone please check these issues:

(1) No credits - for what reason? Seems I am the only one who reported this WU:

https://boinc.bakerlab.org/rosetta/workunit.php?wuid=255385811

(2) A number of validate errors:

https://boinc.bakerlab.org/rosetta/workunit.php?wuid=255112983
https://boinc.bakerlab.org/rosetta/workunit.php?wuid=255132989
https://boinc.bakerlab.org/rosetta/workunit.php?wuid=255098797
https://boinc.bakerlab.org/rosetta/workunit.php?wuid=255085551

Where are the data I sent to the server?

Michael.
President of Rechenkraft.net e.V.

http://www.rechenkraft.net - The world's first and largest distributed computing association. We make those things possible that supercomputers don't.
ID: 63361 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sid Celery

Send message
Joined: 11 Feb 08
Posts: 1981
Credit: 38,373,493
RAC: 10,990
Message 63366 - Posted: 15 Sep 2009, 11:24:58 UTC - in response to Message 63350.  

I've had about half of my "pending" credit get it's granted credit awarded to it, so the servers must be catching up.

I've barely had a third of my backed up credit validated and I'm already up at my 2nd highest day ever! And the day isn't over yet! Holy mackerel! I don't know how big this thing is going to get if I've got another couple of days of this kind of credit. "Money in the bank" indeed. I'm loaded! My RAC has gone up 25% in about 12 hours (ok, after it went down by 20% over the last few days)!

Michael H.W. Weber wrote:
...I was wondering whether something is wrong with it or with the Rosetta@home servers.

It's part of the early attempts to solve the validation slowness issues. Nothing to worry about at your end. You were just unfortunate to start a new system at this time.

Could someone please check these issues:

(1) No credits - for what reason? Seems I am the only one who reported this WU:

https://boinc.bakerlab.org/rosetta/workunit.php?wuid=255385811

(2) A number of validate errors:

https://boinc.bakerlab.org/rosetta/workunit.php?wuid=255112983
https://boinc.bakerlab.org/rosetta/workunit.php?wuid=255132989
https://boinc.bakerlab.org/rosetta/workunit.php?wuid=255098797
https://boinc.bakerlab.org/rosetta/workunit.php?wuid=255085551

Where are the data I sent to the server?

Look at message 63281 from Gen_X_Accord on 11 Sept above and the following few posts. It's the same issue as yours.

It ends with Yifan saying:
"I actually do still use these data. I just need to figure out a way to make them give me back fewer but better structures in the future."

So don't get too anxious, Michael. It's all being worked through very rapidly and you should be seeing some results already if you're anything like me. Unfortunately your computers are hidden, so it's not easy to tell from here.
ID: 63366 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sid Celery

Send message
Joined: 11 Feb 08
Posts: 1981
Credit: 38,373,493
RAC: 10,990
Message 63369 - Posted: 15 Sep 2009, 12:28:10 UTC - in response to Message 63366.  
Last modified: 15 Sep 2009, 12:30:56 UTC

I've barely had a third of my backed up credit validated and I'm already up at my 2nd highest day ever!

Make that 44% of the backlog cleared and highest day ever (and the previous highest one was 3 days worth combined). My RAC has increased from 1011 to 1301 after 4000 credits in the last day.

I'm almost scared to look any more...
ID: 63369 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
AMD_is_logical

Send message
Joined: 20 Dec 05
Posts: 299
Credit: 31,460,681
RAC: 0
Message 63370 - Posted: 15 Sep 2009, 13:48:42 UTC

The Validators are finally catching up, and my pending list has been cut in half.

The home page still lists the TFLOPS as only 17. I don't think it's being calculated correctly in the new Validator configuration.
ID: 63370 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Yifan Song
Volunteer moderator
Project developer
Project scientist

Send message
Joined: 26 May 09
Posts: 62
Credit: 7,322
RAC: 0
Message 63371 - Posted: 15 Sep 2009, 18:10:56 UTC

Thanks for all of your patience. The validator is indeed catching up. TFLOPS is steadily climbing as well. It sums up the last 24 hrs of credit, so it might take another day to get the TFLOPS value back to before.
ID: 63371 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sid Celery

Send message
Joined: 11 Feb 08
Posts: 1981
Credit: 38,373,493
RAC: 10,990
Message 63375 - Posted: 16 Sep 2009, 0:32:43 UTC - in response to Message 63370.  

The home page still lists the TFLOPS as only 17. I don't think it's being calculated correctly in the new Validator configuration.

Boincstats reports 215 TFlops in the 24 hours to 1800GMT. More still to come...
ID: 63375 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Gen_X_Accord
Avatar

Send message
Joined: 5 Jun 06
Posts: 154
Credit: 279,018
RAC: 0
Message 63376 - Posted: 16 Sep 2009, 7:19:31 UTC

Well I'm now down to 1 pending, and that was returned today, so I'd say I'm caught up, I bet a few others get caught up in the next few days as well.
ID: 63376 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Chilean
Avatar

Send message
Joined: 16 Oct 05
Posts: 711
Credit: 26,694,507
RAC: 0
Message 63388 - Posted: 18 Sep 2009, 7:08:02 UTC

The server catched up with my pending credit. I have 0 pending credit.

TFlop rose to 72 as well. Hope it goes back too 100 like before.
ID: 63388 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Previous · 1 · 2 · 3

Message boards : Number crunching : Granted Credit taking forever....



©2024 University of Washington
https://www.bakerlab.org