Rosetta's role in fighting coronavirus

Message boards : News : Rosetta's role in fighting coronavirus

To post messages, you must log in.

Previous · 1 . . . 6 · 7 · 8 · 9 · 10 · 11 · 12 . . . 17 · Next

AuthorMessage
Sid Celery

Send message
Joined: 11 Feb 08
Posts: 1317
Credit: 24,258,185
RAC: 12,236
Message 92666 - Posted: 30 Mar 2020, 22:59:09 UTC - in response to Message 92653.  

As a newer user, I've never seen it under 15k in the last few days - here it is at just above 8k

I know it looks like that ought to be the tasks ready to send (because it's called 'tasks ready to send') but it's actually a pile of crap. The whole server status page is a mess tbh

Scroll down to where it says "Tasks by application". That seems to be the best reflection of where we are. Don't ask why - no-one knows.

Also note the page isn't live. I thought it updated every 15 or 30 minutes, but a time on the page indicates it hasn't been refreshed for nearly 3hrs.

Yeah, you thought this project was the height of technology and sophistication? You'll soon have that idea knocked out of you.

We wait...
ID: 92666 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile yoerik
Avatar

Send message
Joined: 24 Mar 20
Posts: 123
Credit: 59,718
RAC: 570
Message 92668 - Posted: 30 Mar 2020, 23:06:39 UTC - in response to Message 92666.  

As a newer user, I've never seen it under 15k in the last few days - here it is at just above 8k

I know it looks like that ought to be the tasks ready to send (because it's called 'tasks ready to send') but it's actually a pile of crap. The whole server status page is a mess tbh

Scroll down to where it says "Tasks by application". That seems to be the best reflection of where we are. Don't ask why - no-one knows.

Also note the page isn't live. I thought it updated every 15 or 30 minutes, but a time on the page indicates it hasn't been refreshed for nearly 3hrs.

Yeah, you thought this project was the height of technology and sophistication? You'll soon have that idea knocked out of you.

We wait...

well then. That's fun.

Well, I'm glad I have the 5 tasks I do. Thanks for pointing that out - a complete noob moment there, on my part.
ID: 92668 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jim1348

Send message
Joined: 19 Jan 06
Posts: 450
Credit: 21,155,719
RAC: 55,557
Message 92670 - Posted: 30 Mar 2020, 23:08:34 UTC

I was wondering when they would hit the wall. The growth has been phenomenal. They were doing 150,000 GigaFLOPS before the virus hit, and 570,318 GigaFLOPS now.
Maybe they are adding more capacity and had to take the servers down?

I have enough work for a couple of days, but if they would communicate the status to us, it would help, especially for new users who might gravitate elsewhere.
It is a good (actually great) project, and in fact they invented the field they are in, but their public communication skills are less than some other projects I have seen.
ID: 92670 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile yoerik
Avatar

Send message
Joined: 24 Mar 20
Posts: 123
Credit: 59,718
RAC: 570
Message 92672 - Posted: 30 Mar 2020, 23:13:11 UTC - in response to Message 92670.  
Last modified: 30 Mar 2020, 23:14:07 UTC

I was wondering when they would hit the wall. The growth has been phenomenal. They were doing 150,000 GigaFLOPS before the virus hit, and 570,318 GigaFLOPS now.
Maybe they are adding more capacity and had to take the servers down?

I have enough work for a couple of days, but if they would communicate the status to us, it would help, especially for new users who might gravitate elsewhere.
It is a good (actually great) project, and in fact they invented the field they are in, but their public communication skills are less than some other projects I have seen.


I'll be out of work in approx. 8 hours, we'll see what happens.

This growth is really impressive - nearly quadrupling the computer power they've got access to... wowza.
ID: 92672 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
nnneal

Send message
Joined: 11 May 19
Posts: 35
Credit: 4,707,508
RAC: 11,201
Message 92675 - Posted: 30 Mar 2020, 23:46:06 UTC - in response to Message 92670.  

I was wondering when they would hit the wall. The growth has been phenomenal. They were doing 150,000 GigaFLOPS before the virus hit, and 570,318 GigaFLOPS now.
Maybe they are adding more capacity and had to take the servers down?


This seems like as good a time as any to build out more capacity.
ID: 92675 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Admin
Project administrator

Send message
Joined: 1 Jul 05
Posts: 5176
Credit: 0
RAC: 0
Message 92681 - Posted: 31 Mar 2020, 0:46:18 UTC - in response to Message 92670.  

Not only has the coronavirus spiked volunteer interest with our R@h project, but it is also spiking a lot of interest towards R@h within the lab so the communication should definitely get better. A lot has happened in a short amount of time.

1. As mentioned before we spiked ~4x or more in computing in the last week. So far our infrastructure has been handling this okay but we will definitely look into expansion and optimization as we expect many more volunteers to join in the near future. Also, our bandwidth has maxed out on a few occasions but has been okay since.

2. A number of outside groups have approached us to help.

a. The ARM developer community came together and helped port Rosetta for aarch64. The application is being tested on Ralph@home currently and will be released soon along with an announcement.
b. We are in communication with Microsoft to help us integrate and support Rosetta BOINC builds into the Rosetta Commons build and testing infrastructure and possibly with other developments.
c. We are in communication with Charity Engine to possibly use their infrastructure to run GPU machine learning based jobs and to help integrate such capability to R@h.

3. We have plenty of COVID-19 jobs but there may be a pause from time to time to give us a chance to prepare them based on previous results and new targets/methods etc.
4. We hope to update the Rosetta apps that are currently being tested on Ralph soon and plan to do another update in the near future, so 2 application updates are lined up for the near future.
5. With these updates, we hope to recruit more researchers within the lab to use this resource, but in the immediate future, priority will of course be towards COVID-19, specifically for the design of scaffolds and binders to COVID-19 targets as possible therapeutics.

Thanks Everyone!
ID: 92681 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile yoerik
Avatar

Send message
Joined: 24 Mar 20
Posts: 123
Credit: 59,718
RAC: 570
Message 92682 - Posted: 31 Mar 2020, 0:50:06 UTC - in response to Message 92681.  

Not only has the coronavirus spiked volunteer interest with our R@h project, but it is also spiking a lot of interest towards R@h within the lab so the communication should definitely get better. A lot has happened in a short amount of time.

1. As mentioned before we spiked ~4x or more in computing in the last week. So far our infrastructure has been handling this okay but we will definitely look into expansion and optimization as we expect many more volunteers to join in the near future. Also, our bandwidth has maxed out on a few occasions but has been okay since.

2. A number of outside groups have approached us to help.

a. The ARM developer community came together and helped port Rosetta for aarch64. The application is being tested on Ralph@home currently and will be released soon along with an announcement.
b. We are in communication with Microsoft to help us integrate and support Rosetta BOINC builds into the Rosetta Commons build and testing infrastructure and possibly with other developments.
c. We are in communication with Charity Engine to possibly use their infrastructure to run GPU machine learning based jobs and to help integrate such capability to R@h.

3. We have plenty of COVID-19 jobs but there may be a pause from time to time to give us a chance to prepare them based on previous results and new targets/methods etc.
4. We hope to update the Rosetta apps that are currently being tested on Ralph soon and plan to do another update in the near future, so 2 application updates are lined up for the near future.
5. With these updates, we hope to recruit more researchers within the lab to use this resource, but in the immediate future, priority will of course be towards COVID-19, specifically for the design of scaffolds and binders to COVID-19 targets as possible therapeutics.

Thanks Everyone!


Thanks for updating us. That's fantastic, and I think it's fair to say that we all look forward to the good that researchers can do with this data. Keep up the good work.
ID: 92682 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
pball1224

Send message
Joined: 20 Mar 20
Posts: 1
Credit: 610,181
RAC: 157
Message 92688 - Posted: 31 Mar 2020, 1:16:36 UTC - in response to Message 92682.  

If the well is running dry of new tasks, but millions are shown to be currently out in the wild, I think this would indicate that many users have quite large queues. (also evident from some comments I've seen stating folks have a couple days worth of work) With such high compute capacity at the moment, I wonder if the admins would consider limiting, somewhat, the amount of work one client can queue, and that might actually result in getting all work units completed a couple days sooner.
It might be interesting to see how all the active clients for the past few days would distributed across a chart showing them based upon how often they contact the servers, to get a better idea of how many might actually be operating in a semi-offline batch mode, or if they're all working the the default just-in-time configuration, where there's little point in queuing a large backlog of work.
ID: 92688 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile yoerik
Avatar

Send message
Joined: 24 Mar 20
Posts: 123
Credit: 59,718
RAC: 570
Message 92689 - Posted: 31 Mar 2020, 1:24:09 UTC - in response to Message 92688.  

If the well is running dry of new tasks, but millions are shown to be currently out in the wild, I think this would indicate that many users have quite large queues. (also evident from some comments I've seen stating folks have a couple days worth of work) With such high compute capacity at the moment, I wonder if the admins would consider limiting, somewhat, the amount of work one client can queue, and that might actually result in getting all work units completed a couple days sooner.
It might be interesting to see how all the active clients for the past few days would distributed across a chart showing them based upon how often they contact the servers, to get a better idea of how many might actually be operating in a semi-offline batch mode, or if they're all working the the default just-in-time configuration, where there's little point in queuing a large backlog of work.


We'll have to wait and see. A very interesting thing to consider, though. Sparked my curiousity now, as well!
ID: 92689 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile robertmiles

Send message
Joined: 16 Jun 08
Posts: 905
Credit: 10,367,432
RAC: 4,462
Message 92690 - Posted: 31 Mar 2020, 1:24:36 UTC - in response to Message 92147.  

Currently, I have 14 machines with 80+ cores and 14+ GPUs. When I switched over to run one of them to Rosetta, from cancer research (WCG), I did NOT receive anything related to covid19. Whenever I would receive ALL WUs for covid19, I will switch over... Till then I will stick with cancer research. Let me know when covid19 is the priority here. I am NOT going to download the fold@home app. If it is not thru BOINC, I do not run it.
James

There has been a discussion that Folding@home is likely to produce a BOINC version soon, Not ready yet, though.
ID: 92690 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile yoerik
Avatar

Send message
Joined: 24 Mar 20
Posts: 123
Credit: 59,718
RAC: 570
Message 92691 - Posted: 31 Mar 2020, 1:27:16 UTC - in response to Message 92690.  

Currently, I have 14 machines with 80+ cores and 14+ GPUs. When I switched over to run one of them to Rosetta, from cancer research (WCG), I did NOT receive anything related to covid19. Whenever I would receive ALL WUs for covid19, I will switch over... Till then I will stick with cancer research. Let me know when covid19 is the priority here. I am NOT going to download the fold@home app. If it is not thru BOINC, I do not run it.
James

There has been a discussion that Folding@home is likely to produce a BOINC version soon, Not ready yet, though.


Even if so, that would still take ages to finish, as they appear to not have even started. And, considering how hard it was to attempt to get Rosetta to run on GPUs - which it still can't, despite their best efforts - it would be months away at best.
ID: 92691 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Admin
Project administrator

Send message
Joined: 1 Jul 05
Posts: 5176
Credit: 0
RAC: 0
Message 92694 - Posted: 31 Mar 2020, 1:39:39 UTC - in response to Message 92666.  

We are a not for profit academic institution and not a technology company. R@h is based on BOINC which is also a project run by an academic institution.

As an academic research lab our priority is in protein design and structure prediction research. You hopefully can judge our success by the peer reviewed publications and impact from new promising technologies based on our research. Such information is regularly updated on our News.

The BOINC server status page is open source. We are not running the latest version of BOINC however.

The tasks ready to send are indeed the tasks ready to send by our feeder daemons. We try to keep this buffered to at least 15,000 or so whereas the actual queue that researchers submit to is not part of BOINC and is not shown to users other than the "Total queued jobs" on the R@h homepage. When this "tasks ready to send" buffer starts to reduce as it has been mentioned by the attentive participant, that means our internal queue is also low. This may happen from time to time as mentioned in my last post. These stats do not get updated frequently. The server status should be updated hourly so there may be a slowdown gathering the data as you note. The homepage status should be updated every 4 hours. Hope this explains things.
ID: 92694 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Admin
Project administrator

Send message
Joined: 1 Jul 05
Posts: 5176
Credit: 0
RAC: 0
Message 92695 - Posted: 31 Mar 2020, 1:45:43 UTC - in response to Message 92688.  

Very good point. We can consider reducing the "delay_bound" for important time critical jobs.
ID: 92695 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile yoerik
Avatar

Send message
Joined: 24 Mar 20
Posts: 123
Credit: 59,718
RAC: 570
Message 92697 - Posted: 31 Mar 2020, 1:58:07 UTC - in response to Message 92694.  

The tasks ready to send are indeed the tasks ready to send by our feeder daemons. We try to keep this buffered to at least 15,000 or so whereas the actual queue that researchers submit to is not part of BOINC and is not shown to users other than the "Total queued jobs" on the R@h homepage. When this "tasks ready to send" buffer starts to reduce as it has been mentioned by the attentive participant, that means our internal queue is also low. This may happen from time to time as mentioned in my last post. These stats do not get updated frequently. The server status should be updated hourly so there may be a slowdown gathering the data as you note. The homepage status should be updated every 4 hours. Hope this explains things.


Some projects do not even utilize the stats page - so I'm glad that you grant us access to what numbers you do, for our convenience.

Thanks for clarifying this for us, and for your patience as you deal with a huge influx of processing power.
ID: 92697 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Chilean
Avatar

Send message
Joined: 16 Oct 05
Posts: 710
Credit: 25,956,510
RAC: 430
Message 92700 - Posted: 31 Mar 2020, 2:31:48 UTC - in response to Message 92681.  

Not only has the coronavirus spiked volunteer interest with our R@h project, but it is also spiking a lot of interest towards R@h within the lab so the communication should definitely get better. A lot has happened in a short amount of time.

1. As mentioned before we spiked ~4x or more in computing in the last week. So far our infrastructure has been handling this okay but we will definitely look into expansion and optimization as we expect many more volunteers to join in the near future. Also, our bandwidth has maxed out on a few occasions but has been okay since.

2. A number of outside groups have approached us to help.

a. The ARM developer community came together and helped port Rosetta for aarch64. The application is being tested on Ralph@home currently and will be released soon along with an announcement.
b. We are in communication with Microsoft to help us integrate and support Rosetta BOINC builds into the Rosetta Commons build and testing infrastructure and possibly with other developments.
c. We are in communication with Charity Engine to possibly use their infrastructure to run GPU machine learning based jobs and to help integrate such capability to R@h.

3. We have plenty of COVID-19 jobs but there may be a pause from time to time to give us a chance to prepare them based on previous results and new targets/methods etc.
4. We hope to update the Rosetta apps that are currently being tested on Ralph soon and plan to do another update in the near future, so 2 application updates are lined up for the near future.
5. With these updates, we hope to recruit more researchers within the lab to use this resource, but in the immediate future, priority will of course be towards COVID-19, specifically for the design of scaffolds and binders to COVID-19 targets as possible therapeutics.

Thanks Everyone!


Please post this on the homepage! Very informative.

Thanks for the update!
ID: 92700 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Kim. Shinhyoung

Send message
Joined: 6 Apr 19
Posts: 5
Credit: 91,866
RAC: 6
Message 92702 - Posted: 31 Mar 2020, 3:01:48 UTC

I'm not getting new tasks since yesterday.
Is it something problem on my machine?
Or... does it mean that R@H is getting exploded with new participants?
If the latter is the case, I'm very happy to see my malfunctioning client.
ID: 92702 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile yoerik
Avatar

Send message
Joined: 24 Mar 20
Posts: 123
Credit: 59,718
RAC: 570
Message 92703 - Posted: 31 Mar 2020, 3:05:34 UTC - in response to Message 92702.  

I'm not getting new tasks since yesterday.
Is it something problem on my machine?
Or... does it mean that R@H is getting exploded with new participants?
If the latter is the case, I'm very happy to see my malfunctioning client.


Nothing wrong with your machine. Everyone is in the same boat. Their processing capabilities have quadrupled over the last week.
ID: 92703 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
James Lee*

Send message
Joined: 28 Jan 17
Posts: 6
Credit: 2,770,691
RAC: 75
Message 92708 - Posted: 31 Mar 2020, 5:08:12 UTC

It is good to see such interest in this project that we can actually deplete the queue. My WU buffers are only set for 2 hours (instead of days) so when I run out of WUs, I just have another project that will pick up the slack until there is work available. I just set the priority of this project 100 times greater than the secondary project, and my computers will always stay running, but with this project as my main priority.
Please keep us posted on if and when GPU tasks become available. I have about 14 GPUs that would love to run this project as well.

Thanks,
James
ID: 92708 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Gorgan Stefan

Send message
Joined: 25 May 08
Posts: 4
Credit: 33,725
RAC: 0
Message 92716 - Posted: 31 Mar 2020, 6:50:17 UTC - in response to Message 92688.  
Last modified: 31 Mar 2020, 6:52:12 UTC

I was surprised when I saw a comment from someone who had queue for a couple of days. I joined the project about 10 days ago and I usually got 2-3 extra tasks besides the 12 running tasks. My tasks take 7-8 hours to complete so I never got a large queue. Wonder why there's such a big difference.

Also since yesterday I didn't receive any new tasks and I'm now running the last 5 tasks.
ID: 92716 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1113
Credit: 4,713,064
RAC: 5,101
Message 92717 - Posted: 31 Mar 2020, 8:22:25 UTC - in response to Message 92681.  

Not only has the coronavirus spiked volunteer interest with our R@h project, but it is also spiking a lot of interest towards R@h within the lab so the communication should definitely get better.


?? IPD is not interested a lot in Rosetta@Home?
ID: 92717 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Previous · 1 . . . 6 · 7 · 8 · 9 · 10 · 11 · 12 . . . 17 · Next

Message boards : News : Rosetta's role in fighting coronavirus



©2020 University of Washington
https://www.bakerlab.org