team Challenge
https://www.boincstats.com/stats/challenge/team/chat/1074

Chances are very high that we're gonna finish last lol, but I just signed our tiny, but precious, team up!

If any of y'all are teamless, you're more than welcome to join!

---

Just added my 384 threads after being gone for 2 years.... 11 servers 24/7. I don't care about the names. I'll run anything that comes down the network link... First units should be coming back in about 4 hours from the faster machines. Slowest machines are about 8 hours. All 1600 WUs downloaded should be back in about 24 to 28 hours

That's awesome. Thank you for coming back!

My computer does not receive new jobs, maybe they are over. Waiting from China for i5-4690 instead of G1840.

My computer does not receive new jobs, maybe they are over. Waiting from China for i5-4690 instead of G1840.

According to the Server Status page, R@H is basically out of WU to send. Over 1.4 MILLION WUs out in the wild being crunched. I don't recall a time when there were so many WUs distributed out in this project.

---

According to the Server Status page, R@H is basically out of WU to send. Over 1.4 MILLION WUs out in the wild being crunched. I don't recall a time when there were so many WUs distributed out in this project.

Indeed!
On one side im so happy to see so much people joined, fighting togehter in the biggest war in the human history.
On the other side, Im very sad that we are joined for that reason.

---

I was draining down my 24 thread 32GB machines and had a mix of Rosetta and other work. 2 of the 3 machines were well into the swap file but I was to catch them before they ran out. Trimmed back some work and they are now working just fine. Once the other work drains off I will open them up again. The other machine was out of swap and totally thrashing (disk light on continuously and couldn't log on). Reboot and trimmed back some work like the others. On my 128 thread 256GB machine, it filled up the root filesystem as I only had 50GB allocated to it and when Rosetta started up with about 500MB per slot, BOINC died. I was able to still logon so was able to shrink some LVs and extend the root LV. All good now. Everything is running as expected.

According to the Server Status page, R@H is basically out of WU to send. Over 1.4 MILLION WUs out in the wild being crunched. I don't recall a time when there were so many WUs distributed out in this project.

Indeed!
On one side im so happy to see so much people joined, fighting togehter in the biggest war in the human history.
On the other side, Im very sad that we are joined for that reason.

As a newer user, I've never seen it under 15k in the last few days - here it is at just above 8k. It's great to see that we've come together - I just hope they stick around, so they can fight other diseases. Whether with Rosetta, or other projects.

As a newer user, I've never seen it under 15k in the last few days - here it is at just above 8k

I know it looks like that ought to be the tasks ready to send (because it's called 'tasks ready to send') but it's actually a pile of crap. The whole server status page is a mess tbh

Scroll down to where it says "Tasks by application". That seems to be the best reflection of where we are. Don't ask why - no-one knows.

Also note the page isn't live. I thought it updated every 15 or 30 minutes, but a time on the page indicates it hasn't been refreshed for nearly 3hrs.

Yeah, you thought this project was the height of technology and sophistication? You'll soon have that idea knocked out of you.

We wait...

---

As a newer user, I've never seen it under 15k in the last few days - here it is at just above 8k

I know it looks like that ought to be the tasks ready to send (because it's called 'tasks ready to send') but it's actually a pile of crap. The whole server status page is a mess tbh

Scroll down to where it says "Tasks by application". That seems to be the best reflection of where we are. Don't ask why - no-one knows.

Also note the page isn't live. I thought it updated every 15 or 30 minutes, but a time on the page indicates it hasn't been refreshed for nearly 3hrs.

Yeah, you thought this project was the height of technology and sophistication? You'll soon have that idea knocked out of you.

We wait...

well then. That's fun.

Well, I'm glad I have the 5 tasks I do. Thanks for pointing that out - a complete noob moment there, on my part.

I was wondering when they would hit the wall. The growth has been phenomenal. They were doing 150,000 GigaFLOPS before the virus hit, and 570,318 GigaFLOPS now.
Maybe they are adding more capacity and had to take the servers down?

I have enough work for a couple of days, but if they would communicate the status to us, it would help, especially for new users who might gravitate elsewhere.
It is a good (actually great) project, and in fact they invented the field they are in, but their public communication skills are less than some other projects I have seen.

---

I was wondering when they would hit the wall. The growth has been phenomenal. They were doing 150,000 GigaFLOPS before the virus hit, and 570,318 GigaFLOPS now.
Maybe they are adding more capacity and had to take the servers down?

I have enough work for a couple of days, but if they would communicate the status to us, it would help, especially for new users who might gravitate elsewhere.
It is a good (actually great) project, and in fact they invented the field they are in, but their public communication skills are less than some other projects I have seen.

I'll be out of work in approx. 8 hours, we'll see what happens.

This growth is really impressive - nearly quadrupling the computer power they've got access to... wowza.

I was wondering when they would hit the wall. The growth has been phenomenal. They were doing 150,000 GigaFLOPS before the virus hit, and 570,318 GigaFLOPS now.
Maybe they are adding more capacity and had to take the servers down?

This seems like as good a time as any to build out more capacity.

Not only has the coronavirus spiked volunteer interest with our R@h project, but it is also spiking a lot of interest towards R@h within the lab so the communication should definitely get better. A lot has happened in a short amount of time.

1. As mentioned before we spiked ~4x or more in computing in the last week. So far our infrastructure has been handling this okay but we will definitely look into expansion and optimization as we expect many more volunteers to join in the near future. Also, our bandwidth has maxed out on a few occasions but has been okay since.

2. A number of outside groups have approached us to help.

a. The ARM developer community came together and helped port Rosetta for aarch64. The application is being tested on Ralph@home currently and will be released soon along with an announcement.
b. We are in communication with Microsoft to help us integrate and support Rosetta BOINC builds into the Rosetta Commons build and testing infrastructure and possibly with other developments.
c. We are in communication with Charity Engine to possibly use their infrastructure to run GPU machine learning based jobs and to help integrate such capability to R@h.

3. We have plenty of COVID-19 jobs but there may be a pause from time to time to give us a chance to prepare them based on previous results and new targets/methods etc.
4. We hope to update the Rosetta apps that are currently being tested on Ralph soon and plan to do another update in the near future, so 2 application updates are lined up for the near future.
5. With these updates, we hope to recruit more researchers within the lab to use this resource, but in the immediate future, priority will of course be towards COVID-19, specifically for the design of scaffolds and binders to COVID-19 targets as possible therapeutics.

Thanks Everyone!

---

Not only has the coronavirus spiked volunteer interest with our R@h project, but it is also spiking a lot of interest towards R@h within the lab so the communication should definitely get better. A lot has happened in a short amount of time.

1. As mentioned before we spiked ~4x or more in computing in the last week. So far our infrastructure has been handling this okay but we will definitely look into expansion and optimization as we expect many more volunteers to join in the near future. Also, our bandwidth has maxed out on a few occasions but has been okay since.

2. A number of outside groups have approached us to help.

a. The ARM developer community came together and helped port Rosetta for aarch64. The application is being tested on Ralph@home currently and will be released soon along with an announcement.
b. We are in communication with Microsoft to help us integrate and support Rosetta BOINC builds into the Rosetta Commons build and testing infrastructure and possibly with other developments.
c. We are in communication with Charity Engine to possibly use their infrastructure to run GPU machine learning based jobs and to help integrate such capability to R@h.

3. We have plenty of COVID-19 jobs but there may be a pause from time to time to give us a chance to prepare them based on previous results and new targets/methods etc.
4. We hope to update the Rosetta apps that are currently being tested on Ralph soon and plan to do another update in the near future, so 2 application updates are lined up for the near future.
5. With these updates, we hope to recruit more researchers within the lab to use this resource, but in the immediate future, priority will of course be towards COVID-19, specifically for the design of scaffolds and binders to COVID-19 targets as possible therapeutics.

Thanks Everyone!

Thanks for updating us. That's fantastic, and I think it's fair to say that we all look forward to the good that researchers can do with this data. Keep up the good work.

If the well is running dry of new tasks, but millions are shown to be currently out in the wild, I think this would indicate that many users have quite large queues. (also evident from some comments I've seen stating folks have a couple days worth of work) With such high compute capacity at the moment, I wonder if the admins would consider limiting, somewhat, the amount of work one client can queue, and that might actually result in getting all work units completed a couple days sooner.
It might be interesting to see how all the active clients for the past few days would distributed across a chart showing them based upon how often they contact the servers, to get a better idea of how many might actually be operating in a semi-offline batch mode, or if they're all working the the default just-in-time configuration, where there's little point in queuing a large backlog of work.

If the well is running dry of new tasks, but millions are shown to be currently out in the wild, I think this would indicate that many users have quite large queues. (also evident from some comments I've seen stating folks have a couple days worth of work) With such high compute capacity at the moment, I wonder if the admins would consider limiting, somewhat, the amount of work one client can queue, and that might actually result in getting all work units completed a couple days sooner.
It might be interesting to see how all the active clients for the past few days would distributed across a chart showing them based upon how often they contact the servers, to get a better idea of how many might actually be operating in a semi-offline batch mode, or if they're all working the the default just-in-time configuration, where there's little point in queuing a large backlog of work.

We'll have to wait and see. A very interesting thing to consider, though. Sparked my curiousity now, as well!

Currently, I have 14 machines with 80+ cores and 14+ GPUs. When I switched over to run one of them to Rosetta, from cancer research (WCG), I did NOT receive anything related to covid19. Whenever I would receive ALL WUs for covid19, I will switch over... Till then I will stick with cancer research. Let me know when covid19 is the priority here. I am NOT going to download the fold@home app. If it is not thru BOINC, I do not run it.
James

There has been a discussion that Folding@home is likely to produce a BOINC version soon, Not ready yet, though.

Currently, I have 14 machines with 80+ cores and 14+ GPUs. When I switched over to run one of them to Rosetta, from cancer research (WCG), I did NOT receive anything related to covid19. Whenever I would receive ALL WUs for covid19, I will switch over... Till then I will stick with cancer research. Let me know when covid19 is the priority here. I am NOT going to download the fold@home app. If it is not thru BOINC, I do not run it.
James

There has been a discussion that Folding@home is likely to produce a BOINC version soon, Not ready yet, though.

Even if so, that would still take ages to finish, as they appear to not have even started. And, considering how hard it was to attempt to get Rosetta to run on GPUs - which it still can't, despite their best efforts - it would be months away at best.

We are a not for profit academic institution and not a technology company. R@h is based on BOINC which is also a project run by an academic institution.

As an academic research lab our priority is in protein design and structure prediction research. You hopefully can judge our success by the peer reviewed publications and impact from new promising technologies based on our research. Such information is regularly updated on our News.

The BOINC server status page is open source. We are not running the latest version of BOINC however.

The tasks ready to send are indeed the tasks ready to send by our feeder daemons. We try to keep this buffered to at least 15,000 or so whereas the actual queue that researchers submit to is not part of BOINC and is not shown to users other than the "Total queued jobs" on the R@h homepage. When this "tasks ready to send" buffer starts to reduce as it has been mentioned by the attentive participant, that means our internal queue is also low. This may happen from time to time as mentioned in my last post. These stats do not get updated frequently. The server status should be updated hourly so there may be a slowdown gathering the data as you note. The homepage status should be updated every 4 hours. Hope this explains things.

---

Very good point. We can consider reducing the "delay_bound" for important time critical jobs.

---