According to the website status page at https://boinc.bakerlab.org/rosetta/rah_status.php, everything is hunky dory, but it's quite clear the server is not accepting completed work (from some hours ago). Perhaps this is part of some less focused strangeness that has been going on over the last few days, but if so, then there should be some kind of announcement about the problem, and I can't find that anywhere, either. On the top page, the latest news is more than a month old, and the last-listed tweet is about 4 months old.

Well, the system is still clearly out of order, and the website is still clearly incorrect in its status reports. No finished work being accepted by the Baker Lab side, and no fresh work units coming down. At least 12 hours since my original report here, and it was already several hours of brokenness at that time...

Allo? Anyone there?

P.S. Even the Twitter account appears to be dead as of March?

You may want to keep an eye on the Problems and Technical Issues with Rosetta@home thread. Several users have reported the same problem there and one of the project team members has replied to say they are investigating.

krypton wrote:

Thanks for the reports!! We are looking at this now.

Okay, now about that website Server Status Page... It is still showing that the project is all green and that is clearly all wrong. Whatever is wrong should be detected and indicated there. It's also nice if you include an estimated repair time. Communication is good, eh?

I have to repeat my point, even including the bad joke: If you don't communicate more effectively, you are liable to cause all sorts of rumors to appear.

The bad joke was my proposed rumor about an NSA trainee bollixing your website. This is NOT really a funny idea, because a rogue task (AKA work unit) could do ALL sorts of bad things, probably including hijacking your computer's camera to take embarrassing photos.

The people running these BOINC projects need to take some responsibility for providing accurate and timely information about the status of their projects. My own observations actually suggest that something has been going south from around the 22nd of July, but it clearly fell off the edge of the earth yesterday (or maybe the day before that).

For what it is worth, I suspected the change around the 22nd may have involved some attempt to fix the "Computation Error" tasks. My theory was that they nipped some of the sub-projects that were causing those errors, and the change was causing a significant drop in the statistics.

However, because of how poorly the project managers communicate, I wasn't really expecting any clarification from them. Hello, people? We know you aren't NSA professionals, but still...

P.S. Perhaps I should feel some personal culpability here, insofar as I may have been an inadvertent contributor to the design of BOINC. However, if I had been asked MUCH more politely to contribute more, then I hope that proper resource security of the client would have been one of my major concerns. Therefore, I disclaim and proclaim "It ain't my fault!"

P.P.S. If you delete the joke again, then maybe I'll stop taking the rumor as a joke. We seem to be back again to the need for improved communications skills, eh?

P.P.P.S. I'm not really Canadian, though it was an increasingly attraction optional rumor during the Dubya years (of the big dick Cheney).

I have 19 uploads pending like this:

... Upload: retry in 01:10:53 (project backoff 00:26:40)

As shanen says, the status page still reports what appears to be misinformation.

The Problems and Technical Issues page reports issues from 2011.
Any idea yet on what is going on today?

I got the same problem Since July 28.
Thanks for help.

---

I got the same problem Since July 28.
Thanks for help.

---

Same problem for me also. No work units being uploaded and no units received since July 28.

[quote]I have 19 uploads pending like this:

... Upload: retry in 01:10:53 (project backoff 00:26:40)

I have the same problem. It's been going on for at least a week now. I have 24 uploads pending and no downloads in my queue.

---

I have 19 uploads pending like this:
... Upload: retry in 01:10:53 (project backoff 00:26:40)

I have the same problem. It's been going on for at least a week now. I have 24 uploads pending and no downloads in my queue.

Today 13 downloads arrived for me. Rosetta worked through them and now it is idle again. I have 32 uploads pending.

The Problems and Technical Issues page reports issues from 2011.
Any idea yet on what is going on today?

Check the link I gave earlier. It is a sticky thread from another part of the forum, so has information from different time periods. The early posts are from 2011 while the latest posts are about this issue.

This Q&A section of the forum isn't visited much so you are unlikely to get answers in the short term. Most of the discussion is taking place in the Number Crunching section.

Strange as it may seem, the Server Status Page is actually correct. If your computer was inside the firewall at the University of Washington, you would not be aware of any issue. The problem is the internet connection from the campus being throttled to the point where uploads and downloads from us are timing out. This is not monitored on the Server Status Page.

Based on what I see in the Number Crunching section of the Message Boards, I do not expect any resolution until Monday since the Rosetta staff have been away and I expect the UW IT staff will only be back on Monday. In addition, that's Pacific Time so it will likely only be about 15h00 UTC before resolution can be expected. On top of this, when resolved, the routers and switches will get hammered with data.

---

Warped

...The early posts are from 2011 while the latest posts are about this issue.

I later saw that - thanks!

Well my CPU rate shot up a while ago and I found Rosetta was working through 19 new downloads.

The 32 completed tasks had been uploaded!
Maybe things are settling down...

Problem now diagnosed:

krypton wrote:

Some good news and bad news:

Good news: The servers and UW network is working normally. We got a
HUGE spike in new users/computers connected to the R@H project.

Bad news: We didn't get a heads up notice and so were not prepared to
handle soo much traffic at once. These computers are still in the
process of downloading Rosetta/Database. As Ananas suggested, it's an
issue with the number of allowed concurrent connections per server.

Good news: Once all these new computers get a copy of rosetta/database
(which is a large single download), everything will go back to normal.
We will be getting more servers, to prevent this from happening in the
future.

Once we know who these new users are, we'll post something on the front page.

Once again, thank you for all the feedback, these were very helpful in
debugging the issue.

-Krypton.

David E K wrote:

Yep, I'm currently optimizing the number of connections on all our servers. Looks like they can keep up without too much load/memory usage so far. These servers are pretty old and I'm sure we'll upgrade soon hopefully.

Polian wrote:

Looks like a DoS attack to me, to be honest:

I just picked a random new user ID, 680000 and went up by one from there

Example new users:

https://boinc.bakerlab.org/rosetta/show_user.php?userid=680000
https://boinc.bakerlab.org/rosetta/show_user.php?userid=680001
https://boinc.bakerlab.org/rosetta/show_user.php?userid=680002
https://boinc.bakerlab.org/rosetta/show_user.php?userid=680003
https://boinc.bakerlab.org/rosetta/show_user.php?userid=680004
https://boinc.bakerlab.org/rosetta/show_user.php?userid=680005
https://boinc.bakerlab.org/rosetta/show_user.php?userid=680006

David E K wrote:

Hmm, I also suspected this but the IP's from the logs were coming from various places. Maybe I'll have to disable new users for now until we figure things out.

David E K wrote:

I was told by Matthew Blumberg at Gridrepublic that the new users are real crunchers and that they "started a new marketing campaign via charityengine.com." So I re-enabled the account creation for these users. Our servers may get sluggish again but hopefully things will settle down as the new user rates decrease. And hopefully optimizing the connections on our servers will help. In the future, we hope to get more servers.

This issue coincided with the annual RosettaCon meeting (so most of the Baker lab members were out of town), a final ramp up in CASP targets, and me going on a family camping vacation where I had no phone reception. Normally, I would have been able to react faster to debug and help diffuse the situation.

I am sorry for any inconvenience and the fact that it took a few days to finally make some progress.