Michael espoused ...

Do we really have to have these same complaints every time there is a server problem?

Michael, which complaints are you referring to? I think that the comments here run in two basic categories - the first being the fact that the project is down, and the second is that once again, as in the past, there is almost no status being passed back to the community.

You are right, it is what it is, and when it comes to balancing hardware resources against a finite pile of money, compromises have to be made. In my career I have worked in more than a few data centers whose equipment is so old the place could have been designated a historic landmark.

I am familiar with the challenges of keeping out-dated equipment up and running.

However, there is no excuse for the lack of communications with the community providing computing resources to the project. The last published update was days ago.

The community of Rosetta users is a diverse group - it ranges from the casual user who puts Rosetta up on his home PC to those who convert their back rooms into a mini-data center.

We all crunch Rosetta because we believe that what the project is trying to accomplish is important. However, how can be continue to believe that what we are doing is important if the project leadership continues to treat the community like an expendable resource, not worthy of the few minutes required to keep us informed?

Well, I'll step down off my soap box now with just one further thought: keep crunching brother - we are all family, even when we disagree.

CH

[rant]
You know, I left the Seti project and attached to this one because of server based problems. Now it's looking to me like the same shoestring mentality is ruling this project too, as concerns the servers and database. I don't know much about setting up the hardware to run a project such as this, but I would think that primary consideration would be given to make a robust data handling system with redundancies and it looks to me like academia in general doesn't seem to think (to me at least) that making sure your infrastructure is robust enough to take into account the possibility of equipment failure bringing an entire project to a screeching halt. Spend more money on your project infrastructure before you go on dreaming new methods to crunch numbers or whatever, you all. I have seen this with this project, Seti and also the Folding@Home project run by Stanford, although theirs seems to be more a problem with dreaming up new clients and never getting the bugs worked out of their older DC clients.

I hope that you all can get this back up and running soon. Otherwise, I guess I will move on to someone else's project, since it ticks me off to have something I am donating my time to by running their project (and paying the electricity for to run their project on my computers) not take some precautions to keep their project going by providing redundancy in their project's infrastructure to handle hardware failure.
[/rant]

Strange attitudes to have when "donating" your time (by donating time I mean letting your computer stay on). A catastrophic failure is no fun. It takes a lot of work, time, and stress. I believe there are only one or two guys that maintain the servers. I think their time is better served by writing a line or two, as they did, as to the status rather than write an essay on causes and effects and have everyone second guess their methods, actions, and infrastructure on the forums - which is exactly what would happen.

If you can't upload, just keep crunching until you can, what's wrong with that?

...I think it would be useful if the forum were hosted elsewhere from the other equipment, but I don't have any idea if that's practical.

The problem might have been different, but the symptoms appear to have been similar:

My system has been out of Rosetta work for 8 hours but finally managed to get a new task about 30 minutes ago.

This isn’t your OP, but it is a part of a post in your thread.

Although this is part of but a single post, I am fairly certain that I recall other posts on other threads that have similarly reported a complete exhaustion of buffered work, followed by an idle period lasting for a number of hours before new work was received after Rosetta server outages. I will leave it to others, if they are sufficiently interested, to seek the specific posts that report idle time waiting for work.

Why, for example, did this particular poster run out of work while nobody else appeared to do so at the same time? My machine was busy during the period that posts were made to your thread.

Are you sure that you are not describing a distinction without a difference?

As you are quoting me there I might as well respond. Though the "symptoms" may sound similar, it is indeed a completely different problem. The issues you have had relate to problems connecting to the server. The issue described in the other thread was that the work queue ran out of tasks for a short period of time.

The Project team are usually quite good at refilling the work queue on time, so I would guess that they were just caught by surprise by the jump in project speed from the normal 100 Teraflops to the current 121 Teraflops. A 1/5 increase in speed probably emptied the queue a lot faster than they were expecting.

You have implied that my system was idle while Rosetta was short of work, but that wasn't really the case. When my system couldn't get work from Rosetta it just downloaded an extra task from Poem@home and crunched on that for a while. Both projects aim to improve our understanding of proteins, so it doesn't matter to me which one is running.

In answer to your question of why I ran out of Rosetta work and you didn't, it is simply a matter of buffer sizes. You mentioned above that you have a 2 day buffer so an 8 hour shortage of work would probably not have been noticed. However my system is powered down quite often, which can confuse BOINC's calculations on the size of buffer to maintain and lead to missed deadlines, so I keep the buffer at minimal levels.

Deesy58 -

I think that it is safe to say that while the end result was the same - a problem with getting new work units out to the community, the problem described in this thread and the "Houston we have a problem" thread were completely different in nature.

Otherwise I would not have started a new thread.

My system has been out of Rosetta work for 8 hours but finally managed to get a new task about 30 minutes ago.

The problem described in the "Houston we have a problem" thread centered completely on the fact that the reservoir to available work units had dropped to zero for a period of 6 to 8 hours or so. Communications between the BOINC client and the project server were up and functional.

The problem described in this thread seemed to center around a network issue in the project facility - and judging from the fact that when things came back up my browser could hit bakerlab.org but the BOINC client could not it appeared that maybe the BOINC client cached the "old" IP address when it was brought up and that some time during all this that address changed.

The results of ping, nslookup, and trace route commands seem to support this. Was it a change in network configuration or maybe DHCP got in the middle, I don't know for sure, and likely never will.

But since once I could hit the name server and resolve bakerlab.org again, BOINC still would not connect until after a restart it is logical to assume that BOINC does indeed cache the address instead of doing a lookup each time.

I admit that it is speculation, and that I don't have the facts to conclusively state that this is the exact scenario. However, I have had my hands deep in the bowels to many an IP stack and feel comfortable that this was a logical conclusion to draw.

So because the problems were different in nature, your references to the other thread seem a little weak and unrelated to me. But that is just my two cents worth and I don't think either of us were made privy to the technical details by the project.

Have a good night and don't stress so much over past problems - I think in both these cases it is clear the problem was on the "project side" and not with your system, my system, or Sid's system.

I guess you assume (incorrectly) that EVERYBODY who experiences work outages posts on THIS thread.

No, but you can list all the other threads reporting the same issue if you like, or the individual posts. Perhaps I missed them.

Do you actually think that there might be multiple versions of BOINC, and that my version might be different from yours?

Of course there are and running on a wide range of platforms too.

Q.E.D.

Don't we all run the same basic version of BOINC and Rosetta that are automatically updated when necessary?

Rosetta yes, Boinc no, it certainly isn't updated when necessary unless you manually download a new version.

Okay. I am running Version 5.10.28. Is there a newer/better version available for Windows?

From the first link in the quote above you will see the currently recommended version of BOINC for Windows 2000/XP/Vista/7 is 6.10.58. It comes in 32 bit and 64 bit versions.

I cannot provide either positive or negative commentary on the latest version as my system is stable with version 6.4.5. I expect that I won't be upgrading until either my system starts to become unstable or I hear of a new feature that may be useful to me.

In msg 1 you said you had 14 tasks ready to upload, 5 of which had failed. Where did you get 20 from? If you'd actually had 20 it might well have been enough. Personally I keep 2 days worth of 8 hour tasks too, which on my unattended Vista quad machine running 247 is around 24 tasks. I don't know how many cores you run (2?), but it doesn't seem to add up.

Looking through this thread I see, in my absence, only one other person confirmed your problem and they found they had to restart Boinc locally to restore connectivity. There's a hint.

Do you actually think that there might be multiple versions of BOINC, and that my version might be different from yours?

Of course there are and running on a wide range of platforms too.

Don't we all run the same basic version of BOINC and Rosetta that are automatically updated when necessary?

Rosetta yes, Boinc no, it certainly isn't updated when necessary unless you manually download a new version.

If you believe that I have somehow acquired a defective version of BOINC, you should say so, and you should tell me (and everybody else) how to correct it.

Just as well I didn't say so then. There are more issues than just the software and platform to consider, like with any software or connectivity issue.

It appears that you are making unwarranted assumptions, again.

I don't think so. You're saying the servers were up but you couldn't connect and you're assuming that's because the servers weren't actually up. But the only person reporting the same as you restarted Boinc locally and the problem went away. I know the servers were up because I connected several times when you were still suffering a problem and I happen to know I don't have a magic key that lets me in and keeps you out. If it really was a lottery I'd expect occasional failures to connect for me and occasional successes for you. That didn't happen for me (I'm just not that lucky, unfortunately) and you never did quite detail answers to my question about what errors you had and at what times...

Thanks for that. First, as I pointed out before, these are Boinc defaults, not Rosetta defaults (though I doubt that makes a lot of difference in itself tbh). More importantly, defaults in this situation are lowest common denominator across all projects, not tailored to meet every eventuality of this one - especially as a 3 hour runtime and a 0.25 buffer would hardly be recommended as a panacea for all Boinc projects by anyone. That's enough, but 3rd, as has been pointed out already, no project can guarantee uptime, so putting all eggs in one project's basket is going to result in a problem eventually. If Rosetta goes down for a month, we're all running out of Rosetta tasks, though I'll still be running 24/7 here.

Would it be an improvement to limit the number of work units that were distributed to each user for some period of time after a server outage - say 24 hours? Why not ration the work until such time as the system has completely recovered from the outage. It's difficult to see how such an approach would not be more efficient.

Actually no. If it was a case of the server struggling to meet the demands on it (which I don't accept at all btw, but just say mod.sense is right) a user would get insufficient tasks on an eventual successful connection, so it'll just come back again and again to fill the rest of its buffer, resulting in more hits even after connection was successful, not less.

Where this strategy would help is if there were only a few tasks to grab and it was better if everyone got something to get them started. That wasn't the case from what I can recall. (Just seen your convoluted analogy. Very good, but typically it's the one that didn't apply in this situation).

For the record, these weren't 'surplus' tasks - simply the rate my unattended quad machine completed tasks over that 36 hour period (plus refilling the buffer). No doubt I had some of the early 2.16 tasks that crashed on start-up among those.

This is the issue that nobody has been able to explain to me, or to anybody else who has experienced the same problem.

It's sounding more like a local connectivity issue the more I read. Did you try restarting Boinc andor your computer at any point when you saw the servers were reporting as up or is this thread really all down to you blaming someone else before checking at your end? That would explain everything, wouldn't it?

Anyway, I'm sure everything's solved now. Until the inevitable next time... ;)

I believe the implication was that perhaps you were having a problem more specific to your machine then the project in general (such as being handed a small pile of tasks that fail immediately).

no queue at all -- solely a matter of luck

Yes, the host will be short of work very quickly in such a case and then be vying for a scheduler request with everyone else. Odds being no better nor worse at getting new tasks.

In rereading your original posts, I believe the extended backlog on the servers was due to the tasks that had a new optional parameter specified, which was no longer supported by the v2.16 version (the failures you mentioned). So a lot of the work that was being sent out did not keep the machines busy. The tasks immediately failed. The machine immediately required more work and the cycle continued.

Because Rosetta does not guarantee to provide you with work at all times. BOINC includes the option to subscribe to other useful projects to keep your computer busy when your main project is experiencing difficulties. That you have chosen to work exclusively with Rosetta is admirable, but you do have to accept the pitfall that you have no backup and that you will likely be one of the first to experience any difficulties and see them last longer than other users.

People who have been able to select more than one project encounter problems more rarely as their systems adjust automatically to compensate.

Because a one size fits all approach doesn't work. One example is that a default setting of a 10 day buffer would be problematic for people with intermittent connections. If you have a set of tasks with a 12 day turn around but only connect to the network once every 7 days, you will return some of your tasks on the 7th day but the rest will expire on day 12; when you reconnect to the network to upload your tasks on day 14 you will find that all work between day 7 and day 10 has expired and is effectively wasted.

For some people a 10 day work buffer may be an ideal solution, which is why you can customise your settings to match your own circumstances.

I am really unclear why you presume other users were requesting, and being granted piles of surplus tasks while you were without work.

The whole site was down for about 10 hours on Saturday 9th. When it came back up, my logs show no problem uploading or downloading straight away (sending 6, receiving 8 WUs). I have no idea why your machine didn't dial in for a further 24 hours, but Boinc gets funny sometimes. Your buffer is inadequate, as we discussed before, but apparently you know best so I assume that was intentional. No problems on an unattended machine here because I don't fail to plan. Funny how that keeps on working for me.

Before you posted for the first time my unattended machine had connectedul'ddl'd 5 times successfully over the previous 3.5 hours and, it seems, did so again 11 more times by the time your machine got its first tasks.

...ah! NOW you are coming around to where some of the original conversation was focused. If you can crunch 2 days worth of work in 9 hours, it clearly indicates that you did not truly have 2 days of work; right? Simple as that. And that means the BOINC client did not request enough to truly maintain a 2 day cache. And that is exactly the sort of thing other people were observing throughout the BOINC community, and changes were made to BOINC to refine the work-fetch rules to avoid idle CPUs.

If I understand you right, you are suggesting that the backlog will be cleared sooner if the bank limits withdraws to $250 per customer per some time limit. But to do so, now each teller has to verify the time limit, one which varies for each specific customer, before completing a transaction, and the customer has to make multiple transactions just to get the $1,000 they came for. Won't this make each transaction take slightly longer? And wouldn't that make it take longer to get on top of the backlog?

The approach, and this is from the Berkeley server code, is instead to try to make best use of each contact with the client. "best" here meaning send them everything they need. You don't know when they will be able to connect again. This might be the only work request the machine is allowed to make all week. Any client requesting "more work then it could possibly process" is already refused. In general, if everything is running well on the client side, no such requests are ever made.

I believe what you are suggesting though is to only send one task per CPU for example.

The idea has some merit, indeed I've had thoughts along those lines myself, but the code changes to determine when to enter this server conservation mode add additional overhead and are quite complex, and the potential benefits are fairly minimal.

There are a certain number of database hits that occur for each scheduler request, regardless of the amount of work being sent. So if you are doing 20 IOs already, to send out 1 task, why not do 5 more and send out 6 tasks and fulfill the entire request? Your alternative is to process multiple server hits, doing the 20 IOs multiple times to send out 6 tasks.

You could argue that there are multiple server hits occurring now as well, and that is certainly true. But when a web server is backlogged and requests are timing out, the server basically doesn't even actually see the ones it couldn't get to in time, so they bare no cost to the server's performance. So the approach taken by Berkeley yields the least demand for resources on the server overall. The number of requests actually processed is minimized by this approach.

No project will have work available all of the time. With a two day buffer, you have already mitigated your risk of not having work during an outage. Unfortunately for you, it would seem your two days was up (i.e. your need for more work occurred) during an outage. If the average outage is 6 hours and the average server recovery time is 2 hours, your 2 day buffer already reduced your odds of encountering a server backlog to 1 in 6. Given those (roughly historical, yet guesstimated) numbers, 5 out of 6 times you would be completely unaware of a 6hr outage when carrying a 2 day buffer of work.

If you picture a bank, with 1000 customers demanding to withdraw their savings immediately, only with no lines... just a free-for-all on how and when a customer reaches a teller, that is what the world is like for an internet server (of any kind, not just BOINC). So while the bank is open (i.e. servers are running), your request may timeout before it gets a reply. So, in addition to cases where BOINC clients sit idle, not requesting more work when you would think they should, the response times from an overloaded server are always highly variable. It just boils down to random chance. When a request is received, it is fulfilled the same way as when the servers are not busy (i.e. the request, if it is completed, gets assigned all of the work necessary to fill it, assuming enough work is available). So the small percentage of requests that complete in the chaotic bank lobby, are handled with the same care and attention to detail as when the bank is not busy.

Updating to the project only puts your hat in the ring more times (assuming the updates are not so frequent that you get the last request too recent messages). If the server is still busy, you still only have a 1% chance of getting lucky. The good news is that the servers are generally able to work through such backlogs in just a few hours, and so many people don't even notice there was an outage.

dessy58, in the future, please do not respond to posts you find not particularly helpful. Often when server outages occur, various users have different observations over time as things recover. One user stating their observations after you have stated yours should not be taken as any contradiction nor expectation on what you observe. If you string together 5 or 6 such factual observations, you can often see progress with time on getting things back to normal. And so it is commonplace to make such posts in threads such as this one.

When facts about errors, and retries are omitted from problem descriptions the reader is left to presume many things; especially in a project where you can specify a preference for tasks to run anywhere from an hour to 24hrs.

If the reader frequents a number of BOINC project message boards, they often fill in missing details with similar problems they are familiar with. There are a number of issues where BOINC's core client is not requesting work from projects even when cores are idle.

BOINC versions do not automatically up date themselves, so every client machine can be different. Indeed there are scores of versions possible.

When you have a number of rapid failures in a row, the BOINC core client can get confused about how long to expect tasks to take to complete and has trouble requesting a proper amount of work to match the desired network preferences. It appears a batch of v2.16 tasks that were sent out failed on startup. These have since been removed. When such pervasive problems are encountered, the servers get bottlenecked trying to replace and reissue the failing tasks.

No, project servers do not contact the attached machines when they recover from an outage... in order or otherwise. Project servers never contact the attached machines, the architecture is always client-pull, not server-push. Depending upon how many times your machine tried to contact the project, the delay time until the next request gets increasingly large, that may explain any time gap where your machine did not attempt to contact the project for a few hours (if that occurred).

# of tasks in your buffer isn't sufficient information. But you know that, so I leave it to you to fill in the blanks.

The rest isn't relevant if Boinc doesn't call for tasks. Before you posted for the first time my unattended machine had connectedul'ddl'd 5 times successfully over the previous 3.5 hours and, it seems, did so again 11 more times by the time your machine got its first tasks. How many times in the 14hr period (not more) before you received your first tasks did you get the reply "Internet access OK - project servers may be temporarily down" or "Scheduler request completed: got 0 new tasks"?

The first answer I think will be 'none' because the task server was up when the website returned and you posted after that. The second answer, from what you've said, might be once, but seeing as you received many tasks as soon as your machine reported it may also be 'none'. In which case it's a different problem that may be entirely local.

I can't see where I've said no-one else should have a problem because I didn't. What I'm seeking to clarify is whether Boinc on your machine didn't uldl tasks because it didn't actually ask for any. Why that should be, I can't say, but Boinc does get it's task requests wrong a lot (something I see regularly, if not often) so it wouldn't surprise me. Again, that would be a local problem.

The whole site was down for about 10 hours on Saturday 9th. When it came back up, my logs show no problem uploading or downloading straight away (sending 6, receiving 8 WUs). I have no idea why your machine didn't dial in for a further 24 hours, but Boinc gets funny sometimes. Your buffer is inadequate, as we discussed before, but apparently you know best so I assume that was intentional. No problems on an unattended machine here because I don't fail to plan. Funny how that keeps on working for me.

On your computation error issues, all those mem_widd tasks show the same failure and Yifan has confirmed the problem is at Rosetta's end - see the 2.16 thread.