Thanks Feet1st !!!!!

Exactly the kind of help I was in need of.

-Sid

Insidious, I was just wondering which optimized client you are using on your X2 3800's. I don't know if you will see this, but this was your most recent posting. If you don't feel like telling everyone, then please email it to me at motoguzzie123@aol.com

It looks like he is using
5.5.0 on two of them (most probably crunch3rs old one)
and 5.4.11 on the 3rd one (looks like standard client)

Ah, sorry then,

OK
go to the main rosetta page,
click on "your account"
click on "Computers on this account View computers "
Under this column "Computer ID", select the puter you want to merge or delete
Scroll to the bottom of the page, and if this wasn't disabled by the projects you'd see:

Click to merge this computer
or
Click to delete this host.

tony

It'll take a while.

HOW ?

Thanks for taking the time to convince your boss. We appreciate it. Tell you boss I/we appreciate it! If it comes down to it, we'll suck up to him/her!

We'll have 'host merge' back soon. As to deleting, the system will allow you delete the hosts once all "Results" for that computer have been purged. It'll take a while.

Right now the best option seems to be to get the real bugs out of the system, and then tackle the checkpointing issue. But in the face of the short time remaining before the beginning of CASP, increased checkpointing may be beyond reach at this time. Far better to have a stable application, with some use restrictions, than to have an application that checkpoints a lot but fails routinely for other reasons.

Hi Moderator9 and all,

I want to give you some updates on the long jobs / stucked jobs issue based on your suggestions and discussion here:

1, We've tracked down the bugs which were causing some jobs stucked at 1.04% and been testing the fixes on ralph since yesterday.

2, We've coded up rosetta to do more frequent checkpointing in the modeling process. Now for the large jobs, we are expecting less than 30 minutes for the time between two check points. This code has been tested locally, and will be tested on ralph within a couple of days.

3, Rhiju has coded a watchdog thread for rosetta which will terminate the stucking jobs and return the intermediate results. see his post at this thread. This will be tested on ralph within a couple of days too.

We think these measures will greatly improve the stability of R@H, and make your crunching effort much more enjoyable!

Thanks for the quick replys!

So do I need to have my switch time > time for the entire WU to complete?
Isn't there any checkpointing?

-Sid

ALL Work Units will checkpoint at the completion of a model. For some Work Units this means every 5 minuets, for larger ones this could mean 5 or six hours. Also ALL Work units will complete AT LEAST one model no matter how you set your user selectable time setting

The BEST answer if you can do it, is to set your preferences to keep the application in memory during a swap. You could try to set the swap time to 4+ hours, but there is no guarantee that that will make it to a checkpoint. It depends on the size of the protein.

Also keep in mind that "keep in memory" only works if you do not turn your machine off, or stop BOINC for some reason, as these actions would also remove the application from memory.

This issue has to be adressed ASAP. Many cycles go directly to the trash can because of this. An improved checkpointing system should be #1 priority on the TO DO list of the development team of Rosie.

Regards.

I totally agree with you Sid. It is not acceptable to have a job running for hours without a checkpoint. We realized that this is a mistake and have had those largescale jobs canceled from the boinc server. If your computer still has the largescale jobs running or queued please abort them - the new short jobs are waiting to be sent!

In the future we will be extremely cautious to not let this happen again - we are coding and testing some solutions to make the program checkpoint more often. It is of our highest priority at this moment to make sure that ever minute of your precious computer-on time spent on the Rosetta@home project can contribute to the scientific goals we are trying to achieve together.

I should add that for users who crunch Rosetta 24/7 or have "leave in memory" on, you can choose to let the largescale jobs currently in your computers keep running. These results are still of great interest to us!

Thanks Bin!

Knowing the project IS sensitive to this kind of concern and IS making an effort to shape this project in a productive way, is plenty to keep me supporting such an interesting and valuable research endeavor!

-Sid

I get the impression from your response that you have misinterpreted the information I tried to provide, to imply that I had in some way said the project does not think this is important. Nothing could be further from the truth. The original question was why did the Work Unit run so long without a checkpoint (see thread title). I never said this was an issue the project was ignoring, or not trying to fix. I just tried to explain WHY it was working the way it was.

I totally agree with you Sid. It is not acceptable to have a job running for hours without a checkpoint. We realized that this is a mistake and have had those largescale jobs canceled from the boinc server. If your computer still has the largescale jobs running or queued please abort them - the new short jobs are waiting to be sent!

In the future we will be extremely cautious to not let this happen again - we are coding and testing some solutions to make the program checkpoint more often. It is of our highest priority at this moment to make sure that ever minute of your precious computer-on time spent on the Rosetta@home project can contribute to the scientific goals we are trying to achieve together.

I should add that for users who crunch Rosetta 24/7 or have "leave in memory" on, you can choose to let the largescale jobs currently in your computers keep running. These results are still of great interest to us!

This is ridiculous. A simple home-cruncher, leaving his/her PC on for only a few hours per day, could get stuck on one of those WUs indefinitely!

It does sound like a rather stringent requirement.

Is this going to be the norm. from here on out?

The plan is to try to have it checkpoint more often, or at least try to dump everything to a memory image at program swaps. But there is a lot of data involved, and interrupting the model effects the model outcome adversely.

This is ridiculous. A simple home-cruncher, leaving his/her PC on for only a few hours per day, could get stuck on one of those WUs indefinitely!

yes you're right.
Rosetta will only save after every completed model. With these large models, you have to complete them in one go or start over from the beginning when you unload the project (switch to another project for a little time, reboot your computer, you get the idea).

Sid, are your estimated times going down? Closer to actual? If so it is working. As others have pointed out, letting it go into panic mode will get the estimates closer faster.

Also, what sort of time frame are you looking at for your resource balance? If its daily, then Bonic in general may be a lost cause for you, if its longer term balance, it will sort itself out over time.

As a side note SETI will futz with your completion times as well with the various angle ranges of the WU, and will be more pronounced when enhanced goes live.


Also a member of the TeAm. :)

That is a very accurate re-iteration of the issue I am trying to describe. (the idling of another BOINC project in favor of Rosetta for a day or so)
If it were only a matter of a single instance of this occurance I wouldn't think too much of it. The trouble is that this particular machine has gone into this cycle 2 times now. The first time, I aborted the excess work units and the machine was fine for a while but overloaded itself again after a few days. So, this time I reset the project and again, after a few days I found that it had overloaded itself once again. I aborted about a half-dozen of the pending WUs and now it is happy... but it is frustrating keep watching Rosetta push the other project aside. (I like the other project too)

Most of your problem is right here. The continual deletion of WU's and resetting the project keep BOINC from adjusting the duration correction factor properly to the new WU size.

I know you don't want to hear this, but leave BOINC alone, and you'll have fewer problems in the long run. Yes, you'll have days where one project monopolizes the machine (I've had this with Leiden and Sztaki recently.) But in the end, the adjustment factor will kick in, and the long-term debts will accrue correctly and you will have days with NO work for Rosetta. In the end, everything will balance out. But by micro-managing, you're probably making it worse, not better.

If you've set a preference for how long to crunch a WU, then it will try to crunch about that long. Note that the setting will take effect next time boinc contacts the rosetta server.

If you haven't set a preference, the WU will use it's built-in default value. This is 2 hr for current WUs and 8 hr for older ones.

The estimated crunch time that boinc displays has absolutely no effect on how long the WU will actually take.

See the FAQ for more details.

I have left the settings at default. Obviously if I had changed them, I wouldn't be complaining that there is an issue.

-Sid

On the three machines I have been observing, only one was forced into DCF mode when the new time setting became available. At first I tried to manually intervene but this only produced a temporary fix for the problem, and required me to constantly tinker with the machine. When I decided to allow the machine to sort itself out, I set the time to 4 hours, and the time between contacts to the server to .25 days. In less than 24 hours the system stabilized. I was then able to raise the connection time in increments over two days (about 5 adjustments total) and it is now running very well.

You could probably make larger adjustments than I did in the connect time to make it happen faster, but the point is that the system MUST be allowed to correct itself over time. BOINC doe snot have any information about the actual length of the WUs and so it must adjust to them over time. This same situation occurs on other projects when shorter WUs are replaced by longer ones. BOINC is designed to work this out for itself.

If you've set a preference for how long to crunch a WU, then it will try to crunch about that long. Note that the setting will take effect next time boinc contacts the rosetta server.

If you haven't set a preference, the WU will use it's built-in default value. This is 2 hr for current WUs and 8 hr for older ones.

The estimated crunch time that boinc displays has absolutely no effect on how long the WU will actually take.

See the FAQ for more details.