anyway--main question--are people seeing more stuck work units now than
7-10 days ago?

Rom (or someone) should probably do an analysis to see what (if any) common factors there are for the errored units, and the overall frequency. Knock on wood (although with limited sampling), I have kept my run time at 8 hours and have not had any problems with 4.98.

---

Regards,
Bob P.

my first (linux box) .... stuck at 1.04%
TRUNCATE_TERMINI_FULLRELAX_1enh__433_38_0
a.

---

Just got my first stuck WU. Yay me :(

Anyway its.

https://boinc.bakerlab.org/rosetta/workunit.php?wuid=13923483

It's currently at 8+49 CPU time. Stuck at 1.042%

It has exceeded both the default run time and my run time setting.

I have suspended the WU. Bonic 5.2.13. Please advise as to what to do with this WU.

---

I have a couple of these on my Linux system. I would appreciate some help on a clean way to abort these on Linux. I have been hacking client_state.xml and deleting files in the slots directory. There has to be a better way.

Warning! PRODUCTION_ABINITIO_CENTROID_PACKING_1ctf__429_247_0 was started at 2006-04-09 20:52:34 but has not finished!

Warning! HBLR_1.0_2reb_426_994_0 was started at 2006-04-09 23:07:09 but has not finished!

Warning! 7449_largescale_large_fullatom_relax_dec7449_1_05_6.pdb_431_53_0 was started at 2006-04-09 20:58:18 but has not finished!

Warning! PRODUCTION_ABINITIO_CENTROID_PACKING_1vls__428_262_0 was started at 2006-04-09 21:19:28 but has not finished!

Warning! 7485_largescale_large_fullatom_relax_dec7485_1_05_8.pdb_432_129_0 was started at 2006-04-09 21:50:01 but has not finished!

Warning! TRUNCATE_TERMINI_FULLRELAX_1ptq__433_587_0 was started at 2006-04-11 17:55:43 but has not finished!

---

Running on Windows XP Pro, I have a WU stuck at 1.04%. The graphics appears to be locked up; nothing is moving even though the CPU utilization clock keeps ticking. The WU in questions is:

TRUNCATE_TERMINI_FULLRELAX_1ptq_433_906_0

I have suspended it after 1:34:22 of run time. The other WUs progress past that point in a few minutes.

---

Work unit aborted at 48% - CPU time used ~24 hours. Time needed to completion only going up. Nothing moved in the graphics.

WU Name "FA_RLXpt_hom003_1ptq__361_156_3" - Application "rosetta 4.98"
Workunit = 11684527; Result ID = 16802748; System = Intel P4 3.0GHz, Win-XP SP 2

The workunit still reports "in progress" at the time of writing this message.
The workunit was aborted manually ("Aborted via GUI RPC").

---

Just again had a WU that was running for more than 6 hours at 1.17% and when I checked it again another one had started which is running for 45 minutes now at 1.06% but I cannot find that other wu in my results.
Better testdrive a project like this more thoroughly before letting so many people waste their money.
If I go on this month it wil be the last anyway.
Rather fed up with it.
No fun at all anymore.

---

Hi guys... thanks very much for reporting these errors. Work units with 1b3a, 1enh, 2tif, and 1ptq appear to be wreaking havoc throughout boinc. Sorry for the trouble -- this won't happen again, as we are increasing the stringency of our local tests that precede submission to boinc. Please MANUALLY ABORT work units with 1b3a, 1enh, 2tif, or 1ptq in the title!

---

Hi,
I just aborted TRUNCATE_TERMINI_FULLRELAX_2tif_433_796_0 after 11+ hours (11:50:43) at 1.042% complete. Stage=Full Atom Relaxation, Model=1 and Step= 245292. The links are:

RESULT: https://boinc.bakerlab.org/result.php?resultid=17040622
WORKUNIT: https://boinc.bakerlab.org/workunit.php?wuid=13969890.

Unfortunately, I was unable to download the 4.98 PDB for Windows so I can't help you there but this was running under Win2K (v5.0 Build 2195, SP4) on a Pentium R4 3.20Ghz machine. As noted earlier, Rosetta was 4.98.

OOPS! Just caught the blurb about 4.83 and 4.98 being identical!

Hope this helps you find the little bugger!

---

Thanks for the post -- the percent complete is particularly interesting. The reports of 1.04%, 1.042%, 1.17% are telling us that similar work units are getting stuck at rather different points along their simulations. Its helping us focus on where to look for the bug. Please keep posting information on stuck jobs!

Hi,
I just aborted TRUNCATE_TERMINI_FULLRELAX_2tif_433_796_0 after 11+ hours (11:50:43) at 1.042% complete. Stage=Full Atom Relaxation, Model=1 and Step= 245292. The links are:

RESULT: https://boinc.bakerlab.org/result.php?resultid=17040622
WORKUNIT: https://boinc.bakerlab.org/workunit.php?wuid=13969890.

Unfortunately, I was unable to download the 4.98 PDB for Windows so I can't help you there but this was running under Win2K (v5.0 Build 2195, SP4) on a Pentium R4 3.20Ghz machine. As noted earlier, Rosetta was 4.98.

Hope this helps you find the little bugger!

---

As for the post, no problem! We are all exploring our own little corner of the universe and sharing info is just another way of getting us closer to a better understanding of the "bigger picture". Being a M/F Sys Prog makes it easier for me to understand what you guys need to find the "speed bumps". <grin>. I can even relate to the "smack" sound that will occur when you do eventually find it.

Not knowing anything about your algorithm, I have to rely on an intuitive guess as to where the problem might be. I would imagine that you are probably working on a some kind of descent down a "decision tree" and when you get to a dead end, you have to climb back up a level and pursue the next "branch". My guess is that the program is getting "into a bind" when the structure that is being analyzed is complex enough that the process "loses track" of its previous choices and gets into a loop re-analyzing the same sequence of molecules. Just my guess.

Anyways, good luck in finding it. I suspect that you are "almost there".

Thanks for the post -- the percent complete is particularly interesting. The reports of 1.04%, 1.042%, 1.17% are telling us that similar work units are getting stuck at rather different points along their simulations. Its helping us focus on where to look for the bug. Please keep posting information on stuck jobs!

Hi,
I just aborted TRUNCATE_TERMINI_FULLRELAX_2tif_433_796_0 after 11+ hours (11:50:43) at 1.042% complete. Stage=Full Atom Relaxation, Model=1 and Step= 245292. The links are:

RESULT: https://boinc.bakerlab.org/result.php?resultid=17040622
WORKUNIT: https://boinc.bakerlab.org/workunit.php?wuid=13969890.

Unfortunately, I was unable to download the 4.98 PDB for Windows so I can't help you there but this was running under Win2K (v5.0 Build 2195, SP4) on a Pentium R4 3.20Ghz machine. As noted earlier, Rosetta was 4.98.

Hope this helps you find the little bugger!

---

the only change is that we increased the default run time from 2 hours to 4 hours ...

I doubt that's the problem. I think part of the problem is that there have been some bad WUs released recently (the ones Rhiju posted about).

Another problem is that the bug requiring "keep in memory" has been fixed. That means a lot of people are setting "keep in memory" to "no". There are places in some WUs that require more than an hour to get to the next checkpoint, so with the default switching time of one hour the WU will keep dropping back to the last checkpoint indefinitly.

Found a bug! David Baker and I just tracked down the problem with these 4 workunits. Its a stupid infinite loop that only occurs with proteins with lengths of exactly 44 residues using one particular mode of Rosetta -- somehow no one in our group had ever looked at a protein exactly that size! So TallGuy-13088, you predicted right ...

Please do abort these workunits (below); otherwise, your client will continue to crunch the jobs until it times out (about 48 hours on a Windows machine). The good news is that we will give credit to all the jobs that time out, and are increasing the rigor of in-house testing to prevent this from happening in the future. And this little adventure helped us track down a pernicious bug in our code. Unfortunately, we don't yet have fixes for *all* the stuck jobs, though -- please continue to post info on other jobs that stop moving. It helps!

Jobs that should be aborted:
TRUNCATE_TERMINI_FULLRELAX_1enh__433
TRUNCATE_TERMINI_FULLRELAX_1b3aA_433
TRUNCATE_TERMINI_FULLRELAX_1ptq__433
TRUNCATE_TERMINI_FULLRELAX_2tif__433

---

About to abort WU FA_RLXpt_hom004_1ptq__361_308_2 it's stuck at 50.242% Stage full atom relax Model 9 Step 205901 its at 7 hours 43 minutes and counting. Here it is https://boinc.bakerlab.org/rosetta/workunit.php?wuid=11760498
Using Win XP home edition service pack 2 client id 62881.

---

CONGRATS! Just remember, given a choice between "lucky" and "good", ALWAYS choose LUCKY ... with enough luck, you may eventually get good <grin>!

Found a bug! David Baker and I just tracked down the problem with these 4 workunits. Its a stupid infinite loop that only occurs with proteins with lengths of exactly 44 residues using one particular mode of Rosetta -- somehow no one in our group had ever looked at a protein exactly that size! So TallGuy-13088, you predicted right ...

Please do abort these workunits (below); otherwise, your client will continue to crunch the jobs until it times out (about 48 hours on a Windows machine). The good news is that we will give credit to all the jobs that time out, and are increasing the rigor of in-house testing to prevent this from happening in the future. And this little adventure helped us track down a pernicious bug in our code. Unfortunately, we don't yet have fixes for *all* the stuck jobs, though -- please continue to post info on other jobs that stop moving. It helps!

Jobs that should be aborted:
TRUNCATE_TERMINI_FULLRELAX_1enh__433
TRUNCATE_TERMINI_FULLRELAX_1b3aA_433
TRUNCATE_TERMINI_FULLRELAX_1ptq__433
TRUNCATE_TERMINI_FULLRELAX_2tif__433

---

Here's an odd one...

Rosetta 4.98, WU 7449_largescale_large_fullatom_relax_dec7449_1_08_2.pdb_431_25_0 running with BOINC 5.2.13 on Windows XP 64-bit SP1 on an Athlon 64 3200+ with 512MB RAM. I also have SETI@home on that machine.

Starts up, 50% done, 2 hours CPU time used, runs for about an hour, at the end of that time it's still about 50% done, but has 3 hours CPU time; swaps out... SETI runs for an hour and swaps out... and then Rosetta swaps in again, 50% done, 2 hours (!) CPU time used. Caught this one because the accepted protein shape is pretty uncommon (looks sort of like a lollipop).

Shall I kill it or do you want me to keep watching it for a while? It's been on here for three days now, which means ballpark 36 hours, but I think I have only 2 hours credit for it...

---

Very good news. Keep up the good work everyone!

---

Here's an odd one...

Rosetta 4.98, WU 7449_largescale_large_fullatom_relax_dec7449_1_08_2.pdb_431_25_0 running with BOINC 5.2.13 on Windows XP 64-bit SP1 on an Athlon 64 3200+ with 512MB RAM. I also have SETI@home on that machine.

Starts up, 50% done, 2 hours CPU time used, runs for about an hour, at the end of that time it's still about 50% done, but has 3 hours CPU time; swaps out... SETI runs for an hour and swaps out... and then Rosetta swaps in again, 50% done, 2 hours (!) CPU time used. Caught this one because the accepted protein shape is pretty uncommon (looks sort of like a lollipop).

Shall I kill it or do you want me to keep watching it for a while? It's been on here for three days now, which means ballpark 36 hours, but I think I have only 2 hours credit for it...

cwangersky, these are very big WUs which take a loooong time per model, on some P4s they might even take more than 2hr PER MODEL, so unless you have "Leave in mem when pre-empted"=YES, the PC can't complete even 1 model in 2hr before Rosetta gets swapped out to run SETI and your PC starts the WU from 0 again...

Solution: increase "time between swaps" to e.g. 4hr or (if your PC has lots of RAM and/or run few BOINC projects) set "leave in mem when preempted"=YES I would choose the latter.

This very example is why Rosetta needs a BigWU flag in preferences IMHO...

AMD's explained it in a previous comment:

Another problem is that the bug requiring "keep in memory" has been fixed. That means a lot of people are setting "keep in memory" to "no". There are places in some WUs that require more than an hour to get to the next checkpoint, so with the default switching time of one hour the WU will keep dropping back to the last checkpoint indefinitly.

---

Best UFO Resources
Wikipedia R@h
How-To: Join Distributed Computing projects that benefit humanity

aborted wu

After over 9.5 hours this one was still at 1.04% and showing 16 more hours to go. I manually aborted this unit.

Result ID 16987331
Name TRUNCATE_TERMINI_FULLRELAX_1b3aA_433_297_0
Workunit 13923431

---

Glad to see the Truncate_Termini units were cancelled. I just noticed one of my machines was working on one of those units for a lil more than 29 hours. https://boinc.bakerlab.org/rosetta/workunit.php?wuid=13918811

---