Isn't the biggest failure mode still the "stuck at 1%" issue? Do those even get reported as errors, outside of the forum? All the ones I have had have neede to be aborted, after failing to run. Doesn't that just show as "aborted by user", effectively hiding the scope of the problem?

Actually the “%1 bug” only accounts for roughly 5% of the overall failure cases reported per day. It is by far the biggest failure case from the community perspective though as it requires manual intervention.

0xC0000005 and the ERR_NESTED_UNHANDLED_EXCEPTION_DETECTED errors together were accounting for 60% of the reported errors per day. I tackled these first as they seemed at the time manifestations of the same fundamental problem and they accounted for the biggest piece of the pie.

The next biggest heavy hitter is exit code 1; this is a program defined error. This just required that the project change its error logging from stdout to stderr so that it’ll show up in the result log reported back to the server. That work item will be finished in the next few days.

Next after that one is 0xC000000D, which seems to have a reoccurring theme that stackwalker failed to initialize during a stack dump. I’ve added some extra messages to the BOINC API to try and track this one down.

Now we get to the ERR_ABORTED_VIA_GUI error; this 1% error is really nasty. Unfortunately the pdb file was not deployed with the 4.82 release so trying to get stack traces from the community while it is stuck in the loop it is in isn’t really doable. I have started the investigation with members of the Ralph community to try and track this down since they have access to the pdb file for 4.93. You can track the progress being made here.

I hope this clears up some stuff for the community.

---

----- Rom
My Blog

Actually the “%1 bug” only accounts for roughly 5% of the overall failure cases reported per day.
==========

Like you said whats reported Rom, if a lot of people are like me we quit reporting them a long time ago. I didn't see any point in reporting them any more because it's the same thing over & over. I know I aborted at least 5 or 6 Stuck 1% WU's today alone & it's like that ever day ... :/

That is to say, what is reported to the server. When somebody aborts a workunit, it gets reported to the server as ERR_ABORTED_VIA_GUI.

If the workunit eventually exceeds its allocated CPU time it is reported as ERR_RSC_LIMIT_EXCEEDED.

So unless you are resetting the project everytime, I get to see it. :)

---

----- Rom
My Blog

---

If you've got a system that consistently has problems with being stuck at 1% then please join Ralph and help them identify the cause.
==========

Good Idea, I'll do that as soon as I get some free time ... ;)

This is the third stuck at 1% WU in two days(that I know of, I happened to be on that Machine ATM) that I've aborted. It only shows 10 hours but BOINC showed 59 hours...

stderr out <core_client_version>5.2.13</core_client_version>
<message>aborted via GUI RPC
</message>
<stderr_txt>
# random seed: 2739161
# cpu_run_time_pref: 36000
# cpu_run_time_pref: 36000
# random seed: 2739161

</stderr_txt>

---

Join the Teddies@WCG

umm, how bout this one, FA_RLXbq_hom019_1bq9A_359_191_0 running 11 hours, 45.13% done, time to complete is running backwards, now 6:39 hours.
also im getting many computation errors on this system

---

---

Just aborted 4 more. Really hope this gets fixed soon, we've just wasted over 5 days of CPU time! Good luck Rom.

8.8 hours
https://boinc.bakerlab.org/rosetta/workunit.php?wuid=11584596

18.2 hours
https://boinc.bakerlab.org/rosetta/workunit.php?wuid=11551106

41.4 hours
https://boinc.bakerlab.org/rosetta/workunit.php?wuid=11460182

71.0 hours
https://boinc.bakerlab.org/rosetta/workunit.php?wuid=11330309

---

And another.

https://boinc.bakerlab.org/rosetta/workunit.php?wuid=11627337

This and the 4 I mentioned below were all stuck on 1%.

---

Look at the errors on this one

https://boinc.bakerlab.org/rosetta/result.php?resultid=13858838

Someone may want to take a look at the results on this one as well, there's plenty of them and it isn't the 1% "stuck" problem.

https://boinc.bakerlab.org/rosetta/results.php?hostid=181476

I've stopped Rosetta on this machine, as it would run through a ton of jobs and client error them until it gave the message "daily quota met".

Harder

---

I don't know exactly what is going on. for each work unit, we have now close to the targeted 10,000 successful completions, so there are clearly no systematic errors affecting all instantces of a wu. I would love to know how many failures of the sort you had there have been. It is possible that for certain random number seeds very rare rosetta bugs are encountered--this would have to be at less than 1 in 100 since we don't see them in our in house tests. so question: what fraction of your WU have this problem?

we can search for rosetta bugs by starting runs in house with the random number seed and command line from your run. we are doing this now

David,

Many moons ago, in a different thread, you gave instructions for manually restarting a "1% stuck" WU. I've got on one one of my systems, do you want me to restart it, and is there anything else that I can do to help identify the problem? Things like taking a snapshot of the rosetta and slots/0 folders, zipping it up and making it available for you to download, or anything else that might help.

---

HB_BARCODE_30_1bk2__351_7729_0 is stuck! After 5 hours of crunching still at 1% :(

---

These HB_BARCODE_30 were stuck in a slot for ~24 hours without progressing. Some had ~25min CPU time; at least one had ~55min CPU.

It's really annoying that I lost around 4-cpu-days of work because of these four.

dag

https://boinc.bakerlab.org/rosetta/result.php?resultid=14185477
https://boinc.bakerlab.org/rosetta/result.php?resultid=14184543
https://boinc.bakerlab.org/rosetta/result.php?resultid=14100307
https://boinc.bakerlab.org/rosetta/result.php?resultid=14099752

---

dag
--Finding aliens is cool, but understanding the structure of proteins is useful.

https://boinc.bakerlab.org/rosetta/result.php?resultid=14068243

Rather strange : I did not, repeat, did not abort it myself.
Didn't touch the machine, it runs on it own.
Had more of these and still don't know what happens.

---

Another one, aborted after 16 hours stuck on 1%

https://boinc.bakerlab.org/rosetta/workunit.php?wuid=11717839

This one was a bit different - it was stuck on 30.19% after 8 hours. After restarting BOINC, it reset back to 38 mins CPU time and 30.19% and got stuck again.

https://boinc.bakerlab.org/rosetta/workunit.php?wuid=11743501

It's getting increasingly frustrating having to babysit this project all the time. Fingers crossed for those working on a fix.

---

I'm experiencing a lot of stuck WU's with FA_RLX****
I'm now at the point, if a WU is at 1 percent after 1 hour, i'm manually aborting it... i want credit for my cpu time :(

---

This unit stuck for 9 hours:

FA_RLXpt_hom003_1ptq__361_234

Brought up the graphics screen (I dont run any graphics) and it was all froze except for the cpu clock was still counting.

Resetting boinc did no good. Ended up aborting it.

out of 183 results i have 4 errors of the frozen or 1 to 15 percent type.

Cheers all!!!

---

this one stuck at one percent.

HBLR_1.0_1dtj_332_2576

https://boinc.bakerlab.org/rosetta/workunit.php?wuid=9848871

had it crunching for about 350000sek/1500creds :( Aborted it. Seems someone did crunch it eventually. No luck for me tho.
/anton

---