I'm getting a 1% freeze at step 22183 on this WU:

FA_RLXbq_hom030_1bq9A_359_467_0
https://boinc.bakerlab.org/rosetta/result.php?resultid=13795176

Stopping and re-starting BOINC will cause the WU to begin again from step 0 but it always halts at the same place. Aborting unit.

---

This project used to be bullet-proof - what's changed....?

believe me, we are trying!

GREAT NEWS - Perhaps it might be an idea to let people know there is a problem and to stop making work available until it's fixed - that'll take the pressure off you guys.

What sort of percentage of the work returned to you is being trashed by this bug?

Would imagine it's fairly high - although, if it was an epidemic failure, would assume you would have stopped sending out work before now.





The error rate is still very machine specific--many machines appear to have virtually no errors. Aside from a few faulty work units, which we've stopped, we don't think there are new errors or bugs--just the same old problems.

The exciting news is that the Boinc consultant we have hired, Rom, has made an improvement in how
the rosetta process terminates that seems to really have made a difference on Ralph. the problem seems to have been not any bug in the rosetta code, but a problem in how the rosetta process shuts itself down when the processor starts doing something else (hence the leave in memory bug, etc.).

here is his latest email:

Public Project Failure Rate by Type (Top 3):
-164 (0xffffff5c) ERR_NESTED_UNHANDLED_EXCEPTION_DETECTED
4052
39.25%
-1073741819 (0xc0000005) Unknown error number
2293
22.21%
1 Unknown error number
1284
12.44%

Ralph Failure Rate By Type (Top 3):
-186 (0xffffffffffffff46) ERR_RESULT_DOWNLOAD
35
31.25%
1 Unknown error number
20
17.86%
-529697949 (0xffffffffe06d7363) Unknown error number
18
16.07%

The percentages above signify the percentage of times that exit code was given relative to the total number of non-zero exit codes given.

As you can see, the 0xc0000005Ís and the NESTED_UNHANDLED_EXCEPTIONS have dropped off of the top three biggest offenders.



I believe we are making progress.

----- Rom








But surely you must be going into damage limitation mode by now. Can you afford to lose lots of crunchers?
NO! that is why we are investing everything in fixing the problems now


regards and good luck.

Tim

---

3/4/2006 6:51:19 AM|rosetta@home|Starting result ABINITli_hom010_1lis__322_14_1 using rosetta version 482


this job has been running for over 10 12 days now.........it's been on 84.89% for at least 24 hrs now.......maybe a lot longer ?? its still using cpu power so i assume it's doing something ?? runtime is listed as 256 12 hours so far...and still counting up

any advice as to what to do with it would be welcome.

restarted boink .............unit reset to 13 hours then stuck again for several hours at the same point..............aborted now..............imho not worth sending to anyone else

---

46 years dc so far

join team FaDbeens
join us

---

Hello,

I just aborted WU: FA_RLXac_hom017_1acf_359_113_0 after two BOINC Mgr. restarts and about 50+ (combined) hours of processing. All three attempts stalled at STEP: 23171. The last ACCEPTED RMSD was 14.75 and the last ACCEPTED ENERGY was 15.47414. Each attempt showed activity for about 1 minute and hung at 1% complete with the ELAPSED TIME continuing to increment. Rosetta version was 4.82 under Win2K. Just an FYI in case someone wants to debug the workunit.

---

The exciting news is that the Boinc consultant we have hired, Rom, has made an improvement in how
the rosetta process terminates that seems to really have made a difference on Ralph. the problem seems to have been not any bug in the rosetta code, but a problem in how the [b}rosetta process shuts itself down when the processor starts doing something else[/b] (hence the leave in memory bug, etc.).

Hi David,

Well that ties in with an observation I can make, which I've seen once or twice.

I have noticed that a Rosetta work units "fail" when my 3GHz P4/HT switches from one project to another - so there seem to issues when the Rosetta process seems to be "suspended" by BOINC as it then switches over to another project - (I'm running BBC CCE as a second simultaneous BOINC project on the same PC - this "switch over problem" tends to occur when one "CPU" switches out of working on a Rosetta WU and then switches over to the CCE WU).

Maybe this is a help - but seems Rom is on the right trail.


regards

Tim

(edit) typo

So the top 3 errors on Rosetta aren't the top 3 errors on Ralph. Great news. How far down on the Rosetta list are the top 3 errors that Ralph is having?

The ERR_NESTED_UNHANDLED_EXCEPTION_DETECTED no longer appears on the list at all, and the 0xC0000005 only accounts for 6 of the 49 errors reported in the last 24 hours.

If the data of Ralph is any indication about how the application is going to behave on the public project it should result in a 60%-70% in error rate for the public project.

----- Rom

---

----- Rom
My Blog

Isn't the biggest failure mode still the "stuck at 1%" issue? Do those even get reported as errors, outside of the forum? All the ones I have had have neede to be aborted, after failing to run. Doesn't that just show as "aborted by user", effectively hiding the scope of the problem?

---

Proudly Banned from Predictator@Home and now Cosmology@home as well. Added SETI to the list today. Temporary ban only - so need to work harder :)



"You can't fix stupid" (Ron White)

Isn't the biggest failure mode still the "stuck at 1%" issue? Do those even get reported as errors, outside of the forum? All the ones I have had have neede to be aborted, after failing to run. Doesn't that just show as "aborted by user", effectively hiding the scope of the problem?

Actually the “%1 bug” only accounts for roughly 5% of the overall failure cases reported per day. It is by far the biggest failure case from the community perspective though as it requires manual intervention.

0xC0000005 and the ERR_NESTED_UNHANDLED_EXCEPTION_DETECTED errors together were accounting for 60% of the reported errors per day. I tackled these first as they seemed at the time manifestations of the same fundamental problem and they accounted for the biggest piece of the pie.

The next biggest heavy hitter is exit code 1; this is a program defined error. This just required that the project change its error logging from stdout to stderr so that it’ll show up in the result log reported back to the server. That work item will be finished in the next few days.

Next after that one is 0xC000000D, which seems to have a reoccurring theme that stackwalker failed to initialize during a stack dump. I’ve added some extra messages to the BOINC API to try and track this one down.

Now we get to the ERR_ABORTED_VIA_GUI error; this 1% error is really nasty. Unfortunately the pdb file was not deployed with the 4.82 release so trying to get stack traces from the community while it is stuck in the loop it is in isn’t really doable. I have started the investigation with members of the Ralph community to try and track this down since they have access to the pdb file for 4.93. You can track the progress being made here.

I hope this clears up some stuff for the community.

---

----- Rom
My Blog

Actually the “%1 bug” only accounts for roughly 5% of the overall failure cases reported per day.
==========

Like you said whats reported Rom, if a lot of people are like me we quit reporting them a long time ago. I didn't see any point in reporting them any more because it's the same thing over & over. I know I aborted at least 5 or 6 Stuck 1% WU's today alone & it's like that ever day ... :/

That is to say, what is reported to the server. When somebody aborts a workunit, it gets reported to the server as ERR_ABORTED_VIA_GUI.

If the workunit eventually exceeds its allocated CPU time it is reported as ERR_RSC_LIMIT_EXCEEDED.

So unless you are resetting the project everytime, I get to see it. :)

---

----- Rom
My Blog

---

If you've got a system that consistently has problems with being stuck at 1% then please join Ralph and help them identify the cause.
==========

Good Idea, I'll do that as soon as I get some free time ... ;)

This is the third stuck at 1% WU in two days(that I know of, I happened to be on that Machine ATM) that I've aborted. It only shows 10 hours but BOINC showed 59 hours...

stderr out <core_client_version>5.2.13</core_client_version>
<message>aborted via GUI RPC
</message>
<stderr_txt>
# random seed: 2739161
# cpu_run_time_pref: 36000
# cpu_run_time_pref: 36000
# random seed: 2739161

</stderr_txt>

---

Join the Teddies@WCG

umm, how bout this one, FA_RLXbq_hom019_1bq9A_359_191_0 running 11 hours, 45.13% done, time to complete is running backwards, now 6:39 hours.
also im getting many computation errors on this system

---

---

For those who may be interested, Rom has posted information about Rosetta Work Unit errors and the status of the ongoing work to fix the bugs in Rosetta on his "Blog".

---

Moderator9
ROSETTA@home FAQ
Moderator Contact

Just aborted 4 more. Really hope this gets fixed soon, we've just wasted over 5 days of CPU time! Good luck Rom.

8.8 hours
https://boinc.bakerlab.org/rosetta/workunit.php?wuid=11584596

18.2 hours
https://boinc.bakerlab.org/rosetta/workunit.php?wuid=11551106

41.4 hours
https://boinc.bakerlab.org/rosetta/workunit.php?wuid=11460182

71.0 hours
https://boinc.bakerlab.org/rosetta/workunit.php?wuid=11330309

---

And another.

https://boinc.bakerlab.org/rosetta/workunit.php?wuid=11627337

This and the 4 I mentioned below were all stuck on 1%.

---

Look at the errors on this one

https://boinc.bakerlab.org/rosetta/result.php?resultid=13858838