Had a WU taking ~70 hours which has not errored out due to long processing
https://boinc.bakerlab.org/rosetta/workunit.php?wuid=8363123

---

This WU took 165 hours before it finally decided that it was running for too long...

This is not a max time error. This is a 1% hang issue. so I have moved the post to the proper thread. While the WU did finally fail, the cause was the system aborting it after 165 hours. If I am not mistaken Dr. Kim has said he will grant the credit in this case.

---

Moderator9
ROSETTA@home FAQ
Moderator Contact

Could someone please explain the difference between "max cpu time exceeded" and otherwise "hung/stuck WUs" to me?

I mean, let's say I have a WU "stuck", loaded in memory but somehow not actually running -shown with "top" command and "ps" shows them as "SN"=stopped,nice- (I've had a few such situations under Linux).

If user doesn't intervene to "kill" the stuck Rosetta task manually (so BOINC re-runs the same WU with only diff the random seed, apparently), would it abort on its own after X days have passed?

In short my question is: do the "Max CPU time exceeded" WUs actually consume 100% CPU cycles during the X days they kept "running" until they reached their TTL?. Or could it be just "stuck" WUs which simply hit their TTL?

PS: I'm thouroughly confused about the definitions of the various issues (bugs) we're trying to track and I read the R@H forums everyday for the last month.

---

Best UFO Resources
Wikipedia R@h
How-To: Join Distributed Computing projects that benefit humanity

Could someone please explain the difference between "max cpu time exceeded" and otherwise "hung/stuck WUs" to me?

I mean, let's say I have a WU "stuck", loaded in memory but somehow not actually running -shown with "top" command and "ps" shows them as "SN"=stopped,nice- (I've had a few such situations under Linux).

If user doesn't intervene to "kill" the stuck Rosetta task manually (so BOINC re-runs the same WU with only diff the random seed, apparently), would it abort on its own after X days have passed?

In short my question is: do the "Max CPU time exceeded" WUs actually consume 100% CPU cycles during the X days they kept "running" until they reached their TTL?. Or could it be just "stuck" WUs which simply hit their TTL?

PS: I'm thouroughly confused about the definitions of the various issues (bugs) we're trying to track and I read the R@H forums everyday for the last month.

Max time wu run normally and only fail as result of hitting the maximiun time alloted for them to run when the project sent them out.

Hung work units run but the progress never increases. Usually they stick at 1% complete but it can happen anywhere. While they may fail usually they are aborted or restarted by the user.

---

Moderator9
ROSETTA@home FAQ
Moderator Contact

I've had a few of these on different machines:

2/19/2006 10:16:41 PM|rosetta@home|Aborting result PRODUCTION_ABINITIO_1gvp__250_35_2: exceeded CPU time limit 50195.312500
2/19/2006 10:16:41 PM|rosetta@home|Unrecoverable error for result PRODUCTION_ABINITIO_1gvp__250_35_2 (Maximum CPU time exceeded)
2/19/2006 10:16:42 PM||request_reschedule_cpus: process exited

Is there I can do to prevent this from occuring?

---

Join the Teddies@WCG

Owlie forgot to mention that the ones above were also 4.82 with the CPU time set to 4 days......

Owlie forgot to mention that the ones above were also 4.82 with the CPU time set to 4 days......

Thanks Scribbles... I set half my machines to 2 days and left the others at 4...

I got one here, result:
http://www.boinc.bakerlab.org/rosetta/result.php?resultid=11775907
WU (4.82):
http://www.boinc.bakerlab.org/rosetta/workunit.php?wuid=6142533
client:
http://www.boinc.bakerlab.org/rosetta/show_host_detail.php?hostid=128060
error:
<core_client_version>5.2.13</core_client_version>
<message>Maximum CPU time exceeded</message>
<stderr_txt>
# random seed: 216581
# cpu_run_time_pref: 28800
</stderr_txt>

MrS

---

Scanning for our furry friends since Jan 2002

One more time :
NO_SIM_ANNEAL_BARCODE_30_1n0u_251_13883_1
NO_SIM_ANNEAL_BARCODE_30_1dtj_251_13882_1
NO_SIM_ANNEAL_BARCODE_30_1n0u_251_13879_1
PRODUCTION_ABINITIO_2chf__250_2426_1
PRODUCTION_ABINITIO_2vik__250_1994_1
PRODUCTION_ABINITIO_2chf__250_2400_1
PRODUCTION_ABINITIO_1dhn__250_911_0


This 7 wu's with an exit code "-177"

---

Thanks much for reporting these CPU time errors. It looks like
we were able to largely solve the problem in the jobs submitted
after January. We reduced the number of structures per work unit and
extending the max CPU time; none of these later jobs appear to
have given the error.

We're now setting up jobs for the updated application. As David
will explain in a note soon, we're now tapping the BOINC resources
to unleash the powerful information available in sequence "homologues"
(sequences related to the target protein and thus expected
to have nearly the same fold). Very exciting!

These next jobs should hopefully be even less likely to
trigger the max CPU time error.
We now are allowing you to set the maximum time you want
your computer to crunch (default 8 hours) before
returning us structures, rather than asking for a specific
number of structures back. So far seems to have worked on the
test server -- please do report any further Max CPU time errors
here!

---

Whoa now, what is this???

I set my cpu time for 24 hours and I get a max cpu time exceeded after 10 hours.

Here is the WU, and here is the pertinent info:

CPU time 36185.368987

stderr out

<core_client_version>5.2.14</core_client_version>
<message>Maximum CPU time exceeded
</message>
<stderr_txt>
# random seed: 910501
# cpu_run_time_pref: 86400

</stderr_txt>

Validate state Invalid
Claimed credit 101.245443882576
Granted credit 0
application version 4.81

---

I have had two 'max cpu time exceeded' errors reported since upgrading to 4.82. It seems to have been caused by setting my 'target cpu run time' to 4hrs whilst these two WUs were already at 6+ hours of progress, or at least they both errored out shortly after I changed that value.

These are the WUs in question:

https://boinc.bakerlab.org/rosetta/result.php?resultid=11797455
https://boinc.bakerlab.org/rosetta/result.php?resultid=11796520

---

I have had two 'max cpu time exceeded' errors reported since upgrading to 4.82. It seems to have been caused by setting my 'target cpu run time' to 4hrs whilst these two WUs were already at 6+ hours of progress, or at least they both errored out shortly after I changed that value.

These are the WUs in question:

https://boinc.bakerlab.org/rosetta/result.php?resultid=11797455
https://boinc.bakerlab.org/rosetta/result.php?resultid=11796520

I suspect this is a WU related issue. It is possible that the bounds limit has not been set right for these to accommodate the new time settings. I will bring it to the attention of the project team.

---

Moderator9
ROSETTA@home FAQ
Moderator Contact

Just as I suspected these latest Max time errors are a WU related issue please see this from David Kim on this subject -

"The max time errors are due to an older batch of work units. I cancelled that batch and also updated all the rsc_fpops_bound values to a fairly high value so as to not reach the limit in 4 days.

It is difficult though to guarantee not reaching the limit since it also depends on the clients benchmark...

...In the future we will try to prevent sending out the work units that take a long time to produce a single model. The previous batches of ab initio runs have a filter being used that actually ignores structures that do not fit the filtering criteria, thus for some proteins many structures are being modeled before reaching one that passes the filters. We are going to turn the filters off for future batches and filter them ourselves as a post process.

Thanks,

David K

So While there may be a very few more of these that come out of longer queues, for the most part these Max time errors should now stop very soon.

If you see any please keep reporting them here on this thread.

---

Moderator9
ROSETTA@home FAQ
Moderator Contact

Already any progress in granting credits for MCTE WU's ??
Or did I miss it somewhere ?

---

Already any progress in granting credits for MCTE WU's ??
Or did I miss it somewhere ?

As was reported before, it will be AT LEAST mid-March before the project team can deal with the credit granting process for this class of WU failures, and maybe longer. They did say they would grant the credit in due course, but they are focused on fixing run time errors at this time. The cause of the Max time errors has been isolated and fixed so people should not see any more of them. But the credit granting process takes time.

---

Moderator9
ROSETTA@home FAQ
Moderator Contact

Already any progress in granting credits for MCTE WU's ??
Or did I miss it somewhere ?

As was reported before, it will be AT LEAST mid-March before the project team can deal with the credit granting process for this class of WU failures, and maybe longer. They did say they would grant the credit in due course, but they are focused on fixing run time errors at this time. The cause of the Max time errors has been isolated and fixed so people should not see any more of them. But the credit granting process takes time.

Thanks. I'm not visiting these forums on a regularly base, so I must've missed.

---