Not getting any python work

Message boards : Number crunching : Not getting any python work

To post messages, you must log in.

1 · 2 · 3 · 4 . . . 9 · Next

AuthorMessage
Profile Greg_BE
Avatar

Send message
Joined: 30 May 06
Posts: 5652
Credit: 5,622,096
RAC: 58
Message 102927 - Posted: 12 Oct 2021, 8:00:27 UTC

I find this odd, 9,000+ pythons and 24,000+ 4.2 tasks, yet all I get is 4.2
I got python when it first started.

I reset the project, but still no python.

Any ideas as to why the drought on python now, when I had them in the past?
Did something change in the requirements?
ID: 102927 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1669
Credit: 6,555,854
RAC: 2,999
Message 102928 - Posted: 12 Oct 2021, 8:16:01 UTC - in response to Message 102927.  

Any ideas as to why the drought on python now, when I had them in the past?
Did something change in the requirements?


Is virtualization enabled in bios?
What's the log of boinc Manager? There is something like:
 No WSL found.
VirtualBox version: 6.1.26


P.S.
I have the same problem on my notebook and i cannot find the solution
ID: 102928 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
mikey
Avatar

Send message
Joined: 5 Jan 06
Posts: 1886
Credit: 6,120,065
RAC: 19
Message 102929 - Posted: 12 Oct 2021, 11:44:52 UTC - in response to Message 102927.  

I find this odd, 9,000+ pythons and 24,000+ 4.2 tasks, yet all I get is 4.2
I got python when it first started.

I reset the project, but still no python.

Any ideas as to why the drought on python now, when I had them in the past?
Did something change in the requirements?


The only thing that worked for me was to keep aborting all the regular Rosetta tasks and finally Rosetta would start sending me some Python tasks, I now have over half a dozen on each of my pc's and no regular Rosetta tasks.

DAMN I wish they would at least TRY a preferences switch and see what happens, they can always turn it back off next week!!!!
ID: 102929 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile dcdc

Send message
Joined: 3 Nov 05
Posts: 1828
Credit: 107,716,341
RAC: 9,980
Message 102931 - Posted: 12 Oct 2021, 12:13:40 UTC - in response to Message 102929.  
Last modified: 12 Oct 2021, 12:13:51 UTC


DAMN I wish they would at least TRY a preferences switch and see what happens, they can always turn it back off next week!!!!


It might be a lot of work to implement.
ID: 102931 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1669
Credit: 6,555,854
RAC: 2,999
Message 102932 - Posted: 12 Oct 2021, 14:31:26 UTC - in response to Message 102931.  


DAMN I wish they would at least TRY a preferences switch and see what happens, they can always turn it back off next week!!!!


It might be a lot of work to implement.


Mmmm, no.
It's very easy on boinc server to implement 2 different apps
ID: 102932 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Greg_BE
Avatar

Send message
Joined: 30 May 06
Posts: 5652
Credit: 5,622,096
RAC: 58
Message 102934 - Posted: 12 Oct 2021, 18:22:36 UTC - in response to Message 102928.  

Any ideas as to why the drought on python now, when I had them in the past?
Did something change in the requirements?


Is virtualization enabled in bios?
What's the log of boinc Manager? There is something like:
 No WSL found.
VirtualBox version: 6.1.26


P.S.
I have the same problem on my notebook and i cannot find the solution


I run LHC ATLAS. That is a Vbox project.
ID: 102934 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Greg_BE
Avatar

Send message
Joined: 30 May 06
Posts: 5652
Credit: 5,622,096
RAC: 58
Message 102935 - Posted: 12 Oct 2021, 18:29:18 UTC - in response to Message 102932.  


DAMN I wish they would at least TRY a preferences switch and see what happens, they can always turn it back off next week!!!!


It might be a lot of work to implement.


Mmmm, no.
It's very easy on boinc server to implement 2 different apps


Because Python is new, they haven't bothered updating their webpage to reflect that as a soul project choice or a opt out choice. They are always a bit slow to change anything.
I think they figure we will find a way to opt out or to do like you do and force the server to send other work via aborts and such. We always find a way to get our machines to do what they need to do with little or no input from the Baker Lab.

And yes it is easy to add the option to add or subtract or run solo the Python work. WCG runs how many projects under it's server? 5 or more? And you can opt in or out of all of them, plus the usual CPU/GPU preferences. Same with Einstein, you can opt in or out of projects and CPU/GPU.

Not sure what Baker Lab's excuse is, other than they don't want to change anything.
ID: 102935 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jim1348

Send message
Joined: 19 Jan 06
Posts: 881
Credit: 52,257,545
RAC: 1,128
Message 102936 - Posted: 12 Oct 2021, 19:15:56 UTC - in response to Message 102935.  

My only real problem with the pythons is the "virtualbox unmanageable" error that frequently occurs.

I don't think we can do anything about it on our end. They have to fix it on their end.
There was a post somewhere that it is due to changes made by Oracle on virtualbox, and the project needs a new wrapper. That is not our thing.
ID: 102936 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
mikey
Avatar

Send message
Joined: 5 Jan 06
Posts: 1886
Credit: 6,120,065
RAC: 19
Message 102937 - Posted: 12 Oct 2021, 23:13:57 UTC - in response to Message 102935.  


DAMN I wish they would at least TRY a preferences switch and see what happens, they can always turn it back off next week!!!!


It might be a lot of work to implement.


Mmmm, no.
It's very easy on boinc server to implement 2 different apps


Because Python is new, they haven't bothered updating their webpage to reflect that as a soul project choice or a opt out choice. They are always a bit slow to change anything.
I think they figure we will find a way to opt out or to do like you do and force the server to send other work via aborts and such. We always find a way to get our machines to do what they need to do with little or no input from the Baker Lab.

And yes it is easy to add the option to add or subtract or run solo the Python work. WCG runs how many projects under it's server? 5 or more? And you can opt in or out of all of them, plus the usual CPU/GPU preferences. Same with Einstein, you can opt in or out of projects and CPU/GPU.

Not sure what Baker Lab's excuse is, other than they don't want to change anything.


And PrimeGrid runs over a dozen different sub-projects all at the same time with no problems, they manage their priorities thru task availability, ie a push over there and they wind up the Server to produce alot of those and fewer of the others kinds. They also use Challenges to get users to hyper focus on a particular type of task for x amount of time.
ID: 102937 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
mikey
Avatar

Send message
Joined: 5 Jan 06
Posts: 1886
Credit: 6,120,065
RAC: 19
Message 102938 - Posted: 12 Oct 2021, 23:15:52 UTC - in response to Message 102936.  

My only real problem with the pythons is the "virtualbox unmanageable" error that frequently occurs.

I don't think we can do anything about it on our end. They have to fix it on their end.
There was a post somewhere that it is due to changes made by Oracle on virtualbox, and the project needs a new wrapper. That is not our thing.


Are you giving it at least 8gb of ram? If not it may crash because the tasks are STILL asking for that much ram but only using about 60meg per task on my Windows11 laptop when actually running.
ID: 102938 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jim1348

Send message
Joined: 19 Jan 06
Posts: 881
Credit: 52,257,545
RAC: 1,128
Message 102939 - Posted: 13 Oct 2021, 0:03:22 UTC - in response to Message 102938.  
Last modified: 13 Oct 2021, 0:09:27 UTC

Are you giving it at least 8gb of ram? If not it may crash because the tasks are STILL asking for that much ram but only using about 60meg per task on my Windows11 laptop when actually running.
I have 64 GB on a Ryzen 3900X, and they stop anyway.

That happens on almost all VirtualBox projects. Only LHC does it right, because they use an updated wrapper. I don't know if it was from Oracle, or their own thing.
They mentioned it on some forum a while ago. If you search for "virtualbox unmanageable", you might find it. It was a comment directed to Dave Anderson no less, but I don't know if he saw it.
I think he mentioned that it may take an update to BOINC to fix the other projects.

PS - Are you saying that you don't have the problem?
ID: 102939 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
mikey
Avatar

Send message
Joined: 5 Jan 06
Posts: 1886
Credit: 6,120,065
RAC: 19
Message 102942 - Posted: 13 Oct 2021, 12:04:53 UTC - in response to Message 102939.  

Are you giving it at least 8gb of ram? If not it may crash because the tasks are STILL asking for that much ram but only using about 60meg per task on my Windows11 laptop when actually running.
I have 64 GB on a Ryzen 3900X, and they stop anyway.

That happens on almost all VirtualBox projects. Only LHC does it right, because they use an updated wrapper. I don't know if it was from Oracle, or their own thing.
They mentioned it on some forum a while ago. If you search for "virtualbox unmanageable", you might find it. It was a comment directed to Dave Anderson no less, but I don't know if he saw it.
I think he mentioned that it may take an update to BOINC to fix the other projects.

PS - Are you saying that you don't have the problem?


I do not have the problem on my AMD Ryzen 7 4800 model laptop with 16gb of ram but am only running 1 task at a time. I AM having the problem you mentioned on my MacBook Pro with 16gb of ram but it could be trying to run multiple units at once so for now it's on another project. My MacPro desktop with 24gb of ram running 2 tasks at once seems to be doing just fine as well. I do not set any variables for Virtual Box when I install it on my computers, I just leave it set at the defaults and Boinc handles the rest.
ID: 102942 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jim1348

Send message
Joined: 19 Jan 06
Posts: 881
Credit: 52,257,545
RAC: 1,128
Message 102943 - Posted: 13 Oct 2021, 12:54:17 UTC - in response to Message 102942.  

OK, that is useful. It may happen only when running multiple work units (or at least more than two).
In that case, smaller memory may be better. You can't use an app_config to limit the number of work units until they get the download bug fixed.

I can run Rosetta in a second BOINC instance and limit it to one or two work units at a time, but that affects both the pythons and the non-pythons.
They need to give us some way to select them.

Thanks.
ID: 102943 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Bryn Mawr

Send message
Joined: 26 Dec 18
Posts: 333
Credit: 9,389,292
RAC: 2,285
Message 102944 - Posted: 13 Oct 2021, 15:05:04 UTC - in response to Message 102943.  

OK, that is useful. It may happen only when running multiple work units (or at least more than two).
In that case, smaller memory may be better. You can't use an app_config to limit the number of work units until they get the download bug fixed.

I can run Rosetta in a second BOINC instance and limit it to one or two work units at a time, but that affects both the pythons and the non-pythons.
They need to give us some way to select them.

Thanks.


You say that you *can’t* use app_config to limit the number of work units but I’ve been doing so for a couple of years with zero problems.

Each of my projects has :-

<app_config>
<project_max_concurrent>N</project_max_concurrent>
</app_config>


and it limits the processing as required with no runaway downloads.
ID: 102944 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jim1348

Send message
Joined: 19 Jan 06
Posts: 881
Credit: 52,257,545
RAC: 1,128
Message 102945 - Posted: 13 Oct 2021, 16:14:16 UTC - in response to Message 102944.  
Last modified: 13 Oct 2021, 16:20:19 UTC

You say that you *can’t* use app_config to limit the number of work units but I’ve been doing so for a couple of years with zero problems.

Each of my projects has :-

<app_config>
<project_max_concurrent>N</project_max_concurrent>
</app_config>

and it limits the processing as required with no runaway downloads.

Have you tried that on Rosetta with multiple pythons running at once, and with the regular Rosettas running?

To limit the pythons, you would have to use

<app>
<name>rosetta_python_projects</name>
<max_concurrent>X</max_concurrent>
</app>

Then, along with the regular Rosettas, it gets confused. I have seen it on several other projects, or combinations of projects, where different types of work units are present.
It might work on single types of work units, but you don't need it so much there, since you can just limit the number of cores running.

And it is rather sporadic. You may not have encountered it yet, but that is no guarantee of the future.
It did not happen in the older BOINC versions either, but it started a couple of years ago for me.

You can read about it on the LHC forum, and the cited bug report:
https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5726
https://github.com/BOINC/boinc/issues/4322
ID: 102945 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Grant (SSSF)

Send message
Joined: 28 Mar 20
Posts: 1380
Credit: 13,693,695
RAC: 136
Message 102947 - Posted: 14 Oct 2021, 5:38:44 UTC - in response to Message 102944.  
Last modified: 14 Oct 2021, 5:39:12 UTC

You say that you *can’t* use app_config to limit the number of work units but I’ve been doing so for a couple of years with zero problems.
If you check out several of the other threads that Greg_BE has posted to, you would see that using max_concurrent has caused huge amounts of problems in the past- basically the Scheduler allocates 100s (or more) Tasks, when only a few are actually needed.
While it might not cause problems with your systems & projects, it does with Gre_BE here at Rosetta on one of his systems.
As jim1348 posted, he has also had similar issues, when using project_max_concurrent.

There is a bug (rare as it is) with max_concurrent/project_max_concurrent that has yet to be resolved.
Grant
Darwin NT
ID: 102947 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Bryn Mawr

Send message
Joined: 26 Dec 18
Posts: 333
Credit: 9,389,292
RAC: 2,285
Message 102948 - Posted: 14 Oct 2021, 8:56:55 UTC - in response to Message 102947.  

You say that you *can’t* use app_config to limit the number of work units but I’ve been doing so for a couple of years with zero problems.
If you check out several of the other threads that Greg_BE has posted to, you would see that using max_concurrent has caused huge amounts of problems in the past- basically the Scheduler allocates 100s (or more) Tasks, when only a few are actually needed.
While it might not cause problems with your systems & projects, it does with Gre_BE here at Rosetta on one of his systems.
As jim1348 posted, he has also had similar issues, when using project_max_concurrent.

There is a bug (rare as it is) with max_concurrent/project_max_concurrent that has yet to be resolved.


I have obviously seen the bug reports you reference and they usually take the form don’t do it, it always causes problems.

For the sake of balance and to inform those tying to track down the bug I wanted to register the fact that, in some circumstances, it works ok and is stable.
ID: 102948 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1669
Credit: 6,555,854
RAC: 2,999
Message 102953 - Posted: 14 Oct 2021, 13:09:06 UTC - in response to Message 102939.  
Last modified: 14 Oct 2021, 13:35:54 UTC

That happens on almost all VirtualBox projects. Only LHC does it right, because they use an updated wrapper.


Uh? Which version of wrapper are we using on R@H??
The latest official wrapper from boinc developers is 26203 (September 2020)
ID: 102953 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jim1348

Send message
Joined: 19 Jan 06
Posts: 881
Credit: 52,257,545
RAC: 1,128
Message 102954 - Posted: 14 Oct 2021, 13:39:54 UTC - in response to Message 102953.  

Uh? Which version of wrapper are we using on R@H??
The latest official wrapper from boinc developers is 26203 (September 2020)

Don't know. LHC may have rolled their own.
ID: 102954 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1669
Credit: 6,555,854
RAC: 2,999
Message 102955 - Posted: 14 Oct 2021, 14:07:36 UTC - in response to Message 102954.  

Don't know. LHC may have rolled their own.

From LHC folder in C:ProgramData seems that the LHC wrapper is 26198ab7.
I don't know if it is official or personalized
ID: 102955 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
1 · 2 · 3 · 4 . . . 9 · Next

Message boards : Number crunching : Not getting any python work



©2022 University of Washington
https://www.bakerlab.org