Not getting any python work

Message boards : Number crunching : Not getting any python work

To post messages, you must log in.

Previous · 1 . . . 5 · 6 · 7 · 8 · 9 · Next

AuthorMessage
Profile Greg_BE
Avatar

Send message
Joined: 30 May 06
Posts: 5658
Credit: 5,670,291
RAC: 2,328
Message 103480 - Posted: 22 Nov 2021, 22:31:59 UTC - in response to Message 103478.  

That's a great suggestion. I'll add that functionality.


Please can you post back when you have implemented something so we can check our idle machines - I've got a few that aren't getting any Rosetta tasks at the moment and I have no idea why.



1) There are only 9 tasks in 4.2, so you probably are not getting anything from that
2) If you had any problems in the past running Python, you probably got black listed by the python scheduler (see my quote from ADMIN from earlier)

If 2 applies then your are SOL.
ID: 103480 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile dcdc

Send message
Joined: 3 Nov 05
Posts: 1829
Credit: 114,437,372
RAC: 56,040
Message 103481 - Posted: 22 Nov 2021, 22:57:03 UTC

And please reduce the VB tasks RAM requirements in the VB image too!

Unless anyone has seen anything to the contrary, they don't need anywhere near 8GB each - 1GB would probably be more than enough. It's massively limiting the number of tasks that can be run - I've typically for 3 cores running python tasks and 13 idle at the moment.
ID: 103481 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
.clair.

Send message
Joined: 2 Jan 07
Posts: 274
Credit: 26,399,595
RAC: 3
Message 103482 - Posted: 23 Nov 2021, 1:07:30 UTC - in response to Message 103481.  
Last modified: 23 Nov 2021, 1:12:35 UTC

And please reduce the VB tasks RAM requirements in the VB image too!
Unless anyone has seen anything to the contrary, they don't need anywhere near 8GB each - 1GB would probably be more than enough.
It's massively limiting the number of tasks that can be run - I've typically for 3 cores running python tasks and 13 idle at the moment.

Oooo . I`ve found a big one .
Peak working set size 464.75 MB
Peak swap size 492.55 MB
Peak disk usage 3,549.37 MB . . . .
https://boinc.bakerlab.org/rosetta/result.php?resultid=1451421574
I don`t care about disk use, there`s plenty about,
ID: 103482 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Admin
Project administrator

Send message
Joined: 1 Jul 05
Posts: 4805
Credit: 0
RAC: 0
Message 103494 - Posted: 23 Nov 2021, 19:23:41 UTC

I added an option near the bottom of your host "details" page to "Skip" or "Allow" Vbox VM jobs if your host has the boinc vm client.

I'll talk to the researchers that are currently running VM jobs about reducing the memory requirements. This will take some time since they will have to update the VM image, test it, and cancel and resubmit their jobs.
ID: 103494 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Falconet

Send message
Joined: 9 Mar 09
Posts: 350
Credit: 1,002,599
RAC: 188
Message 103495 - Posted: 23 Nov 2021, 19:59:07 UTC - in response to Message 103494.  
Last modified: 23 Nov 2021, 20:01:59 UTC

I added an option near the bottom of your host "details" page to "Skip" or "Allow" Vbox VM jobs if your host has the boinc vm client.

I'll talk to the researchers that are currently running VM jobs about reducing the memory requirements. This will take some time since they will have to update the VM image, test it, and cancel and resubmit their jobs.



Sounds good.
Thanks.

Edit: Next to the skip/allow option, there is also the option to delete the machine. It's already present before on merge option.
ID: 103495 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
.clair.

Send message
Joined: 2 Jan 07
Posts: 274
Credit: 26,399,595
RAC: 3
Message 103496 - Posted: 23 Nov 2021, 20:33:56 UTC - in response to Message 103494.  
Last modified: 23 Nov 2021, 20:37:04 UTC

I added an option near the bottom of your host "details" page to "Skip" or "Allow" Vbox VM jobs if your host has the boinc vm client.

I'll talk to the researchers that are currently running VM jobs about reducing the memory requirements.
This will take some time since they will have to update the VM image, test it, and cancel and resubmit their jobs.

Thank you , the memory problem realy is backing the RPP jobs against a wall.
If they can get it down to 2GB or less per VM they will get a lot more done.
ID: 103496 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Greg_BE
Avatar

Send message
Joined: 30 May 06
Posts: 5658
Credit: 5,670,291
RAC: 2,328
Message 103498 - Posted: 24 Nov 2021, 20:40:23 UTC

ADMIN, I would love to come back, but your slash and burn edit to the python scheduler black listed my system which is apparently why I had to start this thread in the first place. Because of that code, you limited me to 4.2. Now that you can not generate any 4.2 and blacklisted my system in the early stages of this Python adventure, there is no work for me, which is why I left.

I gave you guys 15 years and you block me without saying anything until way later with what? 2 or 3 lines?
That's not a good thing.

I have gotten Vbox to work just fine over on QuChemPed and I have run LHC Atlas for a long time without problems. So I don't think it was really my system that was at fault, rather I think it was errors on your early Python tasks that caused the error you are blocking.

If you don't want me back then I might as well delete any association with this project and say nice knowing you.
ID: 103498 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Michael Goetz
Avatar

Send message
Joined: 17 Jan 08
Posts: 12
Credit: 179,114
RAC: 0
Message 103499 - Posted: 24 Nov 2021, 22:33:17 UTC - in response to Message 103498.  

ADMIN, I would love to come back, but your slash and burn edit to the python scheduler black listed my system which is apparently why I had to start this thread in the first place. Because of that code, you limited me to 4.2. Now that you can not generate any 4.2 and blacklisted my system in the early stages of this Python adventure, there is no work for me, which is why I left.

I gave you guys 15 years and you block me without saying anything until way later with what? 2 or 3 lines?
That's not a good thing.

I have gotten Vbox to work just fine over on QuChemPed and I have run LHC Atlas for a long time without problems. So I don't think it was really my system that was at fault, rather I think it was errors on your early Python tasks that caused the error you are blocking.

If you don't want me back then I might as well delete any association with this project and say nice knowing you.


It looks like we can turn that "blacklist" flag on and off ourselves now.
Want to find one of the largest known primes? Try PrimeGrid. Or help cure disease at WCG.

ID: 103499 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Greg_BE
Avatar

Send message
Joined: 30 May 06
Posts: 5658
Credit: 5,670,291
RAC: 2,328
Message 103500 - Posted: 24 Nov 2021, 23:28:48 UTC - in response to Message 103499.  

ADMIN, I would love to come back, but your slash and burn edit to the python scheduler black listed my system which is apparently why I had to start this thread in the first place. Because of that code, you limited me to 4.2. Now that you can not generate any 4.2 and blacklisted my system in the early stages of this Python adventure, there is no work for me, which is why I left.

I gave you guys 15 years and you block me without saying anything until way later with what? 2 or 3 lines?
That's not a good thing.

I have gotten Vbox to work just fine over on QuChemPed and I have run LHC Atlas for a long time without problems. So I don't think it was really my system that was at fault, rather I think it was errors on your early Python tasks that caused the error you are blocking.

If you don't want me back then I might as well delete any association with this project and say nice knowing you.


It looks like we can turn that "blacklist" flag on and off ourselves now.



How?
My understanding of his post was that he put a code in the scheduler to eliminate systems that generated a certain kind of error. If it is in the scheduler, then how do you get around that? Once it knows you IP and account, I don't think you can undo that.
ID: 103500 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Michael Goetz
Avatar

Send message
Joined: 17 Jan 08
Posts: 12
Credit: 179,114
RAC: 0
Message 103501 - Posted: 24 Nov 2021, 23:34:55 UTC - in response to Message 103500.  

ADMIN, I would love to come back, but your slash and burn edit to the python scheduler black listed my system which is apparently why I had to start this thread in the first place. Because of that code, you limited me to 4.2. Now that you can not generate any 4.2 and blacklisted my system in the early stages of this Python adventure, there is no work for me, which is why I left.

I gave you guys 15 years and you block me without saying anything until way later with what? 2 or 3 lines?
That's not a good thing.

I have gotten Vbox to work just fine over on QuChemPed and I have run LHC Atlas for a long time without problems. So I don't think it was really my system that was at fault, rather I think it was errors on your early Python tasks that caused the error you are blocking.

If you don't want me back then I might as well delete any association with this project and say nice knowing you.


It looks like we can turn that "blacklist" flag on and off ourselves now.



How?
My understanding of his post was that he put a code in the scheduler to eliminate systems that generated a certain kind of error. If it is in the scheduler, then how do you get around that? Once it knows you IP and account, I don't think you can undo that.


Go to the details page for that host. Scroll to the bottom. You'll see, on the last line, a setting "VirtualBox VM jobs", and a blue and red button labelled "Allow". Click on that button. That's it. That turns off the blacklist from the scheduler.

The button will change to "Skip". You can click on the "skip" button if you ever want to turn off the Vbox tasks.
Want to find one of the largest known primes? Try PrimeGrid. Or help cure disease at WCG.

ID: 103501 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile markfw

Send message
Joined: 26 Jan 07
Posts: 5
Credit: 284,410,415
RAC: 244,064
Message 103505 - Posted: 25 Nov 2021, 1:41:42 UTC

By default, its skip. on the details page, but I get no tasks unless I install virtualbox, and then it only loads 16 tasks for a 128 thread 7742 with 128 gig ram.

What is wrong ?
ID: 103505 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Grant (SSSF)

Send message
Joined: 28 Mar 20
Posts: 1467
Credit: 14,331,566
RAC: 16,669
Message 103510 - Posted: 25 Nov 2021, 5:01:48 UTC - in response to Message 103505.  
Last modified: 25 Nov 2021, 5:02:18 UTC

By default, its skip. on the details page, but I get no tasks unless I install virtualbox, and then it only loads 16 tasks for a 128 thread 7742 with 128 gig ram.

What is wrong ?
Not enough RAM.
You need 8GB RAM per Python Task (they don't use anywhere near that amount, but that is the hard minimum requirement that has been set for those Tasks). So if you get just over 1TB of RAM in that system, you'll be able to process Python Tasks on all available cores & threads.
Or just keep waiting for some more Rosetta 4.20 work to be released.
Grant
Darwin NT
ID: 103510 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Greg_BE
Avatar

Send message
Joined: 30 May 06
Posts: 5658
Credit: 5,670,291
RAC: 2,328
Message 103513 - Posted: 25 Nov 2021, 7:51:50 UTC - in response to Message 103501.  
Last modified: 25 Nov 2021, 7:59:18 UTC

ADMIN, I would love to come back, but your slash and burn edit to the python scheduler black listed my system which is apparently why I had to start this thread in the first place. Because of that code, you limited me to 4.2. Now that you can not generate any 4.2 and blacklisted my system in the early stages of this Python adventure, there is no work for me, which is why I left.

I gave you guys 15 years and you block me without saying anything until way later with what? 2 or 3 lines?
That's not a good thing.

I have gotten Vbox to work just fine over on QuChemPed and I have run LHC Atlas for a long time without problems. So I don't think it was really my system that was at fault, rather I think it was errors on your early Python tasks that caused the error you are blocking.

If you don't want me back then I might as well delete any association with this project and say nice knowing you.


It looks like we can turn that "blacklist" flag on and off ourselves now.



How?
My understanding of his post was that he put a code in the scheduler to eliminate systems that generated a certain kind of error. If it is in the scheduler, then how do you get around that? Once it knows you IP and account, I don't think you can undo that.


Go to the details page for that host. Scroll to the bottom. You'll see, on the last line, a setting "VirtualBox VM jobs", and a blue and red button labelled "Allow". Click on that button. That's it. That turns off the blacklist from the scheduler.

The button will change to "Skip". You can click on the "skip" button if you ever want to turn off the Vbox tasks.



I don't see anything in BOINC or on my preferences page to match what you are talking about.
I think that option is only for people that are not black listed.
ID: 103513 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Falconet

Send message
Joined: 9 Mar 09
Posts: 350
Credit: 1,002,599
RAC: 188
Message 103514 - Posted: 25 Nov 2021, 9:58:37 UTC - in response to Message 103513.  
Last modified: 25 Nov 2021, 9:59:17 UTC

Go to my account, devices on this account, click on details on the host you want to run Python on, scroll down to "VirtualBox VM jobs". You should see either "Allow" or "Skip".
Click on Allow to allow VB jobs, click on Skip to forbid VB jobs.

The option only appears on devices that have VirtualBox installed.
ID: 103514 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Michael Goetz
Avatar

Send message
Joined: 17 Jan 08
Posts: 12
Credit: 179,114
RAC: 0
Message 103515 - Posted: 25 Nov 2021, 12:02:54 UTC - in response to Message 103513.  

ADMIN, I would love to come back, but your slash and burn edit to the python scheduler black listed my system which is apparently why I had to start this thread in the first place. Because of that code, you limited me to 4.2. Now that you can not generate any 4.2 and blacklisted my system in the early stages of this Python adventure, there is no work for me, which is why I left.

I gave you guys 15 years and you block me without saying anything until way later with what? 2 or 3 lines?
That's not a good thing.

I have gotten Vbox to work just fine over on QuChemPed and I have run LHC Atlas for a long time without problems. So I don't think it was really my system that was at fault, rather I think it was errors on your early Python tasks that caused the error you are blocking.

If you don't want me back then I might as well delete any association with this project and say nice knowing you.


It looks like we can turn that "blacklist" flag on and off ourselves now.



How?
My understanding of his post was that he put a code in the scheduler to eliminate systems that generated a certain kind of error. If it is in the scheduler, then how do you get around that? Once it knows you IP and account, I don't think you can undo that.


Go to the details page for that host. Scroll to the bottom. You'll see, on the last line, a setting "VirtualBox VM jobs", and a blue and red button labelled "Allow". Click on that button. That's it. That turns off the blacklist from the scheduler.

The button will change to "Skip". You can click on the "skip" button if you ever want to turn off the Vbox tasks.



I don't see anything in BOINC or on my preferences page to match what you are talking about.
I think that option is only for people that are not black listed.


I did not say anything about the preferences page. On any other project, that's where the control would be. But here the change is on the host detail page. Go the your list of computers and select the "details" link for the computer you want to change. There's a different setting for each computer.

Also, I believe that if you don't have vbox installed the server doesn't show that button. The control appears to only be there for computers that have vbox installed.

You definitely CAN turn off the blacklisting. My computer was blacklisted, and now it's not.

The correct page for changing the control for your c omputer is https://boinc.bakerlab.org/rosetta/show_host_detail.php?hostid=3433065 There should be a gaudy blue and red skip/allow button near the bottom of the page beneath the "update location" button.
Want to find one of the largest known primes? Try PrimeGrid. Or help cure disease at WCG.

ID: 103515 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile dcdc

Send message
Joined: 3 Nov 05
Posts: 1829
Credit: 114,437,372
RAC: 56,040
Message 103517 - Posted: 25 Nov 2021, 14:16:47 UTC

Good find - I wouldn't have seen that for a long time!

So there options I've got are either:
Skip: For systems already running VB tasks succesfully
Allow: For systems that have been blocked from VB tasks
Missing (no button there): presumably for systems that can't run VB tasks (no virtualisation, no VirtualBox installed, service install etc.)
ID: 103517 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Greg_BE
Avatar

Send message
Joined: 30 May 06
Posts: 5658
Credit: 5,670,291
RAC: 2,328
Message 103522 - Posted: 25 Nov 2021, 20:17:26 UTC - in response to Message 103515.  
Last modified: 25 Nov 2021, 20:18:04 UTC

ADMIN, I would love to come back, but your slash and burn edit to the python scheduler black listed my system which is apparently why I had to start this thread in the first place. Because of that code, you limited me to 4.2. Now that you can not generate any 4.2 and blacklisted my system in the early stages of this Python adventure, there is no work for me, which is why I left.

I gave you guys 15 years and you block me without saying anything until way later with what? 2 or 3 lines?
That's not a good thing.

I have gotten Vbox to work just fine over on QuChemPed and I have run LHC Atlas for a long time without problems. So I don't think it was really my system that was at fault, rather I think it was errors on your early Python tasks that caused the error you are blocking.

If you don't want me back then I might as well delete any association with this project and say nice knowing you.


It looks like we can turn that "blacklist" flag on and off ourselves now.



How?
My understanding of his post was that he put a code in the scheduler to eliminate systems that generated a certain kind of error. If it is in the scheduler, then how do you get around that? Once it knows you IP and account, I don't think you can undo that.


Go to the details page for that host. Scroll to the bottom. You'll see, on the last line, a setting "VirtualBox VM jobs", and a blue and red button labelled "Allow". Click on that button. That's it. That turns off the blacklist from the scheduler.

The button will change to "Skip". You can click on the "skip" button if you ever want to turn off the Vbox tasks.



I don't see anything in BOINC or on my preferences page to match what you are talking about.
I think that option is only for people that are not black listed.


I did not say anything about the preferences page. On any other project, that's where the control would be. But here the change is on the host detail page. Go the your list of computers and select the "details" link for the computer you want to change. There's a different setting for each computer.

Also, I believe that if you don't have vbox installed the server doesn't show that button. The control appears to only be there for computers that have vbox installed.

You definitely CAN turn off the blacklisting. My computer was blacklisted, and now it's not.

The correct page for changing the control for your c omputer is https://boinc.bakerlab.org/rosetta/show_host_detail.php?hostid=3433065 There should be a gaudy blue and red skip/allow button near the bottom of the page beneath the "update location" button.


How freaking amazing!
But:

If VM jobs cannot run due VirtualBox errors, this host will be flagged again to skip VM jobs.
Ok, what errors then. Needs to be more specific.
J got 4 pythons now.
Curious. I backed down to 5.4.22 to get QuChem to run clean. So this should also allow Python to run clean.
ID: 103522 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Greg_BE
Avatar

Send message
Joined: 30 May 06
Posts: 5658
Credit: 5,670,291
RAC: 2,328
Message 103523 - Posted: 25 Nov 2021, 20:21:50 UTC - in response to Message 103517.  
Last modified: 25 Nov 2021, 20:22:25 UTC

Good find - I wouldn't have seen that for a long time!

So there options I've got are either:
Skip: For systems already running VB tasks succesfully
Allow: For systems that have been blocked from VB tasks
Missing (no button there): presumably for systems that can't run VB tasks (no virtualisation, no VirtualBox installed, service install etc.)



Well about time he did this. Why the *^*%$ didn't he do this when they started python?
Seriously.

But did you notice their disclaimer/threat?
If VM jobs cannot run due VirtualBox errors, this host will be flagged again to skip VM jobs.
But in usual RAH fashion, they are not specific.
ID: 103523 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Greg_BE
Avatar

Send message
Joined: 30 May 06
Posts: 5658
Credit: 5,670,291
RAC: 2,328
Message 103524 - Posted: 25 Nov 2021, 20:27:31 UTC
Last modified: 25 Nov 2021, 20:31:16 UTC

There is one thing missing still, ADMIN needs to change the preference page to reflect how many cores a person wishes to allocate. You need this to avoid Vbox environment instability.
Again at Qchem I found I can only run 3 tasks without it becoming unstable.
RAH does not allow you to control how many cores or how many tasks to run.
So if it tries to run all 4 at the same time and the environment becomes unstable, I have no way to control that scenario. Hopefully BOINC will know what to do.

So 2 RAH in Vbox, 1 Quchem in Vbox and SiDock was also running but finished.
ID: 103524 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Bryn Mawr

Send message
Joined: 26 Dec 18
Posts: 373
Credit: 10,598,406
RAC: 8,429
Message 103525 - Posted: 25 Nov 2021, 22:39:46 UTC - in response to Message 103523.  

Good find - I wouldn't have seen that for a long time!

So there options I've got are either:
Skip: For systems already running VB tasks succesfully
Allow: For systems that have been blocked from VB tasks
Missing (no button there): presumably for systems that can't run VB tasks (no virtualisation, no VirtualBox installed, service install etc.)



Well about time he did this. Why the *^*%$ didn't he do this when they started python?
Seriously.

But did you notice their disclaimer/threat?
If VM jobs cannot run due VirtualBox errors, this host will be flagged again to skip VM jobs.
But in usual RAH fashion, they are not specific.


Seriously? Does nothing ever satisfy you?

You’ve been banging on for weeks about not getting RPP WUs, now you have some but no thank you, no well done - just *^*%$ and more complaints!
ID: 103525 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Previous · 1 . . . 5 · 6 · 7 · 8 · 9 · Next

Message boards : Number crunching : Not getting any python work



©2024 University of Washington
https://www.bakerlab.org