Message boards : Number crunching : Not getting any python work
Previous · 1 · 2 · 3 · 4 · 5 . . . 9 · Next
Author | Message |
---|---|
Greg_BE Send message Joined: 30 May 06 Posts: 5691 Credit: 5,859,226 RAC: 7 |
It's not max concurrent I was having trouble with. That is a job limiting function. I was doing CPU allocation per project. <ncpu>x</ncpu) that screwed me up. I used max concurrent in LHC and that had no problems. I don't need that function anymore in LHC since they upgraded their preferences page so you can set the number of cpu's you want. They took away the need for max_concurrent because again they updated their web preferences page so you can do that your self. IF AND WHEN RAH does this...(most likely never) then we won't be having this discussion. |
Greg_BE Send message Joined: 30 May 06 Posts: 5691 Credit: 5,859,226 RAC: 7 |
back to the topic I started this thread with.... I just aborted 21 tasks for 4.20 in the last 24hrs. I still get only 4.20 tasks and no python. Any other ideas? |
Greg_BE Send message Joined: 30 May 06 Posts: 5691 Credit: 5,859,226 RAC: 7 |
OK, that is useful. It may happen only when running multiple work units (or at least more than two). *exactly* Used this in LHC before they upgraded their preference page. Never had run away downloads. 1 task at a time is how I had it set. But way back when I had to use ncpu's as well, but that is handled by their preferences page as well. So in my OLD app_config (no need to use it now, but for example purposes here) I used max_concurrent for limiting to 1 task at a time (no runaway downloads) and ncpu was set to 4 since that is what ATLAS likes on my system. But as we established in another conversation, that when you try to do ncpu here in RAH then you get slammed with 50-100 tasks that you will never be able to complete in time. I have not tried max_concurrent here, so no idea if there are any problems or not. I can only base my experience with that function from LHC. |
Greg_BE Send message Joined: 30 May 06 Posts: 5691 Credit: 5,859,226 RAC: 7 |
back to the topic I started this thread with.... I still had some residual in my vbox from a long time ago. Cleaned that out and now lets see what happens. |
Werinbert Send message Joined: 22 Jul 13 Posts: 4 Credit: 1,001,196 RAC: 0 |
There was a lull due to lack of work, but python tasks are aplenty now... Yet I still am not getting any python tasks. I tried aborting the 4.20 tasks, that didn't work. I know that I can process python tasks as I was doing so week or so ago. |
Jim1348 Send message Joined: 19 Jan 06 Posts: 881 Credit: 52,257,545 RAC: 0 |
Maybe now there is a minimum memory requirement. All of my machines that got them had at least 48 GB, though I have not tried with less. |
Greg_BE Send message Joined: 30 May 06 Posts: 5691 Credit: 5,859,226 RAC: 7 |
4,900+ python tasks and I get nothing. I don't get it. I still have 60% memory free of 24 Gigs. I don't get memory errors and I don't think anything has been published about Python memory requirements. |
Jim1348 Send message Joined: 19 Jan 06 Posts: 881 Credit: 52,257,545 RAC: 0 |
I don't get memory errors and I don't think anything has been published about Python memory requirements. (1) I have gotten them on every machine I have tried (four of them), and they all have at least 48 GB of memory. (2) You have gotten none with less memory. (3) You apparently imagine that they publish requirements for their work units. Where have you seen them for any of them? PS - The obvious thing to do, as I indicated before, is to detach from your other projects to clear out more memory. Maybe it would work with what you have. Your current approach is not achieving results. |
Werinbert Send message Joined: 22 Jul 13 Posts: 4 Credit: 1,001,196 RAC: 0 |
I have 32GB of memory. Two weeks ago I was getting tasks without a problem. Now I am not getting any. I doubt that a single task requires 48GB to run. |
Jim1348 Send message Joined: 19 Jan 06 Posts: 881 Credit: 52,257,545 RAC: 0 |
I have 32GB of memory. Two weeks ago I was getting tasks without a problem. Now I am not getting any. I doubt that a single task requires 48GB to run. The ones that I have running now on Ubuntu are using less than 2 GB. But they show as 8 GB on BoincTasks, which probably means that they reserve that much. Since you used to get them and now don't, it seems likely that they have imposed additional restrictions on the required minimum amount of memory for download. That is not uncommon in various projects, in case you have not run into it before. |
Jim1348 Send message Joined: 19 Jan 06 Posts: 881 Credit: 52,257,545 RAC: 0 |
By the way, you aren't missing much. Even on a Ryzen 3900X with 96 GB of memory, and limiting it to using only 50% of the cores, I still get "VM job unmanageable" suspensions. That means that no more work can be downloaded until the suspension lifts. That is about 24 hours, or a reboot. I have removed VirtualBox from two of my machines to eliminate the pythons, reassigned another machine, and may take VBox out on the last one too. They could be more trouble than they are worth. Rosetta needs to get its act together. |
Greg_BE Send message Joined: 30 May 06 Posts: 5691 Credit: 5,859,226 RAC: 7 |
I have 32GB of memory. Two weeks ago I was getting tasks without a problem. Now I am not getting any. I doubt that a single task requires 48GB to run. Again, no error messages, no information that I know of indicating what kind of machine is needed. Big black hole of information. |
Greg_BE Send message Joined: 30 May 06 Posts: 5691 Credit: 5,859,226 RAC: 7 |
By the way, you aren't missing much. Even on a Ryzen 3900X with 96 GB of memory, and limiting it to using only 50% of the cores, I still get "VM job unmanageable" suspensions. That means that no more work can be downloaded until the suspension lifts. That is about 24 hours, or a reboot. Interesting. RAH getting its act together? hah! Now your dreaming! I think our role in the project is leftovers and low level stuff. All the rest is run on their neural network and high end systems. Kind of reminds me of TAC. That's a joke of a project for BOINC users. |
Jim1348 Send message Joined: 19 Jan 06 Posts: 881 Credit: 52,257,545 RAC: 0 |
I think our role in the project is leftovers and low level stuff. I am sure they do a lot of their own stuff in-house now, with the AI they are using. But it doesn't follow that we are doing low-level stuff. It may be everything that is sent for processing by groups that do not have much of an in-house capability. So it could be good work. But they don't tell us. |
Greg_BE Send message Joined: 30 May 06 Posts: 5691 Credit: 5,859,226 RAC: 7 |
<--that is the thing about this project. No communication. It's a shame. They used to.I think our role in the project is leftovers and low level stuff. |
Jim1348 Send message Joined: 19 Jan 06 Posts: 881 Credit: 52,257,545 RAC: 0 |
I have attached a Ryzen 3600 with 48 GB of memory, and another Ryzen 3600 with 32 GB of memory. Both are on Ubuntu 20.04.3, and have VirtualBox installed. Both picked up the pythons right away; the 32 GB machine downloaded five pythons, and four regular Rosettas in the first batch. I limited them to 9 virtual cores running at a time to try to prevent the "VM job unmanageable" errors. So far, I have not gotten any errors on the 48 GB machine, but am just starting on the 32 GB machine. So if you are not getting any pythons with 32 GB there is a problem somewhere. I usually just wipe out the OS and start over if I can't find it. It is so fast now that it is easier than spending much time on it Good luck. |
Scottie McKinley Send message Joined: 14 Jan 21 Posts: 5 Credit: 2,419,120 RAC: 144 |
I couldn't get pythons regardless of settings. I just did a full disk wipe and reinstall of Windows 10 with the Boinc client dated October 21 (I think). Now pythons are running without me changing any settings. Weird that it works now. |
Jim1348 Send message Joined: 19 Jan 06 Posts: 881 Credit: 52,257,545 RAC: 0 |
One last thought for Win10. If you have ever enabled "Hyper-V", be sure to disable it (in "Turn Windows features on or off"). It is incompatible with VirtualBox. |
Greg_BE Send message Joined: 30 May 06 Posts: 5691 Credit: 5,859,226 RAC: 7 |
One last thought for Win10. If you have ever enabled "Hyper-V", be sure to disable it (in "Turn Windows features on or off"). For me, I Hyper V is not on. Never has been. There is no way I am wiping out my OS to get some stubborn tasks. There has to be something else holding up Python tasks. Something that can be done without wiping out windows and spending hours reinstalling and re configuring. |
Grant (SSSF) Send message Joined: 28 Mar 20 Posts: 1639 Credit: 16,792,442 RAC: 9,617 |
There has to be something else holding up Python tasks.BOINC Manager, Options, Event log options, Work_fetch_debug, CPU_sched_debug & similar, rr_simulation etc. Enable those (or just one of those) options (expect a huge amount of output in the Event log) to see what the Manager is actually doing- what values it is working with & the results it's producing and decisions it's making. Grant Darwin NT |
Message boards :
Number crunching :
Not getting any python work
©2024 University of Washington
https://www.bakerlab.org