Yeah, you're right, I need way more ram.

Python tasks need like what? 6Gib of RAM each?

Maybe...
I'm thinking that's what they insist on to start, and so VBox will take it from the OS and reserve it for that Task. But from the looks of several of your completed Python Tasks, their actual RAM usage is less than 100MB. So in theory, it should be possible for many of them to run at the same time.
VBox reserves 8GB for one Task, it doesn't need it all so when the next Task tries to start, it only needs to pull another 100MB from the OS for two Tasks to be running, with 8GB still available between them. Another Task starts up, only another 100MB is needed from the OS, etc, etc. (Of course i haven't the slightest idea if that's how VBox actually works).

What each Task does require (according to your completed Tasks), is 7.5GB+ of storage space. It looks like each Task needs it's own copy of the VM image in order to run (i'm wondering if this is a repeat of the previous issue with Rosetta 4.xx Tasks when they ran copying all the support files every time a new one started & then deleting them when done instead of just linking to them, or if it's a issue with how VirtualBox works).

I've given BOINC 280GB of free disc space. I'll check the actual ram usage of VB Python tasks, it's not that high IIRC

Thanks for that.
While each Task doesn't use much RAM in it's Virtual Machine, i'm curious as to how much RAM VBox is actually taking from the OS to run them. Likewise disk usage- 8GB+ for every VM (Python Task) that is running isn't good either.



Edit-

boinc_cages_IL_2727241_31732_0 uses 30.5MB of RAM.

As reported by Windows Task Manager? What about the Virtual Box files that are running (i think there are 3 of them- VirtualBox Manager, VirtualBox Interface, VirtualBox Global Interface) How much RAM are they using?

According to task manager, for that specific task, it's around 30MB for the VirtualBox Headless Frontend, 3.0MB for the VirtualBox Interface, 0.1MB for the VirtualBox Global Interface, and 0.1-1MB for two more instances of the VirtualBox Headless Frontend that I don't know the purpose of. In total, it's 35MB, give or take. I'll take a look at some other tasks once this one completes. That task is taking a really long time to run, but it appears to be actually running.

Yes, It's sub-optimal that each task takes around 8 GiB of storage to run. However, I don't think it's possible for them to share storage like the improved Rosetta 4.20. Not even LHC does that, and they should be one of the best vBox projects on BOINC. Since the base memory of the VMs are 6144MB, I'm assuming they insist on having 6GB of free RAM to run. Perhaps someone well-versed in vBox tasks can chime in?

I could plug in a 2TB hard drive just for BOINC like a crazy person (I have a 1TB HDD idling in my system, but I don't want to use that since it's really, really old), but increasing the RAM is not within my budget. Because of that, I really wish they make the RAM requirements more sane. Why make the RAM requirements 6GB if the vast majority of tasks don't appear to take over 1GB of RAM?

Thank you. I feel a bit stupid - on the same page!

According to task manager, for that specific task, it's around 30MB for the VirtualBox Headless Frontend, 3.0MB for the VirtualBox Interface, 0.1MB for the VirtualBox Global Interface, and 0.1-1MB for two more instances of the VirtualBox Headless Frontend that I don't know the purpose of. In total, it's 35MB, give or take.

Just a bit less than 7.5GB.

Yes, It's sub-optimal that each task takes around 8 GiB of storage to run. However, I don't think it's possible for them to share storage like the improved Rosetta 4.20. Not even LHC does that, and they should be one of the best vBox projects on BOINC. Since the base memory of the VMs are 6144MB, I'm assuming they insist on having 6GB of free RAM to run. Perhaps someone well-versed in vBox tasks can chime in?

Not using the same disk image is probably related to security.
But if the Tasks and VBox itself are only using less than 1GB of RAM (which they appear to be well under), then allowing BOINC to use 8GB of storage per Task being processed should allow more than 1 Task to run on a system with plenty of cores/threads if it's given access to that extra storage.
Have any extra Python Tasks started up with the extra storage space made available?

Why make the RAM requirements 6GB if the vast majority of tasks don't appear to take over 1GB of RAM?

Agreed.
From what i've been able to find online generally is that VirtualBox itself needs 500MB, and if the Tasks are all using less than 100MB, then there's no need to put 7.5GB RAM as a minimum requirement.

If a few more people that have got Python Tasks running, particularly those with 2 or more at the same time, could let us know just how much RAM is being used in total for all the running Python Tasks, and how much storage space is being used, we might be able to point it out to the Project and get the minimum requirements changed to something more realistic, letting people get more work done instead of having cores/threads sitting idle while doing just a single Python Task or two.

---

Grant
Darwin NT

Have any extra Python Tasks started up with the extra storage space made available?

All the tasks got downloaded when there was at least 280GB of free disc space. Still, only one runs at a time with all the other ones waiting. Sometimes, BOINC will pause a task to start a new one.

Occasionally, BOINC will pause a task to download a new one and run that, even if said task is close to the deadline, but that's besides the point.

I just realized Rosetta is eating up 20GB of storage. Yikes, that's...bad... I just let BOINC have my entire 2TB media hard drive sans 250GB since I cannot possibly fill it up. In my case, that's a tad over 300 GB. I've never been able to get more than one python task running simultaneously. I'm not short on storage, I'm short on memo...what was I sayin'?

I've never been able to get more than one python task running simultaneously.

Are you sure you have set the CPU(s) usage to 100%, in 'Processing preferences'?

You are running the Pythons so it makes sense that most of Pythons are waiting to run.
However, with 16 GB of memory, you should be able to run 2 Pythons.
At least, my Ryzen 1400 can run 2 pythons with 16 GB.

---

From what I have seen running here @home, it is the insane amount of memory [7.45GB in task/properties] the RPP tasks demand that stops us from running them
I have lots of disk space, they download ok, on a 32GB system 16 cpu, it usualy runs three, once it ran fore for a while,
{in BM memory is set to use 99% when active, 100% when idle}
then I think BM kacked it, and sent the forth into `waiting` probably coz I was now active/using the system, checking to see what these &*$"@:~ python tasks where doing
If the Admin can get the massive memory demand down then they will get a lot more RPP done
But `we` know it, and we , here on the forum just want them to RTFM [forum actualy] :)
Then they will know why all the RPP are clogging up the que.

Now that I have written this pointless post I will go and rehabilitate by banging my head on the wall , and scream at a lump of rock . . . . .

Looks like more Rosetta 4.2 work.
Received 4 west_nile tasks.

---

One more data point:
aagb-SAR-PHE-ACHC13T-mB3PHG_pp_2_2704977_1_1
According to task manager, VirtualBox Global Interface uses 0.5MB, VirtualBox Interface uses 2.3 MB, VirtualBox Headless Frontend uses 28.8, two other instances of VirtualBox Headless Frontend both use less than 1MB.

Still can only get 1 Rosetta Python to run despite me actually have more than 7GB available (my settings mean that less than 6GB is available to BOINC). I guess these tasks really want 6GB of available RAM to run. The most I can do in theory is indeed 2.

Why do these tasks have such absurd RAM requirements? Can anyone point me to a Python task that actually used more than 1GB of ram? I have 4 WCG tasks running, one ARP & three MCM, one Rosetta Python task, one YouTube video running, two chrome tabs open, and a generous smacking of background processes, I still have 7GB of free RAM.

nevermind

Nevermind.

I literally just got 4.20 work. All west_nile except for 1 zika. Guess we're working the flaviviruses.
Admin made an appearance on the forums - https://boinc.bakerlab.org/rosetta/forum_thread.php?id=14771&postid=103413

---

I have lots of disk space, they download ok, on a 32GB system 16 cpu, it usualy runs three, once it ran fore for a while,
{in BM memory is set to use 99% when active, 100% when idle}

When you've next got 3 (or even 4) running at a time, could you please check in Windows Task manager to see just how much RAM is actually being used by those Tasks & Vbox in total?
Completed Tasks are reporting less than 100MB being used, Tomcat's posts show that only a few hundred MB is being used (if that) by the Tasks & Vbox combined for a single Task when running. If that is the case, the 7.5GB+ minimum requirement is beyond excessive & counter productive. 750MB would appear to be more than enough.

---

Grant
Darwin NT

Have any extra Python Tasks started up with the extra storage space made available?

All the tasks got downloaded when there was at least 280GB of free disc space. Still, only one runs at a time with all the other ones waiting. Sometimes, BOINC will pause a task to start a new one.

Oh well, it was worth a try. Might as well put things back the way they were
Thanks for your efforts.



If more people with running Python Tasks could let us know just how much RAM Vbox & the Tasks are actually using (amount of RAM and number of Tasks), maybe we can get the minimum RAM requirement changed to something more realistic, and get more work done & make better use of the resources people have made available.
And it might help with the issue of some Tasks sitting there & not running, and old VMs accumulating & needing to be cleared out manaully.

---

Grant
Darwin NT

If more people with running Python Tasks could let us know just how much RAM Vbox & the Tasks are actually using (amount of RAM and number of Tasks), maybe we can get the minimum RAM requirement changed to something more realistic, and get more work done & make better use of the resources people have made available.

To help answer this, I have been running 12 pythons on a Ryzen 3900X for an hour. This is an Ubuntu 20.04.3 machine with 96 GB of memory.
https://boinc.bakerlab.org/rosetta/show_host_detail.php?hostid=6155880
These are the only BOINC tasks running.

BoincTasks shows the memory reserved per work unit:
https://postimg.cc/hzKhjPrz

free shows the amount of memory used and available:
https://postimg.cc/MMtHj2Ps

top shows the amount of memory used per work unit as a percentage of the total:
https://postimg.cc/m1DDYXKT

I hope this helps you to convince them that they can reduce the amount that they require.
They have other problems too, but if people can't run them, they will never find them.

---

To help answer this, I have been running 12 pythons on a Ryzen 3900X for an hour. This is an Ubuntu 20.04.3 machine with 96 GB of memory.
https://boinc.bakerlab.org/rosetta/show_host_detail.php?hostid=6155880

Exclent.
I'm not familiar with LINUX, so if you could help me out further that would be great.

BoincTasks shows the memory reserved per work unit:
https://postimg.cc/hzKhjPrz

So each Python Task reserves 7,629.39GB RAM (so 7.7GB is close enough).

free shows the amount of memory used and available:
https://postimg.cc/MMtHj2Ps

So out of 96GB of total system RAM, only just over 20GB is actually in use- for the entire system (OS, BOINC, all other running programmes, services, etc, etc).

top shows the amount of memory used per work unit as a percentage of the total:
https://postimg.cc/m1DDYXKT

So with the current batch of Python work, the most a single Task is actually using is only 1.5% of the total available system RAM. Let's say 1.7% to give it a nice large safety margin.
So that should give us 1.64GB (96492.28MB * 1.7%) is all that's really needed per Task to run (the actual in use value being less than 1.5GB). Which is only a little over 300MB more than the minimum per Task processed RAM requirement i advocate for Rosetta 4.20 Tasks to avoid running in to memory limitation issues.

So they can reduce their minimum RAM requirement by 6.06GB, with no adverse effects (and from what little reading up i've done on VBox, it will ask for more RAM from the Host OS above & beyond a VMs minimum requirement setting if it needs it anyway, if it can't get it, then that's when Virtual Memory comes in to play).

I hope this helps you to convince them that they can reduce the amount that they require.

We can only hope...

They have other problems too, but if people can't run them, they will never find them.

I suspect that many of those issues could be related to it hitting that excessive minimum RAM requirement. Get rid of that, and there's a chance quite a few of those other issues will go away as well. Only one way to find out.


Thanks for your efforts.

---

Grant
Darwin NT

You are welcome. I think your analysis is correct of the Linux part.

But they must have picked such a large amount for a reason (or at least I hope they had a reason).
So I would allow them 4 GB just to get the ball rolling, and reduce that later if necessary.
But maybe they aren't Linux experts either (not that I am, but I can use it).

Lots of luck.

---

I have lots of disk space, they download ok, on a 32GB system 16 cpu, it usualy runs three, once it ran fore for a while,
{in BM memory is set to use 99% when active, 100% when idle}

When you've next got 3 (or even 4) running at a time, could you please check in Windows Task manager to see just how much RAM is actually being used by those Tasks & Vbox in total?
Completed Tasks are reporting less than 100MB being used, Tomcat's posts show that only a few hundred MB is being used (if that) by the Tasks & Vbox combined for a single Task when running. If that is the case, the 7.5GB+ minimum requirement is beyond excessive & counter productive. 750MB would appear to be more than enough.

I had no 4.2 work , so RPP got the lot to play with [no other projects running, qued etc]
At that time I did, and if I was`nt wrong , the largest was 115MB
A VM built in 750MB would plenty from anything I have seen, one gerbil bite if they want to be wild ;)
By the look of it the RPP tasks actual use is the smallest amount of memory of any R@H task in years.

Looks like more Rosetta 4.2 work.
Received 4 west_nile tasks.

And it looks like that batch is done.
Back to Python only again (apart from the odd RB Task or resends).

---

Grant
Darwin NT

I give up on trying to get Python
Your app_config.xml file refers to an unknown application 'rosetta_python_projects'. Known applications: 'rosetta'

That's from that other guy's script in the python thread.
If they can't keep 4.2 running then I'm dumping this project after 15 years of crunching.

I give up on trying to get Python
Your app_config.xml file refers to an unknown application 'rosetta_python_projects'. Known applications: 'rosetta'

That's from that other guy's script in the python thread.
If they can't keep 4.2 running then I'm dumping this project after 15 years of crunching.

That message is because you haven't received a Python task yet.

---