Memory Usage in Beta 5.80

Message boards : Number crunching : Memory Usage in Beta 5.80

To post messages, you must log in.

AuthorMessage
DJStarfox

Send message
Joined: 19 Jul 07
Posts: 145
Credit: 1,235,310
RAC: 70
Message 46265 - Posted: 15 Sep 2007, 13:46:03 UTC

As I posted here:
https://boinc.bakerlab.org/rosetta/forum_thread.php?id=3552&nowrap=true#46202

And another user posted here:
https://boinc.bakerlab.org/rosetta/forum_thread.php?id=3552&nowrap=true#46263

Evidently, the Rosetta Beta 5.80 consumes significantly more RAM while running than previous version 5.79.
1) Has anyone experienced this?
2) Does anyone know what the requirements increase is?
3) Would any of the programmers be willing to optimize the application a little more to decrease the memory requirement back to 200M per instance?
ID: 46265 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Michael B
Avatar

Send message
Joined: 13 Feb 06
Posts: 19
Credit: 306,566
RAC: 0
Message 46278 - Posted: 15 Sep 2007, 15:57:46 UTC


I've noticed that each WU is consuming 250 megs of ram.

So I have 500 megs of ram in use just for Rosetta.


Wow.
ID: 46278 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
The_Bad_Penguin
Avatar

Send message
Joined: 5 Jun 06
Posts: 2751
Credit: 4,271,025
RAC: 0
Message 46289 - Posted: 15 Sep 2007, 17:43:16 UTC
Last modified: 15 Sep 2007, 17:46:50 UTC

I'm running a quadcore, with four cores of Rosetta.

From TaskManager, I can see that 3 cores of Rosetta are using about 75MB each, and 1 core is using about 250 MB.

From BoincManager, I can see that I am running:
3 wu's: 1g4u_boinc_minimize2_score12_capri14_dock_fixbackbone...
1 wu: 1gu4u_boinc_capri14_dock_fixbackbone_pose_loops...

Ergo, I am "assuming" from my Sesame Street days that "one of these things is not like the others", and that it is the "pose_loops" which is utilizing about three times the memory as the "score12" wu's...

FWIW...
ID: 46289 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Mod.Sense
Volunteer moderator

Send message
Joined: 22 Aug 06
Posts: 4018
Credit: 0
RAC: 0
Message 46292 - Posted: 15 Sep 2007, 17:55:35 UTC

The project has always had two potential WU memory sizes. They only send WUs that have high memory requirements to machines that have more then about 500MB of memory. So, what you MAY be observing is simply that they have released a batch of high memory tasks. And so this would not be a change, nor a bug, nor anything that needs to be optimized. It is simply a type of memory intensive work that the project wants to study further.

Having said that, I do not know for certain if these WUs were flagged as "high memory required" or not. So, I'm just trying to inform you that it seems normal. If someone with a 256MB machine told me they had a task consuming 200+MB in task manager, then it would indicate the WUs may not have been properly tagged to only go to high memory systems.
Rosetta Moderator: Mod.Sense
ID: 46292 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Michael B
Avatar

Send message
Joined: 13 Feb 06
Posts: 19
Credit: 306,566
RAC: 0
Message 46300 - Posted: 15 Sep 2007, 19:20:18 UTC - in response to Message 46292.  

The project has always had two potential WU memory sizes. They only send WUs that have high memory requirements to machines that have more then about 500MB of memory. So, what you MAY be observing is simply that they have released a batch of high memory tasks. And so this would not be a change, nor a bug, nor anything that needs to be optimized. It is simply a type of memory intensive work that the project wants to study further.

Having said that, I do not know for certain if these WUs were flagged as "high memory required" or not. So, I'm just trying to inform you that it seems normal. If someone with a 256MB machine told me they had a task consuming 200+MB in task manager, then it would indicate the WUs may not have been properly tagged to only go to high memory systems.


That answer works for me :0 I don't mind that much ram in use...I gots meh plenty ;P

thanks :)

ID: 46300 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Adam Gajdacs (Mr. Fusion)

Send message
Joined: 26 Nov 05
Posts: 13
Credit: 2,316,967
RAC: 3,032
Message 46348 - Posted: 16 Sep 2007, 12:18:00 UTC
Last modified: 16 Sep 2007, 12:24:02 UTC

There is a recurring problem however in connection with multi-core systems and these "high memory requirement" WUs, which would probably need some co-operation between RAH and the folks at BOINC to resolve by modifying the scheduler logic and/or introducing more WU flags regarding expectable memory use.

I'm talking about the situation where a Rosetta WU runs on one core, and another should be started on the other core, but the combined memory use hits the memory boundary set by user preferences for the project, so it's put into the "Waiting for memory" or simply the "Waiting to run" status, then the scheduler starts another Rosetta WU which happens to be also a high memory use one, it hits the memory use cap too, put into the "Waiting for memory" status, so a third needs to be started, and so on.

Right now I have 3 Rosetta WUs in memory, two using 370MB VM each and a third with 310MB, but only one is able to run because of memory preferences, the other two are just sort of deadlocking each other as they're keeping their committed VM in use without being able to actually run so that one of them could be cleared from memory once finished and then the other one get a chance to get processed too.

I wonder if there would be a way for the project or the scheduler to check the following conditions:
- Does the client asking for new work have any high-memory WUs assigned already?
- Is it a multi-CPU system with more than one CPUs enabled for BOINC?
- Would the combined memory use for "CPUs allowed to be used" x mem requirement of one WU be higher than the allowed memory use?

If they are true, then only low-memory WUs should be sent/requested/accepted as new work to that client for the project as long as it didn't return the current high-memory one to stop the scheduler from keep starting and then suspending high-memory WUs (since it has nothing else to process for this specific project), filling up available memory with "zombie" WUs that can not run because of each other.
ID: 46348 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile (_KoDAk_)

Send message
Joined: 18 Jul 06
Posts: 109
Credit: 1,859,263
RAC: 0
Message 46360 - Posted: 16 Sep 2007, 13:32:14 UTC

https://boinc.bakerlab.org/rosetta/result.php?resultid=105599697
uplaod = 14 MB

OMG !!!!!!!!!!!!!!
ID: 46360 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile (_KoDAk_)

Send message
Joined: 18 Jul 06
Posts: 109
Credit: 1,859,263
RAC: 0
Message 46363 - Posted: 16 Sep 2007, 14:19:07 UTC

https://boinc.bakerlab.org/rosetta/result.php?resultid=105599697
Upload = 14 Mb
OMG !!!!!!!!
ID: 46363 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Christoph

Send message
Joined: 10 Dec 05
Posts: 57
Credit: 1,512,386
RAC: 0
Message 46366 - Posted: 16 Sep 2007, 15:03:50 UTC - in response to Message 46363.  
Last modified: 16 Sep 2007, 15:04:27 UTC

https://boinc.bakerlab.org/rosetta/result.php?resultid=105599697
Upload = 14 Mb
OMG !!!!!!!!

Weird. This one created 1630 models and he got more than 560 credits for 1 workunit!
ID: 46366 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Jim

Send message
Joined: 15 Oct 06
Posts: 22
Credit: 5,410,546
RAC: 0
Message 46367 - Posted: 16 Sep 2007, 15:33:48 UTC - in response to Message 46348.  

There is a recurring problem however in connection with multi-core systems and these "high memory requirement" WUs, which would probably need some co-operation between RAH and the folks at BOINC to resolve by modifying the scheduler logic and/or introducing more WU flags regarding expectable memory use.

I'm talking about the situation where a Rosetta WU runs on one core, and another should be started on the other core, but the combined memory use hits the memory boundary set by user preferences for the project, so it's put into the "Waiting for memory" or simply the "Waiting to run" status, then the scheduler starts another Rosetta WU which happens to be also a high memory use one, it hits the memory use cap too, put into the "Waiting for memory" status, so a third needs to be started, and so on.


I have experienced the same condition you described above on my single core Win OS machines. To solve this problem I have installed a memory management tool and set it to scrub the memory every 15 minutes.
The scrub option frees up unused memory and returns it to the main memory pool so the Win OS can use it. I don't have to run this tool on the Linux box since that OS doesn't leak memory like the Win type OS does.
The utility I use is called Memturbo, (I have no interest or anything else to do with the company or product, I just use it.) I don't know if it will work with a dual or quad core machine. The bottom line is that I have not had this problem or any WU failures since I done this over 24 hours ago.
For example on my Intel based P4 2.8Ghz machine I have about 486 Meg of usable RAM. It would drop to 5 Meg of RAM free in about 30 minutes. With the memory being scrubbed every 15 minutes the free RAM figure now sets at over 100 Meg. That worked, no more "waiting or memory" messages.

Good luck, hope this helps.

Right now I have 3 Rosetta WUs in memory, two using 370MB VM each and a third with 310MB, but only one is able to run because of memory preferences, the other two are just sort of deadlocking each other as they're keeping their committed VM in use without being able to actually run so that one of them could be cleared from memory once finished and then the other one get a chance to get processed too.

I wonder if there would be a way for the project or the scheduler to check the following conditions:
- Does the client asking for new work have any high-memory WUs assigned already?
- Is it a multi-CPU system with more than one CPUs enabled for BOINC?
- Would the combined memory use for "CPUs allowed to be used" x mem requirement of one WU be higher than the allowed memory use?

If they are true, then only low-memory WUs should be sent/requested/accepted as new work to that client for the project as long as it didn't return the current high-memory one to stop the scheduler from keep starting and then suspending high-memory WUs (since it has nothing else to process for this specific project), filling up available memory with "zombie" WUs that can not run because of each other.

ID: 46367 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
DJStarfox

Send message
Joined: 19 Jul 07
Posts: 145
Credit: 1,235,310
RAC: 70
Message 46368 - Posted: 16 Sep 2007, 15:37:43 UTC - in response to Message 46292.  

The project has always had two potential WU memory sizes. They only send WUs that have high memory requirements to machines that have more then about 500MB of memory. So, what you MAY be observing is simply that they have released a batch of high memory tasks. And so this would not be a change, nor a bug, nor anything that needs to be optimized. It is simply a type of memory intensive work that the project wants to study further.

Having said that, I do not know for certain if these WUs were flagged as "high memory required" or not. So, I'm just trying to inform you that it seems normal. If someone with a 256MB machine told me they had a task consuming 200+MB in task manager, then it would indicate the WUs may not have been properly tagged to only go to high memory systems.


Adam brings up a good point:
https://boinc.bakerlab.org/rosetta/forum_thread.php?id=3564&nowrap=true#46348

The real trouble started when R@H app needed more than 256MB RAM. There's a natural multiplier relationship between 256 and number of cores. Machines with 512MB RAM (2 cores or more) or 1GB RAM (4 cores or more) are probably the hardest hit by the memory requirement change. These systems may have to run R@H partially out of swap space (to use all cores), which would be very slow.

If it works, then it's a very useful feature to only assign these particular class of WU to high memory machines.

HOWEVER...
R@H staff must be very careful when adjusting the "high memory system" definition, as this not only affects users' performance but also how many of these WU get processed per day (due to limited resources). I would say, conservatively, if a WU needs 300MB of resident memory, then the target client must have at least 600MB of RAM. So, the formula would be: WU size*2 = RAM available on target machine in order to be issued the high-memory WU.
ID: 46368 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Paul

Send message
Joined: 29 Oct 05
Posts: 193
Credit: 65,736,681
RAC: 2,734
Message 46388 - Posted: 16 Sep 2007, 20:51:14 UTC

We discussed similar problems in the: Problems with Rosetta version 5.80 thread

At present, the configuration most affected is: Windows XP with Quad Core.

It would appear that large memory work units on multi-core systems require massive amounts of memory. Even with 2GB of RAM and 2GB of swap, my Q6600 system returns failures either due to not enough memory or due to watchdog timers. The watchdog timers may be related to the XP memory manager.

Can any of you verify if your configs fall into the XP and multiCore or quad core config?

We also noticed that it happens more if you are doing 100% R@H and no other projects.

thx
Thx!

Paul

ID: 46388 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
DJStarfox

Send message
Joined: 19 Jul 07
Posts: 145
Credit: 1,235,310
RAC: 70
Message 46403 - Posted: 17 Sep 2007, 2:36:32 UTC - in response to Message 46388.  

Can any of you verify if your configs fall into the XP and multiCore or quad core config?


I first discovered it on my dual CPU Linux system w/ 512MB RAM. This system is only running R@H.
ID: 46403 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [AF>DoJ] supersonic

Send message
Joined: 12 Dec 05
Posts: 2
Credit: 4,025,201
RAC: 0
Message 47195 - Posted: 29 Sep 2007, 8:42:13 UTC


Hello to everyone,
I'm using BOINC manager 5.10.20 and my units Rosetta Beta 5.80 are each using almost 200 Mo of RAM.

The problem is that I have a Quad Core, therefore Rosetta is using permanantly 800 Mo of RAM, wich is almost half of my 2 Go...

Is there a way to 'ask' for smaller WUs ?

Because when I launch heavy applications such as Virtual PC, I need to turn BOINC down.

Thanx
ID: 47195 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Luuklag

Send message
Joined: 13 Sep 07
Posts: 262
Credit: 4,171
RAC: 0
Message 47212 - Posted: 29 Sep 2007, 18:52:20 UTC - in response to Message 47195.  


Hello to everyone,
I'm using BOINC manager 5.10.20 and my units Rosetta Beta 5.80 are each using almost 200 Mo of RAM.

The problem is that I have a Quad Core, therefore Rosetta is using permanantly 800 Mo of RAM, wich is almost half of my 2 Go...

Is there a way to 'ask' for smaller WUs ?

Because when I launch heavy applications such as Virtual PC, I need to turn BOINC down.

Thanx



if you set your preferences to use only 3 cores, and use max xxx Mo (Mo is the wierd frence word for Mb) boinc and R@h should see that you dont have that much memmory available, so you shouldn't get the WU flagged as "only for high memmory"

Luuklag
ID: 47212 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Mod.Sense
Volunteer moderator

Send message
Joined: 22 Aug 06
Posts: 4018
Credit: 0
RAC: 0
Message 47255 - Posted: 1 Oct 2007, 3:12:56 UTC

If you also work with other projects, and limit BOINC's memory usage, then it will figure out that at least one core needs to be doing work for a project that uses less memory. It might attempt to run a fourth Rosetta task and then cross the memory threshold, and then it will mark it as "waiting for memory". It might then attempt to fire up another Rosetta task if that is where the debts stack up and matches your resource share. Since that next Rosetta task might be a low memory task, that might work out fine for you. If not, BOINC will hold it once your threshold is crossed.

So you could achieve the desired result by limiting memory only, and not limiting number of CPUs. That would leave another CPU open for a low memory Rosetta task, or another low memory BOINC project.

Unfortunately, the direct answer to your question is "no", you cannot change your machine's profile to make the project think you have less memory, nor designate a preference for low memory tasks. I believe the high memory tasks are sent based strictly on the existence of the memory. So, even if you configured BOINC to only use 40% of memory, even when idle, I don't think it is sophisticated enough to avoid sending you the high memory tasks.

Overall, the high memory tasks are a fairly small percentage of Rosetta work. But when they do send high memory tasks, yes, the high memory machines tend to see a batch of them all at once.
Rosetta Moderator: Mod.Sense
ID: 47255 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Ingleside

Send message
Joined: 25 Sep 05
Posts: 107
Credit: 1,514,472
RAC: 0
Message 47294 - Posted: 1 Oct 2007, 20:58:03 UTC - in response to Message 47255.  

Unfortunately, the direct answer to your question is "no", you cannot change your machine's profile to make the project think you have less memory, nor designate a preference for low memory tasks. I believe the high memory tasks are sent based strictly on the existence of the memory. So, even if you configured BOINC to only use 40% of memory, even when idle, I don't think it is sophisticated enough to avoid sending you the high memory tasks.

The Scheduling-server has a little bit of logic, there it uses the preference so if example a wu is marked of needing 512 MB and your preference says only allowed to use 500 MB, you won't get the wu.

But, overall the Scheduling-server is stuped as far as memory goes, since it won't take into account multi-cpu, and it will happily assign 4x 499 MB-wu's to a quad-core.

This would leave it up to BOINC-client halting the tasks, then total memory-usage exceeds 500 MB, potentially meaning only 1 of the cores can run, while no usable wu's for the 3 other cores if not attached to another project.

"I make so many mistakes. But then just think of all the mistakes I don't make, although I might."
ID: 47294 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote

Message boards : Number crunching : Memory Usage in Beta 5.80



©2024 University of Washington
https://www.bakerlab.org