New Memory Requirements?

Message boards : Number crunching : New Memory Requirements?

To post messages, you must log in.

AuthorMessage
Profile Chris Holvenstot
Avatar

Send message
Joined: 2 May 10
Posts: 220
Credit: 9,106,918
RAC: 0
Message 66610 - Posted: 19 Jun 2010, 11:40:24 UTC

Did we get a new batch of work with some higher memory requirements? Over the past day I have frequently seen tasks waiting for memory. I had not seen this before on my systems.

A sample task name would be: rb_06_15_205_770_rs_stg0_lrlxjcst_t000__casp9_SAVE_ALL_OUT_21438_15644_0

It wants about 800 meg (which on a six core system adds up quickly)

If this is going to be the new norm I need to run to Fry's ASAP

ID: 66610 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Murasaki
Avatar

Send message
Joined: 20 Apr 06
Posts: 303
Credit: 511,418
RAC: 0
Message 66612 - Posted: 19 Jun 2010, 14:15:40 UTC
Last modified: 19 Jun 2010, 14:16:40 UTC

Some of the targets we are being asked to calculate for the CASP9 competition have huge structures that require a great deal of memory. How many targets are going to be that big will probably not be known until the last target is released by the competition organisers.

There will still be lots of work with more reasonable memory requirements as part of the normal research activities, but I expect this won't be the last batch of high-memory work units we see.

If you can afford more memory and are willing to upgrade to help Rosetta then great. However I will be keeping my current configuration for now and just continue processing whatever Rosetta chooses to fire at me.
ID: 66612 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Bikermatt

Send message
Joined: 12 Feb 10
Posts: 20
Credit: 10,552,445
RAC: 0
Message 66613 - Posted: 19 Jun 2010, 14:44:13 UTC

Chris
I am playing with a new box I just built and it has been running win7 64 for the last 24 hours or so.

Task manager shows all of the rosetta work units vary between the high 200s to the low 400s for memory usage. The box is not running any of the casp tasks right now though and as Murasaki said, I remember reading some where that those tasks would be using more memory.

Anyway, the same box was running Linux for a few hours before I loaded win and I seem to remember quite a few of the tasks running up into the 600s, but I was not looking at task names so those could have been casp tasks.

The machine was running a lot better with Linux so I will be reloading this morning after it finishes up its last tasks. I will let you know for sure after it is reloaded.

Also, on boxes that I use for crunching only, I set the memory usage up to 90% for "in use" and "idle". I noticed sometimes that if a box was right at the limit it would start waiting for memory even when I was just checking the system out.

I also noticed on my six core box that when I added 3 GT240s for running GPU grid tasks it started waiting for memory when trying to run nine tasks at once (duh). I had to bump it up to eight GB it it now will run all nine tasks without issue.

Four GB for six cores had been working well for me on my six core box so went I built this most recent machine I planned on 4 GB for every six cores.

I'm planning on building another 24 core box at the end of this summer and with more tasks out there that use higher memory I think I will just go for a GB per core from now on.

It seems like quite often a machine will download quite a few of the same types of tasks in a row. I think this has to do with how the tasks are generated.

I have noticed quite often when one of my machines has to download a lot of tasks at the same time, they are quite often the same type of task.

Anyway, if you get a whole batch of the higher memory requiring tasks less than one GB per core might not cut it anymore.
ID: 66613 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Chris Holvenstot
Avatar

Send message
Joined: 2 May 10
Posts: 220
Credit: 9,106,918
RAC: 0
Message 66615 - Posted: 21 Jun 2010, 0:05:33 UTC

Matt - of course Linux runs batter <sly grin>
ID: 66615 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
mikey
Avatar

Send message
Joined: 5 Jan 06
Posts: 1894
Credit: 8,769,835
RAC: 5,482
Message 66621 - Posted: 21 Jun 2010, 10:53:32 UTC - in response to Message 66613.  

Also, on boxes that I use for crunching only, I set the memory usage up to 90% for "in use" and "idle". I noticed sometimes that if a box was right at the limit it would start waiting for memory even when I was just checking the system out.

I also noticed on my six core box that when I added 3 GT240s for running GPU grid tasks it started waiting for memory when trying to run nine tasks at once (duh). I had to bump it up to eight GB it it now will run all nine tasks without issue.


The newer version of Boinc have a setting to turn this off, go to the Boinc Manager, Advanced, Preferences, processor usage and change the setting 'while processor usage is less than' to zero, put the number not the word in there. By default this is set to 25, and if your pc does anything that uses the processor Boinc will not crunch until you have more free time. It will say 'waiting for memory' if this is a problem, it is for most of us. This is a setting for those that want to do something else while they crunch and Boinc is not backing off enough.
ID: 66621 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
mikey
Avatar

Send message
Joined: 5 Jan 06
Posts: 1894
Credit: 8,769,835
RAC: 5,482
Message 66622 - Posted: 21 Jun 2010, 10:54:24 UTC - in response to Message 66615.  

Matt - of course Linux runs batter <sly grin>


You mean it makes brownies too?!
ID: 66622 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Bikermatt

Send message
Joined: 12 Feb 10
Posts: 20
Credit: 10,552,445
RAC: 0
Message 66624 - Posted: 21 Jun 2010, 15:12:15 UTC

I just found the 24 core box running 5 casp 9 tasks out of 24. They were all using around 600MB per task so I must have seen a few of them running the other day.

The Box is using 13GB right now, it runs around 10 GB with no casp tasks. I will definitely go with one GB per core from now on.

Linux is really running great on the box also. I have actually just started using Linux in the last few months.

I tinkered a little bit with it eight years ago when I was a computer tech but never really got into it. Today I have one xp and two win7 licenses,as far as I am concerned I will never never need to buy a windows license again.
ID: 66624 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Matthew Love
Avatar

Send message
Joined: 24 Nov 05
Posts: 73
Credit: 44,948
RAC: 0
Message 66631 - Posted: 21 Jun 2010, 21:35:56 UTC - in response to Message 66621.  

Also, on boxes that I use for crunching only, I set the memory usage up to 90% for "in use" and "idle". I noticed sometimes that if a box was right at the limit it would start waiting for memory even when I was just checking the system out.

I also noticed on my six core box that when I added 3 GT240s for running GPU grid tasks it started waiting for memory when trying to run nine tasks at once (duh). I had to bump it up to eight GB it it now will run all nine tasks without issue.


The newer version of Boinc have a setting to turn this off, go to the Boinc Manager, Advanced, Preferences, processor usage and change the setting 'while processor usage is less than' to zero, put the number not the word in there. By default this is set to 25, and if your pc does anything that uses the processor Boinc will not crunch until you have more free time. It will say 'waiting for memory' if this is a problem, it is for most of us. This is a setting for those that want to do something else while they crunch and Boinc is not backing off enough.


What reason did they give for adding this feature?
ID: 66631 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
mikey
Avatar

Send message
Joined: 5 Jan 06
Posts: 1894
Credit: 8,769,835
RAC: 5,482
Message 66635 - Posted: 22 Jun 2010, 11:30:41 UTC - in response to Message 66631.  

Also, on boxes that I use for crunching only, I set the memory usage up to 90% for "in use" and "idle". I noticed sometimes that if a box was right at the limit it would start waiting for memory even when I was just checking the system out.

I also noticed on my six core box that when I added 3 GT240s for running GPU grid tasks it started waiting for memory when trying to run nine tasks at once (duh). I had to bump it up to eight GB it it now will run all nine tasks without issue.


The newer version of Boinc have a setting to turn this off, go to the Boinc Manager, Advanced, Preferences, processor usage and change the setting 'while processor usage is less than' to zero, put the number not the word in there. By default this is set to 25, and if your pc does anything that uses the processor Boinc will not crunch until you have more free time. It will say 'waiting for memory' if this is a problem, it is for most of us. This is a setting for those that want to do something else while they crunch and Boinc is not backing off enough.


What reason did they give for adding this feature?


Dr. A does not have to give answers and normally does not to us peons, we are not worthy. The Boinc Developers mailing list sometimes gets answers but not always. Dr A has been doing Boinc for a long time and has developed some notions that not everyone else agrees with, but Boinc runs and for the most part works just fine. A guy named Rom, also a programmer and works for Dr. A, is MUCH more communicative but takes his orders from Dr. A, so doesn't always get his way. He had been on the boards before but usually leaves after ending up trying to defend Dr. A's decisions. How can anyone defend what their boss is thinking when the boss doesn't always include you in those thoughts?! People on the boards don't always understand that and it has gotten ugly before.

Anyway back on topic, has anyone had any more problems with memory issues since going to version 6.10.56? Has anyone used the next Beta version 6.10.57? Does it give the same memory issues?
ID: 66635 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Chris Holvenstot
Avatar

Send message
Joined: 2 May 10
Posts: 220
Credit: 9,106,918
RAC: 0
Message 66641 - Posted: 22 Jun 2010, 21:12:17 UTC - in response to Message 66635.  


Anyway back on topic, has anyone had any more problems with memory issues since going to version 6.10.56? Has anyone used the next Beta version 6.10.57? Does it give the same memory issues?


The rotund guy in the back of the room raises his hand ...

When I initially asked the question I was running 6.10.56. Have not seen this repeat on the system I bumped up to 8 gig. I have the memory sicks to do another tonight. Maybe tomorrow if I come down with a terminal case of lazyness.

I am going to guess that a good rule of thumb is going to be "install 1 gig per core"

Thanks for giving me some insight into what you have seen.
ID: 66641 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
mikey
Avatar

Send message
Joined: 5 Jan 06
Posts: 1894
Credit: 8,769,835
RAC: 5,482
Message 66652 - Posted: 23 Jun 2010, 12:07:31 UTC - in response to Message 66641.  


Anyway back on topic, has anyone had any more problems with memory issues since going to version 6.10.56? Has anyone used the next Beta version 6.10.57? Does it give the same memory issues?


The rotund guy in the back of the room raises his hand ...

When I initially asked the question I was running 6.10.56. Have not seen this repeat on the system I bumped up to 8 gig. I have the memory sicks to do another tonight. Maybe tomorrow if I come down with a terminal case of lazyness.

I am going to guess that a good rule of thumb is going to be "install 1 gig per core"

Thanks for giving me some insight into what you have seen.


I think that is a good rule of thumb for Windows especially and probably Linux just to make it easier. Linux handles less memory MUCH better than Windows but that means it may handle more memory better too.
ID: 66652 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Mod.Sense
Volunteer moderator

Send message
Joined: 22 Aug 06
Posts: 4018
Credit: 0
RAC: 0
Message 66655 - Posted: 23 Jun 2010, 16:50:11 UTC

I would say that sounds like a good guideline, regardless of whether Rosetta is being run. And I would also say that while the formal requirements have not changed, and the actual usage varies, that should be enough to assure all cores keep crunching even if trudging through an unfortunate combination of tasks.
Rosetta Moderator: Mod.Sense
ID: 66655 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
darwincollins

Send message
Joined: 1 Oct 09
Posts: 7
Credit: 5,586,679
RAC: 0
Message 66700 - Posted: 27 Jun 2010, 7:18:00 UTC - in response to Message 66624.  

I just found the 24 core box running 5 casp 9 tasks out of 24.


What type of box is it that it can handle 24 cores? Please describe. Thanks
ID: 66700 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile dcdc

Send message
Joined: 3 Nov 05
Posts: 1829
Credit: 116,047,707
RAC: 65,482
Message 66702 - Posted: 27 Jun 2010, 10:58:18 UTC - in response to Message 66700.  

I just found the 24 core box running 5 casp 9 tasks out of 24.


What type of box is it that it can handle 24 cores? Please describe. Thanks

I believe Bikermatt has a 2x12 core Opteron box...
ID: 66702 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote

Message boards : Number crunching : New Memory Requirements?



©2024 University of Washington
https://www.bakerlab.org