How might memory effect R@h processing?

Message boards : Number crunching : How might memory effect R@h processing?

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Stephen "Heretic"

Send message
Joined: 2 Apr 20
Posts: 21
Credit: 11,028
RAC: 0
Message 93444 - Posted: 5 Apr 2020, 0:18:20 UTC

. . I finally have some completed tasks and I delved into the result files for information.

. . The first two results had the following errors ...

    - Unhandled Exception Record -
    Reason: Access Violation (0xc0000005) at address 0x019F2BE0 write attempt to address 0x017D7EC9

    - Unhandled Exception Record -
    Reason: Access Violation (0xc0000005) at address 0x01AC2BE0 write attempt to address 0x017D7EC9



. . While different source addresses both were to the same destination address. Both ran debug twice but I cannot make sense of the results, I am not a programmer. Anyone able to suggest what the cause might be? Do I need to fix anything, should I be worried by this? The third completed task did not have any errors.

Stephen

?

ID: 93444 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
MarkJ

Send message
Joined: 28 Mar 20
Posts: 72
Credit: 25,010,478
RAC: 383
Message 93447 - Posted: 5 Apr 2020, 1:10:27 UTC - in response to Message 93444.  

. . I finally have some completed tasks and I delved into the result files for information.

. . The first two results had the following errors ...

    - Unhandled Exception Record -
    Reason: Access Violation (0xc0000005) at address 0x019F2BE0 write attempt to address 0x017D7EC9

    - Unhandled Exception Record -
    Reason: Access Violation (0xc0000005) at address 0x01AC2BE0 write attempt to address 0x017D7EC9


You’d better run memtest86+ on it. If it’s overclocked then reset to factory defaults.

I had one machine that did that last week. Never had a problem with it before. I ran memtest86+ on it. Narrowed it down to a pair of memory sticks (the i7 in question had 4 x 8GB sticks). I’ve since replaced all 4 sticks and sent the dud pair back under warranty.
BOINC blog
ID: 93447 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Stephen "Heretic"

Send message
Joined: 2 Apr 20
Posts: 21
Credit: 11,028
RAC: 0
Message 93450 - Posted: 5 Apr 2020, 1:34:14 UTC - in response to Message 93447.  

You’d better run memtest86+ on it. If it’s overclocked then reset to factory defaults.
I had one machine that did that last week. Never had a problem with it before. I ran memtest86+ on it. Narrowed it down to a pair of memory sticks (the i7 in question had 4 x 8GB sticks). I’ve since replaced all 4 sticks and sent the dud pair back under warranty.


. . I never overclock, I'm not that adventurous. But it sounds like a memtest session might be a good idea. If it fails I might have to fork out some dollars for a pair of 8GB sticks, it is running just one x 8GB at the moment.

. . Thanks

Stephen

. .
ID: 93450 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Grant (SSSF)

Send message
Joined: 28 Mar 20
Posts: 1467
Credit: 14,334,313
RAC: 16,402
Message 93451 - Posted: 5 Apr 2020, 1:40:01 UTC - in response to Message 93450.  

it is running just one x 8GB at the moment.
That would slow the whole system down quite a bit with anything requiring access to system memory as it wouldn't be able to use Dual Channel mode with only the 1 module.
Dual channel systems, always use modules in pairs, Quad channel, always use them in groups of 4, etc.
Grant
Darwin NT
ID: 93451 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Stephen "Heretic"

Send message
Joined: 2 Apr 20
Posts: 21
Credit: 11,028
RAC: 0
Message 93454 - Posted: 5 Apr 2020, 2:37:25 UTC - in response to Message 93451.  

. . Thanks Grant but I did know that. This system is using a low end MSI MoB with only 2 memory slots, to be honest I would have to do research to even be sure it can do dual channel mode, though I expect it does. But if I have to replace the one stick it came with I would go all out and get a pair.

Stephen

:)
ID: 93454 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile robertmiles

Send message
Joined: 16 Jun 08
Posts: 1223
Credit: 13,824,497
RAC: 2,340
Message 93455 - Posted: 5 Apr 2020, 2:38:08 UTC - in response to Message 93450.  

You’d better run memtest86+ on it. If it’s overclocked then reset to factory defaults.
I had one machine that did that last week. Never had a problem with it before. I ran memtest86+ on it. Narrowed it down to a pair of memory sticks (the i7 in question had 4 x 8GB sticks). I’ve since replaced all 4 sticks and sent the dud pair back under warranty.


. . I never overclock, I'm not that adventurous. But it sounds like a memtest session might be a good idea. If it fails I might have to fork out some dollars for a pair of 8GB sticks, it is running just one x 8GB at the moment.

. . Thanks

Stephen

. .

Having two memory sticks instead of just one tends to increase the speed of programs somewhat. Even more with four, if your motherboard can handle four properly.
ID: 93455 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Klimax

Send message
Joined: 27 Apr 07
Posts: 38
Credit: 2,453,992
RAC: 0
Message 93463 - Posted: 5 Apr 2020, 7:41:40 UTC - in response to Message 93381.  

[snip]

Yes, smt/hyper threading is off. The point is though, on my Intel laptops, the speed has remained the same between 4.07 to 4.12. On my tr it's dropped over 60% regardless of what I try, I can't put it much simpler than that.

Note that all of your Intel laptops have a much lower number of cores, and will therefore have much less of a problem with too many cores trying to share the limited speed path to main memory. I can't put it much simpler than that, either.



4.07 is the reference yes. 4.12 is 60% slower than that, I don't know what is so difficult to understand, it's has nothing to do with memory or anything else. You dont have a 64/128 chip to only run it on 5 cores to keep the same productivity because of a software change. Is there a mod/developer that can possibly comment on this issue?

Resource contention, nothing less, nothing more. Nothing exotic or unknowable. ZEN architecture is strong on memory bandwidth but bad on memory latency. Even if you have relatively larger cache there will be still lots of memory access that has to be fulfilled from main memory. Because of that too many core doing access will overload memory bus and their performance will drop of sharply. (Especially when latency of memory access is already high)

I suggest you experiment with number of concurrent task for RAH to find equilibrium. Alternately, you could try to see if you can get memory to higher frequency without worsening its timing. Also ensure that there is no swapping. (IIRC each 4.12 task needs about 2GB of RAM which amounts to about 128GB of RAM in use beside all other processes already running)
ID: 93463 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jim1348

Send message
Joined: 19 Jan 06
Posts: 881
Credit: 52,257,545
RAC: 0
Message 93469 - Posted: 5 Apr 2020, 8:44:02 UTC - in response to Message 93358.  
Last modified: 5 Apr 2020, 9:06:20 UTC

4.12 is running very poorly on AMD.

I get in on the middle of this discussion, but how are you measuring that?
My Ryzens (2600, 3600, 3700X) are running fine. The scores on 4.12 are at least as good as 4.07 and 4.08.

And on the Ryzen 3600, the stderr_txt shows exactly the same "Device peak FLOPS 5.95 GFLOPS" for both 4.07 and 4.12, for example.
For the Ryzen 2600, it is 5.05 GFLOPS in both cases.

It is about the same on my i7-8700. Whatever the problem, I don't think it is due to the Ryzens.

EDIT: I have all machines running all cores on Ryzens, reserving one for a GPU (at least until the recent shortage; most of them still are).
So if it were a loading effect, it would show up.

EDIT 2: Your Threadripper handles cache differently than the Ryzens. That may have something to do with what you are seeing.
ID: 93469 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
strongboes

Send message
Joined: 3 Mar 20
Posts: 27
Credit: 5,394,270
RAC: 0
Message 93472 - Posted: 5 Apr 2020, 9:31:27 UTC - in response to Message 93469.  
Last modified: 5 Apr 2020, 9:32:30 UTC

I'm making the opinion from a number of factors, but a hard figure is the average processing rate which is found on the computer details/application details page. 4.07 was regularly hitting 30 on the covid design tasks, 4.12 and 11.5 is the limit.

4.07 I was averaging 1 credit per core, per average 11.5 seconds compute time. 4.12 is running at something like 40-50 seconds average for 1 credit, the best I've noted was around 34 secs.

I don't know if it's an accurate number to go on because you can run 100 tasks with the same name, 30 of them will run 10 decoys (for example) finish within 0.5% of time of each other, and the credit awarded can vary by 300 points, some will get 50, others rarely nearly 400.

I'm assuming most people don't pay as much attention to these numbers as I have done and so haven't noticed.
ID: 93472 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Mod.Sense
Volunteer moderator

Send message
Joined: 22 Aug 06
Posts: 4018
Credit: 0
RAC: 0
Message 93507 - Posted: 5 Apr 2020, 15:50:21 UTC - in response to Message 93472.  
Last modified: 5 Apr 2020, 15:51:11 UTC

I suspect there are tasks that are hitting the watchdog timeout (I seem to see a few every time I look at someone's hosts and WUs), and that this may explain poor credit/hour reports. The irregular granted credit then rolls up in to your other summary numbers. So, without something more directly comparable, I wouldn't take these as symptoms specific to any particular CPU type.
Rosetta Moderator: Mod.Sense
ID: 93507 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Keith Myers
Avatar

Send message
Joined: 29 Mar 20
Posts: 95
Credit: 289,903
RAC: 0
Message 93508 - Posted: 5 Apr 2020, 15:55:37 UTC - in response to Message 93469.  


It is about the same on my i7-8700. Whatever the problem, I don't think it is due to the Ryzens.

EDIT: I have all machines running all cores on Ryzens, reserving one for a GPU (at least until the recent shortage; most of them still are).
So if it were a loading effect, it would show up.

EDIT 2: Your Threadripper handles cache differently than the Ryzens. That may have something to do with what you are seeing.

It is a problem on my 3950X and 2920X AMD processors. I barely cracked 3.0 GFLOPS on my cpus with 12 cores running.
The 4.07 app was faster than the 4.08 app. I would have loved if the performance had reached the 5 GFLOPS rating you are getting on your Ryzen 7's.
ID: 93508 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jim1348

Send message
Joined: 19 Jan 06
Posts: 881
Credit: 52,257,545
RAC: 0
Message 93521 - Posted: 5 Apr 2020, 17:13:17 UTC - in response to Message 93508.  
Last modified: 5 Apr 2020, 17:23:43 UTC

It is a problem on my 3950X and 2920X AMD processors. I barely cracked 3.0 GFLOPS on my cpus with 12 cores running.

Interesting. I just happen to have a new Ryzen 3950X which I use on Folding as of yesterday. It is doing great, with around 550 k PPD on the new Covid-19s (Ubuntu 18.04.4).
But it has only 16 GB of memory (though fast 3200 MHz 14-14-14-31), and so it is not suitable for Rosetta. I guess I lucked out on that one.

EDIT: That is on 30 out of 32 threads. I reserve two for a GTX 1070, which adds another 850 k PPD.
ID: 93521 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Keith Myers
Avatar

Send message
Joined: 29 Mar 20
Posts: 95
Credit: 289,903
RAC: 0
Message 93542 - Posted: 5 Apr 2020, 19:00:26 UTC - in response to Message 93521.  

It is a problem on my 3950X and 2920X AMD processors. I barely cracked 3.0 GFLOPS on my cpus with 12 cores running.

Interesting. I just happen to have a new Ryzen 3950X which I use on Folding as of yesterday. It is doing great, with around 550 k PPD on the new Covid-19s (Ubuntu 18.04.4).
But it has only 16 GB of memory (though fast 3200 MHz 14-14-14-31), and so it is not suitable for Rosetta. I guess I lucked out on that one.

EDIT: That is on 30 out of 32 threads. I reserve two for a GTX 1070, which adds another 850 k PPD.

I froze the computer up solid through 3 reboots once I added Rosetta. Every time that BOINC started the computer froze with the disk access light solid. I didn't realize that the 22 cpu tasks I ran for SETI in %cpu usage was going to try and run 22 Rosetta tasks with just 16GB of memory installed.

I finally was able to stop BOINC from loading so I could manually edit the global and override preferences files to knock the number of cpu threads down to a value where the Rosetta startup wouldn't consume all my memory. I eventually stole memory from another SETI mothballed computer to add another 16GB to the computer I was running Rosetta on.
ID: 93542 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Stephen "Heretic"

Send message
Joined: 2 Apr 20
Posts: 21
Credit: 11,028
RAC: 0
Message 93581 - Posted: 5 Apr 2020, 23:57:55 UTC - in response to Message 93542.  
Last modified: 6 Apr 2020, 0:06:17 UTC

I froze the computer up solid through 3 reboots once I added Rosetta. Every time that BOINC started the computer froze with the disk access light solid. I didn't realize that the 22 cpu tasks I ran for SETI in %cpu usage was going to try and run 22 Rosetta tasks with just 16GB of memory installed.
I finally was able to stop BOINC from loading so I could manually edit the global and override preferences files to knock the number of cpu threads down to a value where the Rosetta startup wouldn't consume all my memory. I eventually stole memory from another SETI mothballed computer to add another 16GB to the computer I was running Rosetta on.


. . That is exactly the problem I have been suffering. I would like to run 2 x Rosetta tasks on my i5-6400 and support the GTX1050 running E@H but I only have 8 GB Ram. It is OK with the 32 bit tasks, I have 2 CPU cores committed to Rosetta and it mainly runs just one task, but sometimes there must be enough memory free and it will run 2. The good thing is that when memory demands increase it simply suspends one of the running Rosetta tasks and keeps on ticking, no harm, no foul. But when I have 64 bit tasks it goes to hell in a handbasket. At the moment I have it running OK by suspending the other 64bit Rosetta tasks so that they do NOT attempt to run. It works just fine like that but I have to check the machine more often than I should need to. I would love to know just how much ram is needed to keep 2 64 bit Rosetta tasks happy and still support the GPU. This MoBo only has 2 memory slots so I cannot go past 16GB even if I decide to invest more dollars in this project. I never had this problem running S@H. It ran 3 + 1 as smooth as silk. So in the end, with 32Gb Ram were you able to support all 22 tasks it was trying to run? Or did have to restrict it to any particular number?

Stephen

? ?
ID: 93581 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Grant (SSSF)

Send message
Joined: 28 Mar 20
Posts: 1467
Credit: 14,334,313
RAC: 16,402
Message 93582 - Posted: 6 Apr 2020, 0:07:04 UTC - in response to Message 93581.  
Last modified: 6 Apr 2020, 0:10:23 UTC

At the moment I have it running OK by suspending the other 64bit Rosetta tasks so that they do NOT attempt to run. It works just fine like that but I have to check the machine more often than I should need to. I would love to know just how much ram is needed to keep 2 64 bit Rosetta tasks happy and still support the GPU.
On my system, of 2 Tasks being processed by Rosetta v4.12 windows_x86_64, one is using 750MB of RAM, the other 320MB of RAM.
It's similar to CPU & GPU work on Seti- there are no CPU or GPU tasks, just tasks. Same with the Rosetta v4.12 windows_intelx86 and Rosetta v4.12 windows_x86_64 applications. The Task is allocated to the Application when it is downloaded.

The amount of required RAM depends on the Task, not the application being used.


This MoBo only has 2 memory slots so I cannot go past 16GB even if I decide to invest more dollars in this project.
It doesn't support 16GB DIMMs?
Grant
Darwin NT
ID: 93582 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Stephen "Heretic"

Send message
Joined: 2 Apr 20
Posts: 21
Credit: 11,028
RAC: 0
Message 93584 - Posted: 6 Apr 2020, 0:32:24 UTC - in response to Message 93582.  

I would love to know just how much ram is needed to keep 2 64 bit Rosetta tasks happy and still support the GPU.

On my system, of 2 Tasks being processed by Rosetta v4.12 windows_x86_64, one is using 750MB of RAM, the other 320MB of RAM.

. . Well at those levels I should be AOK as is, the E@H tasks do not require more than 2GB of system ram so there is 5 GB or more left for Rosetta. But it aint happening. How are you seeing how much system ram each task is using?

It's similar to CPU & GPU work on Seti- there are no CPU or GPU tasks, just tasks. Same with the Rosetta v4.12 windows_intelx86 and Rosetta v4.12 windows_x86_64 applications. The Task is allocated to the Application when it is downloaded.

. . Thanks Grant, but I did know that.

The amount of required RAM depends on the Task, not the application being used.

. . That may be so but it does not fit the pattern I am seeing, so if it is not a memory insufficiency then there is something about the 64 bit app that does NOT like my machine. I need a way to preclude the 64 bit app from running to confirm this over a larger sample of tasks.

This MoBo only has 2 memory slots so I cannot go past 16GB even if I decide to invest more dollars in this project.
It doesn't support 16GB DIMMs?

. . It is an old cheapy MSI MoBo. I have very little info but I am pretty sure when I bought it the max was 16GB.

Stephen

:(
ID: 93584 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Grant (SSSF)

Send message
Joined: 28 Mar 20
Posts: 1467
Credit: 14,334,313
RAC: 16,402
Message 93585 - Posted: 6 Apr 2020, 0:48:07 UTC - in response to Message 93584.  
Last modified: 6 Apr 2020, 0:49:20 UTC

How are you seeing how much system ram each task is using?
Task Manager, Processes tab, Memory column.
It can vary by several hundred MB depending on where in the Task it is at.
Although Process Explorer is probably better as it shows how much is also reserved, but not using at the time- it's values are higher than those shown in Task Manager.
A Task using 730MB in Task Manager shows as 760MB Private/ 783MB Working Set in Process Explorer.


This MoBo only has 2 memory slots so I cannot go past 16GB even if I decide to invest more dollars in this project.
It doesn't support 16GB DIMMs?
. . It is an old cheapy MSI MoBo. I have very little info but I am pretty sure when I bought it the max was 16GB.
Check the motherboard manual, it should list what capacity & speed DIMMs are supported.
Grant
Darwin NT
ID: 93585 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Tom M

Send message
Joined: 20 Jun 17
Posts: 85
Credit: 3,975,236
RAC: 48,790
Message 93587 - Posted: 6 Apr 2020, 1:03:17 UTC - in response to Message 93585.  

How are you seeing how much system ram each task is using?
Task Manager, Processes tab, Memory column.
It can vary by several hundred MB depending on where in the Task it is at.
Although Process Explorer is probably better as it shows how much is also reserved, but not using at the time- it's values are higher than those shown in Task Manager.
A Task using 730MB in Task Manager shows as 760MB Private/ 783MB Working Set in Process Explorer.


This MoBo only has 2 memory slots so I cannot go past 16GB even if I decide to invest more dollars in this project.
It doesn't support 16GB DIMMs?
. . It is an old cheapy MSI MoBo. I have very little info but I am pretty sure when I bought it the max was 16GB.
Check the motherboard manual, it should list what capacity & speed DIMMs are supported.


Some of the ram merchants have a ram tester that will recommend the highest level of ram that a MB will support.

If you don't have a lot of cpu cores/threads running Rosetti@Home you should be able to run multiples of the task even if it is running up near 2GB per task on a 16GB ram MB. Lately mine have all been a lot smaller.

Tom
Help, a tagline is missing..... Help, a tagline is......... Help, a........ Hel.....
ID: 93587 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Keith Myers
Avatar

Send message
Joined: 29 Mar 20
Posts: 95
Credit: 289,903
RAC: 0
Message 93591 - Posted: 6 Apr 2020, 1:30:29 UTC - in response to Message 93581.  

So in the end, with 32Gb Ram were you able to support all 22 tasks it was trying to run? Or did have to restrict it to any particular number?

I knocked it down to 16 x86_64 cpu tasks so I could run the three gpu tasks. If I tried to run more, I knocked gpu tasks offline with out of memory-postponed messages. That was with use 90% of memory while idle, 85% while active preferences. Was consuming a little shy of 30GB. The host is my daily driver.
ID: 93591 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile robertmiles

Send message
Joined: 16 Jun 08
Posts: 1223
Credit: 13,824,497
RAC: 2,340
Message 93592 - Posted: 6 Apr 2020, 1:31:03 UTC - in response to Message 93581.  
Last modified: 6 Apr 2020, 1:45:30 UTC

I froze the computer up solid through 3 reboots once I added Rosetta. Every time that BOINC started the computer froze with the disk access light solid. I didn't realize that the 22 cpu tasks I ran for SETI in %cpu usage was going to try and run 22 Rosetta tasks with just 16GB of memory installed.
I finally was able to stop BOINC from loading so I could manually edit the global and override preferences files to knock the number of cpu threads down to a value where the Rosetta startup wouldn't consume all my memory. I eventually stole memory from another SETI mothballed computer to add another 16GB to the computer I was running Rosetta on.


. . That is exactly the problem I have been suffering. I would like to run 2 x Rosetta tasks on my i5-6400 and support the GTX1050 running E@H but I only have 8 GB Ram. It is OK with the 32 bit tasks, I have 2 CPU cores committed to Rosetta and it mainly runs just one task, but sometimes there must be enough memory free and it will run 2. The good thing is that when memory demands increase it simply suspends one of the running Rosetta tasks and keeps on ticking, no harm, no foul. But when I have 64 bit tasks it goes to hell in a handbasket. At the moment I have it running OK by suspending the other 64bit Rosetta tasks so that they do NOT attempt to run. It works just fine like that but I have to check the machine more often than I should need to. I would love to know just how much ram is needed to keep 2 64 bit Rosetta tasks happy and still support the GPU. This MoBo only has 2 memory slots so I cannot go past 16GB even if I decide to invest more dollars in this project. I never had this problem running S@H. It ran 3 + 1 as smooth as silk. So in the end, with 32Gb Ram were you able to support all 22 tasks it was trying to run? Or did have to restrict it to any particular number?

Stephen

? ?

The Rosetta tasks seem to run best with 2 GB per task, plus more for the CPU portion of any GPU task, plus more for the operating system. For 8 GB, two Rosetta tasks running at once is about the limit. If you have enough virtual cores, 16 GB should raise the limit to about 6 tasks.

This assumes that you don't let any BOINC tasks stay in memory while they are not running. Expect lower numbers if you do.

Filling both slots with the same model of memory board gives a little more speed than mismatched memory boards. You might also check if you can get faster memory boards that are compatible with your computer.

Some models of computers are compatible with memory boards that contain more memory.

Back when I first started with BOINC, I used this site to find compatible types of memory:

https://www.crucial.com/store/systemscanner

I now order any new computers with the largest and fastest memories that are compatible with that model.
ID: 93592 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
1 · 2 · Next

Message boards : Number crunching : How might memory effect R@h processing?



©2024 University of Washington
https://www.bakerlab.org