Hi folks, I was just monitoring one of my systems (XP, 5.4.9, P4 2.8GHz, 2GB RAM) as it was crunching away and noticed that my rosetta task was generating a lot of page faults. Over 7 million of them in the last 24 hours, far exceeding any other task. What could cause this? I have over 1.3GB of RAM open right now, yet I can still see the page faults slowly increasing. I have no limit to the amount of virtual memory that Rosetta can use - it's using ~140MB right now.

Additionally, if I were to find a way to reduce that would it help Rosetta?

Thanks in advance. I searched for "page faults" on the forum but found nothing.

-Ford

---

Team Starfire World BOINC

Hi, FP.

From Alex Nichol's Virtual Memory in Windows XP,
What are Page Faults?

Only those parts of the program and data that are currently in active use need to be held in physical RAM. Other parts are then held in a swap file (as it’s called in Windows 95/98/ME: Win386.swp) or page file (in Windows NT versions including Windows 2000 and XP: pagefile.sys). When a program tries to access some address that is not currently in physical RAM, it generates an interrupt, called a Page Fault. This asks the system to retrieve the 4 KB page containing the address from the page file (or in the case of code possibly from the original program file). This, a valid page fault, normally happens quite invisibly. Sometimes, through program or hardware error, the page is not there either. The system then has an "Invalid Page Fault" error. This will be a fatal error if detected in a program: if it is seen within the system itself (perhaps because a program sent it a bad request to do something), it may manifest itself as a "blue screen" failure with a STOP code: consult the page on STOP Messages on this site.

If there is pressure on space in RAM, then parts of code and data that are not currently needed can be "paged out" in order to make room, the page file can thus be seen as an overflow area to make the RAM behave as if it were larger than it is.

I looked at three Windows XP computers and found that iPod was the leader and had about 16,000,000 page faults. Rosetta was second at 1.5 to 6 Million and not proportional to the amount of physical RAM.

Interestingly, rosetta_5.35_windows_intelx86.exe had ~600,000 page faults (increasing) on Windows Vista with 415 MB of 1 GB RAM available. However, it was only 11th on the list. See screenshot. It appears that Windows Vista does a better job than XP of running Rosetta. (Beware the Law of Isolated Observation.) The computer running Vista has an AMD XP 2500+ (Barton) processor.

Do you propose to change the way the operating system handles page faults? How? Rewrite rosetta_5.35_windows_intelx86.exe? It isn't a problem.

---

Hi Vestor, thanks for the info!

Do you propose to change the way the operating system handles page faults? How? Rewrite rosetta_5.35_windows_intelx86.exe? It isn't a problem.

Not at all, I guess I was just surprised that the application pages so much when there's so much RAM freely available for it to use - and I was looking for more info on why it does that and whether there was anything I could do outside of rewriting anything to help reduce that.

I'm surprised that it isn't a problem. Paging is slow, RAM is fast. I suppose I am missing something here - can you help me understand why reducing this won't speed up rosetta?

"When a program tries to access some address that is not currently in physical RAM, it generates an interrupt, called a Page Fault. This asks the system to retrieve the 4 KB page containing the address from the page file "

Thanks again,

-F.

---

Team Starfire World BOINC

I don't think you would see an improvement because "Only those parts of the program and data that are currently in active use need to be held in physical RAM."

RAID 0 or putting the page files on a secondary hard drive will speed the data. From the same reference by Alex Nichol:

Where do I set the placing and size of the page file?

At Control Panel | System | Advanced, click Settings in the “Performance” Section. On the Advanced page of the result, the current total physical size of all page files that may be in existence is shown. Click Change to make settings for the Virtual memory operation. Here you can select any drive partition and set either ‘Custom’; ‘System Managed’ or ‘No page file’; then always click Set before going on to the next partition.

He really said, "No paging file." I tried running that way on another project without experiencing troubles. With 2 GB RAM and Windows XP, you could try it and see how you like it.

Anyone care to comment on no paging file?

---

In C a program can influence paging for the most used allocated structures. There are flags like MOVEABLE and DISCARDABLE which modify the Windows behaviour towards allocated RAM. I'm not sure about C++.

Not all RAM should be allocated not MOVEABLE though as long as a few results still bail out with their requirements on RAM, or many boxes with 512MB or less will be out.


On early Unix machines it sometimes made quite a difference to allocate a pool of RAM in one go and have an own alloc manager that takes care of it - that kind of tweaking made some programs a lot faster but it can only be done if the program can figure out how much RAM it will need quite early.

I haven't used that technique for quite some time now as the machines nowadays are way faster than they have been in the time of 32016 CPUs. I have no idea if the work to implement it would still pay off today.

In C a program can influence paging for the most used allocated structures. There are flags like MOVEABLE and DISCARDABLE which modify the Windows behaviour towards allocated RAM. I'm not sure about C++.

Not all RAM should be allocated not MOVEABLE though as long as a few results still bail out with their requirements on RAM, or many boxes with 512MB or less will be out.


On early Unix machines it sometimes made quite a difference to allocate a pool of RAM in one go and have an own alloc manager that takes care of it - that kind of tweaking made some programs a lot faster but it can only be done if the program can figure out how much RAM it will need quite early.

I haven't used that technique for quite some time now as the machines nowadays are way faster than they have been in the time of 32016 CPUs. I have no idea if the work to implement it would still pay off today.

Rosetta does fairly little runtime allocation of memory (in the sense that it's all allocated at startup - where it does a lot of that!).

Also, I think allocating memory as movable/discardable is an Windows concept that disappeared when Windows 98 came in, as it was a 16-bit concept - and the Win32 API doesn't actually have such a concept at all...

[Unless we're talking about physical locking, which is something that device drivers would do to prevent the memory mapping code to "page-out" some block of memory - if you have a PCI-device that wants to directly read memory, it would make life very weird if that particular piece of memory got removed from RAM and stored to the disk "under the feet" of the PCI device.]

Further, page-faults are most likely not causing a huge amount of problems. The paging system will sometimes mark pages not present just to measure the frequency they are used [I don't KNOW if Windows uses this technique - but it's possible] - it does this to make sure that pages that are less frequently used can be paged out, in preference of paging out pages that are frequently used... There is very little overhead in taking a page-fault for such non-present pages, because the page is actually there, it's just marked as not there so that the OS can get a trap when it gets used, and just mark it present again.

So, one way to use this would be to walk through all the pages at some slow speed, marking each one not-present. Later on when the OS finds that pages need to be paged out, it will first look for pages that haven't been accessed since they were marked not present, and use those before ones that have been re-marked as present.

--
Mats

---

DISCARDABLE is a default attribute for ressource memory. For RAM allocation (GlobalAlloc), GMEM_NODISCARD, GMEM_DISCARDABLE, GMEM_MOVEABLE and GMEM_FIXED have still been supported flags for NT4 and 9x versions, maybe XP ignores those now.

DISCARDABLE is a default attribute for ressource memory. For RAM allocation (GlobalAlloc), GMEM_NODISCARD, GMEM_DISCARDABLE, GMEM_MOVEABLE and GMEM_FIXED have still been supported flags for NT4 and 9x versions, maybe XP ignores those now.

From Microsoft Website:

Note: The global functions are slower than other memory management functions and do not provide as many features. Therefore, new applications should use the heap functions. However, the global functions are still used with DDE, the clipboard functions, and OLE data objects.
<snip>
The following values are obsolete, but are provided for compatibility with 16-bit Windows. They are ignored.

GMEM_DDESHARE
GMEM_DISCARDABLE
GMEM_LOWER
GMEM_NOCOMPACT
GMEM_NODISCARD
GMEM_NOT_BANKED
GMEM_NOTIFY
GMEM_SHARE

GMEM_MOVABLE changes the semantic of GlobalAlloc in that it gives back a handle rather than a memory address, so it has to be "supported" for backwards compatibility with applications - but I bet that it doesn't actually change the address ever - with 4GB, 64GB or 1024TB of addressable memory, it's pretty useless to move a block of memory - and the heap managers that are implemented in Win32 API don't support moving memory within the heap [also per Microsoft web-site, although I'm too lazy to search for that page] - it makes the management of the heap much easier, even if it also increases the problems of fragmenting the heap and potentially growing the application.

GMEM_FIXED is also recognized, but then the meaning of it is pretty much the same as the what VirtualAlloc does in the first place - allocate a piece of memory at a fixed address. Not to mention that it's pretty darn hard to IGNORE a flag value that is "zero", when all other flags are added to that value... ;-)

Further:

Windows Me/98/95: The heap managers are designed for memory blocks smaller than four megabytes. If you expect your memory blocks to be larger than one or two megabytes, you can avoid significant performance degradation by using the VirtualAlloc or VirtualAllocEx function instead.

So although the flags are still available, they are (pretty much) meaningless, and different functions are supposed to be used for the purpose of allocating memory in modern versions of Windows.

Also, if you did actually write code with "movable" allocation, you'd not get a page-fault, but a segment-not-present fault, since the old (16-bit) API used segmentation to keep track of memory allocation - although some virtual memory technology was ALSO used in Windows 3.x, in a hybdrid fashion.

Note, whilst this may sound like "besserwissing", I'm just trying to get a fair view of what can/can't be done in Rosetta, and bringing up old-fashioned mechanisms that aren't meaningfull in the current generation of Windows doesn't quite help here... ;-)

--
Mats

---

Sorry, tried to post last night, but lost my net connection. Faulting has been discussed before in two main threads.
thread 1
thread 2

In the ole days, I saw some WUs doing 1,000 faults per SECOND! So, they've come a long ways :)

One thing that came up during that discourse was that there are hard and soft page faults. So they don't all actually result in a disk IO. With as much memory as you have, unless you are running other memory intensive applications, I'd have to guess most are not causing disk IOs.

---

Add this signature to your EMail:
Running Microsoft's "System Idle Process" will never help cure cancer, AIDS nor Alzheimer's. But running Rosetta@home just might!
https://boinc.bakerlab.org/rosetta/

...Note, whilst this may sound like "besserwissing", ...

No problem with someone knowing something better than me - one cannot learn from people knowing less. Even though I din't have to do much on Windows machines in my job since a few years, the time might come that I will need it again.

He really said, "No paging file." I tried running that way on another project without experiencing troubles. With 2 GB RAM and Windows XP, you could try it and see how you like it.

Unfortunately, this isn't an option. The other applications I run use the page file frequently.

What I'm gathering from the other notes is that it isn't necessarily impacting performance. I don't quite understand why yet, but I'm getting there. :)

...it isn't necessarily impacting performance. I don't quite understand why yet, but I'm getting there. :)

...if guesses are correct, it isn't impacting your performance, because it isn't actually waiting for anything to come in from disk to resolve the fault.

---

Add this signature to your EMail:
Running Microsoft's "System Idle Process" will never help cure cancer, AIDS nor Alzheimer's. But running Rosetta@home just might!
https://boinc.bakerlab.org/rosetta/

it isn't actually waiting for anything to come in from disk to resolve the fault.

I've only seen one other application do this as much as Rosetta - sqlservr.exe. And only when it ran out of RAM, and it slowed to a crawl completing queries when that happened. After feeding it more memory, the same reports would take about 1/50th of the amount of time to complete. Therefore I was inclined to view this as a sign that Rosetta was doing something wrong by paging so much when over half my RAM was free.

What you are saying (or guessing) is that Rosetta is just paging for the hell of it. It's putting 4KB on the hard drive and then checking to make sure those 4KB are there a few hundred times a second without actually needing those 4KB. I don't know enough about the nuts and bolts of this, but isn't the action of checking on it that much spending resources? Or is the amount of effort it spends doing that so miniscule as to not even mention?

The gears in my head are churning... thanks for bearing with me as I learn this.

---

Team Starfire World BOINC

There are essentially three types of page-faults:

1. Soft page-faults: Page-fault caused by data not being available to the current process, but it's in memory - easy and quick to fix, just map it to the correct location and mark it present. This case includes the OS periodically marking pages as not present to keep statistics, if such a function exists in the OS.

A common scenario in applicatios is to ask for a much larger chunk of memory than it would normally use - say a text-editor may ask for 2MB to hold a file that is 5KB - and there are good, bad and ugly reasons for doing so. Since this is so common, the OS will "lazy" give memory to the application - when the application uses a piece of memory that has been given to it, it's ACTUALLY given to the application. There is a big pool of pre-allocated pages for this purpose, so it's very fast to give it to the application - but before it's first used, it's getting a page-fault to tell the OS that it's being used.

I suspect this is a common case in Rosetta, assuming that it's using dynamically allocated memory at least once in a while. For example, it's very possible that the max number of atoms (or some other small unit) of the protein is a fixed number - so every time we load a protein, it's got to fit in this max number. Let's say for arguments sake this number is 500. If you then have to calculate a molecule with only 50 atoms, 90% of the space is never touched. But each time Rosetta creates space for a molecule-copy [which may happen every now and again], it creates space for 500 atoms - because it's much easier to do it that way...

There are also "copy-on-write" pages, where a page is shared between different applications until it has been written to, at which point the original data is copied to a new page. You can imagine that a DLL may contain a lot of data that is initialized but rarely written, so as long as it's not been written to, the DLL can use the same memory location for multiple applications. When some application writes to the data, before the actual write happens, a page-fault is issued to the OS, and the OS creates a application specific page of memory that can be written as much as the application would like - all other applications still see the old data in the original page, and thus are unaffected. Again, the cache of ready-to-use pages are used to satisfy the application at this time, and it's just a case of copying the actual data within the memory.

There are several other variants of "soft page faults", but this gives an example of how it works.

2. Hard Page-fault caused by data having been stored on disk - takes time to read the data back from disk, but hopefully this doesn't happen very often.

3. Error Page-fault caused by bad adddresses. These are the ones that "kill" Rosetta (or any other application). This is where some address asked for by the application isn't actually available (at all) to the application... The OS uses the page-table (the data that describes which bits of memory are where, and if the page-table contains a "not present", gives a page-fault). All the memory that isn't to be used by this application is marked "not present".

Of course, the act of page-faulting the processor, and executing even the simplest code in the page-fault handler WILL take some amount of processor time. But as long as it's relatively rare and usually soft page faults, then it's not too expensive. The ones that take a long time to solve are the "hard page faults", because the OS has to read the data back from the disk (and not unlikely also make some more space available by writing some other bit of memory out to the disk - but often this can be hidden by having a small cache of ready-to-use pages, as described above, and later on writing some old, rarely used, page to the disk).

The page-fault-handler for soft faults is in the range of microseconds in time, reading the hard-disk will be in the millisecond range, so relatively speaking, it's 1000x slower to take a "hard" pagefault than a "soft". Timing of error-page-faults is irrellevant for any normal application, as it usually means suicide to perform such operatons (the application can of course be recovered from by certain types of error-handling, but it's usually a bug in the code when that happens in the first place, so trying to cope with it in the code isn't likely to help - exiting nicely with some good debug information may be a good idea tho').

--
Mats

---

What you are saying (or guessing) is that Rosetta is just paging for the hell of it.

Applications can only "cause" paging to happen by using huge amounts of memory, aside from soft-faults as discussed in my post just a few seconds ago. And I'd be willing to bet that all faults in Rosetta are soft-faults.

--
Mats

---

Mats,
. . or anyone else I suppose. Does turning off the paging file do anything to performance? I have 2gb of RAM, and according to task manager, my peak Commit Charge was less than 800mb. I definately don't notice the machine running any slower, if anything programs that have been opened and close open faster from then on. I don't see how having anything write to disk can be beneficial if indeed I have a 1gb of memory 'buffer'.

Thanks for any thoughts,
Ethan

---

Does turning off the paging file do anything to performance?

...only if you are physically using it... in which case, you'd be hard pressed if you turned it off.

Look at it this way, the system wants to run as fast as it can. I only pushes things out to the swap file when it MUST do so to accomodate demands for memory, perhaps from other applications. So the new application "steals" the pages of another application that may have been sitting idle for several minutes. The a timer trips in the first application and it needs to run again. It can run up to the point that some of it's variables or program instructions reside on a page that is no longer in memory. Then it must wait for the disk read of the swap file before it can proceed.

Many applications create large tree structures, but only actively use a portion of it. The old 80/20 rule at work. That 20% of the tree that is actively used is part of the "working set" of the application. i.e. you won't get much done, unless you've at least got the pages where that 20% of the tree reside in memory. This is why it is the working set that is key to understanding the performance implications of more memory or other means to try and reduce faulting.

Ethan, you run a new application, say MS-Word. It must go read the entire program from disk... you read a document and close the application. On a machine with plenty of memory, you come back, even 10 minutes later, and open another MS-Word document... the program (.exe and .dll) is still in memory! This is why it runs much faster the second time. It only has to go to disk to read the document itself the second time.

Ford, you mentioned writing 4K to disk and then checking it all the time to see if it's there. The application doesn't KNOW where that 4K is, the operating system takes care of that. And so the only way to force it to be read back would be to utilize some of the variables stored in that memory, and yes; applications utilize their variables. Once you "touch" the page, it stays around a while. If a page must be force out to swap file to accomodate something else, it will be the least-recently-used page. Since you JUST touched your page, it's at the END of the line to be swapped out. So, instead of yours, a page from an application that has been idle for some time will be pushed to the swap file to make room to accomodate the current activity.

---

Add this signature to your EMail:
Running Microsoft's "System Idle Process" will never help cure cancer, AIDS nor Alzheimer's. But running Rosetta@home just might!
https://boinc.bakerlab.org/rosetta/

Ethan, you run a new application, say MS-Word. It must go read the entire program from disk... you read a document and close the application. On a machine with plenty of memory, you come back, even 10 minutes later, and open another MS-Word document... the program (.exe and .dll) is still in memory! This is why it runs much faster the second time. It only has to go to disk to read the document itself the second time.

Actually, no. Windows (and Linux) uses something called "on-demand loading", so if you start some large application, it will load the application (and it's DLL's) as and when it's needed - so if you never used the "print" function in the application for the first two hours, and then go to print, it will load up that bit of code into memory when it's needed. This is one version of using paging - the application is given it's space in memory, and when the code jumps to a page that isn't present (i.e. hasn't been loaded yet), it page-faults, and the OS finds that "Oh, we need this page now", grabs a page out of the cache, and loads that bit of the application.

This has the great benefit for large applications that you don't need to load up many megabytes, when only, say 20% of the code is actually used in the common cases (if you don't use the "draw" function in Word, the code doesn't get loaded ...).

Ford, you mentioned writing 4K to disk and then checking it all the time to see if it's there. The application doesn't KNOW where that 4K is, the operating system takes care of that. And so the only way to force it to be read back would be to utilize some of the variables stored in that memory, and yes; applications utilize their variables. Once you "touch" the page, it stays around a while. If a page must be force out to swap file to accomodate something else, it will be the least-recently-used page. Since you JUST touched your page, it's at the END of the line to be swapped out. So, instead of yours, a page from an application that has been idle for some time will be pushed to the swap file to make room to accomodate the current activity.

That is exactly what I tried to say - thanks for clarifying...

--
Mats

---

Yes Mats, perhaps I implied more than I intended. Once the application is ended, I believe the dlls it used and etc. are probably still around in memory, and then when you use the same portion of the application (i.e. when you just load MS-Word to display a document) a second time, it's much faster because we don't have to go to disk to bring in the same code that was just used a moment ago to display a different document. Since the system being discussed has lots of memory, it ends up having useful remnants of things around like this and better performance, and less wear on disk drives.

I think we're all on the same "page" now... "page!" get it? I crack myself up :)

---

Add this signature to your EMail:
Running Microsoft's "System Idle Process" will never help cure cancer, AIDS nor Alzheimer's. But running Rosetta@home just might!
https://boinc.bakerlab.org/rosetta/

As an FYI - I turned off the page file two days ago and the computer has never been faster. . not that it was slow, it's just 'snappier'. I'm not saying it's something people should do, but I haven't seen a downside. I'm running two threads of rosetta (dual core), a bunch of IE windows, word, picasa, itunes, and mstsc to a couple work machines.

---