Excessive workunit fetch

Message boards : Number crunching : Excessive workunit fetch

To post messages, you must log in.

Previous · 1 · 2

AuthorMessage
Profile Grant (SSSF)

Send message
Joined: 28 Mar 20
Posts: 1721
Credit: 18,356,357
RAC: 25,250
Message 103353 - Posted: 16 Nov 2021, 21:33:13 UTC - in response to Message 103340.  
Last modified: 16 Nov 2021, 21:35:05 UTC

The regular Rosetta tasks also reserve 8gb of ram per task
Complete and utter rubbish.

Stop making things up.



how much the Virtual Memory Size is and the Working set size is, you care about the one that shows the most memory.
No you don't.
Virtual memory is just that- virtual memory. It is used if there isn't enough physical RAM. The Working Set size is the one that matters.

If the amount of Virtual Memory required becomes an issue, then you've got a much bigger problem with your system as a whole due to lack of free disk space than some value reported for a particular Rosetta Task.
Grant
Darwin NT
ID: 103353 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Mad_Max

Send message
Joined: 31 Dec 09
Posts: 209
Credit: 26,393,873
RAC: 19,562
Message 103653 - Posted: 2 Dec 2021, 0:52:52 UTC
Last modified: 2 Dec 2021, 0:54:40 UTC

It is a confirmed BOINC (not Rosetta) bug with excessive workunit fetch for a project if <max_concurrent> setting is used.

It was identified about year ago or so.
One of the latest bug reports to BOINC developer: https://github.com/BOINC/boinc/issues/4322

And finally a patch to this problem is in work now: https://github.com/BOINC/boinc/pull/4592
It will be included in one of the future BOINC releases.

For now there are few workarounds until new fixed version of BOINC available:
1 - avoid using <max_concurrent> setting completely
OR
2 - set work cache size to a really low values
OR
3 - use R@H with other projects with stable WUs supply and WITHOUT <max_concurrent> setting applied to this "spare" projects (only to R@H)

Because bug itself is a wrong calculation of amount of work BOINC already have in queue for projects limited by <max_concurrent> setting.
It just count only up to <max_concurrent> number of tasks in queue and ignores the rest. So calculated work queue size does not increase no matter how many tasks BOINC has already loaded for a such project. And falls into an endless loop of getting new work.
ID: 103653 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Grant (SSSF)

Send message
Joined: 28 Mar 20
Posts: 1721
Credit: 18,356,357
RAC: 25,250
Message 103655 - Posted: 2 Dec 2021, 5:48:42 UTC - in response to Message 103653.  

And finally a patch to this problem is in work now: https://github.com/BOINC/boinc/pull/4592
It will be included in one of the future BOINC releases.
Glad to see there might finally be a fix for this issue.
Grant
Darwin NT
ID: 103655 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [AF>Le_Pommier] Jerome_C2005

Send message
Joined: 22 Aug 06
Posts: 44
Credit: 1,258,039
RAC: 0
Message 103702 - Posted: 4 Dec 2021, 14:43:14 UTC

Thanks for this input Mad Max.

I may give it a try, next year when FB is over :)
ID: 103702 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Greg_BE
Avatar

Send message
Joined: 30 May 06
Posts: 5691
Credit: 5,859,226
RAC: 0
Message 103750 - Posted: 6 Dec 2021, 20:57:54 UTC

You can get around that by adding the word project_ to the max concurrent.
I have been using this for a few days now and it works perfect.
[project_max_concurrent] N [/project_max_concurrent]
ID: 103750 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Greg_BE
Avatar

Send message
Joined: 30 May 06
Posts: 5691
Credit: 5,859,226
RAC: 0
Message 103751 - Posted: 6 Dec 2021, 20:59:59 UTC - in response to Message 103353.  

The regular Rosetta tasks also reserve 8gb of ram per task
Complete and utter rubbish.

Stop making things up.



how much the Virtual Memory Size is and the Working set size is, you care about the one that shows the most memory.
No you don't.
Virtual memory is just that- virtual memory. It is used if there isn't enough physical RAM. The Working Set size is the one that matters.

If the amount of Virtual Memory required becomes an issue, then you've got a much bigger problem with your system as a whole due to lack of free disk space than some value reported for a particular Rosetta Task.



Grant, hes not that far off. 7629.39 physical per task and 103.13 virtual (cages tasks)
ID: 103751 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Mad_Max

Send message
Joined: 31 Dec 09
Posts: 209
Credit: 26,393,873
RAC: 19,562
Message 103762 - Posted: 7 Dec 2021, 21:37:36 UTC - in response to Message 103751.  
Last modified: 7 Dec 2021, 21:43:55 UTC

He said it about regular (non Python/virtual box) task. Regular R@H tasks RAM usage lies in 0.7-3 Gb range for almost all tasks (for >95% of tasks) and in 0.7-1.5 Gb usually (for >70%).
So it is very far from the 8 GB per tasks.
ID: 103762 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [AF>Le_Pommier] Jerome_C2005

Send message
Joined: 22 Aug 06
Posts: 44
Credit: 1,258,039
RAC: 0
Message 105860 - Posted: 7 Apr 2022, 17:21:35 UTC

If I can read github correctly, it is said that this patch will be published in boinc 7.20.0 ??
ID: 105860 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Mad_Max

Send message
Joined: 31 Dec 09
Posts: 209
Credit: 26,393,873
RAC: 19,562
Message 106180 - Posted: 11 May 2022, 0:38:59 UTC

Yes, it already included in BOINC ver 7.20.0 (and later).

But v. 7.20 itself is not finished yet.
ID: 106180 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Previous · 1 · 2

Message boards : Number crunching : Excessive workunit fetch



©2024 University of Washington
https://www.bakerlab.org