Disk usage limit exceeded

Message boards : Number crunching : Disk usage limit exceeded

To post messages, you must log in.

AuthorMessage
Jean-David Beyer

Send message
Joined: 2 Nov 05
Posts: 188
Credit: 6,430,914
RAC: 5,659
Message 108369 - Posted: 17 May 2023, 18:43:51 UTC

I get this error message with this task.

Task 1531200508

Peak working set size 	356.58 MB
Peak swap size 	427.38 MB
Peak disk usage 	2,437.40 MB

<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
[code]<stderr_txt>
command: ../../projects/boinc.bakerlab.org_rosetta/rosetta_beta_6.00_x86_64-pc-linux-gnu 
@yT1NLvkg_flpp3_16_binder_15_cyclic_152_dldesign_0_cycle2.cyclic_redesigned.r1_0001.flags
 -nstruct 10000
 -cpu_run_time 28800 -boinc:max_nstruct 20000 
-checkpoint_interval 120 
-database minirosetta_database
 -in::file::zip minirosetta_database.zip -boinc::watchdog
 -boinc::cpu_run_timeout 36000
 -run::rng mt19937
Using database: database_0f7f01a1b07/database

</stderr_txt>
]]>

My machine is like this.

Computer 5910575
Computer information

CPU type 	GenuineIntel
Intel(R) Xeon(R) W-2245 CPU @ 3.90GHz [Family 6 Model 85 Stepping 7]
Number of processors 	16

Operating System 	Linux Red Hat Enterprise Linux
Red Hat Enterprise Linux 8.7 (Ootpa) [4.18.0-425.13.1.el8_7.x86_64|libc 2.28]
BOINC version 	7.20.2
Memory 	128345.24 MB
Cache 	16896 KB
Swap space 	15992 MB
Total disk space 	488.04 GB
Free Disk Space 	473.91 GB


With that much free disk space, and the relatively modest amount used by the task, what are they complaining about?
ID: 108369 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Grant (SSSF)

Send message
Joined: 28 Mar 20
Posts: 1680
Credit: 17,850,115
RAC: 22,492
Message 108370 - Posted: 18 May 2023, 7:09:18 UTC - in response to Message 108369.  

With that much free disk space, and the relatively modest amount used by the task, what are they complaining about?
It doesn't matter how much disk space you have, if you don't let BOINC use it.
And it could just be a hard coded disk usage limit in the Rosetta application that got exceeded.
Grant
Darwin NT
ID: 108370 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Conan
Avatar

Send message
Joined: 11 Oct 05
Posts: 150
Credit: 4,198,375
RAC: 1,306
Message 108371 - Posted: 18 May 2023, 9:53:26 UTC
Last modified: 18 May 2023, 9:54:45 UTC

It should not of used so much disk space.

I run Fedora Linux and Rosetta Beta work units only use less than 30 MB of disk space and 1.6 GB to 1.8 GB of RAM.

This has been consistent for nearly all beta work units.

They use so much memory that my 32 GB RAM on a 24 thread computer gets almost 100% used as well as most of my 8.6 GB Swap space.

Boinc actually limits running threads so it can keep going. Anyway must work as no errors.

Conan
ID: 108371 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jean-David Beyer

Send message
Joined: 2 Nov 05
Posts: 188
Credit: 6,430,914
RAC: 5,659
Message 108372 - Posted: 18 May 2023, 13:56:26 UTC - in response to Message 108370.  

With that much free disk space, and the relatively modest amount used by the task, what are they complaining about?

It doesn't matter how much disk space you have, if you don't let BOINC use it.
And it could just be a hard coded disk usage limit in the Rosetta application that got exceeded.


But I DO let Boinc use it; At least 420 GBytes of it. I have entire partition on one of my hard drives where all the Boinc stuff is and it has 420 GBytes allocated. And only about 11 GBytes are being used at the moment.

I notice that the other user who attempted to run this task failed the same way I did.
Him:
Task 1531192066
Name 	yT1NLvkg_flpp3_16_binder_15_cyclic_152_dldesign_0_cycle2.cyclic_redesigned.r1_0001_SAVE_ALL_OUT_2959796_604_0


Me:
Task 1531200508
Name 	yT1NLvkg_flpp3_16_binder_15_cyclic_152_dldesign_0_cycle2.cyclic_redesigned.r1_0001_SAVE_ALL_OUT_2959796_604_1


I have two running at the moment and they are not hogging memory:

$ ps -F 516894 538315
UID          PID    PPID  C    SZ   RSS PSR STIME TTY      STAT   TIME CMD
boinc     516894   18695 94 111050 372104 8 01:57 ?        RNl  449:17 ../../projects/boinc.bakerlab.org_rosetta/rosetta_beta_6.00_x86_64-pc
boinc     538315   18695 98 110291 368636 14 08:56 ?       RNl   54:50 ../../projects/boinc.bakerlab.org_rosetta/rosetta_beta_6.00_x86_64-pc

ID: 108372 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jean-David Beyer

Send message
Joined: 2 Nov 05
Posts: 188
Credit: 6,430,914
RAC: 5,659
Message 108373 - Posted: 18 May 2023, 14:05:30 UTC - in response to Message 108371.  

It should not of used so much disk space.

I run Fedora Linux and Rosetta Beta work units only use less than 30 MB of disk space and 1.6 GB to 1.8 GB of RAM.

This has been consistent for nearly all beta work units.

They use so much memory that my 32 GB RAM on a 24 thread computer gets almost 100% used as well as most of my 8.6 GB Swap space.

Boinc actually limits running threads so it can keep going. Anyway must work as no errors.


I am running Red Hat Enterprise Linux release 8.7 (Ootpa). My machine is ID: 5910575.
My machine has 128 GBytes of RAM, so I am not running out of that. I am using very little swap space, and not thrashing.

$ free -hw
              total        used        free      shared     buffers       cache   available
Mem:          125Gi       7.4Gi       3.0Gi       123Mi       430Mi       114Gi       116Gi
Swap:          15Gi        47Mi        15Gi

ID: 108373 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
computezrmle

Send message
Joined: 9 Dec 11
Posts: 63
Credit: 9,680,103
RAC: 0
Message 108374 - Posted: 18 May 2023, 16:26:07 UTC

A disk usage limit can be set by the project via <rsc_disk_bound> (in bytes).
It is thought to protect the disk from being filled by endless loops.
You may check client_state.xml for the current values.

If the error happens regularly the project people may need to increase the value.
Otherwise it may indicate a program error.
ID: 108374 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jean-David Beyer

Send message
Joined: 2 Nov 05
Posts: 188
Credit: 6,430,914
RAC: 5,659
Message 108375 - Posted: 18 May 2023, 17:03:25 UTC - in response to Message 108374.  

A disk usage limit can be set by the project via <rsc_disk_bound> (in bytes).
It is thought to protect the disk from being filled by endless loops.
You may check client_state.xml for the current values.


Which one? There are many different ones. this is a sort -u of them.

    <rsc_disk_bound>100000000.000000</rsc_disk_bound>
    <rsc_disk_bound>104857600.000000</rsc_disk_bound>
    <rsc_disk_bound>160000000.000000</rsc_disk_bound>
    <rsc_disk_bound>2000000000.000000</rsc_disk_bound>
    <rsc_disk_bound>20000000.000000</rsc_disk_bound>
    <rsc_disk_bound>524288000.000000</rsc_disk_bound>
    <rsc_disk_bound>75000000.000000</rsc_disk_bound>

ID: 108375 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jean-David Beyer

Send message
Joined: 2 Nov 05
Posts: 188
Credit: 6,430,914
RAC: 5,659
Message 108376 - Posted: 18 May 2023, 17:43:21 UTC - in response to Message 108374.  

I think I found it:

   <app_name>rosetta_beta</app_name>
    <version_num>600</version_num>
    <rsc_fpops_est>80000000000000.000000</rsc_fpops_est>
    <rsc_fpops_bound>100000000000000000.000000</rsc_fpops_bound>
    <rsc_memory_bound>3500000000.000000</rsc_memory_bound>
      <rsc_disk_bound>2000000000.000000</rsc_disk_bound>

They allow more RAM than disk?
ID: 108376 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
majortotositetop1

Send message
Joined: 22 May 23
Posts: 4
Credit: 0
RAC: 0
Message 108381 - Posted: 22 May 2023, 7:25:57 UTC

I’m still learning from you, but I’m trying to achieve my goals.

메이저사이트
카지노사이트
사설토토
메이저사이트
카지노사이트
사설토토
ID: 108381 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
majortotositetop1

Send message
Joined: 22 May 23
Posts: 4
Credit: 0
RAC: 0
Message 108382 - Posted: 22 May 2023, 7:29:12 UTC

I definitely enjoy reading everything that is posted on your blog.

메이저사이트
카지노사이트
사설토토
메이저사이트
카지노사이트
사설토토
ID: 108382 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sid Celery

Send message
Joined: 11 Feb 08
Posts: 2124
Credit: 41,228,077
RAC: 11,048
Message 108390 - Posted: 30 May 2023, 5:25:08 UTC - in response to Message 108376.  

I think I found it:

   <app_name>rosetta_beta</app_name>
    <version_num>600</version_num>
    <rsc_fpops_est>80000000000000.000000</rsc_fpops_est>
    <rsc_fpops_bound>100000000000000000.000000</rsc_fpops_bound>
    <rsc_memory_bound>3500000000.000000</rsc_memory_bound>
      <rsc_disk_bound>2000000000.000000</rsc_disk_bound>

They allow more RAM than disk?

Yeah, each individual task sets its own requirement, so it'll come up multiple times in the client state file

But didn't we have this before and found it to be a problem with Boinc rather than Rosetta?
I used to put no limit (blank) on maximum disk size set aside for Boinc, but it ended up needing a high set figure because of some kind of miscalculation, so if I put 500Gb then tasks went through ok, even if the unlimited disk space turned out to be more than that.

I may be misremembering how we resolved this, but it was something to do with that, however nonsensical the solution seemed to be.
ID: 108390 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
.clair.

Send message
Joined: 2 Jan 07
Posts: 274
Credit: 26,399,595
RAC: 0
Message 108398 - Posted: 30 May 2023, 20:30:29 UTC
Last modified: 30 May 2023, 20:51:54 UTC

It waz like this

If you want to have something in all three boxes in boinc manager this setting also works and gets the maximum use of the disk with all three boxes used , if you [whoever you are that is reading this] insist on it .
Use no more than - 500 GB - Just set this to something BIG
Leave at least - 0.01 GB free
Use no more than - 99.9 % of total

from my thread from February 2022 when we were dealing with greedy python`s ;-)

https://boinc.bakerlab.org/rosetta/forum_thread.php?id=14903&postid=105215

I went as far as putting those settings in my web preferences so that any new computer I attach to ANY project gets them automatically , problem fixed PERMANTLY .
until next time something screws up . . . .
After all this is BOINC .
nnnnnn.........
ID: 108398 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sid Celery

Send message
Joined: 11 Feb 08
Posts: 2124
Credit: 41,228,077
RAC: 11,048
Message 108400 - Posted: 31 May 2023, 1:54:18 UTC - in response to Message 108398.  

It waz like this

If you want to have something in all three boxes in boinc manager this setting also works and gets the maximum use of the disk with all three boxes used , if you [whoever you are that is reading this] insist on it .
Use no more than - 500 GB - Just set this to something BIG
Leave at least - 0.01 GB free
Use no more than - 99.9 % of total

from my thread from February 2022 when we were dealing with greedy python`s ;-)

https://boinc.bakerlab.org/rosetta/forum_thread.php?id=14903&postid=105215

I went as far as putting those settings in my web preferences so that any new computer I attach to ANY project gets them automatically , problem fixed PERMANTLY .
until next time something screws up . . . .
After all this is BOINC .
nnnnnn.........

I must've read that at the time and did much the same.
500Gb
1Gb
<blank>
ID: 108400 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote

Message boards : Number crunching : Disk usage limit exceeded



©2024 University of Washington
https://www.bakerlab.org