New jobs in Rosetta Python projects

Message boards : Number crunching : New jobs in Rosetta Python projects

To post messages, you must log in.

1 · 2 · 3 · Next

AuthorMessage
dubadmin
Volunteer moderator
Project developer
Project scientist

Send message
Joined: 17 Jan 24
Posts: 1
Credit: 0
RAC: 0
Message 108787 - Posted: 17 Jan 2024, 6:10:47 UTC

We will be releasing new jobs using the Rosetta Python projects application with RoseTTAFold All-Atom, our state-of-the-art deep learning based application that predict the structures of proteins in complex with ligands such as DNA and other biological molecules and also models post-translational modifications. As a reminder, this application will be released in the form of virtual machine. To help us with testing and gathering performance statistics, we ask that you do not abort related work units.
ID: 108787 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Grant (SSSF)

Send message
Joined: 28 Mar 20
Posts: 1548
Credit: 15,862,044
RAC: 20,299
Message 108788 - Posted: 17 Jan 2024, 6:55:14 UTC

Will they make use of Multiattach mode disk images?
The biggest problem (of many) with Python tasks making use of VMs is that each & every one requires massive amounts of disk space & the disk I/O needed when starting each new Task. So the disk I/O and space required for 32core/64thread and larger systems is more than just somewhat ridiculous.
Grant
Darwin NT
ID: 108788 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1905
Credit: 8,764,369
RAC: 10,476
Message 108789 - Posted: 18 Jan 2024, 13:54:20 UTC - in response to Message 108788.  
Last modified: 18 Jan 2024, 13:56:28 UTC

Will they make use of Multiattach mode disk images?


In Ralph no. They are still using the old virtualbox wrapper (26203) without multiattach mode.
In December the boinc team released a new version (26207).....
ID: 108789 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1905
Credit: 8,764,369
RAC: 10,476
Message 108790 - Posted: 19 Jan 2024, 8:14:47 UTC

I see there is over 500 wus of python app.
With the old version (June 2021).
ID: 108790 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1905
Credit: 8,764,369
RAC: 10,476
Message 108792 - Posted: 19 Jan 2024, 8:25:47 UTC - in response to Message 108789.  

In December the boinc team released a new version (26207).....

Here you can find the latest vbox wrapper and some guides
ID: 108792 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1905
Credit: 8,764,369
RAC: 10,476
Message 108793 - Posted: 19 Jan 2024, 8:52:35 UTC - in response to Message 108789.  

In December the boinc team released a new version (26207).....


I wrote to a boinc developer and he said to me that the latest stable is 26206 (26207 is in beta).
This is why is not here, in release notes
ID: 108793 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
kksplace

Send message
Joined: 12 May 19
Posts: 7
Credit: 5,303,601
RAC: 9
Message 108794 - Posted: 19 Jan 2024, 19:13:07 UTC - in response to Message 108787.  

You asked not to abort these tasks. I received four today. They all show "Postponed: Communication with VM Hypervisor failed." after about 12 seconds. Should I just leave them or abort them?
ID: 108794 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Ryan Munro

Send message
Joined: 19 Nov 15
Posts: 16
Credit: 13,478,595
RAC: 6,589
Message 108795 - Posted: 20 Jan 2024, 2:25:15 UTC

I am getting the error "Postponed: Communication with VM Hypervisor failed" as well, Linux Mint 21.3 running the latest version of Virtualbox, any one know of a fix?
ID: 108795 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1905
Credit: 8,764,369
RAC: 10,476
Message 108796 - Posted: 20 Jan 2024, 12:03:09 UTC

No problems here, with my Windows 11 machines.

A little note: in my 12-cores cpu i started, manually, 2/3 wus at the same time (and not twelve), 'cause disk and ram usage
ID: 108796 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1905
Credit: 8,764,369
RAC: 10,476
Message 108797 - Posted: 21 Jan 2024, 18:24:23 UTC

A strange beaviour.
Every time i finished all my wus and donwloaded other wus, i re-downloaded also the .vdi file (5 gb)
ID: 108797 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
mikey
Avatar

Send message
Joined: 5 Jan 06
Posts: 1894
Credit: 8,854,568
RAC: 2,270
Message 108798 - Posted: 23 Jan 2024, 2:22:36 UTC - in response to Message 108787.  

We will be releasing new jobs using the Rosetta Python projects application with RoseTTAFold All-Atom, our state-of-the-art deep learning based application that predict the structures of proteins in complex with ligands such as DNA and other biological molecules and also models post-translational modifications. As a reminder, this application will be released in the form of virtual machine. To help us with testing and gathering performance statistics, we ask that you do not abort related work units.


I know Rosetta hasn't ever done this but these new Python tasks take 7.45gb of memory for EACH task on my Windows pc's, i have pc's with anything from 6 to 32 cpu cores but not a one of them has enough memory to fill every cpu core with a Python task and not lock up the pc to the point only a reboot and reducing the physical number of tasks I can run at Rosetta. Normally it's not a problem with anything EXCEPT the Python tasks so what I'm asking is you add an optional setting to run the Python tasks, be it thru a choice box or thru a box that says ie 'allow beta tasks' that we can then select. The problem with the 2nd option alone is that people can't choose to not run the regular Rosetta tasks while they wait for any Python tasks to come their way so they could easily end up with the same problem I'm having with lack of memory in my desktops.

A simple set of type of tasks choice boxes would make things a whole lot easier on the users end and not really change anything beyond the initial setup for you Admins on your end.
This page shows what I am talking about but most Projects have something similar to it:
https://escatter11.fullerton.edu/nfs/prefs.php?subset=project

In case the link doesn't work, it's to my Preferences page at NFS, here is a copy of one section of it:

Home
Resource share 1
Use CPU
Run only the selected applications 14e Lattice Sieve (lasieved) - uses less than 0.5 GB memory, work may be infrequently available: yes
15e Lattice Sieve for smaller numbers (lasievee_small) - small numbers, uses up to 0.8 GB memory: no
15e Lattice Sieve (lasievee) - medium numbers, uses up to 1 GB memory: no
16e Lattice Sieve for smaller numbers (lasievef_small) - large numbers, uses up to 1 GB memory: no
16e Lattice Sieve V5 (lasieve5f) - largest numbers, uses up to 1.25 GB memory: no
If no work for selected applications is available, accept work from other applications? no

Thank you for reading this and I hope you can discuss and implement this soon .
Thank you for you time
mikey
ID: 108798 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Ryan Munro

Send message
Joined: 19 Nov 15
Posts: 16
Credit: 13,478,595
RAC: 6,589
Message 108799 - Posted: 23 Jan 2024, 8:51:04 UTC

Anyone managed to get the VM jobs working on Linux?
ID: 108799 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Link
Avatar

Send message
Joined: 4 May 07
Posts: 355
Credit: 382,349
RAC: 0
Message 108800 - Posted: 23 Jan 2024, 10:10:36 UTC - in response to Message 108798.  

I know Rosetta hasn't ever done this but these new Python tasks take 7.45gb of memory for EACH task on my Windows pc's, i have pc's with anything from 6 to 32 cpu cores but not a one of them has enough memory to fill every cpu core with a Python task and not lock up the pc to the point only a reboot and reducing the physical number of tasks I can run at Rosetta. Normally it's not a problem with anything EXCEPT the Python tasks
Than limit the amount of concurrent python tasks using an app_confil.xml. This can be even different for each of your computers, depending on the amount of memory.
.
ID: 108800 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
highCon

Send message
Joined: 25 Aug 19
Posts: 1
Credit: 7,114,825
RAC: 2,619
Message 108801 - Posted: 23 Jan 2024, 11:34:47 UTC - in response to Message 108799.  
Last modified: 23 Jan 2024, 11:49:15 UTC

Anyone managed to get the VM jobs working on Linux?

Yes; it works. (Arch Linux with VirtualBox installed)

Do you encounter any problems?

I had these:

    1. Due to the disk-hungry tasks, I had to move my BOINC folder to another partition.
    2. After uploading some successfully crunched tasks, new tasks were suspended with the status "Communication with VM Hypervisor failed" (NS_ERROR_SOCKET_FAIL in the vbox logs). Digging deeper, it helped to clean the <HardDisk> entries in the VirtualBox.xml
    3. Just today, some tasks were running fine, while others had the status "Postponed: VM environment needed to be cleaned up". Perhaps this was caused by the previous intervention (point 2); luckily, restarting boinc-client helped.


Let's see what else will happen.

But it definitely hurts me to see that the 5.5GB compressed vdi file is redownloaded everytime on starting new tasks (when no python-based tasks are currently crunching, which apparently keep the downloaded file alive).

ID: 108801 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Ryan Munro

Send message
Joined: 19 Nov 15
Posts: 16
Credit: 13,478,595
RAC: 6,589
Message 108802 - Posted: 23 Jan 2024, 15:35:29 UTC - in response to Message 108801.  

I get "Postponed: Communication with VM Hypervisor failed" when a unit is downloaded
ID: 108802 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1905
Credit: 8,764,369
RAC: 10,476
Message 108803 - Posted: 23 Jan 2024, 16:46:13 UTC - in response to Message 108798.  

Thank you for reading this and I hope you can discuss and implement this soon .
Thank you for you time
mikey


Do you know that they do not read the forums and, if they read, they do NOTHING for a better usage of the project by users?
ID: 108803 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1905
Credit: 8,764,369
RAC: 10,476
Message 108804 - Posted: 23 Jan 2024, 16:47:30 UTC - in response to Message 108801.  

But it definitely hurts me to see that the 5.5GB compressed vdi file is redownloaded everytime on starting new tasks (when no python-based tasks are currently crunching, which apparently keep the downloaded file alive).


It seems that the project remove the vdi file from the project folder...
And start the download again

(still with old wrapper)
ID: 108804 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
hadron

Send message
Joined: 4 Sep 22
Posts: 57
Credit: 1,301,943
RAC: 7,925
Message 108806 - Posted: 23 Jan 2024, 22:08:25 UTC - in response to Message 108799.  

Anyone managed to get the VM jobs working on Linux?


Is user boinc a member of group vboxuser? Run "sudo groups boinc" to find out. Then, if necessary, use

sudo groupmod -a -U boinc vboxusers

to add boinc to that group.
ID: 108806 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Ryan Munro

Send message
Joined: 19 Nov 15
Posts: 16
Credit: 13,478,595
RAC: 6,589
Message 108807 - Posted: 23 Jan 2024, 23:37:16 UTC - in response to Message 108806.  

Output I get is : "boinc : boinc video render vboxusers" so I am assuming so?
ID: 108807 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
mikey
Avatar

Send message
Joined: 5 Jan 06
Posts: 1894
Credit: 8,854,568
RAC: 2,270
Message 108808 - Posted: 24 Jan 2024, 4:36:14 UTC - in response to Message 108800.  

I know Rosetta hasn't ever done this but these new Python tasks take 7.45gb of memory for EACH task on my Windows pc's, i have pc's with anything from 6 to 32 cpu cores but not a one of them has enough memory to fill every cpu core with a Python task and not lock up the pc to the point only a reboot and reducing the physical number of tasks I can run at Rosetta. Normally it's not a problem with anything EXCEPT the Python tasks


Than limit the amount of concurrent python tasks using an app_confil.xml. This can be even different for each of your computers, depending on the amount of memory.


I DO and said that but the problem is as I said when I get the normal or beta tasks I will want to run as many at a time as I can and I can't do that with the current Prefences setup.
I know I could use an app_config.xml file to limit this or that type of task but have forgotten how to do that and the settings are all project specific and I don't remember how to that here and besides newbies will have no clue and just give up and go elsewhere and that's NOT what anyone wants. PrimeGrid has an elaborate prefences setup that even includes number of cpu's per task in additionto the type of task and how many of that type of task to run so it IS possible to have more and better choices.

Honestly I wish the Developers would just hard code it into the Server coding which would fix places like Gerasim who have totally borked the settings, but that's a whole other problem, as well as help out places like Rosetta. YES Projects should be able to disable the defaults as they do now but sameness can be an advantage for some things.,
ID: 108808 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
1 · 2 · 3 · Next

Message boards : Number crunching : New jobs in Rosetta Python projects



©2024 University of Washington
https://www.bakerlab.org