Multiattach mode disk images

Message boards : Number crunching : Multiattach mode disk images

To post messages, you must log in.

Previous · 1 · 2 · 3 · Next

AuthorMessage
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1994
Credit: 9,623,704
RAC: 9,591
Message 106430 - Posted: 28 Jun 2022, 9:41:43 UTC - in response to Message 106389.  

Now also CMS@Home and Test4Theory have new versions with the new wrapper (in test on boinc lhc-dev site)


New wrapper is now in production with new Atlas@Home app.

When Rosetta?
ID: 106430 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
.clair.

Send message
Joined: 2 Jan 07
Posts: 274
Credit: 26,399,595
RAC: 0
Message 106439 - Posted: 28 Jun 2022, 19:39:51 UTC - in response to Message 106430.  

Now also CMS@Home and Test4Theory have new versions with the new wrapper (in test on boinc lhc-dev site)

New wrapper is now in production with new Atlas@Home app.

When Rosetta?

About the same time I win the Lottery . . . .
ID: 106439 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
computezrmle

Send message
Joined: 9 Dec 11
Posts: 63
Credit: 9,680,103
RAC: 0
Message 106452 - Posted: 30 Jun 2022, 19:04:03 UTC
Last modified: 30 Jun 2022, 20:02:07 UTC

The steps below are for the impatient volunteers who want to test the differencing image method.
Use on own risk.
Expect a higher error rate.

The steps are written for Linux but can be used for Windows if the Windows vboxwrapper is used:
https://boinc.berkeley.edu/dl/vboxwrapper_26204_windows_x86_64.zip


1.
Set Rosetta to "no new tasks" and finish/report all Rosetta work

2.
Reset Rosetta and close BOINC

3.
Change the working directory to to "<BOINC_data_directory>/projects/boinc.bakerlab.org_rosetta/"

4.
Downlod the following files:
https://boinc.berkeley.edu/dl/vboxwrapper_26204_x86_64-pc-linux-gnu
https://boinc-files.bakerlab.org/rosetta/download/boinc_app_v4
https://boinc-files.bakerlab.org/rosetta/download/vbox_job_v4.xml
https://boinc-files.bakerlab.org/rosetta/download/26/AIMNet_vm_v2.vdi.gz

5.
Set executable flags:
chmod +x vboxwrapper_26204_x86_64-pc-linux-gnu
chmod +x boinc_app_v4

Extract AIMNet_vm_v2.vdi from AIMNet_vm_v2.vdi.gz
rename "AIMNet_vm_v2.vdi" to "AIMNet_vm_v2.vdi.bak"
run "vboxmanage clonemedium AIMNet_vm_v2.vdi.bak AIMNet_vm_v2.vdi" (Important: this creates a new UUID for AIMNet_vm_v2.vdi)
chmod +r AIMNet_vm_v2.vdi
move AIMNet_vm_v2.vdi.bak to a different directory


6.
Create the following "app_info.xml"
<app_info>
    <app>
        <name>rosetta_python_projects</name>
        <user_friendly_name>rosetta python projects</user_friendly_name>
        <non_cpu_intensive>0</non_cpu_intensive>
    </app>
    <file_info>
        <name>vboxwrapper_26204_x86_64-pc-linux-gnu</name>
        <executable/>
    </file_info>
    <file_info>
        <name>boinc_app_v4</name>
        <executable/>
    </file_info>
    <file_info>
        <name>vbox_job_v4.xml</name>
    </file_info>
    <file_info>
        <name>AIMNet_vm_v2.vdi</name>
    </file_info>
    <app_version>
        <app_name>rosetta_python_projects</app_name>
        <version_num>103</version_num>
        <avg_ncpus>1.000000</avg_ncpus>
        <plan_class>vbox64</plan_class>
        <api_version>7.7.0</api_version>
        <file_ref>
            <file_name>vboxwrapper_26204_x86_64-pc-linux-gnu</file_name>
            <main_program/>
        </file_ref>
        <file_ref>
            <file_name>boinc_app_v4</file_name>
            <open_name>shared/boinc_app</open_name>
            <copy_file/>
        </file_ref>
        <file_ref>
            <file_name>vbox_job_v4.xml</file_name>
            <open_name>vbox_job.xml</open_name>
        </file_ref>
        <file_ref>
            <file_name>AIMNet_vm_v2.vdi</file_name>
        </file_ref>
        <dont_throttle/>
    </app_version>
</app_info>



7.
Modify vbox_job_v4.xml
<vbox_job>
    <memory_size_mb>6144</memory_size_mb>
    <os_name>Debian_64</os_name>
    <enable_shared_directory/>
    <enable_remotedesktop/>
    <enable_vm_savestate_usage/>
    <disable_automatic_checkpoints/>
    <multiattach_vdi_file>AIMNet_vm_v2.vdi</multiattach_vdi_file>
</vbox_job>


8.
Restart BOINC and get fresh work (start with a small work buffer)


<edit>
May be better to add this to the file_ref section of "app_info.xml" instead of just the filename line:
        <file_ref>
            <file_name>AIMNet_vm_v2.vdi</file_name>
            <open_name>vm_image.vdi</open_name>
        </file_ref>

</edit>
ID: 106452 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1994
Credit: 9,623,704
RAC: 9,591
Message 106465 - Posted: 2 Jul 2022, 9:27:47 UTC - in response to Message 106452.  

The steps below are for the impatient volunteers who want to test the differencing image method.


You-are-great!!!
But i prefer an OFFICIAL app from the project.
ID: 106465 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1994
Credit: 9,623,704
RAC: 9,591
Message 106698 - Posted: 3 Aug 2022, 14:30:20 UTC - in response to Message 106376.  

BOINC recently published vboxwrapper 26204 which introduces the code to deal with differencing images.


LHC-Dev project is testing the 26205 version, with some bugfix
Waiting for Rosetta@Home....
ID: 106698 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jim1348

Send message
Joined: 19 Jan 06
Posts: 881
Credit: 52,257,545
RAC: 0
Message 106699 - Posted: 3 Aug 2022, 15:43:17 UTC - in response to Message 106698.  

Waiting for Rosetta@Home....

There are not that many left.
Total queued jobs: 244,472


Given their glacial (or continental drift) rate of progress, I expect they would wait until the next series to introduce any changes.
If there is a next series at all.
ID: 106699 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
computezrmle

Send message
Joined: 9 Dec 11
Posts: 63
Credit: 9,680,103
RAC: 0
Message 106754 - Posted: 10 Aug 2022, 15:26:46 UTC - in response to Message 106452.  

Based on the experience from LHC@home the suggestion from post https://boinc.bakerlab.org/rosetta/forum_thread.php?id=14930&postid=106452 needs to be slightly refined.


Get vboxwrapper 26205 instead of 26204.
As long as 26205 is not available on https://boinc.berkeley.edu/dl/

get it from LHC@home


Linux version:
http://lhcathome.cern.ch/lhcathome/download/vboxwrapper_26205_x86_64-pc-linux-gnu

Windows version:
http://lhcathome.cern.ch/lhcathome/download/vboxwrapper_26205_windows_x86_64.exe



Replace "vboxwrapper_26204" with "vboxwrapper_26205" in app_info.xml


The tag "<open_name>vm_image.vdi</open_name>" at the bottom of the previous post is now obsolete and shouldn't be used.


Volunteers waiting for an official release from Rosetta may wait forever.
ID: 106754 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
kotenok2000
Avatar

Send message
Joined: 22 Feb 11
Posts: 259
Credit: 497,274
RAC: 1,201
Message 106757 - Posted: 10 Aug 2022, 17:53:52 UTC - in response to Message 106754.  
Last modified: 10 Aug 2022, 17:56:39 UTC

I get this error
10/08/2022 20:51:07 | Rosetta@home | [error] Application file AIMNet_vm_v2.vdi missing signature
10/08/2022 20:51:07 | Rosetta@home | [error] BOINC cannot accept this file
I'm running boinc 7.20.2
My app_info.xml is
<app_info>
    <app>
        <name>rosetta_python_projects</name>
        <user_friendly_name>rosetta python projects</user_friendly_name>
        <non_cpu_intensive>0</non_cpu_intensive>
    </app>
    <file_info>
        <name>vboxwrapper_26205_windows_x86_64.exe</name>
        <executable/>
    </file_info>
    <file_info>
        <name>boinc_app_v4</name>
        <executable/>
    </file_info>
    <file_info>
        <name>vbox_job_v4.xml</name>
    </file_info>
    <file_info>
        <name>AIMNet_vm_v2.vdi</name>
    </file_info>
    <app_version>
        <app_name>rosetta_python_projects</app_name>
        <version_num>103</version_num>
        <avg_ncpus>1.000000</avg_ncpus>
        <plan_class>vbox64</plan_class>
        <api_version>7.7.0</api_version>
        <file_ref>
            <file_name>vboxwrapper_26205_windows_x86_64.exe</file_name>
            <main_program/>
        </file_ref>
        <file_ref>
            <file_name>boinc_app_v4</file_name>
            <open_name>shared/boinc_app</open_name>
            <copy_file/>
        </file_ref>
        <file_ref>
            <file_name>vbox_job_v4.xml</file_name>
            <open_name>vbox_job.xml</open_name>
        </file_ref>
        <file_ref>
            <file_name>AIMNet_vm_v2.vdi</file_name>
			<open_name>vm_image.vdi</open_name>
        </file_ref>
        <dont_throttle/>
    </app_version>
</app_info>
ID: 106757 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
computezrmle

Send message
Joined: 9 Dec 11
Posts: 63
Credit: 9,680,103
RAC: 0
Message 106758 - Posted: 10 Aug 2022, 18:56:56 UTC - in response to Message 106757.  

Remove this line from your app_info.xml
<open_name>vm_image.vdi</open_name>


Backup the following files from your Rosetta project directory:

vboxwrapper_26205_windows_x86_64.exe
AIMNet_vm_v2.vdi (the already cloned one)
app_info.xml
vbox_job_v4.xml (the already modified one)
boinc_app_v4



Disable the client's network access
Reset the Rosetta project (this removes the app_version you got from the project; must be done to avoid getting mixed versions)
Shutdown the client
Restore the backed up files
Check if executable flags are set for vboxwrapper and boinc_app_v4
Start the client
Enable network access
ID: 106758 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jim1348

Send message
Joined: 19 Jan 06
Posts: 881
Credit: 52,257,545
RAC: 0
Message 106762 - Posted: 11 Aug 2022, 19:23:01 UTC

ID: 106762 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
computezrmle

Send message
Joined: 9 Dec 11
Posts: 63
Credit: 9,680,103
RAC: 0
Message 106763 - Posted: 11 Aug 2022, 19:43:19 UTC - in response to Message 106762.  

I suspect that you didn't reset the project.
If so, in client_state.xml there's still a signature of the original app_version files.

A project reset removes them.
ID: 106763 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jim1348

Send message
Joined: 19 Jan 06
Posts: 881
Credit: 52,257,545
RAC: 0
Message 106764 - Posted: 11 Aug 2022, 20:16:49 UTC - in response to Message 106763.  

Thanks, but a reset did not fix it. I still get the download errors.

I am wondering whether it is because I downloaded the files on a Windows machine, and then transferred them to my Ubuntu machine.
Maybe they picked up the wrong permissions or some other attributes?

What I am going to do (probably not today) is to totally detach from all BOINC projects, and then download the files again directly onto the Ubuntu machine.
It should work.
ID: 106764 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
computezrmle

Send message
Joined: 9 Dec 11
Posts: 63
Credit: 9,680,103
RAC: 0
Message 106766 - Posted: 12 Aug 2022, 4:00:14 UTC - in response to Message 106764.  

Files listed in app_info.xml must be present in Rosetta's project dir before the BOINC client starts (including app_info.xml itself).
My guess would be that after the project reset your client contacted the server and got an app_version update from there.

If this happened your client_state.xml will contain <signature_required/> tags related to Rosetta files you mentioned in app_info.xml (like AIMNet_vm_v2.vdi). These would be wrong.

In addition the client must be restarted to read app_info.xml.
ID: 106766 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jim1348

Send message
Joined: 19 Jan 06
Posts: 881
Credit: 52,257,545
RAC: 0
Message 106767 - Posted: 12 Aug 2022, 5:21:11 UTC - in response to Message 106766.  

Files listed in app_info.xml must be present in Rosetta's project dir before the BOINC client starts (including app_info.xml itself).

That could be it. But how do you prevent BOINC from starting? There is no option for that in the GUI. You may be referring to a command line option.
ID: 106767 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
computezrmle

Send message
Joined: 9 Dec 11
Posts: 63
Credit: 9,680,103
RAC: 0
Message 106768 - Posted: 12 Aug 2022, 5:58:49 UTC - in response to Message 106767.  

Pause BOINC's network activity, pause Rosetta or do both before you reset the project and shutdown BOINC.
Then copy the required files back before you restart BOINC.
See:
https://boinc.bakerlab.org/rosetta/forum_thread.php?id=14930&postid=106758


Using the "anonymous platform" is a rather old method, hence not perfectly user friendly.
But once it is correctly set up it is as reliable as an app_version from the server.
ID: 106768 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jim1348

Send message
Joined: 19 Jan 06
Posts: 881
Credit: 52,257,545
RAC: 0
Message 106771 - Posted: 12 Aug 2022, 10:28:38 UTC - in response to Message 106768.  

Pause BOINC's network activity, pause Rosetta or do both before you reset the project and shutdown BOINC.

We are getting closer, but the question is how do you shutdown BOINC? There is no command for it in the GUI.
The only commands are for pausing BOINC and shutting down the network (no new tasks).
(Would a reboot do it?)

I get the following error message at Step 5,
$ sudo vboxmanage clonemedium AIMNet_vm_v2.vdi.bak AIMNet_vm_v2.vdi
[sudo] password for jim:
0%...
Progress state: VBOX_E_FILE_ERROR
VBoxManage: error: Failed to clone medium
VBoxManage: error: Could not create the clone medium '/var/lib/boinc-client/projects/boinc.bakerlab.org_rosetta/AIMNet_vm_v2.vdi'.
VBoxManage: error: VDI: cannot create image '/var/lib/boinc-client/projects/boinc.bakerlab.org_rosetta/AIMNet_vm_v2.vdi' (VERR_ALREADY_EXISTS)
VBoxManage: error: Details: code VBOX_E_FILE_ERROR (0x80bb0004), component MediumWrap, interface IMedium
VBoxManage: error: Context: "RTEXITCODE handleCloneMedium(HandlerArg*)" at line 1071 of file VBoxManageDisk.cpp

So the original .vdi file apparently still exists.

But one thing I did learn is that you can't use VirtualBox installed from a downloaded .deb package.
You have to use VirtualBox installed via "sudo apt install virtualbox".
Otherwise, the "vboxmanage" command will not run at all.
ID: 106771 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
computezrmle

Send message
Joined: 9 Dec 11
Posts: 63
Credit: 9,680,103
RAC: 0
Message 106773 - Posted: 12 Aug 2022, 13:02:42 UTC - in response to Message 106771.  

how do you shutdown BOINC?

In your BOINC Manager (GUI) isnt' there a menu entry like "File -> Shutdown the currently connected client".
(Sorry I'm running the German GUI, hence don't know the exact term)


On Linux you may run "[sudo] systemctl stop boinc-client".


I get the following error message at Step 5,
.
.
.
So the original .vdi file apparently still exists.

The file isn't the problem.
Open your VirtualBox GUI -> Media Manager and remove the disk entry there (it shouldn't be connected to a VM).
You will be asked if you also want to remove the file. Don't remove the file.
ID: 106773 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jim1348

Send message
Joined: 19 Jan 06
Posts: 881
Credit: 52,257,545
RAC: 0
Message 106774 - Posted: 12 Aug 2022, 15:08:20 UTC - in response to Message 106773.  

On Linux you may run "[sudo] systemctl stop boinc-client".
Thanks. That is what I needed for that part.
On Windows, they have "Exit BOINC" in the GUI, which is probably what you wanted there.


Open your VirtualBox GUI -> Media Manager and remove the disk entry there (it shouldn't be connected to a VM).
You will be asked if you also want to remove the file. Don't remove the file.
There is no disk (or file) at all in VBox Media Manager.


And I am getting the same error messages as before.
$ sudo vboxmanage clonemedium AIMNet_vm_v2.vdi.bak AIMNet_vm_v2.vdi
[sudo] password for jim:
0%...
Progress state: VBOX_E_FILE_ERROR
VBoxManage: error: Failed to clone medium
VBoxManage: error: Could not create the clone medium '/var/lib/boinc-client/projects/boinc.bakerlab.org_rosetta/AIMNet_vm_v2.vdi'.
VBoxManage: error: VDI: cannot create image '/var/lib/boinc-client/projects/boinc.bakerlab.org_rosetta/AIMNet_vm_v2.vdi' (VERR_ALREADY_EXISTS)
VBoxManage: error: Details: code VBOX_E_FILE_ERROR (0x80bb0004), component MediumWrap, interface IMedium
VBoxManage: error: Context: "RTEXITCODE handleCloneMedium(HandlerArg*)" at line 1071 of file VBoxManageDisk.cpp


It could be a permissions problem.

I install BOINC using "sudo apt install boinc", which gets it from Ubuntu I suppose.
I don't try to compile it myself, so it is not in the "Home" directory, which may be a difference.

I will give it a rest for a while, and try again later. Thanks for you input.
ID: 106774 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jim1348

Send message
Joined: 19 Jan 06
Posts: 881
Credit: 52,257,545
RAC: 0
Message 106775 - Posted: 12 Aug 2022, 15:31:28 UTC - in response to Message 106774.  

But I did reboot, then a reset, then stopped BOINC and deleted the "AIMNet_vm_v2.vdi" file so that I could try again.

I then ran the "sudo vboxmanage clonemedium AIMNet_vm_v2.vdi.bak AIMNet_vm_v2.vdi" command, and got:
$ sudo vboxmanage clonemedium AIMNet_vm_v2.vdi.bak AIMNet_vm_v2.vdi
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...
Progress state: NS_ERROR_INVALID_ARG
VBoxManage: error: Failed to clone medium
VBoxManage: error: Cannot register the hard disk '/var/lib/boinc-client/projects/boinc.bakerlab.org_rosetta/AIMNet_vm_v2.vdi' {11d8afd8-b2da-4d2b-9e01-6bcb604c710a} because a hard disk '/var/lib/boinc-client/projects/boinc.bakerlab.org_rosetta/AIMNet_vm_v2.vdi' with UUID {f93b7440-2207-4969-bc1f-bbd87433063a} already exists
VBoxManage: error: Details: code NS_ERROR_INVALID_ARG (0x80070057), component VirtualBoxWrap, interface IVirtualBox
VBoxManage: error: Context: "RTEXITCODE handleCloneMedium(HandlerArg*)" at line 1071 of file VBoxManageDisk.cpp

So it does think that a hard disk already exists.
However, it still does not show up in VirtualBox Manager.

There may be something wrong with the VirtualBox installation.
That is beyond me for now.
ID: 106775 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
computezrmle

Send message
Joined: 9 Dec 11
Posts: 63
Credit: 9,680,103
RAC: 0
Message 106776 - Posted: 12 Aug 2022, 15:41:19 UTC - in response to Message 106774.  
Last modified: 12 Aug 2022, 15:43:56 UTC

VERR_ALREADY_EXISTS tells you there's a disk registered with this name/UUID

Try
vboxmanage list hdds

if AIMNet_vm_v2.vdi is in the list, try
vboxmanage closemedium UUID


In addition, check the owner/group of the files.
Compare that with files from other projects.


<edit>
If you run vboxmanage commands or if you use the GUI, use the account of your BOINC user as each user on your box has it's own vbox environment.
</edit>
ID: 106776 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Previous · 1 · 2 · 3 · Next

Message boards : Number crunching : Multiattach mode disk images



©2024 University of Washington
https://www.bakerlab.org