Hi,

I want to test Rosetta@home in a VM (VirtualBox) on our desktop systems. Naive as I am I thought, it is sufficient to just install the boinc package in Ubuntu 10.04 and spread this VM over all the machines. My Problem is now, that I have jobs calculated but the clients can't send them back. I see all my systems in "your computers" but only one has really some credit. Is my approach wrong? And is there a possibility to do this with only one VM image? I mount them immutable, only the dir /var/lib/bonic-client is on a NFS server.
Thanks for any answers.

Cheers
Michael

I'm no expert but can you give the images different host names?

---

is there a possibility to do this with only one VM image?

...not sure I understood that part of the question. Perhaps "do this" is where I am unclear.

I think what you wanted to do was install a BOINC client in your image, deploy the VM to multiple machines, then attach to the project on each machine (or there are BOINC setup files that can attach for you upon first running BOINC). If you attach first and then deploy the image the hosts conflict. By attaching on each machine image, each establishes it's own host ID.

Now that it sounds as though you've done it the other way around, I'd suggest just setting the project to no new work, working through and reporting back what you've downloaded, then detach from the project, then attach to the project again. This should get each machine (except one) a fresh host ID.

...now I see you do have 25 host IDs. What problem are you having in sending results back? Are there some error messages in the BOINC Manager?

How are you seperating machine 1's /var/lib/bonic-client from machine 2's /var/lib/bonic-client??

---

Rosetta Moderator: Mod.Sense

...
...now I see you do have 25 host IDs. What problem are you having in sending results back? Are there some error messages in the BOINC Manager?

"temporarily failed upload of ...: transient upload error" on each VM.

How are you seperating machine 1's /var/lib/bonic-client from machine 2's /var/lib/bonic-client??

Ok here's what I've done. I wanted to test BOINC. I've read that Ubuntu has boinc packages. So I installed Ubuntu 10.04 in VirtualBox 3.1x on my machine. I chose Rosetta and ran some first calculations over the night. After this first test I created a shared folder for VBox and mounted it with rights boinc:boinc to /var/lib/boinc-client, so that I'm indep. of the VM. I copied the contents of the old dir to this new shared folder. Everything seemed to work fine after that. So the next step was, I copied the VM to the Server for NFS export. So I did with the boin-client dir. I started a second VM on a different host (the same VM over NFS in immutable mode, this creates a local diff file). The VM mounted the boinc-client dir (shared folder) which was a copy of the first VM. But I had some problems with the client (forgot what). So I decided to reset the second boinc-client. This cleared the dir for Rosetta and after that the client seemed to work. I first did a copy of the reseted boinc-client dir. This copy I used for all the other clients. Every VM has it's own NFS dir with it's own boinc-client dir. This NFS dir is mounted on boot to the host, and the VM uses it as a shared folder and mounts it to /var/lib/boinc-client.

cheers
Michael

...
...now I see you do have 25 host IDs. What problem are you having in sending results back? Are there some error messages in the BOINC Manager?

"temporarily failed upload of ...: transient upload error" on each VM.

How are you seperating machine 1's /var/lib/bonic-client from machine 2's /var/lib/bonic-client??

Ok here's what I've done. I wanted to test BOINC. I've read that Ubuntu has boinc packages. So I installed Ubuntu 10.04 in VirtualBox 3.1x on my machine. I chose Rosetta and ran some first calculations over the night. After this first test I created a shared folder for VBox and mounted it with rights boinc:boinc to /var/lib/boinc-client, so that I'm indep. of the VM. I copied the contents of the old dir to this new shared folder. Everything seemed to work fine after that. So the next step was, I copied the VM to the Server for NFS export. So I did with the boin-client dir. I started a second VM on a different host (the same VM over NFS in immutable mode, this creates a local diff file). The VM mounted the boinc-client dir (shared folder) which was a copy of the first VM. But I had some problems with the client (forgot what). So I decided to reset the second boinc-client. This cleared the dir for Rosetta and after that the client seemed to work. I first did a copy of the reseted boinc-client dir. This copy I used for all the other clients. Every VM has it's own NFS dir with it's own boinc-client dir. This NFS dir is mounted on boot to the host, and the VM uses it as a shared folder and mounts it to /var/lib/boinc-client.

cheers
Michael

Each running VM will need it's own data directory. This is where things like host ID and state are stored, along with current active tasks and results that have to be uploaded. This information cannot be shared with another active host. So you would have to configure each VM to use a named data directory, or somehow map each machine to a unique data directory.

I think what's happening here is that once you get a host passed initial problems (the problems are likely the conflict with the last host you installed) it has established a new host ID. It may then be seeing completed results from another host and trying to upload them (which can only be done by the host ID they were issued to).

---

Rosetta Moderator: Mod.Sense

Sorry for the second post, your forum had a serious hiccup.

Each running VM will need it's own data directory.

I know the setup is a bit complicated, because I use shared folders feature of VBox to mount a host path, which actually is a NFS share. But I can assure you that every VM has it's own boinc-client dir on the server.

This is where things like host ID and state are stored, along with current active tasks and results that have to be uploaded. This information cannot be shared with another active host. So you would have to configure each VM to use a named data directory, or somehow map each machine to a unique data directory.

The dirs are unique, but at the beginning they were exact copies of the initial VM. Only the Rosetta dir was empty, because I did a reset before deployment.

I think what's happening here is that once you get a host passed initial problems (the problems are likely the conflict with the last host you installed) it has established a new host ID. It may then be seeing completed results from another host and trying to upload them (which can only be done by the host ID they were issued to).

Hmm my VMs seem to calculate different jobs, but maybe the scheduler gets somehow confused.

It probably does get confused.

Now, I have been scratching my head trying to figure out why you are doing this... using multiple VMs on a single machine will allow you to have more tasks in flight at the same time but the overhead of the VMs is going to mean that ultimately you are doing less work than you would if you just had one BOINC installation...

Now, I have been scratching my head trying to figure out why you are doing this... using multiple VMs on a single machine will allow you to have more tasks in flight at the same time but the overhead of the VMs is going to mean that ultimately you are doing less work than you would if you just had one BOINC installation...

I would say this depends on the machinery you use and on the jobs you want to calculate. If the jobs calculate only on one core or they need not that much RAM, I would say that you could do more work with more than one VM. I read a paper on this topic.

But you're getting me wrong. Maybe it's because of my horrible English, I don't know. I have only one VM for all hosts. It gets mounted via NFS and started in readonly mode (immutable mode, local diff). The dir /var/lib/boinc is different for every VM. I use shared folders for that. This mounts a dir on the host into the VM. That dir is on NFS as well, and is mounted by the host on boot. I could mount this dir directly to the VM via NFS, but I'm kinda lazy, and the host already prepares this share, so it does not conflict with other shares. I hope it gets clearer now.

Yesterday I reinstalled the client into a VM, and cleared all old data (rm -rf /etc/boinc-client /var/lib/boinc-client). I created a new project and started one job. But after it finished it still couldn't be uploaded.

I started wireshark an got this when I tried to upload the results:
client: /rosetta_cgi/file_upload_handler
<data server request>
...
<get_file_size>lrm_...</get_file_size>...

server: 200 OK
<data_server_reply>
<status>0</status>
<file_size>0</file_size>...

Next package finishes the connection. Any Ideas? After reinstallation of boinc, old / duplicate data shouldn't be the problem. Do I need a unique hostname?

cheers
Michael

Next package finishes the connection. Any Ideas? After reinstallation of boinc, old / duplicate data shouldn't be the problem. Do I need a unique hostname?

Hostname? No, but they should get unique host IDs assigned when they attach to the project. The host IDs will be shown in the messages when BOINC first starts up.

That portion of your wireshark looks normal. Since the file size on the server is zero, the next interaction should be the client attempting to send the file.

---

Rosetta Moderator: Mod.Sense

Hostname? No, but they should get unique host IDs assigned when they attach to the project. The host IDs will be shown in the messages when BOINC first starts up.

Example:
1287876 ubuntu-mini 1.76 29 AuthenticAMD
AMD Athlon(tm) 64 X2 Dual Core Processor 4800+ [Family 15 Model 107 Stepping 1] Linux
2.6.32-22-generic 10 23 May 2010 17:34:43 UTC

I just clicked "update", or what the name of the button on top in English is named. This VM has one Job I ran today but can't upload it. This Job can't be uploaded. I have 2 clients running today, both have different IDs ("last connected").
The client doesn't try to send anything. I have 20 packets in wireshark 6 DNS and rest TCP (incl. 2 HTTP Messages I already posted). After the 200 OK I see only 4 packets:
17: client ack for packet 16 (the 200 OK)
18: server fin,ack
19: client fin,ack
20: server ack

And the log in the manager shows:
"temporarily failed upload of ...: transient upload error"
"backing off..."

cheers
Michael

I would say this depends on the machinery you use and on the jobs you want to calculate. If the jobs calculate only on one core or they need not that much RAM, I would say that you could do more work with more than one VM. I read a paper on this topic.

Um, no ...

Ok, a Virtual Machine means that on a base computer you create "Virtual" environments because the environment on the base host is not what you want ... not enough memory, different processors, OS, or whatever... or you want to simulate many machines on one machine. In this case you launch multiple VMs that each "seem" to be complete computers. Into these VM you can install whatever programs you want ... Like I can run a VM on my Mac that emulates windows machines ... but to do that I will always have the overhead of the VM, however slight that might be and that eats into the CPU cycles available to do other things ...

Now, because some BOINC Windows programs may be much more efficient due to an optimized compiler it is certainly possible that I would be able to run windows in a VM and get more BOINC production but I have not tried that experiment for any number of reasons ...

But it sounds to me like you are not doing that ... you are trying to set up a common image of a computing environment and then just using that image on multiple and separate machines in lieu of just installing BOINC on the individual machines ... and to that point I still don't understand the interest ... if you install BOINC natively on each machine it will use that machines full capabilities as possibly limited by the constraints allowed by BOINC...

As to the issues I think that the simplest explanation as to why you cannot do the uploads is that in your configuration BOINC does not correctly follow or find the data directory.

But that still brings me back to the simplest question ... if you are trying to run BOINC on 5 machines why don't you want to install BOINC on each machine? How is all this buying you an improvement? Only one copy of BOINC? The disk footprint of BOINC is tiny in today's world ... the biggest part is the data directories and that can't be common for BOINC to work correctly (as BOINC is currently contrived)...

One way there might be to start up each of the VM, install BOINC with each VM copy pointing to its own unique data directory and common BOINC directory ... save each VM with BOINC installed and go from there ... but, I think this is a misuse of VM technology to no essential gain ... you still have 5 separate installs of BOINC albeit each one is a VM that is running on one and only one machine ... meaning that you have no specific gain from creating a VM in the first place ... Because almost be definition virtually all VMs actually have fewer features than the base machines (at least with all the VMs I am aware of or have used ... a Xeon turns into a 486 for example) ...

Unless I completely misread your explanation ...

{edit}

I would say this depends on the machinery you use and on the jobs you want to calculate. If the jobs calculate only on one core or they need not that much RAM, I would say that you could do more work with more than one VM. I read a paper on this topic.

Um, can you point to that paper on-line?

Clarification, if on my windows boxes, I wanted to I could create multiple VMs, in each have a copy of BOINC running and run WUProp and FreeHAL and boost my credit scores and not do much at all ... for Rosetta tasks, running them natively on a single BOINC installation I will use all CPU cycles and memory to run the launched RaH tasks without using any memory or CPU to run the VM emulation. If I create two VM on that same machine and install and run BOINC on them to run RaH, I would get less done because of the overhead of the VM than I do now ... RaH tasks are compute bound, not I/O or memory bound (well, if you have way too little memory you will be all three as Windows thrashes the virtual memory in and out of the disk drive) ...

I would say this depends on the machinery you use and on the jobs you want to calculate. If the jobs calculate only on one core or they need not that much RAM, I would say that you could do more work with more than one VM. I read a paper on this topic.
cheers Michael

I think what you are missing is that even though you have a VM you are still using the pc's cpu to run it, so if you are running Rosetta on the main pc and the VM pc too, you are still using the SAME cpu to do it. You cannot increase the total number of hardware components thru VM. Yes VM works very well for things where you do not use the same parts for both the real pc and the virtual pc, but when the same parts are used for each, both slow down. For example I run a Boinc cpu project and a Boinc project that uses memory only and a Boinc gpu project on my pc's. On the real part of the box I crunch with all three but on the VM box I can only crunch the memory only project. This is because I only have 4 cpu's and I can allocate them to either the real or the VM box for crunching but not both, or they both slow down. If I allocate 4 cpu's to the real pc and I then allocate 1 cpu to the VM part, 1 of those cpu's is shared between them. Memory though is another story, since I have 4 gig of ram I can allocate 1 gig to the VM part and that leaves 3 gig for the real part. If I allocate 3 cpu's to the real part and 1 cpu to the VM part I am not saving anything in Boinc. I am just creating problems.

Does that make sense?

For example I run a Boinc cpu project and a Boinc project that uses memory only and a Boinc gpu project on my pc's. On the real part of the box I crunch with all three but on the VM box I can only crunch the memory only project.

If you are speaking of FreeHAL it is not a memory only project though that is a popular misconception ... the CPU use is low as it is another NCI project like DepSPIDER (Closed) and Anansi (Inactive) or even WUProp and QCN... But, indeed the most common load people notice with FreeHAL is the memory footprint which depending on the version can be fairly high ...

For example I run a Boinc cpu project and a Boinc project that uses memory only and a Boinc gpu project on my pc's. On the real part of the box I crunch with all three but on the VM box I can only crunch the memory only project.

If you are speaking of FreeHAL it is not a memory only project though that is a popular misconception ... the CPU use is low as it is another NCI project like DepSPIDER (Closed) and Anansi (Inactive) or even WUProp and QCN... But, indeed the most common load people notice with FreeHAL is the memory footprint which depending on the version can be fairly high ...

You are indeed correct, FreeHal is in fact a cpu project, it just uses very little cpu due to timing tweaks.

...
Um, can you point to that paper on-line?
...

Paper: Optimizing Grid Site Manager Performance with Virtual Machines

At least I found out why the clients didn't upload finished jobs. After the wireshark results I suspected the network connection as possible source of error. The problem was the ca-bundle.crt file. Since shared folders do not work with links, I simply copied these files. And all the trouble because of this... Now I mount the NFS dirs directly via VM. And I finally could submit (some of) my old jobs.

I think what you are missing is that even though you have a VM you are still using the pc's cpu to run it, so if you are running Rosetta on the main pc and the VM pc too, you are still using the SAME cpu to do it.
...

I don't use the host for rosetta calculations. They are simply linux workstations for students and sometimes a second vm is started for windows courses ;). Until this point it seems stable.

Why I do this? I want to test the environment for all kind of background work, that's why I use the VM.

Cheers
Michael

I don't use the host for rosetta calculations. They are simply linux workstations for students and sometimes a second vm is started for windows courses ;). Until this point it seems stable.

Why I do this? I want to test the environment for all kind of background work, that's why I use the VM.

Cheers Michael

Now that is a GOOD use of VM!! I thought you were trying to increase your Rosetta crunching abilities.

...
Um, can you point to that paper on-line?
...

Paper: Optimizing Grid Site Manager Performance with Virtual Machines

At least I found out why the clients didn't upload finished jobs. After the wireshark results I suspected the network connection as possible source of error. The problem was the ca-bundle.crt file. Since shared folders do not work with links, I simply copied these files. And all the trouble because of this... Now I mount the NFS dirs directly via VM. And I finally could submit (some of) my old jobs.

I think what you are missing is that even though you have a VM you are still using the pc's cpu to run it, so if you are running Rosetta on the main pc and the VM pc too, you are still using the SAME cpu to do it.
...

I don't use the host for rosetta calculations. They are simply linux workstations for students and sometimes a second vm is started for windows courses ;). Until this point it seems stable.

Why I do this? I want to test the environment for all kind of background work, that's why I use the VM.

Cheers
Michael

Thanks for the paper ... interesting read ... and I am glad you found your issue(s) and are up and running. I would note two points however and that is:

First, BOINC tasks with rare exceptions are compute bound tasks and as such are not really addressed by the paper (at least not on my first read) and spawning multiple VMs on a single machine will not do anything to improve the performance of a task assigned to a VM machine as opposed to running it natively. Exceptions are the NCI tasks from the projects I mentioned before, where a person might create one or more VMs to run multiple copies of BOINC and these NCI projects increasing the number of tasks "in flight" and thus the "earning power" of that machine.

Second, BOINC itself is more or less nothing more than a different approach to virtualizing the "system" so that large compute bound tasks can be segmented and processed on weaker machines in a timely manner. GPU Grid which is also doing molecular modeling actually assigns tasks which are part of a larger problem so that they can get reasonable response times on their models ... by that I mean that the 6+ hour task you run is only part of the larger model ... your result creates a new task which after being processed may generate an additional generation ... MW is another project that is doing something similar... the point being, there are differing approaches to slicing up the problems ...

The paper is looking at how to better use the cluster resources to process a job mix and they sponsor a virtualization approach... BOINC looks at a large compute bound problem that cannot be economically processed on limited resources and solves the problem another way ...

No matter ... good luck with your experiments ...

Now that is a GOOD use of VM!! I thought you were trying to increase your Rosetta crunching abilities.

I don't need VMs to fake that, I have enough machines for that. But I only can use them a few weeks. After that my tests are over :(.

...
First, BOINC tasks with rare exceptions are compute bound tasks and as such are not really addressed by the paper (at least not on my first read) and spawning multiple VMs on a single machine will not do anything to improve the performance of a task assigned to a VM machine as opposed to running it natively. Exceptions are the NCI tasks from the projects I mentioned before, where a person might create one or more VMs to run multiple copies of BOINC and these NCI projects increasing the number of tasks "in flight" and thus the "earning power" of that machine.
...

I never said that Rosetta jobs could be used i such a scenario. I only said that I wouldn't say that you generally can't optimize you CPU usage with VMs.

No matter ... good luck with your experiments ...

Thanks. And I know that BOINC offers me a lot of configuration options like "don't use in case of mouse action", which I give away by using it in a VM. But I want to test if other applications work in this environment too, which usually don't offer any of these configurations (like a cluster/grid). I hope starting the VM with nice level 19 is sufficient for this or I'll have to search for a different scheduler.

Cheers
Michael

Thanks. And I know that BOINC offers me a lot of configuration options like "don't use in case of mouse action", which I give away by using it in a VM. But I want to test if other applications work in this environment too, which usually don't offer any of these configurations (like a cluster/grid). I hope starting the VM with nice level 19 is sufficient for this or I'll have to search for a different scheduler.

You should try it with those options disabled... BOINC is pretty good about getting out of the way of other tasks because BOINC and the science applications run at very low priorites ... the one exception is GPU class where if the kernals are not properly tuned you can get a very laggy system ... also the CPU side is run at a slightly higher priority so that the GPUs don't get starved for data because the CPU is patiently waiting for something to give it a slice ...

Even so, I run up to dual GPUs (once had a quad GPU system with two GTX295 cards in it) and have not really seen issues even with the GPUs with rare exceptions where the GPU application was not properly tuned ...