New Docker image with Virtualbox and Nvidia GPU passthrough support

Message boards : Number crunching : New Docker image with Virtualbox and Nvidia GPU passthrough support

To post messages, you must log in.

AuthorMessage
Profile Ryan Ludwig

Send message
Joined: 21 Feb 11
Posts: 4
Credit: 6,609,198
RAC: 0
Message 102915 - Posted: 10 Oct 2021, 21:47:48 UTC
Last modified: 10 Oct 2021, 21:49:08 UTC

Hey R@H fam!

I've been playing around with docker and have created a docker build file that passes through virtualbox and an Nvidia GPU from the host to the container

My PCs (Host: Ubuntu 20.04) running this have been successfully completing the python vbox project WUs along with GPUGrid acemd CUDA WUs so at some level this does work.

I would love any suggestions at improving it from anyone here!

https://github.com/WeLiveInSpace/docker-boinc-vbox-nv_gpu
ID: 102915 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1866
Credit: 8,186,159
RAC: 7,029
Message 102916 - Posted: 11 Oct 2021, 7:01:30 UTC - in response to Message 102915.  

Hey R@H fam!
I've been playing around with docker and have created a docker build file that passes through virtualbox and an Nvidia GPU from the host to the container


Wait, wait
Are you saying you're crunching python wus ON the gpu and not on the cpu?
Results?
ID: 102916 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
bozz4science

Send message
Joined: 2 May 20
Posts: 7
Credit: 228,784
RAC: 0
Message 102917 - Posted: 11 Oct 2021, 9:44:30 UTC - in response to Message 102916.  

Sounds very exciting! Didn't know that you could pass through a virtual box designed CPU-intended WU to an NVIDIA GPU using a docker image. Will take a look later this week and test this on my system! Is there a way to limit tasks to the python tasks only for testing?

P.S. This docker image could potentially also run on Windows, right?
ID: 102917 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1866
Credit: 8,186,159
RAC: 7,029
Message 102919 - Posted: 11 Oct 2021, 12:29:42 UTC - in response to Message 102917.  
Last modified: 11 Oct 2021, 12:30:02 UTC

P.S. This docker image could potentially also run on Windows, right?

Yes. Windows 11, for example, completely supports Docker and also the access to gpu from the virtual machine (Cuda on WSL, AMD on WLS, etc)
ID: 102919 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1866
Credit: 8,186,159
RAC: 7,029
Message 102920 - Posted: 11 Oct 2021, 13:25:12 UTC - in response to Message 102919.  

P.S. This docker image could potentially also run on Windows, right?

Yes. Windows 11, for example, completely supports Docker and also the access to gpu from the virtual machine (Cuda on WSL, AMD on WLS, etc)


My fault.
Also Windows 10 (from 1809 release) supports Docker and "gpu acceleration" on WSL2
ID: 102920 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Ryan Ludwig

Send message
Joined: 21 Feb 11
Posts: 4
Credit: 6,609,198
RAC: 0
Message 102923 - Posted: 12 Oct 2021, 0:33:43 UTC - in response to Message 102916.  
Last modified: 12 Oct 2021, 0:37:24 UTC

Good question! Unfortunately nothing that exciting.

Containers are a great way to rapidly deploy BOINC with all needed dependencies out to your systems. Though by default they have no access to low level devices on the host. This makes containerizing BOINC a bit tricky as you want to A) standardize the BOINC rollout without B) Artificially limiting the type of WU's you can complete.

This is a docker image build that lets you run a containerized BOINC instance without limiting the WU's you are able to complete due to running BOINC within said container.

I threw a README in the git repo. But essentially the setup process for a fresh debian system is...

1. Install base nvidia drivers on host
2. reboot
3. Install nvidia-container-toolkit, virtualbox-dkms, virtualbox
4. reboot
5. clone repo, cd into repo, run docker build
6. Run image (example Run commands in git repo)

You can seamlessly migrate over from a bare-metal BOINC install by modifying the RUN command (Step #6).

Requirements for migration (unsure about Windows, this again may only apply to Linux)
1. Suspend current WU's and completely stop the existing BOINC client. (eg. systemctl stop boinc-client)
2. In the RUN command
--a) point the volume (-v /path/to...) to your existing BOINC data directory
--b) (optional) modify the hostname to match your existing BOINC hostname. <-- This may not work as it will conflict with the host. Regardless I've found that there's some unique identifier in the BOINC data directory as projects transferred over my host average + total credit even though the hostname changed.

Once the build is vetted to work on other systems (IDK about windows and what exactly it allows you to passthrough) I'll create docker.io images so you no don't need to build the image yourself and can instead pull directly from docker.io. :)

I also have docker build files for AMD's ROCM and the generic Intel OpenCL runtimes. Both work and have completed WU's on WCG. Still working on getting repo's setup for them.[/list]
ID: 102923 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Ryan Ludwig

Send message
Joined: 21 Feb 11
Posts: 4
Credit: 6,609,198
RAC: 0
Message 102924 - Posted: 12 Oct 2021, 0:42:30 UTC - in response to Message 102920.  
Last modified: 12 Oct 2021, 0:43:28 UTC

P.S. This docker image could potentially also run on Windows, right?

Yes. Windows 11, for example, completely supports Docker and also the access to gpu from the virtual machine (Cuda on WSL, AMD on WLS, etc)


My fault.
Also Windows 10 (from 1809 release) supports Docker and "gpu acceleration" on WSL2



I didn't know this! Interesting... My build likely will not work in Windows docker containers as Microsoft threw some dependencies in the prebuilt image they are using. Hence the "The container base image must be mcr.microsoft.com/windows:1809 or newer."

I don't have much experience with windows docker deploys but I can look into creating a custom build to suit that use case!
ID: 102924 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1866
Credit: 8,186,159
RAC: 7,029
Message 102926 - Posted: 12 Oct 2021, 7:06:37 UTC

I also have docker build files for AMD's ROCM and the generic Intel OpenCL runtimes. Both work and have completed WU's on WCG. Still working on getting repo's setup for them


I don't have much experience with windows docker deploys but I can look into creating a custom build to suit that use case!


You're doing a GREAT work!!
I hope that the admins of project will use these ideas/codes to use better ours pc (es. gpu/better work distribution/etc)

P.S. If you need help, there is some MS documentation about Docker on Windows, like this. There is also documentations from Docker's site like this and this
ID: 102926 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1866
Credit: 8,186,159
RAC: 7,029
Message 102933 - Posted: 12 Oct 2021, 14:43:50 UTC

Interesting news: WSL will be published on Windows Store and not more as a part of Windows.

There are two big reasons to be excited for this change: You can get access to WSL features faster, and you don’t have to worry about changing your Windows version when getting the latest WSL updates.This change moves those binaries from being part of the Windows image, to instead being part of an application that you install from the Store. This decouples WSL from your Windows version, allowing you to update through the Microsoft Store instead. So now once new features like GUI app support, GPU compute, and Linux file system drive mounting are developed, tested and ready for a release you will get access to it right away on your machine without needing to update your entire Windows OS, or going to Windows Insider preview builds.

ID: 102933 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1866
Credit: 8,186,159
RAC: 7,029
Message 102981 - Posted: 19 Oct 2021, 7:26:35 UTC - in response to Message 102933.  

Seems that Nvidia GPU is widely used with Rosetta code in Baker Lab: Design of proteins
Maybe, one day, on our home computers......
ID: 102981 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Greg_BE
Avatar

Send message
Joined: 30 May 06
Posts: 5664
Credit: 5,711,666
RAC: 1,996
Message 102983 - Posted: 19 Oct 2021, 19:32:44 UTC - in response to Message 102981.  

Seems that Nvidia GPU is widely used with Rosetta code in Baker Lab: Design of proteins
Maybe, one day, on our home computers......



Hah! IF we are lucky..in the next decade.
The paper is quoting a run time of a $880 GPU. 90 minutes. So put that down on our level....at least double if not triple that time.
And a 225W power draw...damn! I would have to get a separate power supply just for that beast.
It scores 38th out of all GPUs and my 1070 scores 64th.
Baker lab uses nice stuff internally, but has yet to adapt their stuff for BOINC GPU work.
GPU has been asked for over and over again and the response is the same or ignored.
ID: 102983 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1866
Credit: 8,186,159
RAC: 7,029
Message 102987 - Posted: 20 Oct 2021, 8:00:06 UTC - in response to Message 102983.  
Last modified: 20 Oct 2021, 8:19:12 UTC

The paper is quoting a run time of a $880 GPU. 90 minutes. So put that down on our level....at least double if not triple that time.

So, what's the problem? I'm crunching Folding@Home on my entry-level gpu with wus longer than 24hs.
And, in the Boinc's world, there are a lot of people with high-end gpus


GPU has been asked for over and over again and the response is the same or ignored.

That's for sure

P.S.
Don't forget that a gpu client will attract a lot of volunteers.
ID: 102987 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
mikey
Avatar

Send message
Joined: 5 Jan 06
Posts: 1894
Credit: 8,767,498
RAC: 6,467
Message 102988 - Posted: 20 Oct 2021, 11:41:13 UTC - in response to Message 102987.  

The paper is quoting a run time of a $880 GPU. 90 minutes. So put that down on our level....at least double if not triple that time.

So, what's the problem? I'm crunching Folding@Home on my entry-level gpu with wus longer than 24hs.
And, in the Boinc's world, there are a lot of people with high-end gpus


GPU has been asked for over and over again and the response is the same or ignored.

That's for sure

P.S.
Don't forget that a gpu client will attract a lot of volunteers.


Not if a task tasks 3 or 4 days to run it though, yes most of us run tasks non stop but our gpu's do get a slight break between tasks even if we are running multiple tasks at once, but running the same task nonstop for 3 or 4 days could burn out a gpu if the settings aren't right.
ID: 102988 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Greg_BE
Avatar

Send message
Joined: 30 May 06
Posts: 5664
Credit: 5,711,666
RAC: 1,996
Message 102994 - Posted: 20 Oct 2021, 19:13:41 UTC - in response to Message 102988.  

The paper is quoting a run time of a $880 GPU. 90 minutes. So put that down on our level....at least double if not triple that time.

So, what's the problem? I'm crunching Folding@Home on my entry-level gpu with wus longer than 24hs.
And, in the Boinc's world, there are a lot of people with high-end gpus


GPU has been asked for over and over again and the response is the same or ignored.

That's for sure

P.S.
Don't forget that a gpu client will attract a lot of volunteers.


Not if a task tasks 3 or 4 days to run it though, yes most of us run tasks non stop but our gpu's do get a slight break between tasks even if we are running multiple tasks at once, but running the same task nonstop for 3 or 4 days could burn out a gpu if the settings aren't right.



But Baker lab does not give us the super highend stuff. We get lower level to mid level stuff.
They have a neural network for deep learning and all that fancy stuff to work on the seriously crazy stuff.

FAH barely uses my 1050 and 1070. But I have yet to find a good science GPU project.
I might just have to reattach to primegrid or something to get things going.


As for this work around, I have a week off in two weeks or so and I can take the time to read through this more deeply and step by step. Right now is browse and comment and move on.
ID: 102994 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1866
Credit: 8,186,159
RAC: 7,029
Message 102995 - Posted: 20 Oct 2021, 20:32:53 UTC - in response to Message 102988.  

Not if a task tasks 3 or 4 days to run it though, yes most of us run tasks non stop but our gpu's do get a slight break between tasks even if we are running multiple tasks at once, but running the same task nonstop for 3 or 4 days could burn out a gpu if the settings aren't right.


A 2080 used in the paper produces approximately 9000 gflops in single precision (i think they use the single) and crunch a wu in 90 minutes.
A 1060 (entry level gpu of 2016) produces approximately 4000 gflps in SP, so a wu takes 200 minutes to finish.
I think it's acceptable. 3/4 days will take an old integrated Intel gpu....

But mine are only speculation, cause we don't know nothing about this app and the possibility to run this in Boinc.
ID: 102995 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
mikey
Avatar

Send message
Joined: 5 Jan 06
Posts: 1894
Credit: 8,767,498
RAC: 6,467
Message 102996 - Posted: 21 Oct 2021, 11:43:55 UTC - in response to Message 102994.  


FAH barely uses my 1050 and 1070. But I have yet to find a good science GPU project.
I might just have to reattach to primegrid or something to get things going.


Try Einstein they have both "Gravitational Wave search O3 All-Sky" and "Gamma-ray pulsar binary search #1 (GPU)" tasks right now
with the 2nd one giving 3 times the credit of the 1st one.
ID: 102996 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Greg_BE
Avatar

Send message
Joined: 30 May 06
Posts: 5664
Credit: 5,711,666
RAC: 1,996
Message 103000 - Posted: 21 Oct 2021, 19:18:15 UTC - in response to Message 102996.  


FAH barely uses my 1050 and 1070. But I have yet to find a good science GPU project.
I might just have to reattach to primegrid or something to get things going.


Try Einstein they have both "Gravitational Wave search O3 All-Sky" and "Gamma-ray pulsar binary search #1 (GPU)" tasks right now
with the 2nd one giving 3 times the credit of the 1st one.


I see what the problem is...a certain power supply software that I was looking at for temp monitoring and some other things is way off on its values. I had a look at MSI afterburner and its showing 98% usage which makes sense because my temps are high enough and I can hear my fans blowing.
ID: 103000 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
mikey
Avatar

Send message
Joined: 5 Jan 06
Posts: 1894
Credit: 8,767,498
RAC: 6,467
Message 103004 - Posted: 22 Oct 2021, 11:14:55 UTC - in response to Message 103000.  


FAH barely uses my 1050 and 1070. But I have yet to find a good science GPU project.
I might just have to reattach to primegrid or something to get things going.


Try Einstein they have both "Gravitational Wave search O3 All-Sky" and "Gamma-ray pulsar binary search #1 (GPU)" tasks right now
with the 2nd one giving 3 times the credit of the 1st one.


I see what the problem is...a certain power supply software that I was looking at for temp monitoring and some other things is way off on its values. I had a look at MSI afterburner and its showing 98% usage which makes sense because my temps are high enough and I can hear my fans blowing.


Yup that does make sense
ID: 103004 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote

Message boards : Number crunching : New Docker image with Virtualbox and Nvidia GPU passthrough support



©2024 University of Washington
https://www.bakerlab.org