Raspberry Pi Cluster

Message boards : Number crunching : Raspberry Pi Cluster

To post messages, you must log in.

AuthorMessage
Profile Carbon-14

Send message
Joined: 24 Feb 18
Posts: 2
Credit: 7,113,337
RAC: 805
Message 101746 - Posted: 6 May 2021, 3:07:32 UTC

I think Rosetta is now issuing work units that will run on Raspberry Pi's (correct? or at least some of them). I've been contributing for a while with Mac and AMD Ryzen systems. I am toying with setting up an RPi cluster of RPi-4B's (4 Gb). Yeah, hardware bought. Set-up pains with Ubuntu, etc. Maybe ready to actually start running.

I found a post or web page about using Docker to do this. Any comments or suggestions about how to get a cluster up and running in a secure fashion would be appreciated. My idea is to see how many RPi's it takes to make one AMD Ryzen 9 3990X and also to compare them on a work/kW-h basis. My desktop tears through work, but uses about 370 W of power... enough to heat up the office and cause complaints. Can a Pi cluster do close to the amount of work but use less power?
ID: 101746 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Grant (SSSF)

Send message
Joined: 28 Mar 20
Posts: 1351
Credit: 13,624,788
RAC: 0
Message 101747 - Posted: 6 May 2021, 6:10:39 UTC - in response to Message 101746.  

Can a Pi cluster do close to the amount of work but use less power?
Short answer, no.

And while there have been many posts over the years asking about making a Cluster of Pis, i've never heard back from anyone that has actually managed to do it. You need the Clustering software to provide a system that can have the BOINC Manager installed on it, that uses all the connected Pi's CPUs and RAM and provides them to the BOINC Manager as a single CPU with lots of cores/threads.
Most people just go for lots & lots of Pis with BOINC installed on them (often using a script to help set things up), and use a manger such as BOINC Tasks to manage them all. If you've got the time and inclination, all well and good. I personally prefer just to have 1 very capable system (such as your Ryzen) and be done with it. Less grief in the long run.
Grant
Darwin NT
ID: 101747 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
mikey
Avatar

Send message
Joined: 5 Jan 06
Posts: 1885
Credit: 6,073,711
RAC: 2,805
Message 101750 - Posted: 6 May 2021, 12:33:58 UTC - in response to Message 101746.  

I think Rosetta is now issuing work units that will run on Raspberry Pi's (correct? or at least some of them). I've been contributing for a while with Mac and AMD Ryzen systems. I am toying with setting up an RPi cluster of RPi-4B's (4 Gb). Yeah, hardware bought. Set-up pains with Ubuntu, etc. Maybe ready to actually start running.

I found a post or web page about using Docker to do this. Any comments or suggestions about how to get a cluster up and running in a secure fashion would be appreciated. My idea is to see how many RPi's it takes to make one AMD Ryzen 9 3990X and also to compare them on a work/kW-h basis. My desktop tears through work, but uses about 370 W of power... enough to heat up the office and cause complaints. Can a Pi cluster do close to the amount of work but use less power?


I did some Rosetta units on my Pi 4 8gb and they worked great but I only let it run 3 at a time, it may have been able to run 4 at a time but no more than that as the Pi 4 is only a quad core machine, add in the steps to add the 32bit stuff and getting the name of the Pi so Rosetta will use it and it worked great. As for replacing your Ryzen not a chance, my suggestion is just to throttle the Ryzen during the day when people are complaining and then crank it up when you leave for the day and then throttle it again the next day.

The User PorkyPies has a cluster of Pi's running and the link to it is here: http://marksrpicluster.blogspot.com/.
Here is a link to his computers here at Rosetta: https://boinc.bakerlab.org/rosetta/hosts_user.php?userid=2134166

The info on his website helped me alot getting mine up and running as did setting up ZRam in some of the memory.
ID: 101750 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sid Celery

Send message
Joined: 11 Feb 08
Posts: 1851
Credit: 34,015,547
RAC: 3,893
Message 101755 - Posted: 7 May 2021, 1:01:50 UTC - in response to Message 101746.  

I think Rosetta is now issuing work units that will run on Raspberry Pi's (correct? or at least some of them).

I've received a task that calls for only 3338Mb (jgSP*) so hopefully those will be suitable for 4Gb PIs - 8Gb ones ought already to be managing to work on tasks, from what I understand
ID: 101755 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
mikey
Avatar

Send message
Joined: 5 Jan 06
Posts: 1885
Credit: 6,073,711
RAC: 2,805
Message 101759 - Posted: 7 May 2021, 12:18:08 UTC - in response to Message 101755.  
Last modified: 7 May 2021, 12:18:30 UTC

I think Rosetta is now issuing work units that will run on Raspberry Pi's (correct? or at least some of them).


I've received a task that calls for only 3338Mb (jgSP*) so hopefully those will be suitable for 4Gb PIs - 8Gb ones ought already to be managing to work on tasks, from what I understand


Yes I have run Rosetta tasks on my 8gb Pi 4, I ran 3 at a time with no problem.
ID: 101759 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
PorkyPies

Send message
Joined: 6 Apr 20
Posts: 45
Credit: 1,650,779
RAC: 0
Message 101776 - Posted: 9 May 2021, 7:27:42 UTC - in response to Message 101746.  
Last modified: 9 May 2021, 7:50:25 UTC

I think Rosetta is now issuing work units that will run on Raspberry Pi's (correct? or at least some of them). I've been contributing for a while with Mac and AMD Ryzen systems. I am toying with setting up an RPi cluster of RPi-4B's (4 Gb). Yeah, hardware bought. Set-up pains with Ubuntu, etc. Maybe ready to actually start running.

I found a post or web page about using Docker to do this. Any comments or suggestions about how to get a cluster up and running in a secure fashion would be appreciated. My idea is to see how many RPi's it takes to make one AMD Ryzen 9 3990X and also to compare them on a work/kW-h basis. My desktop tears through work, but uses about 370 W of power... enough to heat up the office and cause complaints. Can a Pi cluster do close to the amount of work but use less power?

Thanks for the kind words Mikey

I'm using the Raspberry Pi OS 64 bit version. No need to make work for yourself using Ubuntu and Docker. Given its a BOINC project there is no need to run clustering software, just run BOINC on each node and use BOINCtasks to monitor them. You could use Ansible for maintaining them although I don't.

If you want it to be secure I'd suggest using a firewall to close the ports on the Pi. You can allow outgoing connections but only need 2 ports to be open for incoming. You limit them to your BOINCtasks host and the host that will ssh into them for maintenance. In my case its the same machine. I also use a proxy server which is great for times when you have to repeat file transfers (eg OS updates), it doesn't help with downloading work units as each one is unique.

My farm is currently 4 x Pi3 which only run Einstein, 8 x Pi4 8GB which run both Einstein and Rosetta in a 50/50 split and a few support nodes.

Ryzen 3990X is a Threadripper with 128 threads. ARM cores aren't as fast as the Ryzen cores even if they were clocked at the same speed. The Pi4 model B is clocked at 1.5GHz and Ryzen at 2.9GHz (base), so you would need at least 64 Pi4's to get anywhere near it. There is a lot more effort looking after 64 Pi's compared to one Threadripper. As for wattage 64 Pi4's would use around 128 watts in theory so your power bill and heat would be reduced.
MarksRpiCluster
ID: 101776 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
mikey
Avatar

Send message
Joined: 5 Jan 06
Posts: 1885
Credit: 6,073,711
RAC: 2,805
Message 101777 - Posted: 9 May 2021, 11:25:28 UTC - in response to Message 101776.  

Porky Pies wrote:
Thanks for the kind words Mikey


You are very welcome and it is VERY well deserved!!!
ID: 101777 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Carbon-14

Send message
Joined: 24 Feb 18
Posts: 2
Credit: 7,113,337
RAC: 805
Message 101830 - Posted: 16 May 2021, 20:36:03 UTC - in response to Message 101746.  

OK. Thanks for the comments. I eventually got the Pi cluster up and running. Being a new user of Linux and Docker, I'm not really sure what to check for. But so far so good. Of note, I did see an error occurring something like "requires X memory but only Y available." I seemed to have solved that by allocating a 4 GB swap file to augment the 4 GB RAM of each Raspberry Pi.

The point about just running 6 independent Pis as opposed to 6 Pis in a swarm is well-taken and I don't know the answer to that.

FYI: I don't have a Threadripper. That was a typo. It's an AMD Ryzen 9 3900X not 3990X. Wish I had the Threadripper though.

Anyway, more experimentation to come. The RPi POE hat fans are really loud. So next modification is to get rid of those.

Any comments welcomed.
ID: 101830 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
mikey
Avatar

Send message
Joined: 5 Jan 06
Posts: 1885
Credit: 6,073,711
RAC: 2,805
Message 101832 - Posted: 16 May 2021, 23:54:07 UTC - in response to Message 101830.  

OK. Thanks for the comments. I eventually got the Pi cluster up and running. Being a new user of Linux and Docker, I'm not really sure what to check for. But so far so good. Of note, I did see an error occurring something like "requires X memory but only Y available." I seemed to have solved that by allocating a 4 GB swap file to augment the 4 GB RAM of each Raspberry Pi.

The point about just running 6 independent Pis as opposed to 6 Pis in a swarm is well-taken and I don't know the answer to that.

FYI: I don't have a Threadripper. That was a typo. It's an AMD Ryzen 9 3900X not 3990X. Wish I had the Threadripper though.

Anyway, more experimentation to come. The RPi POE hat fans are really loud. So next modification is to get rid of those.

Any comments welcomed.


That's very good!! There is a way to slow down the fan if you have it running on the high speed pins but I don't have the pin layout in front of me right now.
ID: 101832 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
PorkyPies

Send message
Joined: 6 Apr 20
Posts: 45
Credit: 1,650,779
RAC: 0
Message 101867 - Posted: 21 May 2021, 7:38:30 UTC - in response to Message 101830.  
Last modified: 21 May 2021, 7:56:12 UTC

The RPi POE hat fans are really loud. So next modification is to get rid of those.

Any comments welcomed.

Alternatives are:
- Heat sink Pi cases. They can get toasty, but generally do the job with no noise.
- A case that holds multiple Pi's (and has reasonable sized fans) eg the Cloudlet Cluster Case from C4labs.
- Use decent quality fans. The larger the fan the more air it will move and it can run at a lower speed which reduces the noise

I tried the Pimoroni fan shim. The fans seized up on all of them after a couple of months. Don't bother with them.

These days I have three support nodes in the Flirc Pi case (its a heatsink case). I have a couple of the Heatsink (black aluminium) cases. However most of my number crunchers have Noctua 40mm fans which are quiet and last. I have some that have been running 24/7 for years. You just need to clean the dust off regularly.

I designed and printed off my own case to hold 4 Pi side by side and it has the Noctua fans on top for cooling. i refer to it at the Pi^4 case (Pi to the power of four). See this blog post for a recent photo
MarksRpiCluster
ID: 101867 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
PorkyPies

Send message
Joined: 6 Apr 20
Posts: 45
Credit: 1,650,779
RAC: 0
Message 101868 - Posted: 21 May 2021, 7:55:35 UTC - in response to Message 101830.  

FYI: I don't have a Threadripper. That was a typo. It's an AMD Ryzen 9 3900X not 3990X. Wish I had the Threadripper though.

Well 24 threads of x86_64 architecture versus the Pi's. I would say you need about 12 x Pi4 to be able to get close to it.
MarksRpiCluster
ID: 101868 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Endgame124

Send message
Joined: 19 Mar 20
Posts: 63
Credit: 14,855,841
RAC: 12
Message 101949 - Posted: 28 May 2021, 1:05:16 UTC - in response to Message 101868.  

I’ve been using 15 pi’s of various sizes for over a year now. Zram and a decent case is the key to running 4 Rosetta processes on a 4GB pi 4.

I’ve also trimmed back the OS and shut off as much of the unnecessary hardware as possible to aide energy savings.

You can read about some of my adventures here, including some instructions for cutting back your Linux install and shutting off things like hdmi here:

https://hardforum.com/threads/distributed-computing-on-raspberry-pi.1997998/
ID: 101949 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote

Message boards : Number crunching : Raspberry Pi Cluster



©2022 University of Washington
https://www.bakerlab.org