Posts by Endgame124

1) Message boards : Number crunching : Raspberry Pi Cluster (Message 101949)
Posted 28 May 2021 by Endgame124
Post:
I’ve been using 15 pi’s of various sizes for over a year now. Zram and a decent case is the key to running 4 Rosetta processes on a 4GB pi 4.

I’ve also trimmed back the OS and shut off as much of the unnecessary hardware as possible to aide energy savings.

You can read about some of my adventures here, including some instructions for cutting back your Linux install and shutting off things like hdmi here:

https://hardforum.com/threads/distributed-computing-on-raspberry-pi.1997998/
2) Message boards : Number crunching : Running multiple covid projects on multiple machines, how do you personally go about it? (Message 98484)
Posted 11 Aug 2020 by Endgame124
Post:
I’m running 12 raspberry pis and a couple of desktops. One thing I can tell you that doesn’t seem to be mentioned here is running multiple projects on the same host reduces performance on both projects, at least on raspberry pi.

Ex: splitting a pi to run 2 threads of open pandemics and 2 threads of of Rosetta will yield about 450 average credit on Rosetta. Putting all 4 threads on Rosetta will yield around 1100 average credit. The same thing happens on open pandemics.

Given this, I’ve now switched to run only one project per pi.
3) Message boards : Number crunching : ARM Compute Raspberry Pi4 with Ubuntu Mate 20.04 beta1 (Message 98430)
Posted 7 Aug 2020 by Endgame124
Post:
You can easily run 4 threads on a pi 4 4GB. Setup ZRam as covered in the other pi 4 thread on this forum.

I currently have 5 4gb pi 4s running 4 threads without issue running a 150% zram configuration.

Lots of info in this thread for the pi 4:
https://boinc.bakerlab.org/rosetta/forum_thread.php?id=13732&sort_style=8&start=75
4) Message boards : Number crunching : Neocortix supports Rosetta@Home (Message 98387)
Posted 30 Jul 2020 by Endgame124
Post:
Arm support has been with Rosetta since April, but interesting to see that it will soon be available for folding at home. I’m surprised that a pi 4 could hit the return deadline at F@H
5) Message boards : Number crunching : no new tasks? (Message 97563)
Posted 24 Jun 2020 by Endgame124
Post:
As of this morning (US Central time), looks like the well has run dry again.

I'd like to figure out a work share setup that would allow WCG tasks to run to completion when the Rosetta tasks come back. I've never liked the "suspend one to do another" approach - I'd rather have what's in the queue finish up regardless of which project it belongs to.


That probably won't be easy with the massively different task deadlines, one way to try is to change the resource share settings, ie set WCG at 70 and Rosetta at 7 so WCG has a 10 times larger daily credit advantage. You will have to play with the numbers to see if you can get it to work for you though.

I’ve been using an app_config.xml to set 2 processes for Rosetta and no limit on WCG processes. It may not work long term, but on the short term I’m getting 2 Rosetta processes and 2 WCG processes on my raspberry pis when Rosetta has work, and 4 WCG processes when Rosetta has no work
6) Message boards : Number crunching : no new tasks? (Message 97551)
Posted 24 Jun 2020 by Endgame124
Post:
Looks like about 30000 tasks ready to download. Some of my hosts that were out of work have started to pick some up, and I’m guessing I’ll have work across the board by morning.
7) Message boards : Cafe Rosetta : something broke (Message 97513)
Posted 23 Jun 2020 by Endgame124
Post:
I had this problem on one of my raspberry pis. Turns out I forgot to add the extra lines to cc_config.xml to enable arm support.

Maybe this helps?
8) Message boards : Number crunching : Running on a 4GB Raspberry Pi 4 - How to? (Message 97437)
Posted 17 Jun 2020 by Endgame124
Post:
I picked up a pi 4 2GB, and have been running some tests with it with just a small heatsink and a low airflow usb fan blowing on it (no case).

The heatsink I used is an old Zalman ZM-RHS1 - its a super small VGA heatsink from like 10+ years ago. I have probably 30 of them on hand, as I bought them on clearance at something like $1 per box of 8 and only used a couple over the years.

Observations:

1) While in theory the 2GB model should use less power than the 4 GB model, my UPS shows the same 3-4W usage at 100% load. When set to over_voltage=-2, and with USB, HDMI, wifi, etc turned off, it reads a consistant 3W power usage, just like a 4GB model.

2) when turned on its edge (GPIO connector down, USB-C connector up), with the small heatsink and a fan blowing on it, it runs fine even over clocked at 2000 mhz and over volt 4 (max temp seen so far is 69C, and no throttling). I imagine that a more reasonably large heatsink designed for the pi4 would drop the temp in this scenario multiple degrees.

3) with a zram configuration for 100% (effective 4GB of ram), you can run 4 Rosetta processes most of the time. I suspect you could stretch it to 150% (effective 5GB ram) and you would almost always be running 4 Rosetta processes. That said, I think the ideal setup is to split the 2GB model between 2 Rosetta processes and 2 Open Pandemic processes.
9) Message boards : Number crunching : Running on a 4GB Raspberry Pi 4 - How to? (Message 97308)
Posted 9 Jun 2020 by Endgame124
Post:
update on the under volting. It seems to be that there is a minimum voltage set at -2 (going lower has no effect). That minimum is 0.8350V.

I've had success with over_voltage -2 AND arm_freq 1565 - that is about a 4% clockspeed increase while also being under volted. It may actually be very hard to beat this combination of speed and power.

Anyone else able to try to see if they can hit 1565mhz on over_volt=-2?
10) Message boards : Number crunching : Running on a 4GB Raspberry Pi 4 - How to? (Message 97251)
Posted 5 Jun 2020 by Endgame124
Post:
Just playing around a little bit with undervolting all 4 of my pi 4s run at stock speed 1500mhz at -1 voltage. 3 of them seem to run fine at Stock speed at -2 voltage, and one seems to work at -3. At the moment I don’t know if it will be long term stable, or if there was a measurable drop in watts, but it seems to have knocked 4-5 c off the temp.
11) Message boards : Number crunching : Running on a 4GB Raspberry Pi 4 - How to? (Message 97160)
Posted 2 Jun 2020 by Endgame124
Post:
thanks i think i'd get a 8gb one when it becomes available at my local retailer. 1.5GB tends to be seen for the 'larger' wu, but i've at other times seen about 30% of memory used on a 4GB Pi4 with all 3 threads running. Perhaps it is because the wu just started running and like you've mentioned, they tend to increase in size over time. 8 GB would also means one can do away with all that zram stuff, there really isn't a point 'wasting' cpu cycles compressing and uncompressing memory. takes more power returns less points. swap and zram only make sense if it is used to keep 'unused' stuff.

Agreed - With the 8gb pi there is no reason to have zram. However, the way zram works there is minimal impact to have it enabled and unused - a unused zram partition uses like 32kb of ram. If you don’t need to swap it stays at that low level and doesn’t need to compress anything.
12) Message boards : Number crunching : Running on a 4GB Raspberry Pi 4 - How to? (Message 97157)
Posted 2 Jun 2020 by Endgame124
Post:
do share the peak memory consumption that you observe, you can try running rpi-monitor which would capture that.
i'm thinking of getting a piece or two but my local distributor don't have stock yet
it seemed with 4GB and running 3 threads, i don't seem to see 'waiting for memory' situations frequently any more, but of course with 8GB it is more comfortable running all 4 threads
i noted credits earned seem lower with the more recent wu, various from robetta as they are prefixed with rb i'm not sure why

The initial set of jobs this pi started included a pair that used 1.5gb each, and 2 at 480mb each. Just under 4gb total would work on a 4gb pi 4, but these jobs tend to increase in size over time, so I may have already had a job set that could use the extra memory.
13) Message boards : Number crunching : Running on a 4GB Raspberry Pi 4 - How to? (Message 97145)
Posted 2 Jun 2020 by Endgame124
Post:
I ended up going with the standard 32bit Raspbian light for the install on my first 8GB pi 4. I figured that would give the best possible comparison to the 4GB pi 4. Oddly enough it benchmarked slightly slower than a 4GB pi4 at the same clockspeed, but I don't know what the margin of error is on the boinc benchmark test so it may not matter.

The new host is here: https://boinc.bakerlab.org/rosetta/show_host_detail.php?hostid=4525006
14) Message boards : Number crunching : Running on a 4GB Raspberry Pi 4 - How to? (Message 96948)
Posted 31 May 2020 by Endgame124
Post:
Has anyone tried the new Raspberry Pi OS (64 bit) yet?

I upgraded my Pi's a couple of days ago and they got about 30 updates and the kernel version went up. They appear to be running the same as before. I haven't specifically tried doing a "full-upgrade" as they suggest or a clean install of the 64 bit version, which they are saying is a beta version.

I haven’t tried it yet - I’ll be building a clean 64bit install when my first 8GB pi arrives on Wednesday. I’ve heard the Linux kernel 5 has specific improvements for the new Broadcom chip in the pi 4 , but I’m not sure if it will have an impact on Rosetta.

I’m looking at building a kubernetes cluster out of my Pis just to work out the process, but once I do that my results will no longer be comparable.
15) Message boards : Number crunching : Running on a 4GB Raspberry Pi 4 - How to? (Message 96881)
Posted 30 May 2020 by Endgame124
Post:
I'm currently doing the following to save power on my pi:
Disable HDMI:
sudo tvservice --off
Disable Wifi
echo "dtoverlay=disable-wifi" | sudo tee -a /boot/config.txt
Disable Bluetooth and bluetooth services
sudo systemctl disable hciuart
echo "dtoverlay=disable-bt" | sudo tee -a /boot/config.txt
Turn off Power LED:
echo 0 | sudo tee /sys/class/leds/led1/brightness
Have you done any power testing with Ethernet vs Wifi?

I usually blacklist the WiFi and Bluetooth. Not to save power. I have 20 Pi's and I would rather they not use WiFi at all, there wouldn't be much bandwidth left for the other devices. Most of them are Pi3's so are running Einstein work.

cd /etc/modprobe.d
sudo nano raspi-blacklist.conf and add the following
# wifi
blacklist brcmfmac
blacklist brcmutil
# bluetooth
blacklist btbcm

Not sure if they are still correct, I got this from forum posts on raspberrypi.org some years ago. I also remove the pi-bluetooth package.

What is the difference between blacklisting and disabling like in the quoted text?
16) Message boards : Number crunching : Running on a 4GB Raspberry Pi 4 - How to? (Message 96829)
Posted 28 May 2020 by Endgame124
Post:
hi, Pi4 has got a new 8GB version, and more importantly 64 bit OS ;)
https://www.raspberrypi.org/blog/8gb-raspberry-pi-4-on-sale-now-at-75/
https://www.raspberrypi.org/forums/viewtopic.php?f=117&t=275370
https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=275372

it seem to be something to try out, it is Beta though,
this is really on the bleeding edge lol

I'll be ordering one today! I don't expect dramatically different performance from the 4GB model, but every once in a while I need more than 4GB of ram for Rosetta.

Does anyone know if there is planned long term support for ARM in Rosetta? I would hate to buy 3 8GB Pis and have support dropped in a few months.
17) Message boards : Number crunching : Raspberry Pi4 (Message 96825)
Posted 28 May 2020 by Endgame124
Post:
Well, an 8gb pi would sure solve all the problems with running out of memory when using Rosetta. At $75 it’s starting to get a bit expensive though given its still an arm processor.
18) Message boards : Number crunching : Raspberry Pi4 (Message 96780)
Posted 25 May 2020 by Endgame124
Post:
If you have rosetta@home installed on your Raspberry Pi4 (running Raspbian 'buster') but you don't get any work to be processed then check here:
https://www.element14.com/community/people/gam3t3ch/blog/2020/04/09/rosettahome-on-raspbian-pi-4

Adding just a few lines to one of the configuration-files will solve these problems.

I have been running rosetta@home for a couple of weeks now and my Pi4 has never been idle... Enough work to do!

Good luck,
Adam

Check the below thread on the forums. There are detailed discussions on raspbian configuration , boinc configuration, zram, and power saving options for the pi 4:

https://boinc.bakerlab.org/rosetta/forum_thread.php?id=13732
19) Message boards : Number crunching : The most efficient cruncher rig possible (Message 96769)
Posted 24 May 2020 by Endgame124
Post:
What you have measured just highlights what I have mentioned above: you can't trick physics. If you are bumping up the the voltage and disabling a core, you are losing efficiency.

Comparing top scores doesn't make sense.

I've recommended to compare a RAC over many weeks (>2), preferably from the same time interval. ~1000 RAC does sound realistic for a Raspberry Pi 4. dcdc's system doesn't seem to be crunching since long and it's not in continuous operation:

https://www.boincstats.com/stats/-1/host/detail/346676938/lastDays

If it is not cooled properly, it will throttle itself even above 60C, but otherwise the values should match closely between well-cooled devices.

You also didn't seem to have picked up my hint that if you use a titanium ATX PSU to power multiple Pi's then it will vastly increase the performance/watt ratio (compared to ~60-70% wall warts) and that if you purchase 100 Pi devices they will give you a huge discount - I've tried this myself.


Stock pi 4 stabilizes at roughly 1,200 average credit. I think running at 2gh and a slight over volt is the sweet spot, but I need my over clocked pi’s credit to stabilize for another week to verify.

As for running 100 pis off of a single titanium psu, how are you wiring it to create 100 usb c cables? That is a pretty important bit :)
20) Message boards : Number crunching : Running on a 4GB Raspberry Pi 4 - How to? (Message 96724)
Posted 22 May 2020 by Endgame124
Post:
I now have my newest Pi 4 4GB online. This one seems to be stable at the maximum overclock with these settings:

arm_frequency=2147
over_volt=6

With USB, HDMI, Bluetooth, Wifi, and Power LED turned off, the system is consuming 6W as read from my UPS.

This Pi 4 is in a Flirc case, and is the top of a stack of 3 Pi 4s. A single fan is blowing on the stack. Maximum temp seen so far is 54 C

Host details:
https://boinc.bakerlab.org/rosetta/show_host_detail.php?hostid=4453627


Next 20



©2022 University of Washington
https://www.bakerlab.org