Hi all. I was considering undergoing a goal to build an array of computers that use little power, crunch lots of data, and take up little space.

I have not seen any photos on this forum regarding such a task, but my general goal (and although I am a computer expert, not an engineer) is to:

1)Mount 50-100 "computers" in storage racks
2)Each computer would probably be simply a motherboard, 2gig of RAM, and a Quad core chip
3)Each computer would possibly run on a flash drive so as not to waste electricity on a hard drive spinning 24x7 and to also save money (not having to buy a hard drive).
4)I am not sure what/how each machine would run on a Flash drive but I believe numerous Linux systems can. right?
5)I would assume the power supply for each motherboard could be low since there is no cd drive, no hard drive, etc...just a motherboard and a flash drive hanging out of it


Has anyone done this type of project? Any photos? how much electricity is it sucking? etc.

I'm extremely interested in doing this.

-Eric

---

I would search either the Folding@Home forums or SETI forums for "headless" or "diskless". F@H forums may have been wiped since the big post about this, but if you inquired there, I'm sure people could point you in the right direction.

You could also search for diskless on google. This yielded this site (which was last updated in 2002, hehe, but is for diskless Linux):
http://www.schnozzle.org/~coldwell/diskless/

F@H also has monitoring software so you could know if a node goes down (if crunching for F@H). It was called FAHMON.

Anyhow, keep us posted on what you do and how it goes!

---

Sounds like a great project! Do keep us posted.

Depending on what you want to crunch, though, it might be more cost effective to just buy a lot of PS3s and crunch for Folding@home.

But if you want to do Rosetta@home (which I prefer to Folding because of the science but also because it's smaller, so the marginal utility of a new CPU is higher), the Quad-core route is probably the best one. Depending on your time frame, you might want to wait for Intel's six-core chips that are coming in the second half of 2008.

Good luck!

---

i run my WinXP MCE2005 computer on compactflash, and have had a few others that have run XP on CF - you can get XP down to ~350MB quite easily. The problem with CF or netbooting is the lack of a swap file so you need plenty of RAM, otherwise you'll run out of memory. My MCE machine has 1.2GB RAM and never runs out of memory with a single instance of Rosetta running.

I had Damn Small Linux running rosetta too- it needed around 50MB for the installation, plus space for Rosetta (although you can just run it from a network drive).

Bitspit on my team has a netbooting linux cluster which is a bit cheaper to setup, but I don't think it's quite as straightforward to setup or maintain.

If you want any more info then let me know.

---

Sounds like a great project! Do keep us posted.

Depending on what you want to crunch, though, it might be more cost effective to just buy a lot of PS3s and crunch for Folding@home.

But if you want to do Rosetta@home (which I prefer to Folding because of the science but also because it's smaller, so the marginal utility of a new CPU is higher), the Quad-core route is probably the best one. Depending on your time frame, you might want to wait for Intel's six-core chips that are coming in the second half of 2008.

Good luck!

Now, I don't know the main 1-2 differences between RAH and FAH (can anyone list here rather than point me to a webpage with 11 pages of information?), but I hear that the PS3 is essentially equal to about 16 home computers...that's kinda vague...16 crusty Celeron pcs or 16 Core Duo machines?

Since the PS3 is $399 list price, that's obviously cheaper than most home computers that would be somewhat fast. And, the PS3 is easy to administer since it's a relatively small box...and let's not forget that Sony is right behind FAH 100%.

Has anyone ever contacted Sony to see if they would give a discount (say a measely $100) for people like us who would buy them in bulk? I can see that if we just bought 1 or 2, Sony might think we are lying and handing them out to the kiddies. But what if I wanted to buy 50? That would be about $15,000 worth of hardware (at $300 a pop) that I would dedicate to science. Sony can snip the HDMI cable or whatever they want to do if they don't trust me.

I also believe that Sony will release a PS4 (for lack of better word) sometime within the next 3 years since PS3 has been out since Nov 2006. And, I would bet money that the PS4 would support FAH right out of the gate. And if Sony packs in another quantum leap in processing technology, that will be even better.

Any thoughts? Has anyone contacted Sony with such a plan? They've sold 10.5 million PS3s and 1 million of them are on FAH which is amazing.

-Eric

---

Sounds like a great project! Do keep us posted.

Depending on what you want to crunch, though, it might be more cost effective to just buy a lot of PS3s and crunch for Folding@home.

But if you want to do Rosetta@home (which I prefer to Folding because of the science but also because it's smaller, so the marginal utility of a new CPU is higher), the Quad-core route is probably the best one. Depending on your time frame, you might want to wait for Intel's six-core chips that are coming in the second half of 2008.

Good luck!

Now, I don't know the main 1-2 differences between RAH and FAH (can anyone list here rather than point me to a webpage with 11 pages of information?), but I hear that the PS3 is essentially equal to about 16 home computers...that's kinda vague...16 crusty Celeron pcs or 16 Core Duo machines?

Since the PS3 is $399 list price, that's obviously cheaper than most home computers that would be somewhat fast. And, the PS3 is easy to administer since it's a relatively small box...and let's not forget that Sony is right behind FAH 100%.

Has anyone ever contacted Sony to see if they would give a discount (say a measely $100) for people like us who would buy them in bulk? I can see that if we just bought 1 or 2, Sony might think we are lying and handing them out to the kiddies. But what if I wanted to buy 50? That would be about $15,000 worth of hardware (at $300 a pop) that I would dedicate to science. Sony can snip the HDMI cable or whatever they want to do if they don't trust me.

I also believe that Sony will release a PS4 (for lack of better word) sometime within the next 3 years since PS3 has been out since Nov 2006. And, I would bet money that the PS4 would support FAH right out of the gate. And if Sony packs in another quantum leap in processing technology, that will be even better.

Any thoughts? Has anyone contacted Sony with such a plan? They've sold 10.5 million PS3s and 1 million of them are on FAH which is amazing.

-Eric

The PS3 (as with all consoles) is sold at a loss and the money is made on games, so they'd probably charge a premium for crunchers rather than giving a discount! They've also sold their cell manufacturing facilities to Toshiba I believe (although it might be a share scheme), so I'm not sure there will be a PS4...

---

The PS3 (as with all consoles) is sold at a loss and the money is made on games, so they'd probably charge a premium for crunchers rather than giving a discount! They've also sold their cell manufacturing facilities to Toshiba I believe (although it might be a share scheme), so I'm not sure there will be a PS4...

Well, I don't think Sony is getting out of the gaming business if that's what you're trying to say. Sony is an entertainment company and has done very well with their gaming systems over the past 10+ years.

But anyway....

So wouldn't it be cool if the # of PS3s doing FAH got up to 3 or 4 million rather than the current 1? Or even 6 million ultimately by end of 2009 as Sony keeps on selling PS3s (they are still proudly selling the PS2 and I had one of those back in 1998).

---

PS3 vs GPU

PS3s would be an easy way to go, though I am unsure how they respond (if anything changes) when no display is attached. 0% of getting a discount.

GPU folding is much more complex and is twice as effective as PS3 when you count both equipment and power costs. The 2x might become 8x whenever the FAH GPU client for the ATI 3870 (X2) comes out. This is because the 3870 may only eat up 100W while folding (saving $100/year in electric bills), whereas the 1950XT eats up 200W. GPU folding is much more demanding on the user to maintain, at present. I do not recommend dual-slot GPU folding, because the systems are much more prone to fail as far as scientific folding is concerned.


R@H vs F@H

R@H is more concerned about predicting the final structure and interactions between proteins (or proteins and drug candidates). I respect the methodology of R@H more. F@H is more concerned with the process of protein folding and mis-folding. It is believed that Alzheimer's is caused by mis-folding proteins.

(feel free to correct my synopsi)

---

You can google BOINCpe, which can be run on a flash drive.
I have several machines running it, and not a problem at all.
I can also provide you with motherboards. I sell them and know where to get it cheap.

Hi all. I was considering undergoing a goal to build an array of computers that use little power, crunch lots of data, and take up little space.

Great idea - if you get it up and running, keep us all posted - there'd be a lot of interest I think. For example, I too have the space to store such an array, and could afford to gradually add to one, but the electricity bill is becoming the limiting factor for me.

---

Alver Valley Software Ltd - Contributing ALL our spare computing power to BOINC, 24x365.

I'd like to kick start this topic again and keep off the PS3 and GPU topics.

I haven't had any time to really research the best/cost effective way to build such an array...and am looking for your advice.

To me, the EASIEST and FASTEST way to build such an array would be to buy pre-built computers like 20 Dell Inspiron 530 or 20 Mac Minis...the machines would all be ready to go...just plug them in...sure, they would come with some extras that I have no need for such as humumgous hard drives, cd/dvd drives, killer video cards, but, on the other hand I could always try to sell them a year or 2 later and use the money for faster machines.

A more complex solution would be to have a giant case that simply holds tons of motherboards...who knows where the power cases would be...and maybe the OS and software would run on a flash drive.

So...kick starting this topic again, any ideas? I'd like to find "machines" that are under $400 that come with a quad-core chip and 3-4gig of RAM. For example, I recently bought a Dell Insprion 530 with 3gig ram, quad-core chip, 500gig hard drive, dvd/cd drive for $499 to my door. Surely, there must be some mom & pop bare bones systems that I can buy for $100 less with the same RAM and CPU specs. Maybe even $300 if I buy a quantity of 20 or something.


-Eric

---

I'm running what could be called a 10 computer array. I'm using LTSP 4.2 which is a netbooting Linux package. Each system loads off a central server. The server is the only computer that has a hard drive (well except for my laptop but I netboot when it's docked). It does take some work to get things set up, but once the first system is setup, all the others simply get added to a configuration file. The hardest part has been compiling a kernel with SMP support so I could get dual and quad core support. The other major issue is the network card. Don't bother with the motherboard's onboard NIC unless it's Intel. All the others suck and cause problems. I personally like Broadcom's gigabit network cards because they're cheap (around $8) and they work with no effort. Now for my dedicated crunchers.

2007 Intel setup (started with this):
Intel E4400 overclocked to 3.2GHz
1GB generic 667 DDR2
Abit F-I90HD motherboard (do not recommend, I'm replacing these)
Generic 400W power supply
Broadcom gigabit NIC
$5 8MB ATI Rage Pro PCI video card

This uses 115W and gets roughly a 1,050 RAC

2008 Intel setup (switching to this):
Intel Q6600 overclocked to 3.0GHz
Asus P5K-VM motherboard
2GB Corsair XMS2-675 DDR2
Enermax Liberty 400W (80% efficiency plus no extra, unused cables)
Broadcom gigabit NIC
$5 8MB ATI Rage Pro

This uses 145W for a Q6600 B3 stepping (haven't checked G0) and gets around 2400 RAC, although this can go as high as 3,000/day if running primarily Rosetta Mini jobs (large L2 + Linux = crazy high points, granted credit up to 4.5x claimed).

I'm also not using any cases. I screw standoffs onto them and then place them onto cork-covered shelves. Never had any static issues with it and no worries about shorting.

The Q6600 setup does end up close to $450 (before I've spent another $40 on cooling) for one simple reason: I can't buy parts in bulk like Dell. Some of that cost can be lowered by using cheaper memory, a lower end motherboard, skipping the video card and using onboard. Even then, getting below $400 using quality parts will be difficult.

I really should get around to making a web page for this.
/me adds item to rarely checked to do list

if you're doing this i'd definitely go for a setup like BitSpit's rather than a load of full systems. For a given output it'll be cheaper to buy and to run, and you'll be able to overclock/undervolt, both of which make a huge difference to output per $. If netbooting Linux is beyond you, or you want to start out simple, then you can boot Linux or XP from a flash card (installing to a flash drive is the same as installing to a normal HD, although installing to USB is a bit tricker, but you only have to do it once and then copy it for the other machines). You can either:

use small CompactFlash cards or USB memory sticks (256MB is fine for Linux but a bit of a squeeze for XP, so 512MB for XP), and run BOINC from a network folder on a central server with a hard drive.

or

use 1GB flash drives and run BOINC directly from those. Then you have no issues with having a server (such as not being able to reboot it, and having BOINC die on all machines if there's an interruption to the network etc)

BS - how do you get around the problem of no swap file with your setup if all the physical memory is being used?

---

You might consider using 4GB for a quad. I've seen WUs get above 512MB of real memory in the past (not too often, though). I expect that there may be even more such WUs in the future.

I disagree with the previous poster about onboard NICs. I've used several different boards and the onboard NIC always worked fine. The only problem was when the NIC was so new that it wasn't supported by the version of Linux I was using.

For the power supply, make sure it has the "80 Plus" efficiency certification.

Consider having the nodes boot from cheap USB flash drives, and then mount a directory from a file server for running BOINC.

for such a large cluster, you will probably want to have the nodes go through a caching proxy of some sort so that all those big files will only need to be downloaded once.

BS - how do you get around the problem of no swap file with your setup if all the physical memory is being used?

Make sure there's plenty available. :) Seriously, I allocate 512MB per core and set my preferences to allow 99% memory usage. This seems to be sufficient. The Linux version doesn't seem to require as much. I've only seen one job run out. The job crashed and BOINC simply loaded up the next task.

I disagree with the previous poster about onboard NICs. I've used several different boards and the onboard NIC always worked fine. The only problem was when the NIC was so new that it wasn't supported by the version of Linux I was using.

I've noticed the onboard NICs tend to drop packets during the boot process before the kernel driver loads which is why I made my comment. I'd much rather spend a few dollars on a NIC than throttle the server.

I really should get around to making a web page for this.
/me adds item to rarely checked to do list

Bitspit, thanks for the post...a few questions and comments:

1)Yes, would love to see a webpage and maybe a how-to or where you ordered from. I'm tech savvy but I gave up the build-it-yourself model around 1996 when it just wasn't my gig anymore. :)

2)Your current system seems about $50 cheaper than a Dell box with the same quad chip, 3gig of ram (instead of your 1gig), a 500g drive (instead of your 0g drive), a case, cd/dvd drive, keyboard, and mouse and maybe a few other things...so...although Dell might be a slight monopoly, why not just buy a bunch of the same Dell machines? I would understand if it was physical size dimensions issue and/or power sucking issue...care to elaborate? Just wondering. Heck, you could sell the 500g drives for $100 a pop not to mention the dvd/cd drive for $25.

3)Maybe I will start a new thread with this, but has anyone (inside or outside RAH or BOINC in general) considered petitioning to pc manufacturers the idea of allowing RAH/BOINC registered users purchase bare bones machines for a cheap price? A)We would be ordering bare bones machines so I doubt there would be an argument that we are buying these machines for our employers in order to save money. B)Maybe put in a clause that each user needs to order more than 5 at a time. C)It's all for good use. D)It allows the manufacturers to "give back" by giving us good-hearted folks a discount to help cure diseases with our own personal cash and E)Maybe force us participants to buy only on certain days of the month when mass quantities could be ordered (for example, allowing the ordering website to be open on the 15th of the month only and at that time, if more than 500 "pcs" are ordered in total, the price is X...if more than 1000 the price goes to Y...if more than 5000 the price goes to Z...kinda like the Home Shopping Network (at least from my memories from the late 80s). I doubt a giant vendor like Dell or HP is going to get involved because we are probably too small a project and they would rather just "donate" cash or machines to a single body...but I bet mom and pop shops would love to get monthly orders of 1000 units...it would certainly promote their business as well as earn them revenue.


Anyway, overall I'd love to learn a little bit more about where you bought your equipment, how it was set up, some pics, etc. I'm technical, I just haven't played around with diskless systems, Linux operating systems, etc.

-Eric

---

2)Your current system seems about $50 cheaper than a Dell box with the same quad chip, 3gig of ram (instead of your 1gig), a 500g drive (instead of your 0g drive), a case, cd/dvd drive, keyboard, and mouse and maybe a few other things...so...although Dell might be a slight monopoly, why not just buy a bunch of the same Dell machines? I would understand if it was physical size dimensions issue and/or power sucking issue...care to elaborate? Just wondering. Heck, you could sell the 500g drives for $100 a pop not to mention the dvd/cd drive for $25.

I'm using 2GB on my quads, not 1. I'm more concerned about the quality and capability of parts rather than quantity. A stock Q6600 gets around 1650 RAC (at least for me). Pushing it to 3GHz gets around 2400. That's not something as easily done with a cheap Dell. The Dell also wouldn't be designed for 24/7 overclocked use. Building it myself lets me design it how I want.

Anyway, overall I'd love to learn a little bit more about where you bought your equipment, how it was set up, some pics, etc. I'm technical, I just haven't played around with diskless systems, Linux operating systems, etc.

I mostly buy my parts from Mwave, Fry's (they've been selling retail Q6600s for $200 or less), and Directron.

I started doing this netbooting stuff about 4 1/2 years ago with Athlon XPs running Find-a-Drug. Part of it was to simply learn a few things about Linux. Some people download Ubuntu. I take on a crazy project. The setup has been through a couple revisions but it's at a state that mostly satisfies me. I'm probably a little more hardcore with my setup than most would be. I have some networked I/O controllers to monitor power that are also directly wired to the motherboard power switch and reset pins for those times when I need control at the hardware level.

2)Your current system seems about $50 cheaper than a Dell box with the same quad chip, 3gig of ram (instead of your 1gig), a 500g drive (instead of your 0g drive), a case, cd/dvd drive, keyboard, and mouse and maybe a few other things...so...although Dell might be a slight monopoly, why not just buy a bunch of the same Dell machines? I would understand if it was physical size dimensions issue and/or power sucking issue...care to elaborate? Just wondering. Heck, you could sell the 500g drives for $100 a pop not to mention the dvd/cd drive for $25.

I'm using 2GB on my quads, not 1. I'm more concerned about the quality and capability of parts rather than quantity. A stock Q6600 gets around 1650 RAC (at least for me). Pushing it to 3GHz gets around 2400. That's not something as easily done with a cheap Dell. The Dell also wouldn't be designed for 24/7 overclocked use. Building it myself lets me design it how I want.

Anyway, overall I'd love to learn a little bit more about where you bought your equipment, how it was set up, some pics, etc. I'm technical, I just haven't played around with diskless systems, Linux operating systems, etc.

I mostly buy my parts from Mwave, Fry's (they've been selling retail Q6600s for $200 or less), and Directron.

Thanks for the tips, Bitspit. I guess I am still in a kind of dilemma...

1)I don't overclock my chips. I don't really believe in it and usually the manufacturers won't replace it if it breaks.

2)I know Dell doesn't sell the cutting edge products/components for a great price but what they do offer is a very economical system. My Quad is pulling in a RAC of just under 1400...and I'm not running RAH at 100%. I'm also on XP which probably affects the performance (compared to linux) by a tad. So let's say I get about 1500 RAC...not bad compared to your 1650. Although I do love to see my stats go up, I'm not going to lose sleep over an 8% performance degradation if all other things are about equal (see point 4) ). And again, the full computer comes with parts I can sell which would in turn drop the desktop price by 10-15%. Not to mention if I did keep the units intact with all the parts, I could turn around and sell the computer a year later vs. trying to sell motherboards (like your setup) would be a little harder/longer of a process...selling a desktop is easy since there is a big market...selling parts gives me a limited audience.

3)I'm not really looking to squeeze every last clock cycle out of the chip or 1 extra RAC. :)

4)I'd love to buy slim "computers" to put in a reasonable amount of physical space, however, I also don't want to spend a lot of time building pieces when I can get a very comparable desktop for the same price or cheaper. The pro about the desktop is that it's ready to go...a few clicks to install XP and I'm set...no hardware to assemble. The con is that the desktop would surely eat up more physical space and probably a bit more power. I'd love to find a Mac Mini-type pc with comparable performance (to a quad with 3gig ram) for $400-$500...this way the physical space is much smaller than a normal desktop. Again, there are a few factors here for this "array project": a)physical size of the array, b)power the array requires (hence my electric bill), c)performance/cost ratio of the array, d)ease of setup/maintenance.



Since, to me, the dollar amounts are so close...as well as the performance (taking out the overclocking variable), why did you go down the route of building everything separately? Was it purely the love of learning something new and the act of building or was there more to it? I used to build computers all the time from the mid 80s till about 1999 and then just lost the excitement (as well as other things in life that eat up your time as you get older :) ).

Thanks again for the tips.

-Eric

---

1)I don't overclock my chips. I don't really believe in it and usually the manufacturers won't replace it if it breaks.


My Quad is pulling in a RAC of just under 1400

So let's say I get about 1500 RAC...not bad compared to your 1650.

3)I'm not really looking to squeeze every last clock cycle out of the chip or 1 extra RAC. :)

b)power the array requires (hence my electric bill), c)performance/cost ratio of the array

The current Intel chips are sold at a fraction of their potential, and they generally don't break from overclocking unless you really overvolt them, and most motherboards won't let you take them that far anyway.
BitSpit quoted 2400 RAC for his OC'd Q6600 against a stock one at 1650 RAC - that's around 45% more for slightly increased power consumption (assuming you're not overvolting in order to get there). Some Core2 chips are stable at 3.2Ghz while undervolted, which can power consumption and heat substantially.

You'll get a lot more compute power/RAC for your money if you build your own and overclock/undervolt.

---

One other thing you should consider, depending on where you live.

Summer is coming up quick in the US. If you live on the Top side of the world, there will be a lot of waste heat generated by an array of computer systems, which will need to be taken care of.

When talking electric bill, just don't forget about cooling all those PCs running at Max.

---