Posts by Timo

1) Message boards : Number crunching : Problems and Technical Issues with Rosetta@home (Message 87913)
Posted 2 days ago by Profile Timo
Post:
Seems like it's now fixed (only about 1 minute after I posted this!) Nice work :)
2) Message boards : Number crunching : Problems and Technical Issues with Rosetta@home (Message 87912)
Posted 2 days ago by Profile Timo
Post:
No new workunits being downloaded on at least 1 of my boxes. Log says:
"Server Error: Feeder not running."

I suspect this just just happened as the server status page still shows the Feeder as running.
3) Message boards : News : Outage notice (Message 86745)
Posted 26 Jun 2017 by Profile Timo
Post:
I had an issue uploading on a couple of my boxes too, but after investigating I discovered it was due to an old 'host' file that had hard-coded DNS entries for the project servers -- something I did myself a little while back as a work-around when the project's DNS registrar was having problems. Simply removing the rah server entries from the host file fixed the issue.
4) Message boards : Number crunching : Question for Researchers about waiting for results (Message 81503)
Posted 20 Apr 2017 by Profile Timo
Post:
Hi David and team.

Some quick background: Yesterday I spent most all of my work day waiting, I write some \'big data\' blending jobs in Hive&Pig (and starting to learn Spark), but the transformations I\'m working on involve many BILLIONS of records and so even with my company\'s 160+ node Hadoop cluster, some steps of my transformation take a couple of hours to crunch.

This waiting time really slows down my ability to iterate and test some aspects of my logic. Where possible I try to find a subset of data that can serve as a test case but there are some use cases where this is strategy cannot be applied.

So, my question for you is, with the multi-days/weeks long turn around times of rosetta jobs, how the heck do you manage to iterate in your experiments efficiently and perhaps more importantly how do you ensure that you don\'t spend a whole two weeks waiting for a run to complete only to find out that there was a typo in the input sequences somewhere? Secondly, what do you do while waiting for jobs to finish?
5) Message boards : Number crunching : Stuck on uploading is a new problem? (Message 81428)
Posted 14 Apr 2017 by Profile Timo
Post:
I also have a task that is stuck uploading. Will try to post more details about it tomorrow if it\'s still stuck when I wake up. I tried putting back the hosts information in case it\'s a dns problem, also tried flushing the dns resolver cache and removing any host entries too.. Doesn\'t seem to be DNS related.
6) Message boards : Number crunching : Ryzen (Message 81344)
Posted 18 Mar 2017 by Profile Timo
Post:
2 minutes on google led me to this R7 1700:

http://boinc.bakerlab.org/rosetta/results.php?hostid=2225133

and this R7 1800X:

https://boinc.bakerlab.org/rosetta/show_host_detail.php?hostid=3181998

For the 1700, assuming it\'s running 16 threads, from the results returned the RAC is approx 15,210 and for the 1800X it works out to 12,268.

If they\'re not running 16 threads then the numbers above will be scaled too high. The granted credit-per-second is higher on the 1700 which suggests it\'s running a lower non-Rosetta load, or is overclocked quite a lot. Or maybe the 1800X gets powered down more frequently in that period. I\'d guess it\'s due to load though.

If those numbers are correct, then they\'re really good at Rosetta and I want one. Or five.


Great finds! Very exciting to see competition in the higher end again. Hopefully they sell enough of these to recoup their investment and keep up the momentum at their R&D labs in AMD.

A few things I\'ve read about Ryzen have explained that finding the right memory configuration is key to getting the best performance out of these chips as keeping all 16 cores fed requires lots of fast memory bandwidth. Furthermore, some people complained that it was very difficult to find a motherboard+memory combination that actually allowed the memory to run at full speeds, hoping as the motherboard selection starts to mature a bit and more OEMs jump in the ring these issues smooth themselves out. :)
7) Message boards : Number crunching : DNS Problems and Late Work Units (Message 81317)
Posted 12 Mar 2017 by Profile Timo
Post:
[quote] There seems to be a deep assumption in there somewhere that most of the clients are supposed to be running continuously for many hours at a time. (Some of mine do, and others don\'t.)

The client preferences (on the website side) have the option of creating multiple (separate) profile categories. You could set up one with shorter target run-times for your systems that run less frequently and another with longer run-times for systems that run more continuously. The available range is quite large, as low as 1 hour and as high as 24 hours.

Either way, it sounds like in your particular case the amount of work being cached may be set too high. The BOINC manager will adjust the cached amount over time to match the percentage of time BOINC is running per 24 hour period, but it takes time to adjust and any changes to the settings will take more than a few days to get applied. (This is the same reason why many people suggest that any changes in target run-times be made in smaller incremental steps).
8) Message boards : Number crunching : Ryzen (Message 81221)
Posted 24 Feb 2017 by Profile Timo
Post:
Very excited to see some competition in the higher end of the CPU market again. This is a huge win for high performance computing / BOINC / Rosetta etc. too

These chips look to pack an amazing overall amount of compute power inside a surprisingly energy efficient envelope.

AMD unveils Ryzen launch dates, clock speeds, performance, pricing [ExtremeTech.com]
9) Message boards : Number crunching : Unintended consequences of the new credit system? (Message 80931)
Posted 23 Dec 2016 by Profile Timo
Post:
I don't blame R@H for having deadlines at all, in fact I keep a short work buffer (0.3 days on most boxes) and have WCG setup as a backup project with low priority so it will just kick in if R@H goes down. This gives everyone the best of both worlds, the average TAT for work on my machines is less than a day, and my boxes stay busy (with a fallback to WCG) even if R@H goes down.

Quick work turnaround is important to me because I am thinking first and foremost about the researchers using this platform. I work with some high performance computing clusters in my day-job and I can attest first hand that long turn around times for queries/model runs leads directly to slower iteration and slower progress.
10) Message boards : Number crunching : Problems and Technical Issues with Rosetta@home (Message 80786)
Posted 26 Oct 2016 by Profile Timo
Post:
[snip]

some of the simulations being run on R@H can complete in just a few days

DAYS? For one work unit? I've never read anywhere if the researchers prefer a long run-time-per-WU over a short run-time-per-WU so I've always (7 years now) had my preferences set to shorter WUs.
Thanks for the input Timo, I appreciate it.

I've read that they wanted longer workunits selected as a way to reduce the load on the server. I haven't seen whether they changed their minds on that, though.


To be clear, I was advocating for having a shorter work 'buffer' - ie. no need to have 10+ days of work buffered and thus drive up the average 'turn around time' of a given WU (even a short one) to be many days because it gets downloaded onto the end of a really long queue..

By all means, set a longer WU target run length. What I think is probably not a good idea is a 10 day buffer/cache of 1 Hour WUs. Basically, high server load coupled with long turn around time..

As someone who deals with crunching data daily at my day job, I appreciate first hand the frustration of waiting for queries to run that take many hours, I can't imagine being asked to iterate and experiment in an environment where queries take many days or even weeks to complete.. hence why I think a small cache is better for the project just in terms of enabling faster iterative experimentation.
11) Message boards : Number crunching : Problems and Technical Issues with Rosetta@home (Message 80778)
Posted 25 Oct 2016 by Profile Timo
Post:
@LC - Just checking the obvious here, so I don't mean to be insulting by asking such a simple question but it's happened to myself before without even noticing so I thought I would ask/mention... Did you by chance hit the 'Show active tasks' button at the top left of the 'Tasks' pane of BOINC (If so it will currently say 'Show all tasks', try toggling it and seeing if you indeed do have more work buffered).

Secondly, a 10 day buffer (if I read your post right, that is what you had set?) seems really really excessive, some of the simulations being run on R@H can complete in just a few days - why be the one guy slowing down pace of iteration/experimentation by making researchers wait a whole 10 days? I usually keep a 1 day buffer, and have 'Mapping Cancer Markers' (via World Community Grid) as my 'backup' project when R@H is out of work.. just a thought.
12) Message boards : Number crunching : Problems and Technical Issues with Rosetta@home (Message 80616)
Posted 10 Sep 2016 by Profile Timo
Post:
Thanks for the updates David and Serge, looking forward to see how this new hardware improves things next time CE decides to throw a wall of compute at us.

I echo others' chants that this is a great opportunity to update the server software since there will be extended downtime anyways, (if even just updating BOINC server software --> Step-by-step guide to running BOINC server Upgrades here: https://boinc.berkeley.edu/trac/wiki/ToolUpgrade) The aforementioned guide seems to imply that the upgrade can be done rather painlessly and is not destructive to existing data. Of course, would be best to test on Ralph first :)

Good luck! Thanks again for keeping us all in the loop.. Exciting times :)
13) Message boards : Rosetta@home Science : The Protein folding revolution! (Message 80457)
Posted 1 Aug 2016 by Profile Timo
Post:
Watched the video, it\'s super accessible to lay people and does a great job of highlighting a huge swath of what Rosetta does and in particular the co-evolution \'shortcut\' that you\'ve been working so much on Sergey - big congrats!

Are you using co-evolution predictions in this year\'s CASP? I remember it being quite successful in CASP11.

I\'d be curious to learn if/how there has been any iterative refinement of the methodology since CASP11?

PS - You and David are the most engaged baker lab members on the boards here, thanks for taking the time to post and keep us in the loop. :)
14) Message boards : Number crunching : Crowdfounding for new servers? (Message 80425)
Posted 23 Jul 2016 by Profile Timo
Post:

Their name (business name, memorial name) on a series of BOINC work units?


While I think all of your suggested perks are great, this one is especially awesome because it has a very low cost to provide (it\'s just data entry when naming a query) but carries enough value that it could be a perk for a decent price level ($50 or $100 donations, etc.)


Does anyone have experience with crowdfunding projects? Such a project will need someone to advise, and manage it.
...
Does anyone have PR experience that might be able to help create press releases to draw attention to such a campaign?


I think this could go in two basic directions, one would be an open crowdfunding campaign on a public crowdfunding site, in which case yes the PR coordination, etc. would be needed. The benefit is it would potentially bring in much more support, and raise awareness around Rosetta@Home/BOINC potentially leading to more crunchers, the drawback is it would require much more work to create and manage, fulfill perks, etc.

The second direction, which may be simpler (not necessarily better) would be to implement a fundraising thermometer on the front page that is directly linked to whatever payment processing API the admins wish to use (A number of PHP based thermometers exist that interface with PayPal for example, like this one)
The benefit is it would create a specific goal and a way to visualize progress towards that goal, transparency around project funding, along with visualizing one\'s own contribution, while requiring little management work compared to a full blown public crowdfunding campaign. The down side is it would only tap the existing Rosetta@Home user base for support, wouldn\'t include cool perks, etc.
15) Message boards : Number crunching : Crowdfounding for new servers? (Message 80417)
Posted 23 Jul 2016 by Profile Timo
Post:

[quote]
I think the admins and scientists get enough from its contributors and maybe time they should be giving something back so, NO to further contributions.
...
As for your opinion \"A Fool And His Money Are Soon Parted\"


@Betting Slip - Reading through a few pages of the 70 or so posts you have made on these forums I\'m extremely hard pressed to find any where you have said or contributed a single positive thing; Indeed, in most of your posts you sound almost constantly miserable, disgruntled, and incredibly cynical about basically everything.

I really truly feel bad for you. I hope you can find a resolution to whatever it is in your daily life that is the cause of your seemingly constant irritability and negativity.

We\'re all here because we want to make a difference, maybe try to keep sight of that before spewing any more negativity onto these boards.
16) Message boards : Number crunching : Crowdfounding for new servers? (Message 80415)
Posted 22 Jul 2016 by Profile Timo
Post:
I think the admins and scientists get enough from its contributors and maybe time they should be giving something back so, NO to further contributions.


Speak for yourself, I\'ll gladly donate more if a properly organized crowdfunding campaign were held. I think this is a great idea boboviz, hoping the admins take notice and forward it to the appropriate parties.

17) Message boards : Number crunching : Credit granted pending stack (Message 80394)
Posted 18 Jul 2016 by Profile Timo
Post:
Is it possible to see a reporting about donations/grants/whatsoever and how they use money? Or these numbers are not public?
It\'s not polemic, only curiosity


Ok, i do a little research by myself and i find:
In June 2014 the IPD established its Translational Research Center, which is operated with a $1.4 million Opportunity Grant from the Life Sciences Discovery Fund (LSDF). The LSDF funding to the IPD was matched 4-fold by generous contributions from private donors ($3.2 M), UW ($1.4 M), and the Washington Research Foundation ($1 M).
Dr. Pultz was recently awarded a competitive $250K proof of concept matching grant from the LSDF to conduct animal efficacy studies and safety testing of the lead KumaMax variant, an effort which is being matched by an additional $176K in philanthropic donor funding.
In the last update, we mentioned that Governor Jay Inslee’s 2014 Supplemental Budget (4-13-13) included $1 million to support the IPD mission, enabling the UW to recruit a talented computational biologist, Dr. Frank DiMaio
This January, the IPD was awarded $12M in funding by the Bill & Melinda Gates Foundation to support vaccine research.
May 15, 2014. With a very generous $8 M gift from the Washington Research Foundation (WRF), the IPD has launched the WRF-IPD Innovation Fellows Program supporting research partnerships between the IPD and other Seattle-area research institutes or UW departments


Yeap, my 20 dollars are very important.


While those numbers might seem impressive, there\'s no indication of how many of those donations are earmarked directly to the Rosetta@Home program fund. At the same time, $28.426M is still very much a shoestring budget in the bioinformatics world and the things people are stomping their feet for on these forums may not make the cut in terms of priorities when slicing up this particular pie; All said, the fact remains that your $20 donation may be the loudest voice you have in advocating for something like a server upgrade, etc.
18) Message boards : Number crunching : Credit granted pending stack (Message 80382)
Posted 16 Jul 2016 by Profile Timo
Post:

Donation link is on the front page near the top-left, something to consider.

I never paid attention to that part of the site before...


I\'ll also mention that when you donate you are given a box to fill in what you want the money to be spent on. If the program suddenly got flooded with a hundred donations all with the message \"for use for server upgrades to help with the \'project backoff\' issues during peak load times\" perhaps it would be more effective than being a talking head on a forum.


That\'s exactly what I just did...


@ BarryAZ and Dr. Merkwürdigliebe - Thanks for being so awesome. When I wrote the post about donating I was half prepared to be heavily flamed by the \'but we are already donating our computing power!\' crowd.

If the project is in need of funds, be it for upgrading the server or perhaps hiring some help for exploring CPU optimizations or GPU ports or whatever, I think looking into running official crowd-funding campaigns may be a good way to go. \'Fundraising Thermometers\' do wonders for helping projects like this hit a goal :)

Let\'s keep the donations flowing, as my original comment said, you can put your thoughts about the need for server hardware upgrades in the \'what do you want this donation to be spent on\' comment box that you get to fill out when donating, and I recon it will get a lot more official visibility from the project admins than posting aimlessly on this forum. :)
19) Message boards : Number crunching : Credit granted pending stack (Message 80379)
Posted 16 Jul 2016 by Profile Timo
Post:
For all the talking heads on here complaining about lack of server upgrades, consider putting your money where your mouth is.

I donate a couple hundred dollars per year to this project on top of the computing resources I contribute.

Perhaps if more people donated a little money (it doesn\'t have to be hundreds of dollars, if everyone on these boards donated $10 it would add up quickly) perhaps it would afford the project the resources to do things like upgrade the servers, etc.

Donation link is on the front page near the top-left, something to consider.

I\'ll also mention that when you donate you are given a box to fill in what you want the money to be spent on. If the program suddenly got flooded with a hundred donations all with the message \"for use for server upgrades to help with the \'project backoff\' issues during peak load times\" perhaps it would be more effective than being a talking head on a forum.
20) Message boards : Number crunching : Stuck on Uploading (Message 80278)
Posted 24 Jun 2016 by Profile Timo
Post:
My suspicions confirmed;

  • srv1.bakerlab.org at 128.95.160.142 is not working / not accepting uploads
  • srv4.bakerlab.org at 128.95.160.145 is accepting uploads



All tasks that are trying to be uploaded to the first server are failing, while any tasks attempt to be sent to srv4.bakerlab.org successfully upload.



No go for me.

Here is the line I added:
128.95.160.145    srv1.bakerlab.org, in accordance with these instructions.

Here is the Event log:
6/24/2016 4:48:10 PM | rosetta@home | update requested by user
6/24/2016 4:48:15 PM | rosetta@home | Sending scheduler request: Requested by user.
6/24/2016 4:48:15 PM | rosetta@home | Not requesting tasks: too many uploads in progress
6/24/2016 4:48:17 PM | rosetta@home | Scheduler request completed


This fix is not to help with Requesting new tasks, but rather to help get your uploads going again. Judging from the snippet of your log file that you shared, you were attempting to fetch new work..

Go to your transfers tab and click one of your Uploading tasks that is stuck on \'Retry in ....\' and click Retry Now - it should work.

Also, if you want to check that your hosts modification worked, you can attempt to visit this link http://srv1.bakerlab.org/ and it should bring you to the Rosetta@Home homepage :) Alternatively, you can ping srv1.bakerlab.org

Cheers


Next 20



©2017 University of Washington
http://www.bakerlab.org