Are beta testing? Or working?

Message boards : Number crunching : Are beta testing? Or working?

To post messages, you must log in.

AuthorMessage
Profile Paul D. Buck

Send message
Joined: 17 Sep 05
Posts: 815
Credit: 1,812,737
RAC: 0
Message 52189 - Posted: 30 Mar 2008, 19:41:50 UTC

I did not see a thread that addressed this question in a way that explained it to my 2-year old type mind...

The Rosetta Application says Beta, so, are we doing science? Or just playing around?

I know all new applications need to be tested. But what are we testing... how is this advancing the project to finding how to pack the suitcase?

From what little I have been able to glean, and I will admit I have not done an exhaustive search (time and health prohibit), but, I sort of feel like I am, er, wasting my CPU cycles ... as necessary as testing is, well, to me, at least, it is not what makes me feel that we are advancing ... or is it?

Inquiring mind wants to know ... before he might leave again ... :)
ID: 52189 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Astro
Avatar

Send message
Joined: 2 Oct 05
Posts: 987
Credit: 500,253
RAC: 0
Message 52190 - Posted: 30 Mar 2008, 20:10:14 UTC
Last modified: 30 Mar 2008, 20:29:45 UTC

HI Paul,

don't write this in stone, but from my understanding their "Ralph" (rosetta alpha) project is the site for beta/alpha testing applications. The word "Beta" in the wu name is just a name and nothing more from their regular "production" run. Or perhaps it not the app that's being "beta" tested but the wu type itself. Anyway, it's a fair enough question that I don't have a definite answer for.

tony

[edit]OK I just spent the time from my last post to this attempting to find the one post which I kind of think I remember seeing on this, without success.[/edit]
ID: 52190 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile dcdc

Send message
Joined: 3 Nov 05
Posts: 1829
Credit: 114,404,250
RAC: 54,810
Message 52192 - Posted: 30 Mar 2008, 22:10:19 UTC

Paul, the bakerlab have just created the worlds first enzymes that have no natural pre-cursors. The 'beta' reference is probably not the best label - they're not testing the rosetta software for computer compatibility etc, they're testing the how well the software can simulate reality - the project is a software project so the different Rosetta tasks are the science.

You can be sure that your CPU cycles are not being wasted! For confirmation have a look through the more recent posts in David Baker's journal.

HTH
Danny
ID: 52192 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Paul D. Buck

Send message
Joined: 17 Sep 05
Posts: 815
Credit: 1,812,737
RAC: 0
Message 52193 - Posted: 30 Mar 2008, 23:14:57 UTC

Well, they were not wasted in either case in that the idea was to load down my new PC to try to "break" it ... if it is gonna die, I want it to die soon ... and one other one I am giving away to a friend and I did not want to give away a computer that did not work ... SO ...

But it was a curiosity...

I have been trying to wade through stuff and get back to understanding this world ... though is has not been a comfortable ride ... as the testing period is coming to a close it does look like that leaving again is the best bet ... though, not sure yet ...

It has been interesting testing this new beast ... it took just over 3 days to run the RAID Utility's verify test on the RAID array, thankfully the array can be mounted while this is going on (slows r/w speed, though I did not notice it for what else I was doing) ...

I will try to find time to wade through the journal, though I thought I had read it ... and either I was stupid and missed the explanation, or, well, I missed it ... or did not read it ...

Thanks for the tips and I will go look, time has not improved my situation so, I am a LOT slower now than in the past ... so ... may be why I did not find it ...
ID: 52193 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Snags

Send message
Joined: 22 Feb 07
Posts: 198
Credit: 2,797,168
RAC: 648
Message 52195 - Posted: 31 Mar 2008, 0:00:52 UTC - in response to Message 52190.  

HI Paul,

don't write this in stone, but from my understanding their "Ralph" (rosetta alpha) project is the site for beta/alpha testing applications. The word "Beta" in the wu name is just a name and nothing more from their regular "production" run. Or perhaps it not the app that's being "beta" tested but the wu type itself. Anyway, it's a fair enough question that I don't have a definite answer for.

tony

[edit]OK I just spent the time from my last post to this attempting to find the one post which I kind of think I remember seeing on this, without success.[/edit]



Is this post from Mod.Sense what you were looking for?
While the project needs to run under two versions, one will always need to have a unique name from the other. At this point the word "beta" is used to distinguish the two. All the work performed by Rosetta has previously been tested on Ralph.


Or this post from David E K
I didn't pull back the mini tasks because there are bugs in it. I did it in response to user's complaints and the fact that the minirosetta jobs were filling the queue more than expected. Are you talking about the beta in the rosetta application name? We should take that out of the name since it isn't a beta app and is misleading.

I think the general flavor of things here at R@H is that longer more exhaustive testing should be done at Ralph before the program is released to R@H.
The feeling is that "beta" type work should be done at Ralph and then a fresh named release without "beta" be released on R@H. The feeling as I see it, is that work units and programs are not tested enough before being released here. minirosetta for instance should be deeply tested on Ralph and when it is "complete" and work for that program has been tested deeply and when it is 99% tested with work, then release it here. Don't halfway test it on Ralph and then release it here and then pull it back because there are bugs in it.

"If you aren't going to use Ralph as a "real" test site, why don't you shut it down, conserve those resources and continue to test on Rosetta like you do now."

Ralph is invaluable. We catch many bugs and errors with ralph that would otherwise happen on R@h.


/off-topic
And as a side note to Paul: I have been eagerly reading your posts over on the Seti boards. I haven't posted as I am slow to gather my thoughts and my attempts to get them down in writing have so far been far longer than your own posts(!). I just wanted to say your vision is appreciated.

Snags
ID: 52195 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Mod.Sense
Volunteer moderator

Send message
Joined: 22 Aug 06
Posts: 4018
Credit: 0
RAC: 0
Message 52196 - Posted: 31 Mar 2008, 0:08:04 UTC

Yes, Ralph is where the testing is done. The Project Team has been running dual versions of late. One being the last, well they seem to refer to it as "stable" version, although that's not the best wording either. The other having the word "beta" in the app name.

They're running longer timeframes of some trials and want all WUs of a given trial to be run with the same version of code. This gives them a frame of reference where everything in a given trial was run with the same application version. They are also adding new searches for RNA, the enzyme work and enhancements, and these are coming in the "beta" version name. They've been tested on Ralph, but are a newer version then the current trial in progress.

Even more recently, they have been introducing the new "mini" application. This is basically a rewrite of the Rosetta programs. They cleaned up a lot of the code, optimize memory usage, and this will help pave the way for future revisions, enhancements and tackling still larger proteins... and CASP8!
Rosetta Moderator: Mod.Sense
ID: 52196 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Snags

Send message
Joined: 22 Feb 07
Posts: 198
Credit: 2,797,168
RAC: 648
Message 52197 - Posted: 31 Mar 2008, 0:13:30 UTC

Oops. Left this one from Rhiju one out of the previous post:

Another interesting change: we are now able to send out workunits with the older application (5.68) and the newer one (5.70). This is advantageous because the older one is "stable" and lets us carry out long-term experiments such as looking for very rare low energy conformations with hundreds of thousands of workunits over several months. The new one has bug fixes for some modes as well as new features. The new one will be called "rosetta_beta_5.70" -- you can check out this link to see what's crunching.


That's from the Rosetta Application Version Log, a stickied thread in Number Crunching. Many of the later posts for newer versions don't mention beta in the name although that's how they show up on my computer.

Snags
ID: 52197 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Snags

Send message
Joined: 22 Feb 07
Posts: 198
Credit: 2,797,168
RAC: 648
Message 52198 - Posted: 31 Mar 2008, 0:24:02 UTC - in response to Message 52196.  


They're running longer timeframes of some trials and want all WUs of a given trial to be run with the same version of code. This gives them a frame of reference where everything in a given trial was run with the same application version. They are also adding new searches for RNA, the enzyme work and enhancements, and these are coming in the "beta" version name. They've been tested on Ralph, but are a newer version then the current trial in progress.



Does this mean that the tasks run by apps with beta in their name are essentially a rerun of previous tasks in order to test improvements to the algorithm?

Snags
ID: 52198 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Mod.Sense
Volunteer moderator

Send message
Joined: 22 Aug 06
Posts: 4018
Credit: 0
RAC: 0
Message 52199 - Posted: 31 Mar 2008, 1:17:50 UTC - in response to Message 52198.  

Does this mean that the tasks run by apps with beta in their name are essentially a rerun of previous tasks in order to test improvements to the algorithm?

Snags


No. The tasks under the new version may not even be for the same protein. Or, if the same protein, might be using one of the other methods to solve. etc.

But yes, some of the work run is to get a comparison. I mean if you write new code and think it's going to help get better answers faster... how else do you prove whether or not you have achieved your objective? And to what degree?

It's sort of the same reasoning behind why we run searches for proteins where the native structure is already known. We have to prove whether or not we could have predicted that native structure properly.
Rosetta Moderator: Mod.Sense
ID: 52199 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Paul D. Buck

Send message
Joined: 17 Sep 05
Posts: 815
Credit: 1,812,737
RAC: 0
Message 52200 - Posted: 31 Mar 2008, 1:44:27 UTC

Ok, I guess, I am Ok with this but I am still not sure what we are doing with this run.

History lesson (sorry) when I was here two years ago we were, as I recall, running known things for which there was no known 3D representation of the "reality", in other words, we were not sure what we were looking for ... but, each trial we ran we could see if and how close we might be to the truth.

I have looked at the other places and I am still not sure I got my brain around it ... that is, what are we doing NOW ...

I don't suppose it is all that important, this was to be a short term interlude during which I was going to sneak in and sneak out and keep my vow of silence ... alas, the best laid plans ...

Snags,

I thank you for the kind words ... and I have to admit that the overt sense of hostility (not sure if that is quite the right word), but certainly the overall feel of stagnation and loss of vitality that has spread over the world of BOINC like a dark blanket daily tells me that I was wrong to come back and should leave again as soon as possible.

It is not like I came to the understandings published over there overnight ... there is literally YEARS of history behind the concepts I am trying to articulate ( and mostly failing to get across ) to both the participants and the project personnel.

Though only a couple people have made the comment, I know the most common reaction is that they are, as one critic put it, long-winded, non-specific, rants ... Well, this is the second pass through this territory for me.

As I said over there in one of my last posts I see a straw in the wind and a light at the end of the tunnel ... but I will not be surprised if the next thing will be the whistle of the train that runs me over ... again ...

Once more in my life I am having the experience of having predicted the future and returning to find out that I was right ... the user population when I left, if memory serves was over 1.5M ... the stat I saw the other day was 1.3M ... that is not growth, that is not even stagnation, that is slow death ... I may have the numbers wrong, but, after two years gone the numbers should be 2-3M participants ...

Anyway, I promised a couple of people I would do a couple things, and I want to get SAH below EAh in my stats ... and the only long pole I have in the tent now is one CPDN task that will take me ~17 days to clear ... but ... that is the extent of my commitment ... And I might be able to shave that time by rconfiguring the Xeon to work in non-HT mode ... so, though I have not decided for sure ... my take is enjoy the next two weeks ... :)

Then I will be gone ... and by the next time I buy a new computer, BOINC will be gone too ... the self-destruction I predicted 2+ years ago is well underway ... I saw the pre-cursor signs then and now the symptoms are all over the place ... yes, even here ...
ID: 52200 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile dcdc

Send message
Joined: 3 Nov 05
Posts: 1829
Credit: 114,404,250
RAC: 54,810
Message 52201 - Posted: 31 Mar 2008, 12:43:12 UTC - in response to Message 52200.  

Then I will be gone ... and by the next time I buy a new computer, BOINC will be gone too ... the self-destruction I predicted 2+ years ago is well underway ... I saw the pre-cursor signs then and now the symptoms are all over the place ... yes, even here ...

DC isn't going anywhere. Rosetta is tackling one of the biggest problems in science and is making real-life progress beyond what anyone has ever achieved before:

http://www.nature.com/nature/journal/vaop/ncurrent/full/nature06879.html

Why anyone wouldn't want to contribute when their contribution is of minimal cost is beyond me.
ID: 52201 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Paul D. Buck

Send message
Joined: 17 Sep 05
Posts: 815
Credit: 1,812,737
RAC: 0
Message 52204 - Posted: 31 Mar 2008, 15:19:50 UTC - in response to Message 52201.  

Why anyone wouldn't want to contribute when their contribution is of minimal cost is beyond me.

I shan't repeat the long history and my reaction to it. Suffice to say that the last time around I sacrificed what little was left of my sanity. In that there is not much left I submit that there was not far to go. But I shall not repeat the experiment.
ID: 52204 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote

Message boards : Number crunching : Are beta testing? Or working?



©2024 University of Washington
https://www.bakerlab.org