Posts by In Memory of Kimsey M Fowler Sr
|
1)
Message boards :
Number crunching :
Client Errors
(Message 73031)
Posted 9 May 2012 by  In Memory of Kimsey M Fowler Sr
Post: Guys, I'm sorry to say that, but this is really embarrassing. I'm also software developer and I also have to deal with user reported bugs and I cannot imagine having a bug reported 3 months ago, with tons of data to reproduce the issue, and no answer. There's a bug in validation code - where else could it be? I'm just shaking my head in frustration about this too... I'm a former software engineer from an SEI CMM Level 5 software design organization in Seattle. I was dumbstruck by an e-mail back from Rosetta staff yesterday that no further effort will be expended to determine why the Rosetta servers are rejecting WU's. One can assume that the staff doesn't see this problem as widespread enough to make it worth their time to look into. Yesterday's post by David Baker suggests he is thrilled with the available computing power available to the project at the present time. Might I suggest that those of you experiencing this problem consider donating your computing resources to folding@home at Stanford Medical School. Their software is self-contained, stable, doesn't run under BOINC middleware, computes on your choice of CPU and/or GPU, has an excellent working simulation of the protein molecule that can be manipulated with the mouse to view/rotate/enlarge/etc, lots of interesting information that's easily accessible about each protein you're folding and why it is important, and is the world's largest computing network. |
|
2)
Message boards :
Number crunching :
Client Errors
(Message 72988)
Posted 4 May 2012 by In Memory of Kimsey M Fowler Sr
Post: For those waiting for a new version 7.0.27 is out as a Beta but DOES have the fixes in it that can help with everything running at High Priority. Whether it fixes Rosetta's error problems or not is unknown, but since it builds on version 7.0.25 it is unlikely. It does not fix the client error problem. --KMF, Jr. |
|
3)
Message boards :
Number crunching :
Client Errors
(Message 72950)
Posted 1 May 2012 by In Memory of Kimsey M Fowler Sr
Post: Just ran three tasks. Successfully. That was exciting to read so I upgraded to NVIDIA 301.24, Rosetta 3.30, but stuck with BOINC 6.12.34. No successes to report. My first three WU's to complete failed in the usual manner. If I don't soon hear of some progress from the Rosetta project on our problem, I'm considering bailing on Win7/64 and giving Ubuntu a try. Since this machine is totally dedicated to folding, it would be a good learning platform to experiment with their new Linux OS released just last week. Someone on one of the other Rosetta number crunching forums said it was really fast and easy to set up. You can even give Ubuntu a test drive from a CD burned from the downloaded ISO file without committing to installing it on the hard drive... and it's all free! How sweet is that? ---KMF, Jr. |
|
4)
Message boards :
Number crunching :
Client Errors
(Message 72902)
Posted 27 Apr 2012 by In Memory of Kimsey M Fowler Sr
Post: I've only made a preliminary review of the {Wireshark} data so far, but one obvious difference is that when the NVIDIA driver is installed (failure condition), some additional information related to CUDA is included in the outgoing information packets transferred to the server. Some additional packets have been captured from my other machine (the one that's working correctly). CUDA tags are going out in the data uploaded to the Rosetta server from that machine, and this suggests this is not an issue. Rather than continue to rerun the same old WU's saved to disk at breakpoint, a third test was done using seven new WU's run on the problem machine. Their completion times were staggered to keep data uploaded from various WU's clearly segregated. Wireshark was used to sniff packets for three sets of data in failure mode and the remaining four sets in success mode. Wireshark includes a utility to reassemble packet data, and three distinct sets were reassembled for each of the seven WU's and output to text files. When ExamDiff was used to compare similar output files, no differences were observed that did not seem unreasonable. Our problem is only being reported on machines running Win7 64-bit OS, so I'm leaning towards a suspicion that the accompanying 64-bit compatible GPU drivers are somehow inserting some extra bytes somewhere. Perhaps this causes the uploaded data file to fail validation when the server software cannot match the file length and/or MD5 hash... just a possible theory. ---KMF, Jr. |
|
5)
Message boards :
Number crunching :
Client Errors
(Message 72886)
Posted 25 Apr 2012 by In Memory of Kimsey M Fowler Sr
Post: I have a client error too. May I post here? Hi: Yes, certainly, if it's the same problem. Without knowing your work unit number, we cannot look at the full results: The very last line where it says "application version"... was the version number absent? If so then that is a good sign that you are having the same problem. Your computing platform seems to have the same characteristics as well, the i7 processor and Win 7 64-bit anyway. What video card are you using? Here's a seemingly silly question for any of you experiencing this problem: Does your machine have a CD/DVD drive installed? One sort of odd thing with my problem machine is that I do not have one built in. I use a USB DVD drive I plug in when necessary. Thanks. ---KMF, Jr. |
|
6)
Message boards :
Number crunching :
Client Errors
(Message 72878)
Posted 25 Apr 2012 by In Memory of Kimsey M Fowler Sr
Post: Per request from Rocco, Wireshark was used to collect packets moving between my machine and the Rosetta server under both success and failure modes of operation. ExamDiff is being used to identify differences in the packet contents. I've only made a preliminary review of the data so far, but one obvious difference is that when the NVIDIA driver is installed (failure condition), some additional information related to CUDA is included in the outgoing information packets transferred to the server. Here is a snippet extracted from a packet:
<os_name>Microsoft Windows 7</os_name>
<os_version>Home Premium x64 Edition, Service Pack 1, (06.01.7601.00)</os_version>
</host_info>
<disk_usage>
<d_boinc_used_total>2148698524.000000</d_boinc_used_total>
<d_boinc_used_project>2147898783.000000</d_boinc_used_project>
</disk_usage>
<coprocs>
<coproc_cuda>
<count>2</count>
<name>GeForce GTX 580</name>
<req_secs>345601.728000</req_secs>
<req_instances>2.000000</req_instances>
<estimated_delay>0.000000</estimated_delay>
<drvVersion>28562</drvVersion>
<cudaVersion>4010</cudaVersion>
<totalGlobalMem>3221225472</totalGlobalMem>
<sharedMemPerBlock>49152</sharedMemPerBlock>
<regsPerBlock>32768</regsPerBlock>
<warpSize>32</warpSize>
<memPitch>2147483647</memPitch>
<maxThreadsPerBlock>1024</maxThreadsPerBlock>
<maxThreadsDim>1024 1024 64</maxThreadsDim>
<maxGridSize>65535 65535 65535</maxGridSize>
<totalConstMem>65536</totalConstMem>
<major>2</major>
<minor>0</minor>
<clockRate>1800000</clockRate>
<textureAlignment>512</textureAlignment>
<deviceOverlap>1</deviceOverlap>
<multiProcessorCount>16</multiProcessorCount>
</coproc_cuda>
</coprocs>
<result>
<name>heterodimer_design_18_pose_C_abinitio_SAVE_ALL_OUT_46199_4020_0</name>
<final_cpu_time>10643.500000</final_cpu_time>
<final_elapsed_time>11430.226067</final_elapsed_time>
<exit_status>0</exit_status>
<state>5</state>
<platform>windows_x86_64</platform>
<version_num>326</version_num>
<app_version_num>326</app_version_num>
<stderr_out>
<core_client_version>6.12.34</core_client_version>
<![CDATA[
<stderr_txt>
[2012- 4- 8 19:55:12:] :: BOINC:: Initializing ... ok.
[2012- 4- 8 19:55:12:] :: BOINC :: boinc_init()
BOINC:: Setting up shared resources ... ok.
BOINC:: Setting up semaphores ... ok.
BOINC:: Updating status ... ok.
BOINC:: Registering timer callback... ok.
BOINC:: Worker initialized successfully.
Everything between <coprocs> and </coprocs> is added information. Notice that soon after that data comes the version number of Rosetta: <version_num>326</version_num> which is lost in the final WU's web page. I wonder if the server software is robust enough to handle unexpected XML tags? For example, perhaps these tags get output only for the Win 7 64-bit version of the driver. ---KMF, Jr. |
|
7)
Message boards :
Number crunching :
Client Errors
(Message 72775)
Posted 15 Apr 2012 by In Memory of Kimsey M Fowler Sr
Post: Just got a batch of Ralph WUs and I can confirm that my host has been able to successfully complete Ralph while failing here at Rosetta. I set the runtimes to be 1 hr on both projects. I installed Ralph@Home several days ago, but it won't give me any tasks. I've tried the PROJECT, UPDATE button many times. Any suggestions? ---KMF, Jr. |
|
8)
Message boards :
Number crunching :
Client Errors
(Message 72774)
Posted 15 Apr 2012 by In Memory of Kimsey M Fowler Sr
Post: Here are the proposed test steps: The test to run a WU to both a failed and a successful state of completion is done. I generally followed the procedure, but with modifications suggested by Mod.Sense (thank you for the very useful info). The primary WU data files that are generated by my computer look about the same except with some "time description" differences (delta times, not clock times). I don't know if that is significant. The file name BOINC/client_state.xml had generally the same information for both the failed and successful runs. One difference was the file size and MD5 were different, but only due to the difference in the time description values. I looked at other file types in the collection, but couldn't find anything interesting. Rocco's earlier findings from looking at my save BOINC directory didn't pick up any irregularities either. Most of the run details you see reported on the web for each work unit are being extracted from the client_state.xml file. There's only one of these files in the BOINC directory, and it contains details for all of the different WU's you are running or have waiting in the queue. The raw data file for each WU lives in the directory BOINCprojectsboinc.bakerlab.org_rosetta and is named with the long (alphanumeric) name of the WU. It has no file extension, so you need to add ".GZ" if you want to unzip it to see the data. The file is usually deleted after it uploads soon after completion of a WU. The point of this is that what you see on the WU web page primarily comes from the client_state.xml file and not from the raw data file. This begs the questions: 1) is the raw data file getting uploaded at all? 2) is it being modified/damaged during transfer? 3) will failing the MD5 check cause client errors? 4) what server-side software writes the phrase "client error" and what triggers that error? 5) where does the WU web page get the Rosetta version number? If you are interested in looking at the details of the analysis, the pertinent information for one WU I looked can be found in this zip file. If you want additional data files from the BOINC directory, please let me know. ---KMF, Jr. |
|
9)
Message boards :
Number crunching :
Client Errors
(Message 72748)
Posted 13 Apr 2012 by In Memory of Kimsey M Fowler Sr
Post:
I agree with this statement. I cannot speak for the AMD's. Sorry for the delay with the test I proposed several days ago, but it's tax time in the US. The test runs are nearly complete, and I hope to analyze the results this weekend. |
|
10)
Message boards :
Number crunching :
Client Errors
(Message 72718)
Posted 9 Apr 2012 by In Memory of Kimsey M Fowler Sr
Post:
Questions for Rocco or other Rosetta Staff Members: I am planning a new test for which I would appreciate any tips that might be helpful for planning and execution. First of all I think we have all been assuming there is a problem on the client side output from Rosetta or BOINC. What if for example one of the output files that's uploaded to the server at the completion of a WU has a slightly different output format, say an unexpected space or extra character caused by an NVIDIA driver. When the uploaded files are processed by the server, a read statement in the processing software chokes on the extra character and we never see the Rosetta version number for the WU in "Task Details". Am I correct that the server uses our uploaded files as an input to generate the "Task Details" HTML files that we view on the web for each WU? If so, do we know if "Exit status = 0 (0x0)" being already in the preliminary version of the file is there by default at the end of a failed WU, or did our "client error" WU's really successfully complete, but part of the server side software failed in generating the HTML output? Enough rambling about my theories. The goal of the test is to run the same WU twice, first with the machine in a bad state, and a second time with the machine configured for success. Here are the proposed test steps: 1) Determine from Rosetta staff exactly which files are uploaded to the server following execution of a WU and where exactly they reside while waiting to upload. 2) Configure my problem machine so that it will successfully complete WU's without client error. 3) Pick a particular work unit and when it is at about the 50% completion mark, terminate BOINC/Rosetta to force it to be saved at a breakpoint. 4) Copy and save the entire /ProgramData/BOINC directory. 5) Unplug the network cable, restart BOINC/Rosetta, and let the chosen WU complete successfully. 6) Terminate BOINC/Rosetta, again forcing existing WU's into a second breakpoint condition. 7) Copy & save the output files for the chosen WU (note that it cannot upload because the network cable is unplugged). 9) Configure the machine so that subsequent WU's will end with client errors. 8) Restore the entire BOINC directory to the state it was in at the first breakpoint. 10) Let the same WU chosen earlier complete again, but with client errors. 11) Collect the same set of output files before they are uploaded to the server. 12) Use file comparison software to identify differences. Once the differences are identified, it should be much easier to find a solution. Comments/suggestions please? KMF, Jr. |
|
11)
Message boards :
Number crunching :
Client Errors
(Message 72716)
Posted 9 Apr 2012 by In Memory of Kimsey M Fowler Sr
Post: I have received one WU at ralph and crunched it successfully!! Hi Rayburner: Thank you for some positive sounding news! I wonder if you have fiddled with the drivers or anything else since you last had a WU ending in client error? Perhaps you could give Rosetta another try and see what happens. Thus far I have not tried Ralph, but will give it a shot. Kimsey, Jr. |
|
12)
Message boards :
Number crunching :
Client Errors
(Message 72715)
Posted 9 Apr 2012 by In Memory of Kimsey M Fowler Sr
Post: Some additional tests have shown that when my machine was generating good work units with "only one NVIDIA installed", only a subset of the NVIDIA drivers were really installed. Noteworthy among the missing was the CUDA driver. I'm not saying this driver is at the root of the problem, only that the full complement of drivers was not present for successful WU's. |
|
13)
Message boards :
Number crunching :
Client Errors
(Message 72714)
Posted 9 Apr 2012 by In Memory of Kimsey M Fowler Sr
Post: Is there anyone with our problem that does not have all of the following attributes: 1) one or more NVIDIA GPU's, & 2) running Win7 64-bit, & 3) Intel I7 processor ? |
|
14)
Message boards :
Number crunching :
Client Errors
(Message 72704)
Posted 9 Apr 2012 by In Memory of Kimsey M Fowler Sr
Post: If people are willing, signing up to Ralph@home with computers which are experiencing the problem would probably be worth trying. I was going to try Ralph@Home, but when I went to add it as a BOINC project, it wasn't in the list. Is there is a trick to finding it, or has it been temporarily removed because Rosetta 3.26 just got release and there's nothing to test? |
|
15)
Message boards :
Number crunching :
Client Errors
(Message 72700)
Posted 8 Apr 2012 by In Memory of Kimsey M Fowler Sr
Post: This is an update on my troubleshooting activites since my last post five days ago. I've been experimenting under the assumption that there are no problems with Rosetta/BOINC software. My activies have assumed there is a driver or software configuration problem causing "client errors". I've learned from various websites that there can be a complex relationship between among the motherboard chipset drivers, the GPU drivers (a.k.a. the "display adapter drivers" in Windows 7) and the video monitor drivers. All drivers associated with putting something out there for your eyes to see must work together as a team. Another issue is that from a driver perspective, VGA drivers are more forgiving, and DVI drivers are less forgiving/interchangable; DVI monitor drivers are more specific. Here are some of the things I've tested: 1) Plugged the monitor into each of four different DVI ports. This may seem silly, but I read reports of the DVI connector closest to the motherboard being preferred. My GPU's don't have VGA ports, and I am using a VGA cable with adapter. (Are any of you using a real DVI cable???) 2) I installed the DVI driver from the CD included with the HP S2031 flat panel 20" monitor I bought last month at Fry's for only $89. Was HP dumping them at a low price for a reason I wondered? 3) Next I got rid of the flat panel, and replaced it with a 20 year old Dell VGA extracted from the attic. 4) I reinstalled the chipset, GPU, and monitor drivers in different orders. These test were time consuming in waiting for tasks to complete to determine results. None of the above solved the problem. The next idea was to determine if the problem could be pinned to hardware or software. As mentioned above, I have two nearly identical systems with the same motherboards, CPUs, & OS Win 7 64-bit. The main difference is that the good computer has two unbridged EVGA 560 Ti's, and the problem computer has two unbridged EVGA 580's. I wondered what would happen if a mirror image of the hard drive from the good computer was put in place into the problem computer. I tested this using the following procedure: 1) Used Win 7 OS to make an emergency boot recovery CD for the good machine. 2) used Win 7 OS to generate a disk image of the entire hard drive (SSD) from the good machine onto an external hard drive plugged in via USB. 3) Used that emergency boot disk to boot the problem machine. 4) Went to Windows recover in control panels of the problem machine and installed the image from the external hard drive. This process completely wiped all existing files from the problem machine's boot drive. 5) Disconnected the network cable from the wall so the problem computer could not communicated with BOINC, Rosetta, Folding@HOme, or Microsoft pirate hunter. 6) Using built-in Win 7 OS capabilities, I proceeded to change the computer name, the user name, and Win 7 product key to those of the prior installation of the problem computer. BOINC, Rosetta, and Folding@Home were uninstalled, and their data directories deleted. 7) The network cable was plugged in, the machine rebooted, and Win 7 was re-validated with Microsoft. 8) BOINC and Rosetta were reinstalled from a fresh copy. At this point my machine was configured for testing the old hardware with a known good software configuration. The driver for the EVGA GeForce 560's and 580's is the same. I looked under device manager to check the display adapter driver and found that just the one EVGA 580 that the monitor was attached to was present, but it was using the correct NVIDIA driver. Without fiddling with anything else, I immediately launched Rosetta. The WU's returned from this configuration were reported correctly. Sixteen Wu's ran successfully before anything was changed. This might seem like a conclusive test, but to be fair, I must point out that all of the new WU's used Rosetta 3.26, whereas both the good and problem machines had been running 3.24. This introduced an unwanted variable into the test. The next step was to get the second EVGA 580 to be recognized by the computer. I found many, many reports on the web of a second video card not being recognized, and many of those reports mentioned an ASUS motherboard, the same as my installation. Next the NVIDIA drivers were reinstalled using a freshly downloaded file from EVGA. Upon rebooting the machine, device manager reported that both GPU's were now recognized and installed. Subsequent WU's completed by Rosetta reported client errors.... ouch! "Device Manager" was used to uninstalled one GPU, and again WU's completed successfully. Some Notes of Interest: a) I found that reinstallation of the NVIDIA driver can knock out the HP S2031 flat panel monitor driver. b) If a WU begins on a machine that's in a good state, but finishes while the machine is in a bad state, the WU fails with client errors. If a WU's progress reaches 100% while the machine is in a bad state, the WU fails with client errors. If a WU has finished in a bad state, and is "Ready to Report" and the machine is changed into a good state before the report occurs, the WU is reported with client errors. Therefore, I would conclude that the WU only fails with client errors (the type addressed by this forum anyway) if the platform upon which it is running is in a bad state at the moment when it's state of progress reaches 100%, and that its success is independent of the good/bad state at any time other than the 100% mark. I have more tests in progress now, and will try to make another report tonight. In the mean time, can you guys with the problem please respond to these questions: A) Anyone using the HP S1931, 2031, 2231, 2331 monitor series? B) Anyone have their monitor plugged in with a true DVI cable without DVI-to-VGA adapter? C) Is everyone running with two GPU's installed? D) Has anyone else discovered that if the second GPU is "uninstalled" via Device Manager, the problem goes away??? E) Anyone using an SLI bridge between multiple GPU's? Keep looking for the golden Easter egg! |
|
16)
Message boards :
Number crunching :
Client Errors
(Message 72654)
Posted 3 Apr 2012 by In Memory of Kimsey M Fowler Sr
Post: Hi everyone... I don't have any great progress to report, but as we are all anxious for a way to resolve this, I want to give you an update. Rocco Moretti from Rosetta requested some sets of data from my machine collected both while running properly and while running with the client error condition. The data was sent near the end of last week, and this morning he reported that nothing has been found. Analysis is continuing. I'm trying real hard to come up with some ideas of what to do to try to get it running properly, test any theories, software/hardware configurations, etc. The problem boils down to whether or not the NVIDIA GPU driver is installed versus the Windows default driver. Understanding the problem is complicated by the fact that I have two machines running with the same versions of software on the same versions of motherboards and processors, but one machine has never experience the problem and the other has. Did this summary of the problem help you think of anything? It did for me, so now I have a few simple little experiments to try this evening. |
|
17)
Message boards :
Number crunching :
Client Errors
(Message 72587)
Posted 24 Mar 2012 by In Memory of Kimsey M Fowler Sr
Post: Troubleshooting Report: A recent version of the GPU driver was uninstalled and the machine reverted back to the Windows default driver. Rosetta once again started working correctly. Next, a much older July 2011 version of the NVIDIA driver was installed. Rosetta reported client errors again. One thing of note is that if the driver is switched while a Rosetta WU is paused/suspended, the final results (success or failure) is only influenced by the driver loaded at the time the WU completes and uploads. The current recommended version of BOINC is 6.12.34; it has been in place since July 2011. A glaring characteristic of our problem is that it was first reported three days after Rosetta 3.22 was released. With older tasks quickly falling off the bottom of contributer's task lists, it's a little late to do a detailed analysis of what happened when 3.22 replaced the previous version of Rosetta. Spring break is almost over at the University of Washington, so hopefully next week we can get some support. In the mean time, does anyone know if there is a way to set up a test running a pre-Rosetta 3.22 task? If the task will complete normally with an NVIDIA driver installed in the GPU, some change that went into Rosetta 3.22 would likely be the source of the problem. Markus Elfring's suggestion above is noteworthy, and I read his additional comments on the other thread. |
|
18)
Message boards :
Number crunching :
Client Errors
(Message 72583)
Posted 23 Mar 2012 by In Memory of Kimsey M Fowler Sr
Post:
Hi Jerome - I looked at one of your failed WU's, and it's a different problem than we are working on here. Up above in this same thread there is a mention of the same mechanism of failure that you are experiencing:
I see you are using BOINC 7 and have errno=13. You might want to go back to the previous version of BOINC. The latest is not necessarily the greatest! |
|
19)
Message boards :
Number crunching :
Client Errors
(Message 72576)
Posted 22 Mar 2012 by In Memory of Kimsey M Fowler Sr
Post: Troubleshooting Progress Report: At the end of the previous troubleshooting session I had shown that BOINC/Rosetta would work fine until the EVGA GPU driver was installed. Last night I continue testing by uninstalling the EVGA driver and going back to the Windows generic display adapter driver. Once again, with the Windows generic driver installed, BOINC/Rosetta began working correctly. I tested yet a second EVGA driver several hours later. Per suggestion I also installed all other useful applications from the EVGA disk along with all Window 7(64bit) updates. Again BOINC/Rosetta became unable to deliver valid tasks without client errors. Is it time to get the BOINC people involved? I'm fresh out of idea here. The "Help Wanted" sign is hanging in the window! |
|
20)
Message boards :
Number crunching :
Client Errors
(Message 72570)
Posted 21 Mar 2012 by In Memory of Kimsey M Fowler Sr
Post: To Rocco Moretti - can you tell by looking at the uploaded results of a failed task if the data computed/returned is good, but it's only failing because of some more trivial absence of information like the missing application version. Is the upload being prematurely terminated? What exactly is causing the task to be marked as invalid? Thanks. |
Next 20
©2024 University of Washington
https://www.bakerlab.org