Message boards : Number crunching : Problems and Technical Issues with Rosetta@home
Previous · 1 . . . 254 · 255 · 256 · 257 · 258 · 259 · 260 . . . 310 · Next
Author | Message |
---|---|
rakvium Send message Joined: 2 Apr 18 Posts: 4 Credit: 230,190 RAC: 13 |
Thank you for your help and suggestions! It seems like the version update was not crucial though, the problem was probably in /var/lib/boinc-client/ca-bundle.crt file. The CA bundle file was outdated before, now it seems to be a symbolic link to /etc/ssl/certs/ca-certificates.crt file, which was recently updated after re-installing BOINC 7.9.3 and libcurl4-openssl-dev package and uploads go well now. With care, Viktor |
rakvium Send message Joined: 2 Apr 18 Posts: 4 Credit: 230,190 RAC: 13 |
and uploads go well now. Well, downloads didn't. I have managed to update BOINC to 7.20.5 (that wasn't necessary in my case, as I have found later) with the next steps: # stopped the previous version: sudo service boinc-client stop # without update-ca-certificates there was the next error: # Cannot add PPA: 'ppa:~costamagnagianfranco/ubuntu/boinc'. # ERROR: '~costamagnagianfranco' user or team does not exist. sudo apt-get update && sudo apt-get install ca-certificates -y && sudo update-ca-certificates # added the repository with the latest versions sudo add-apt-repository ppa:costamagnagianfranco/boinc # acknowledged the updates and upgraded boinc sudo apt-get update && sudo apt-get upgrade boinc # started the new version: sudo service boinc-client start Nevertheless, the error persisted. It seemed a bit like of a curl problem, so I have decided to check it: $ curl https://boinc-files.bakerlab.org/rosetta/download/c1/KC_12mer_12_hallucinated_93_236.flags curl: (60) SSL certificate problem: unable to get local issuer certificate More details here: https://curl.haxx.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above. So the problem wasn't around the BOINC itself after all. I don't know why exactly my local curl decided not to trust Rosetta@Home files server's PEM certificates. Nevertheless, I saved those certificates to a separate crt file (/usr/local/share/ca-certificates/rosetta.crt) and run sudo update-ca-certificates again. After that downloads started work again for Rosetta@home project for my BOINC client. The problem seems to be solved for me for now. Leaving this here just in case someone stumbles on the same thing. With care, Viktor |
.clair. Send message Joined: 2 Jan 07 Posts: 274 Credit: 26,399,595 RAC: 0 |
Looking through the top hosts list it is interesting to see that some are still returning completed work even though they do have some "not started by deadline" "canceled by server" and "aborted" Others have work units "waiting validation" from October Old relics from "15 Apr 2020, 12:37:56 UTC Not started by deadline - canceled " as database dirt And the rest of us getting on with crunching other projects work I wonder if rosetta is going to run all work out and planning to do a big full project database reset to clean out the cruft ? hint . . . . . And why am I posting this ? Its just something to do on a day like today is , before I attack the accumulated pans and dishes in the kitchen , nnn ;-) |
Jean-David Beyer Send message Joined: 2 Nov 05 Posts: 197 Credit: 6,613,600 RAC: 4,116 |
So the problem wasn't around the BOINC itself after all. That was just the hint I needed. I ran the update-ca-trust command on my RHEL8.6 Linux system, and it seems to work (as far as I can tell). It does not download any Rosetta tasks at the moment, because there are none available. But at least, no error messages. |
rsNeutrino Send message Joined: 22 Mar 20 Posts: 13 Credit: 5,139,722 RAC: 4,579 |
I, too, was not able to download any WU files during the last batch around 15.-21.12.2022 BOINC version 7.20.2 on Ubuntu 22.04.1, fully updated. update-ca-certificates did not help. I did some analysis: I noticed two different URLs used by Rosetta: Root URL for general communication: https://boinc.bakerlab.org/ Old download URL, index with folder and file structure visible: https://boinc.bakerlab.org/rosetta/download/ It seems some time ago Rosetta switched to a new URL for downloads. New download URL, index hidden, shown as offline on the status page: https://boinc-files.bakerlab.org/ Both URLs seem to target the same underlying file system. As an examlple, the following two URLs lead to the same file: https://boinc.bakerlab.org/rosetta/download/0/3stub_cyc_target_1cwa_00081_2_extract_B.zip https://boinc-files.bakerlab.org/rosetta/download/0/3stub_cyc_target_1cwa_00081_2_extract_B.zip Tests on Debian with curl: old URL: curl https://boinc.bakerlab.org/rosetta/download/0/3stub_cyc_target_1cwa_00081_2_extract_B.zip -vI * Trying 128.95.160.156:443... * Connected to boinc.bakerlab.org (128.95.160.156) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * CAfile: /etc/ssl/certs/ca-certificates.crt * CApath: /etc/ssl/certs * TLSv1.0 (OUT), TLS header, Certificate Status (22): * TLSv1.3 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS header, Certificate Status (22): * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS header, Certificate Status (22): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS header, Certificate Status (22): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS header, Certificate Status (22): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS header, Certificate Status (22): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS header, Finished (20): * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.2 (OUT), TLS header, Certificate Status (22): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS header, Finished (20): * TLSv1.2 (IN), TLS header, Certificate Status (22): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384 * ALPN, server accepted to use http/1.1 * Server certificate: * subject: C=US; ST=Washington; O=University of Washington; CN=*.bakerlab.org * start date: Dec 14 00:00:00 2022 GMT * expire date: Dec 14 23:59:59 2023 GMT * subjectAltName: host "boinc.bakerlab.org" matched cert's "*.bakerlab.org" * issuer: C=US; ST=MI; L=Ann Arbor; O=Internet2; OU=InCommon; CN=InCommon RSA Server CA * SSL certificate verify ok. new URL: curl https://boinc-files.bakerlab.org/rosetta/download/0/3stub_cyc_target_1cwa_00081_2_extract_B.zip -vI * Trying 128.95.160.134:443... * Connected to boinc-files.bakerlab.org (128.95.160.134) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * CAfile: /etc/ssl/certs/ca-certificates.crt * CApath: /etc/ssl/certs * TLSv1.0 (OUT), TLS header, Certificate Status (22): * TLSv1.3 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS header, Certificate Status (22): * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS header, Certificate Status (22): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (OUT), TLS header, Unknown (21): * TLSv1.2 (OUT), TLS alert, unknown CA (560): * SSL certificate problem: unable to get local issuer certificate * Closing connection 0 curl: (60) SSL certificate problem: unable to get local issuer certificate More details here: https://curl.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above. I could recreate the same error on multiple independent Rasbpian/Debian installations with the same result. Firefox on Windows and Ubuntu did not complain, shows verified by Internet2. On Windows, Openssl fails on both the new and old URL, probably because it's out of date (2021). wget and curl in powershell did not complain. Edge on Windows downloads the zip without warning, but warns about a missing certificate when opening https://boinc-files.bakerlab.org and clicking on the lock symbol. Openssl tests on Debian, each with the same results: Raspbian (buster): OpenSSL 1.1.1n 15 Mar 2022 Ubuntu (22.04.1 LTS): OpenSSL 3.0.2 15 Mar 2022 new URL: openssl s_client -connect boinc-files.bakerlab.org:443 CONNECTED(00000003) depth=0 C = US, ST = Washington, O = University of Washington, CN = *.bakerlab.org verify error:num=20:unable to get local issuer certificate verify return:1 depth=0 C = US, ST = Washington, O = University of Washington, CN = *.bakerlab.org verify error:num=21:unable to verify the first certificate verify return:1 depth=0 C = US, ST = Washington, O = University of Washington, CN = *.bakerlab.org verify return:1 --- Certificate chain 0 s:C = US, ST = Washington, O = University of Washington, CN = *.bakerlab.org i:C = US, ST = MI, L = Ann Arbor, O = Internet2, OU = InCommon, CN = InCommon RSA Server CA a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256 v:NotBefore: Dec 14 00:00:00 2022 GMT; NotAfter: Dec 14 23:59:59 2023 GMT --- Server certificate -----BEGIN CERTIFICATE----- MIIGpDCCBYygAwIBAgIQLX34hscHGmFRGzZSrcfHojANBgkqhkiG9w0BAQsFADB2 ---snip--- Start Time: 1672334317 Timeout : 7200 (sec) Verify return code: 21 (unable to verify the first certificate) Extended master secret: yes old URL: openssl s_client -connect boinc.bakerlab.org:443 CONNECTED(00000003) depth=2 C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority verify return:1 depth=1 C = US, ST = MI, L = Ann Arbor, O = Internet2, OU = InCommon, CN = InCommon RSA Server CA verify return:1 depth=0 C = US, ST = Washington, O = University of Washington, CN = *.bakerlab.org verify return:1 --- Certificate chain 0 s:C = US, ST = Washington, O = University of Washington, CN = *.bakerlab.org i:C = US, ST = MI, L = Ann Arbor, O = Internet2, OU = InCommon, CN = InCommon RSA Server CA a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256 v:NotBefore: Dec 14 00:00:00 2022 GMT; NotAfter: Dec 14 23:59:59 2023 GMT 1 s:C = US, ST = MI, L = Ann Arbor, O = Internet2, OU = InCommon, CN = InCommon RSA Server CA i:C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA512 v:NotBefore: Sep 19 00:00:00 2014 GMT; NotAfter: Sep 18 23:59:59 2024 GMT 2 s:C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority i:C = GB, ST = Greater Manchester, L = Salford, O = Comodo CA Limited, CN = AAA Certificate Services a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA384 v:NotBefore: Mar 12 00:00:00 2019 GMT; NotAfter: Dec 31 23:59:59 2028 GMT --- Server certificate -----BEGIN CERTIFICATE----- MIIGpDCCBYygAwIBAgIQLX34hscHGmFRGzZSrcfHojANBgkqhkiG9w0BAQsFADB2 --snip-- Start Time: 1672334384 Timeout : 7200 (sec) Verify return code: 0 (ok) Extended master secret: no Here a comparison with an external 3rd party online analysis from SSLLABS: https://www.ssllabs.com/ssltest/analyze.html?d=boinc-files.bakerlab.org ("This server's certificate chain is incomplete. Grade capped to B.") https://www.ssllabs.com/ssltest/analyze.html?d=boinc.bakerlab.org (Rating: A) (Additional test: https://www.digicert.com/help/) As you can see, even they show issues with the certificate chain for the new URL. If I understand the results correctly, the old URL sends both the server certificate (1) and the intermediate certificate (2) (= issuer certificate) to the client (SSLLABS: Certification Paths: "1 - Sent by server, 2 - Sent by server"), completing the chain of trust with the root certificate (3) on the client. The new URL only sends the server certificate, but not the intermediate certificate (SSLLABS: "2 - Extra download", from InCommon RSA Server CA / Internet2). If the client does not already have this intermediate certificate in its trust store, which seems often but not always the case (comparing Firefox vs windows vs Debian), the chain is broken and any connection to boinc-files.bakerlab.org fails. Maybe there are also some automated tricks and workarounds going on, like caching the intermediate after once connecting to boinc.bakerlab.org, so that the client can puzzle the chain together anyway. As others already wrote, it is visible in Rosetta's statistics that something widespread isn't working. Comparing earlier batches with the last batch is noticeble slower in crunshing and returning WUs: https://munin.kiska.pw/munin/Munin-Node/Munin-Node/results_rosetta.html https://www.boincstats.com/stats/14/project/detail/credit So, I think what needs to be done is to recreate the configuration for / copy the intermediate certificate from boinc.bakerlab.org to boinc-files.bakerlab.org, so that it gets sent to clients as well. |
rsNeutrino Send message Joined: 22 Mar 20 Posts: 13 Credit: 5,139,722 RAC: 4,579 |
Screenshots for comparison of the chains, no differences between platforms: URL: boinc.bakerlab.org URL: boinc-files.bakerlab.org |
Jean-David Beyer Send message Joined: 2 Nov 05 Posts: 197 Credit: 6,613,600 RAC: 4,116 |
I ran the update-ca-trust command on my RHEL8.6 Linux system, and it seems to work (as far as I can tell). My system has now been updated to RHEL8.7. Having run update-ca-trust has not hurt anything, but it did not help either. Well, Rosetta still does not admit to having any work units on its web site, but it tried to download some to me today. I think it was only one. It created these files but you can see they are all empty. And they are all marked "downloading." -rw-r--r--. 1 boinc boinc 0 Jan 4 22:18 database_357d5d93529_n_methyl.zip -rw-r--r--. 1 boinc boinc 0 Jan 4 22:18 LiberationSans-Regular.ttf -rw-r--r--. 1 boinc boinc 0 Jan 4 08:27 rb_01_04_474411_469629_ab_t000__robetta.200.11mers.index.gz -rw-r--r--. 1 boinc boinc 0 Jan 4 16:29 rb_01_04_474411_469629_ab_t000__robetta.200.3mers.index.gz -rw-r--r--. 1 boinc boinc 0 Jan 4 08:27 rb_01_04_474411_469629_ab_t000__robetta.200.4mers.index.gz -rw-r--r--. 1 boinc boinc 0 Jan 4 08:27 rb_01_04_474411_469629_ab_t000__robetta.200.5mers.index.gz -rw-r--r--. 1 boinc boinc 0 Jan 4 16:13 rb_01_04_474411_469629_ab_t000__robetta.200.6mers.index.gz -rw-r--r--. 1 boinc boinc 0 Jan 4 08:27 rb_01_04_474411_469629_ab_t000__robetta.200.7mers.index.gz -rw-r--r--. 1 boinc boinc 0 Jan 4 08:27 rb_01_04_474411_469629_ab_t000__robetta.200.8mers.index.gz -rw-r--r--. 1 boinc boinc 0 Jan 4 08:27 rb_01_04_474411_469629_ab_t000__robetta.200.9mers.index.gz -rw-r--r--. 1 boinc boinc 0 Jan 4 18:49 rb_01_04_474411_469629_ab_t000__robetta_FLAGS -rw-r--r--. 1 boinc boinc 0 Jan 4 16:29 rb_01_04_474411_469629_ab_t000__robetta.zip -rw-r--r--. 1 boinc boinc 0 Jan 4 17:31 rosetta_4.20_x86_64-pc-linux-gnu -rw-r--r--. 1 boinc boinc 0 Jan 4 17:31 rosetta_graphics_4.20_x86_64-pc-linux-gnu |
Mr P Hucker Send message Joined: 12 Aug 06 Posts: 1600 Credit: 12,116,986 RAC: 6,010 |
My system has now been updated to RHEL8.7. Having run update-ca-trust has not hurt anything, but it did not help either.There are a very small number of Robetta tasks coming through to my Windows machines too. I think we're the overflow for Robetta's own machines. It seems everyone's asleep and not producing much work - even with WCG, Rosetta, and Asteroids, I sometimes run out of CPU work. Oh well, I'm off to fiddle with some GPUs, I've constructed an 800A 12V ring main on a large bookshelf to plug GPUs in anywhere. A few kilowatt 12V LED power supplies and a couple of reels of car starter motor cable (0 AWG). |
DJStarfox Send message Joined: 19 Jul 07 Posts: 145 Credit: 1,250,162 RAC: 0 |
To whom it may concern, The download server is broken per status page: Download server boinc-files.bakerlab.org Not Running Please fix. Thanks. |
.clair. Send message Joined: 2 Jan 07 Posts: 274 Credit: 26,399,595 RAC: 0 |
The servers over @ Ralph are all running , they could chuck all the work over there if they can`t or be botherd to fix this lot . |
[VENETO] boboviz Send message Joined: 1 Dec 05 Posts: 2002 Credit: 9,790,281 RAC: 4,437 |
The servers over @ Ralph are all running , they could chuck all the work over there if they can`t or be botherd to fix this lot . Ralph is undervalued and underused by the project itself.... |
TankbusterGames Send message Joined: 26 Apr 21 Posts: 1 Credit: 609,542 RAC: 0 |
Wow, this hasn't been fixxed in weeks? I stopped Rosetta when it I couldn't download any tasks couple weeks back, today I restarted and I still can not download 99% of BOINC issues lately have been to SSL certificates. SiDock, World Community Grid, Cosmology, Rosetta, probably some other projects I forgot about.... It's a really bad trend to see honestly. We need to figure out a solution so not every year a different project goes down for some months before the admin can replace some weird certificate or whatever. |
DJStarfox Send message Joined: 19 Jul 07 Posts: 145 Credit: 1,250,162 RAC: 0 |
Let'sEncrypt has been making free and (relatively) easy to acquire SSL certificates available for years now. There's no excuse anymore for not having a valid, publicly trusted SSL certificate. |
Mr P Hucker Send message Joined: 12 Aug 06 Posts: 1600 Credit: 12,116,986 RAC: 6,010 |
Let'sEncrypt has been making free and (relatively) easy to acquire SSL certificates available for years now. There's no excuse anymore for not having a valid, publicly trusted SSL certificate.We all managed just fine before this SSL shit. Best to ditch it altogether. |
Mario W. Send message Joined: 15 Aug 07 Posts: 15 Credit: 290,903 RAC: 0 |
Do you Guys having any Information about getting new Tasks for the Rosetta@Home Calculation? Server Status says "Download Server not started" but i dont know. I dont receive any new Tasks sniff |
Mr P Hucker Send message Joined: 12 Aug 06 Posts: 1600 Credit: 12,116,986 RAC: 6,010 |
Do you Guys having any Information about getting new Tasks for the Rosetta@Home Calculation? Server Status says "Download Server not started" but i dont know. I dont receive any new Tasks sniffI get a few every other day. Just checked now and got: 20604 Rosetta@home Tue, 10/1/2023 11:48:15 AM update requested by user 20605 Rosetta@home Tue, 10/1/2023 11:48:15 AM Sending scheduler request: Requested by user. 20606 Rosetta@home Tue, 10/1/2023 11:48:15 AM Requesting new tasks for CPU 20607 Rosetta@home Tue, 10/1/2023 11:48:18 AM Scheduler request completed: got 0 new tasks 20608 Rosetta@home Tue, 10/1/2023 11:48:18 AM No tasks sent 20609 Rosetta@home Tue, 10/1/2023 11:48:18 AM Project requested delay of 31 secondsSo not broken, just empty. I noticed that on the server status too, a week ago, yet I've had plenty tasks since then. I guess either it's only being switched on when there's work, or it's unreliable, or the status page is wrong. |
Mario W. Send message Joined: 15 Aug 07 Posts: 15 Credit: 290,903 RAC: 0 |
Ah, well just as i posted that. A few Minutes later i received 8 new Tasks. Well, thanks for the Headup anyways :) |
srettie Volunteer moderator Project developer Project scientist Send message Joined: 10 Jul 17 Posts: 9 Credit: 50,961 RAC: 0 |
Hello, I have passed this on to the people in charge. Thank you everyone for posting the issue here. |
.clair. Send message Joined: 2 Jan 07 Posts: 274 Credit: 26,399,595 RAC: 0 |
Hello, I have passed this on to the people in charge. Thank you everyone for posting the issue here. Thank you , things got funky/broke three weeks ago . |
.clair. Send message Joined: 2 Jan 07 Posts: 274 Credit: 26,399,595 RAC: 0 |
I noticed that on the server status too, a week ago, yet I've had plenty tasks since then. I guess either it's only being switched on when there's work, or it's unreliable, or the status page is wrong. Is there any definable pattern as to which systems get work and not , such as Linux or mycrowsoft , operating system version etc ? |
Message boards :
Number crunching :
Problems and Technical Issues with Rosetta@home
©2024 University of Washington
https://www.bakerlab.org