Problems and Technical Issues with Rosetta@home

Message boards : Number crunching : Problems and Technical Issues with Rosetta@home

To post messages, you must log in.

Previous · 1 . . . 90 · 91 · 92 · 93 · 94 · 95 · 96 . . . 264 · Next

AuthorMessage
Kissagogo27

Send message
Joined: 31 Mar 20
Posts: 75
Credit: 1,957,821
RAC: 2,559
Message 101054 - Posted: 4 Apr 2021, 10:06:32 UTC

weird thing, just got a resend but not sure to finish it !

https://boinc.bakerlab.org/rosetta/workunit.php?wuid=1217325166
ID: 101054 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Brian Nixon

Send message
Joined: 12 Apr 20
Posts: 293
Credit: 8,432,366
RAC: 0
Message 101055 - Posted: 4 Apr 2021, 10:30:05 UTC - in response to Message 101054.  
Last modified: 4 Apr 2021, 10:36:08 UTC

This issue was discussed recently in another thread.

The work unit got resent because the first machine hadn’t completed it by its deadline. But 10 minutes later – after you’d started the resend but before you’d finished it – the other host submitted its results. I think you’ll still get credit if you complete it before the deadline, but from the science perspective there’s no point because the results are already in.
ID: 101055 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Kissagogo27

Send message
Joined: 31 Mar 20
Posts: 75
Credit: 1,957,821
RAC: 2,559
Message 101057 - Posted: 4 Apr 2021, 13:38:15 UTC
Last modified: 4 Apr 2021, 13:41:09 UTC

with a 10GB boinc disk space setting, Boinc still send weird messages like this one


04-Apr-2021 15:13:03 [Rosetta@home] Rosetta needs 6675.72 MB RAM but only 4060.49 MB is available for use.
04-Apr-2021 15:21:01 [Rosetta@home] Rosetta needs 252.21MB more disk space. You currently have 8330.86 MB available and it needs 8583.07 MB.


and nothing was downloaded ...

i have to reset the project and this message is gone away, and now Boinc is downloading tasks again . . .

even the first message of lack of memory gone too for this moment ..
ID: 101057 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Kissagogo27

Send message
Joined: 31 Mar 20
Posts: 75
Credit: 1,957,821
RAC: 2,559
Message 101058 - Posted: 4 Apr 2021, 15:52:39 UTC

Haha , they're back
.

04-Apr-2021 15:57:48 [Rosetta@home] Sending scheduler request: To fetch work.
04-Apr-2021 15:57:48 [Rosetta@home] Requesting new tasks for CPU
04-Apr-2021 15:57:51 [Rosetta@home] Scheduler request completed: got 0 new tasks
04-Apr-2021 15:57:51 [Rosetta@home] No tasks sent
04-Apr-2021 15:57:51 [Rosetta@home] Rosetta needs 6675.72 MB RAM but only 4060.49 MB is available for use.
04-Apr-2021 15:57:51 [Rosetta@home] Rosetta needs 134.84MB more disk space. You currently have 8448.22 MB available and it needs 8583.07 MB.
04-Apr-2021 15:57:51 [Rosetta@home] Project requested delay of 31 seconds
04-Apr-2021 15:57:56 [Rosetta@home] General prefs: from Rosetta@home (last modified 04-Apr-2021 15:30:57)
04-Apr-2021 15:57:56 [Rosetta@home] Computer location: home
04-Apr-2021 15:57:56 [---] General prefs: using separate prefs for home
04-Apr-2021 15:57:56 [---] Preferences:
04-Apr-2021 15:57:56 [---] max memory usage when active: 4060.49 MB
04-Apr-2021 15:57:56 [---] max memory usage when idle: 4060.49 MB
04-Apr-2021 15:57:56 [---] max disk usage: 12.00 GB
04-Apr-2021 15:57:56 [---] (to change preferences, visit a project web site or select Preferences in the Manager)
ID: 101058 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Brian Nixon

Send message
Joined: 12 Apr 20
Posts: 293
Credit: 8,432,366
RAC: 0
Message 101063 - Posted: 4 Apr 2021, 19:13:29 UTC - in response to Message 101057.  

This issue was discussed recently in another thread.

As well as claiming they needed 6.6 GB of RAM, the recent work units were configured to require 8.5 GB of disk space. With a preference setting maximum disk usage to 10 GB, and more than 1.5 GB already in use (around 2 GB is normal for R@h), the server was unable to send those tasks and so issued that warning.

Resetting the project didn’t make any difference because the disk space that freed up was immediately consumed again by the smaller tasks you were able to download.
ID: 101063 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Grant (SSSF)

Send message
Joined: 28 Mar 20
Posts: 1400
Credit: 13,693,695
RAC: 0
Message 101065 - Posted: 4 Apr 2021, 22:04:27 UTC - in response to Message 101063.  
Last modified: 4 Apr 2021, 22:08:51 UTC

Resetting the project didn’t make any difference because the disk space that freed up was immediately consumed again by the smaller tasks you were able to download.
Along with the executables & support data files.
Over time as different Tasks are downloaded, those support data files will be re-downloaded & the lack of disk space issue will re-occur (if the configuration issue for certain Work Units hasn't been fixed by then), as you soon found out.


Since the project is out of work again, other than the odd resend, now no one will be able to get any new work.
Hopefully the next batches of work system requirements will be configured more appropriately.
Grant
Darwin NT
ID: 101065 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Grant (SSSF)

Send message
Joined: 28 Mar 20
Posts: 1400
Credit: 13,693,695
RAC: 0
Message 101067 - Posted: 5 Apr 2021, 0:56:12 UTC

A new batch of work has been loaded up- hopefully these have their requirements set properly, and they won't error out in a matter of seconds either.
Grant
Darwin NT
ID: 101067 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
rensie

Send message
Joined: 22 Jan 06
Posts: 3
Credit: 1,480,056
RAC: 0
Message 101068 - Posted: 5 Apr 2021, 4:26:44 UTC

Just to follow up, my primary rosetta cruncher has 6 tasks using 1.6gb of ram total. Looks like they got it figured out.
ID: 101068 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile robertmiles

Send message
Joined: 16 Jun 08
Posts: 1218
Credit: 13,490,219
RAC: 4,500
Message 101069 - Posted: 5 Apr 2021, 5:04:00 UTC - in response to Message 101067.  

A new batch of work has been loaded up- hopefully these have their requirements set properly, and they won't error out in a matter of seconds either.

I got 11 of them. 3 gave errors within seconds, with error messages that probably require access to the source code to interpret. 6 are running normally. The last two haven't started.

At least that is better than mostly fast errors.
ID: 101069 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sid Celery

Send message
Joined: 11 Feb 08
Posts: 1923
Credit: 36,202,734
RAC: 23,753
Message 101071 - Posted: 5 Apr 2021, 13:00:24 UTC - in response to Message 101047.  
Last modified: 5 Apr 2021, 13:05:41 UTC

Returning to my anecdote about a remote PC I have being unable to download any Rosetta tasks, so running its backup project, WCG, 24/7, my local laptop is also doing weird things. It refuses to run a particular Rosetta task, so it's running those it has room for - a combination of WCG and later Rosetta tasks, but only 3 on 4 cores. Now I know it's definitely happening, I've set NNT and suspended all running tasks except for the one problem Rosetta task. It still refuses to run, even as the only task. No tasks are running in my experiment!

So, maintaining NNT, I've found some combination of WCG and Rosetta tasks that'll run together on all 4 cores. I'll work my way through my small cache until all are completed bar the problem task and see if it runs then. If not, I'll finally abort it and just grab fresh tasks.

Bit of a weird one. Even attempting to micromanage tasks doesn't entirely work. No wonder that graph is running so much lower than it was, if I'm any example

Finally got to the end of this.
Last night I had 3 WCG tasks running (2 of which were Africa Rainfall project that use slightly more RAM, but in fact were only using 300Mb each) and my one weird Rosetta nip* task reporting "waiting for memory" on my 4-core laptop.
Looking at my Event log, it was only when the last ARP was wrapping up that sufficient RAM was available for the Rosetta task to begin running. The last ARP task completed 3 minutes later and now the Rosetta task is the only task running.
Looking at the task's properties, it's only using between 271Mb & 292Mb RAM, while earlier complaining that it needed something like 6.6Gb RAM to begin.
I'm going to wait for completion before dragging any more tasks down. Hopefully there are some new tasks available to download at that time.

The task in question is this one
ID: 101071 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
yo2020

Send message
Joined: 2 Jan 21
Posts: 2
Credit: 111,170
RAC: 0
Message 101072 - Posted: 5 Apr 2021, 14:55:54 UTC

It seems the RAM problem is not yet solved. I just updated Rosetta now and got the same complaint about needing 6 GB of RAM and only having 3 GB.
ID: 101072 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sid Celery

Send message
Joined: 11 Feb 08
Posts: 1923
Credit: 36,202,734
RAC: 23,753
Message 101073 - Posted: 5 Apr 2021, 14:59:36 UTC - in response to Message 101071.  

Returning to my anecdote about a remote PC I have being unable to download any Rosetta tasks, so running its backup project, WCG, 24/7, my local laptop is also doing weird things. It refuses to run a particular Rosetta task, so it's running those it has room for - a combination of WCG and later Rosetta tasks, but only 3 on 4 cores. Now I know it's definitely happening, I've set NNT and suspended all running tasks except for the one problem Rosetta task. It still refuses to run, even as the only task. No tasks are running in my experiment!

So, maintaining NNT, I've found some combination of WCG and Rosetta tasks that'll run together on all 4 cores. I'll work my way through my small cache until all are completed bar the problem task and see if it runs then. If not, I'll finally abort it and just grab fresh tasks.

Bit of a weird one. Even attempting to micromanage tasks doesn't entirely work. No wonder that graph is running so much lower than it was, if I'm any example

Finally got to the end of this.
Last night I had 3 WCG tasks running (2 of which were Africa Rainfall project that use slightly more RAM, but in fact were only using 300Mb each) and my one weird Rosetta nip* task reporting "waiting for memory" on my 4-core laptop.
Looking at my Event log, it was only when the last ARP was wrapping up that sufficient RAM was available for the Rosetta task to begin running. The last ARP task completed 3 minutes later and now the Rosetta task is the only task running.
Looking at the task's properties, it's only using between 271Mb & 292Mb RAM, while earlier complaining that it needed something like 6.6Gb RAM to begin.
I'm going to wait for completion before dragging any more tasks down. Hopefully there are some new tasks available to download at that time.

The task in question is this one

With 30 minutes to go, I've allowed new tasks and 11have come down. Stage one successful.
3 of the new tasks attempt to start. Stage two successful.
2 of the new tasks are waiting for memory... Oh

I'm going out for a while. When I return the older task will have completed and I'll see if the new tasks all run ok.
The journey continues...
ID: 101073 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sid Celery

Send message
Joined: 11 Feb 08
Posts: 1923
Credit: 36,202,734
RAC: 23,753
Message 101076 - Posted: 5 Apr 2021, 16:12:10 UTC - in response to Message 101073.  

Returning to my anecdote about a remote PC I have being unable to download any Rosetta tasks, so running its backup project, WCG, 24/7, my local laptop is also doing weird things. It refuses to run a particular Rosetta task, so it's running those it has room for - a combination of WCG and later Rosetta tasks, but only 3 on 4 cores. Now I know it's definitely happening, I've set NNT and suspended all running tasks except for the one problem Rosetta task. It still refuses to run, even as the only task. No tasks are running in my experiment!

So, maintaining NNT, I've found some combination of WCG and Rosetta tasks that'll run together on all 4 cores. I'll work my way through my small cache until all are completed bar the problem task and see if it runs then. If not, I'll finally abort it and just grab fresh tasks.

Bit of a weird one. Even attempting to micromanage tasks doesn't entirely work. No wonder that graph is running so much lower than it was, if I'm any example

Finally got to the end of this.
Last night I had 3 WCG tasks running (2 of which were Africa Rainfall project that use slightly more RAM, but in fact were only using 300Mb each) and my one weird Rosetta nip* task reporting "waiting for memory" on my 4-core laptop.
Looking at my Event log, it was only when the last ARP was wrapping up that sufficient RAM was available for the Rosetta task to begin running. The last ARP task completed 3 minutes later and now the Rosetta task is the only task running.
Looking at the task's properties, it's only using between 271Mb & 292Mb RAM, while earlier complaining that it needed something like 6.6Gb RAM to begin.
I'm going to wait for completion before dragging any more tasks down. Hopefully there are some new tasks available to download at that time.

The task in question is this one

With 30 minutes to go, I've allowed new tasks and 11have come down. Stage one successful.
3 of the new tasks attempt to start. Stage two successful.
2 of the new tasks are waiting for memory... Oh

I'm going out for a while. When I return the older task will have completed and I'll see if the new tasks all run ok.
The journey continues...

And the answer is... no.
Still two new Rosetta tasks running and two more new ones waiting for memory...

It seems we're a way from having a solution or correction
ID: 101076 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Mr P Hucker
Avatar

Send message
Joined: 12 Aug 06
Posts: 1527
Credit: 8,227,427
RAC: 22,985
Message 101077 - Posted: 5 Apr 2021, 17:23:53 UTC - in response to Message 101072.  

It seems the RAM problem is not yet solved. I just updated Rosetta now and got the same complaint about needing 6 GB of RAM and only having 3 GB.
It's not a problem, it's just some tasks needing more RAM. The bigger machines will take those, then you'll get smaller ones. I'm currently running 250-450MB ones, despite having machines with up to 64GB. The only thing they could do is put them in a seperate queue, so you can get smaller tasks for your smaller machine, without having to wait for the big ones to get out of the way. Maybe they didn't realise there were that many people with old machines.
ID: 101077 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
strombergFs

Send message
Joined: 18 Mar 21
Posts: 11
Credit: 150,490
RAC: 0
Message 101079 - Posted: 5 Apr 2021, 19:43:54 UTC

i also got some problems again with the 6gb ram minimum.
After pressing several times Update i got again tasks for my C4s.
Now 5 C4s are running. :-)

The question i have is how often does Rosetta updates automatically and requests new tasks?
If I would not have pressed update, i would have waited however time to get again tasks.

Is there a possibility to tell Rosetta to update every 10mins or similar?

My feeling is that managing Rosetta to use the computer close to 100% is a bit luck.
Its a bit sad because i really want to help and i bought the C4s to run efficiently 100% around the clock.
I do hope that in the near future there are again for many weeks and month enough tasks for <<1GB ram per task so that all the small computers can support Rosetta with their maximum performance.
ID: 101079 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sid Celery

Send message
Joined: 11 Feb 08
Posts: 1923
Credit: 36,202,734
RAC: 23,753
Message 101080 - Posted: 5 Apr 2021, 21:11:05 UTC - in response to Message 101077.  

It seems the RAM problem is not yet solved. I just updated Rosetta now and got the same complaint about needing 6 GB of RAM and only having 3 GB.
It's not a problem, it's just some tasks needing more RAM. The bigger machines will take those, then you'll get smaller ones. I'm currently running 250-450MB ones, despite having machines with up to 64GB. The only thing they could do is put them in a separate queue, so you can get smaller tasks for your smaller machine, without having to wait for the big ones to get out of the way. Maybe they didn't realise there were that many people with old machines.

I wish that were true, but it isn't.
I have a 4-core i3-8350K with 16Gb RAM and lots of disk space but it hasn't been able to download or run any Rosetta tasks - only WCG backup project tasks, of which it now has a dozen.
While my main desktop 16-core 32Gb RAM hasn't had any problem.
If things work, it's like no problem exists. If they don't, it's like nothing will fix it.
ID: 101080 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile robertmiles

Send message
Joined: 16 Jun 08
Posts: 1218
Credit: 13,490,219
RAC: 4,500
Message 101082 - Posted: 5 Apr 2021, 21:22:44 UTC - in response to Message 101080.  

[snip]

I wish that were true, but it isn't.
I have a 4-core i3-8350K with 16Gb RAM and lots of disk space but it hasn't been able to download or run any Rosetta tasks - only WCG backup project tasks, of which it now has a dozen.
While my main desktop 16-core 32Gb RAM hasn't had any problem.
If things work, it's like no problem exists. If they don't, it's like nothing will fix it.

Have you checked how much of the RAM is reserved for the operating system (Windows, Linux. etc.)?

Have you checked your settings for have much RAM and how much disk space BOINC is allowed to use?
ID: 101082 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
mrhastyrib

Send message
Joined: 18 Feb 21
Posts: 90
Credit: 2,528,935
RAC: 38
Message 101083 - Posted: 5 Apr 2021, 21:33:36 UTC - in response to Message 101068.  

Just to follow up, my primary rosetta cruncher has 6 tasks using 1.6gb of ram total. Looks like they got it figured out.


Can I ask a dumb question? Why is it that we have to divine what "they" are doing by observing the vagaries of tasks and RAM? Isn't there a board or site somewhere that explains what "they" are doing when changes occur in the project that affect the donors in significant ways?
ID: 101083 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Brian Nixon

Send message
Joined: 12 Apr 20
Posts: 293
Credit: 8,432,366
RAC: 0
Message 101084 - Posted: 5 Apr 2021, 22:21:44 UTC - in response to Message 101079.  

The question i have is how often does Rosetta updates automatically and requests new tasks?
It’s not a fixed interval. While the client is busy with work, it will not contact the server until it needs more work or has results to report. When it asks for work and does not receive any, the client will back off for ever-increasing lengths of time – up to 1½ days – before asking again. This is to prevent hundreds of thousands of hungry hosts hammering a server that has no work to send out.


Is there a possibility to tell Rosetta to update every 10mins or similar?
Anything you can do in the GUI can be scripted, so the short answer is yes – but it will needlessly burden the server, and should not be necessary.


My feeling is that managing Rosetta to use the computer close to 100% is a bit luck.
At the moment there is an element of luck in whether you get work for Rosetta, because overall there is only a very small amount of work that needs doing, and because some of the work units can only be sent to machines with large amounts of RAM and disk space. If those happen to be all that is available at the moment a smaller machine contacts the server, that machine will get nothing, and back off. You can leave machines running unattended 24×7; many people here have been doing it successfully for years. But it is not reasonable to expect any single project to have 100% uptime, or to need work doing 100% of the time. That is why it is recommended to attach to more than one project.
ID: 101084 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jim1348

Send message
Joined: 19 Jan 06
Posts: 881
Credit: 52,257,545
RAC: 0
Message 101085 - Posted: 5 Apr 2021, 22:32:46 UTC - in response to Message 101083.  

Can I ask a dumb question? Why is it that we have to divine what "they" are doing by observing the vagaries of tasks and RAM? Isn't there a board or site somewhere that explains what "they" are doing when changes occur in the project that affect the donors in significant ways?
(1) You just plan for the worst case if you want to run all the work units. 1.5 GB/work unit should be enough memory. Disk space has never been a problem for me, and I don't know what the minimum is.
(2) This is not a project for laptops, though people may run them. It needs several hours of run time per day, preferably 24/7.
(3) "They" barely acknowledge your existence at all, except some thanks when a project is finished. If that is not your thing, there are other projects.
ID: 101085 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Previous · 1 . . . 90 · 91 · 92 · 93 · 94 · 95 · 96 . . . 264 · Next

Message boards : Number crunching : Problems and Technical Issues with Rosetta@home



©2023 University of Washington
https://www.bakerlab.org