minirosetta_database directory in slot

Message boards : Number crunching : minirosetta_database directory in slot

To post messages, you must log in.

Previous · 1 · 2 · 3 · Next

AuthorMessage
Mod.Sense
Volunteer moderator

Send message
Joined: 22 Aug 06
Posts: 4018
Credit: 0
RAC: 0
Message 95259 - Posted: 23 Apr 2020, 23:49:52 UTC

So, the expanded .zip file would become locally generated file structure.

Locally-generated input files

One (hypothetical) class of files: input files which, if not present, are generated computationally by the app. Such files should be listed (in sched reply) as sticky optional input files with no download URL, and as optional output files (this causes them to be marked as present).

The app must use file locking to ensure that two jobs don't try to generate the file at the same time.

Rosetta Moderator: Mod.Sense
ID: 95259 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sid Celery

Send message
Joined: 11 Feb 08
Posts: 1965
Credit: 38,175,715
RAC: 10,000
Message 95260 - Posted: 23 Apr 2020, 23:50:37 UTC - in response to Message 95233.  

You might also like to have a look at the suggestion that I made on RALPH nearly 12 years ago

Ouch! Lol
ID: 95260 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Admin
Project administrator

Send message
Joined: 1 Jul 05
Posts: 4805
Credit: 0
RAC: 0
Message 95261 - Posted: 24 Apr 2020, 0:12:44 UTC

I of course did look at the suggestion. Like I said, I was not sure if a directory can be placed in the project folder. I assumed it cannot but will see since it appears from the feedback here that it may be possible but I'm still not completely sure.
ID: 95261 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Mod.Sense
Volunteer moderator

Send message
Joined: 22 Aug 06
Posts: 4018
Credit: 0
RAC: 0
Message 95262 - Posted: 24 Apr 2020, 0:20:47 UTC

Explicit client file management

The standard mode of file access on a BOINC client is:

Physical files exist in the projects/URL directory. BOINC keeps track of all physical files, and deletes them when no longer needed.
Logical files are links in the slots/N directory, and refer to physical files. Applications are free to create other files in their slot directory; BOINC doesn't know about these, other than to delete them when the app finishes.

It's also possible for applications to create and access files in the project/URL directory. BOINC doesn't know about these files, and in particular it won't delete them until the user detaches your project. Some cases where you might want to do this:

Your application uses a large number of files, and you supply these as a single archive that is unpacked by your application (for example, using the boinc_zip library). If you store them in your project directory, you'll only need to do this unpacking once.
Your application generates a lot of files that are shared between workunits.

In order to put files in the project directory, you need to know where it is. This can be found by calling boinc_get_init_data(); the directory is in APP_INIT_DATA.project_dir.




GetFileList
To instruct a host to send a list of all persistent files, use the function

request_file_list(int host_id)

or the command line program (run from the project root directory)

request_file_list [ -host_id X ]

If -host_id is absent, get file lists for all hosts. A message is created for the specific host (or all hosts) and added to the msg_to_host table in the database. The upload message included in the next RPC reply to the host.

The file list will be included in the next scheduler RPC request. You must modify the scheduler to parse and store it.



Server config.xml must include:
<msg_to_host/>
If present, check the msg_to_host table on each RPC, and send the client any messages queued for it.
Rosetta Moderator: Mod.Sense
ID: 95262 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Admin
Project administrator

Send message
Joined: 1 Jul 05
Posts: 4805
Credit: 0
RAC: 0
Message 95266 - Posted: 24 Apr 2020, 2:06:12 UTC

Thanks Mod-Sense!
ID: 95266 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Mod.Sense
Volunteer moderator

Send message
Joined: 22 Aug 06
Posts: 4018
Credit: 0
RAC: 0
Message 95268 - Posted: 24 Apr 2020, 4:02:56 UTC
Last modified: 24 Apr 2020, 4:03:33 UTC

Ut, I found the last piece:

Deleting sticky files from clients


Sticky files can be deleted in either of two ways.
Regular expression matching

You can create a file file_delete_regex in your project directory, containing a list of regular expressions, one per line. If a client has a sticky file whose name matches any of these expressions, it will be instructed to delete the file.

Explicit deletion

To delete a sticky file from a particular host, use the function

delete_file(int host_id, const char* file_name)


or the command line program (run from the project root dir)

delete_file -host_id X -file_name Y


delete_file() creates a message for the specific host and adds it to the msg_to_host table. This message instructs the client to delete the file. The message is included in the next scheduler reply to the host. The message XML has the form

<delete_file_info>file_name</delete_file_info>


For this to work, you need to include <msg_to_host/> in config.xml.

Rosetta Moderator: Mod.Sense
ID: 95268 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
funkydude

Send message
Joined: 15 Jun 08
Posts: 28
Credit: 397,934
RAC: 0
Message 95274 - Posted: 24 Apr 2020, 5:17:20 UTC - in response to Message 95245.  

Completely agree. If there's a way, we can do it. I can look into this and test on our ralph project. But this will take some time since it will require an application update.


Happy to see this response from you. I'll use this opportunity to ask for something else: Remove the IP address blocking from registration for ralph (and rosetta).

Please understand that this is an archaic and flawed way of preventing spam. There is nothing unique or static about an IP address. An increasing amount of ISPs are now implementing CGNAT (Carrier Grade NAT) due to IPv4 shortage, which you will be familiar with since you're a university: Many users under a same IP address.

There are multiple ways of preventing spam by not using IP blocking.
SETI chooses to use a Google Captcha (https://setiathome.berkeley.edu/signup.php)
GPUGRID chooses to force you to register via the BOINC client (https://www.gpugrid.net/create_account_form.php)
I believe Cloudflare even offers services for free which include filtering methods.

I do not know what version of the BOINC web software you are using, perhaps the captcha is an in built optional feature now?

If you can do this, I'd be happy to sign up to Ralph and help you test shared data (if you implement it)
If you don't do this, please, at least improve the error message to stop misleading people with the "incorrectly formatted email address" error.

Thanks.
ID: 95274 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Admin
Project administrator

Send message
Joined: 1 Jul 05
Posts: 4805
Credit: 0
RAC: 0
Message 95290 - Posted: 24 Apr 2020, 9:43:17 UTC

I think I have a working app version now. I'll probably push it out for testing on ralph soon. As for the captcha, I can add it to the registration / email forms to replace the internal spam filtering when I get a chance but that's definitely low priority.
ID: 95290 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sid Celery

Send message
Joined: 11 Feb 08
Posts: 1965
Credit: 38,175,715
RAC: 10,000
Message 95304 - Posted: 24 Apr 2020, 13:21:54 UTC - in response to Message 95261.  

I of course did look at the suggestion

I'm sure, but you have to admit it was funny.

From another perspective, way to hold a grudge! Lol
ID: 95304 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Raistmer

Send message
Joined: 7 Apr 20
Posts: 49
Credit: 794,064
RAC: 3
Message 95309 - Posted: 24 Apr 2020, 14:30:36 UTC - in response to Message 95290.  

I think I have a working app version now. I'll probably push it out for testing on ralph soon. As for the captcha, I can add it to the registration / email forms to replace the internal spam filtering when I get a chance but that's definitely low priority.

Great! Joining to Ralph then and awaiting app to test.
ID: 95309 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Admin
Project administrator

Send message
Joined: 1 Jul 05
Posts: 4805
Credit: 0
RAC: 0
Message 95316 - Posted: 24 Apr 2020, 18:17:11 UTC - in response to Message 95304.  

Can I give you and Keith "Bad Mutha" badges? :)

But seriously, if anyone provides specifics about what can be implemented, how, and why (for what benefit to the project and volunteers), AND I am aware of such feedback (place a huge thanks to Mod.Sense here), it's extremely helpful and more likely to be addressed.
ID: 95316 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sid Celery

Send message
Joined: 11 Feb 08
Posts: 1965
Credit: 38,175,715
RAC: 10,000
Message 95322 - Posted: 24 Apr 2020, 21:18:37 UTC - in response to Message 95316.  

Can I give you and Keith "Bad Mutha" badges? :)

It's apparent to me you guys are busting a gut every day - more than we realise, as bcov's post indicated - so I'm glad you have the time to pick up the quality material to lighten the load occasionally.

It seems every forum post has its day eventually.
Like a 5 year old one about 48hr tasks sending so much data back the task errors out. And then the same happening with new 36hr tasks too.
I notice these things - and I did say in advance...
ID: 95322 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Raistmer

Send message
Joined: 7 Apr 20
Posts: 49
Credit: 794,064
RAC: 3
Message 95470 - Posted: 28 Apr 2020, 10:00:25 UTC - in response to Message 95290.  

I think I have a working app version now. I'll probably push it out for testing on ralph soon.


Any ETA for that? And will it be announced here or on Ralph boards?
ID: 95470 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Mod.Sense
Volunteer moderator

Send message
Joined: 22 Aug 06
Posts: 4018
Credit: 0
RAC: 0
Message 95503 - Posted: 28 Apr 2020, 21:35:03 UTC - in response to Message 95470.  

Well, it would be a new version on Ralph (4.17 announcement today ), and then later, a new version on Rosetta@home.
Rosetta Moderator: Mod.Sense
ID: 95503 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Admin
Project administrator

Send message
Joined: 1 Jul 05
Posts: 4805
Credit: 0
RAC: 0
Message 95506 - Posted: 28 Apr 2020, 22:00:16 UTC

Still working out some kinks. I also included more frequent checkpointing for the comparative modeling protocol which lacked checkpoints within a model trajectory. If all goes well after another app updated and testing phase on Ralph, then I'll phase in the new app version on R@h, one or two platforms per day to prevent excessive network IO.
ID: 95506 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Grant (SSSF)

Send message
Joined: 28 Mar 20
Posts: 1467
Credit: 14,327,617
RAC: 16,631
Message 95700 - Posted: 1 May 2020, 23:29:00 UTC

I'm pretty sure it was announced that Rosetta Mini was being depreciated, but now there is a whole bunch of new Rosetta Mini work out there. I was wondering if the Rosetta Mini application will now be updated to reduce the disk usage as well?
Grant
Darwin NT
ID: 95700 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Raistmer

Send message
Joined: 7 Apr 20
Posts: 49
Credit: 794,064
RAC: 3
Message 95704 - Posted: 1 May 2020, 23:44:35 UTC - in response to Message 95700.  

I'm pretty sure it was announced that Rosetta Mini was being depreciated, but now there is a whole bunch of new Rosetta Mini work out there. I was wondering if the Rosetta Mini application will now be updated to reduce the disk usage as well?

BTW, what the difference between Rosetta and Rosetta Mini ? What part is "mini" ?
ID: 95704 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
CIA

Send message
Joined: 3 May 07
Posts: 100
Credit: 21,059,812
RAC: 0
Message 95705 - Posted: 1 May 2020, 23:46:45 UTC - in response to Message 95704.  

I'm pretty sure it was announced that Rosetta Mini was being depreciated, but now there is a whole bunch of new Rosetta Mini work out there. I was wondering if the Rosetta Mini application will now be updated to reduce the disk usage as well?

BTW, what the difference between Rosetta and Rosetta Mini ? What part is "mini" ?


https://boinc.bakerlab.org/rosetta/forum_thread.php?id=4394#56015
ID: 95705 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Admin
Project administrator

Send message
Joined: 1 Jul 05
Posts: 4805
Credit: 0
RAC: 0
Message 95709 - Posted: 2 May 2020, 0:11:07 UTC

We are trying to deprecate the app. There will not be an update. It looks like a researcher submitted some jobs recently but this will be the last batch.
ID: 95709 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Grant (SSSF)

Send message
Joined: 28 Mar 20
Posts: 1467
Credit: 14,327,617
RAC: 16,631
Message 95712 - Posted: 2 May 2020, 0:14:11 UTC - in response to Message 95709.  

We are trying to deprecate the app. There will not be an update. It looks like a researcher submitted some jobs recently but this will be the last batch.
Ah, no worries.
Thanks for the update.
Grant
Darwin NT
ID: 95712 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Previous · 1 · 2 · 3 · Next

Message boards : Number crunching : minirosetta_database directory in slot



©2024 University of Washington
https://www.bakerlab.org