Deleting old data files
5 Jan 2005 2:55:52 UTC
Topic 187205
(moderation:
I've been crunching about 1.5 months now and E@H has soaked up 126MB of disk space so far. I understand the data files are used for multiple WUs, but I have to ask..... is there a plan for deleting old data files that are "no longer needed"?
Deleting old data files
)
In fact this is something I am working on right now. Unfortunately as it stands currently, BOINC has no working mechanism to 'clean out' un-needed files on a local host. We are going to be using a feature of BOINC called "scheduling locality" which will remove files for which no more work is available. I'll post some instructions soon about removing old files 'by hand' so we don't hog so much space on your disk.
Bruce
Director, Einstein@Home
My BOINC free space is
)
My BOINC free space is currently 29GB. :)
Cheers,
PeterV.
If you are interessted, BOINC
)
If you are interessted, BOINC folder 100MB large, E@h uses 87% of this.
Greetings from Germany
Basti
Join Ad Astra
> In fact this is something I
)
> In fact this is something I am working on right now. Unfortunately as it
> stands currently, BOINC has no working mechanism to 'clean out' un-needed
> files on a local host. We are going to be using a feature of BOINC called
> "scheduling locality" which will remove files for which no more work is
> available. I'll post some instructions soon about removing old files 'by
> hand' so we don't hog so much space on your disk.
>
> Bruce
>
Bruce, I understand, that in short time this can only be done manually, but please, keep in mind, for people running several boxes this will be a lot work to do. At the moment, I have 16 boxes attached to E@H. So, it would be fine, if you put this cleaning on the long term todo-list for BOINC-development.
Greetings from Germany
Yeti
Supporting BOINC, a great concept !
> In fact this is something I
)
> In fact this is something I am working on right now. Unfortunately as it
> stands currently, BOINC has no working mechanism to 'clean out' un-needed
> files on a local host. We are going to be using a feature of BOINC called
> "scheduling locality" which will remove files for which no more work is
> available. I'll post some instructions soon about removing old files 'by
> hand' so we don't hog so much space on your disk.
>
> Bruce
Were the instructions ever posted? I found the data file directory and I still have about 64MB of space taken up by what appears fo data from November/December. Is it as simple as deleting these old data files or is there some sort of "inventory" that needs to be updated as well?
Thanks for the help. It'll make the backups for all these core-client updates a little smoother if I can get rid of "unnecessary" files.
> > available. I'll post
)
> > available. I'll post some instructions soon about removing old files
> 'by hand' so we don't hog so much space on your disk.
> Were the instructions ever posted? I found the data file directory and I still
> have about 64MB of space taken up by what appears fo data from
> November/December. Is it as simple as deleting these old data files or is
> there some sort of "inventory" that needs to be updated as well?
I apologize -- we have been so busy trying to get BOINC working properly, to get our application working properly, and to get the data prepared properly, that this completely slipped my mind.
The data files that you should KEEP are from the directory
projects/einstein.phys.uwm.edu/
The files are:
earth
sun
einstein_4.7X*
Config*
config*
H1_FFFF.F
Any other files can be deleted (you are welcome to check back with me about this first). Another way to tell what's no longer needed is to look in the client_state.xml file at the top level. It will contain a bunch of sections. Any file which is NOT named in a section of some can be removed.
For what it's worth, I've put a lot of work into improving the BOINC (locality) scheduler so that hosts get work primarily for the files that they already have resident on them, and files that are no longer needed get deleted. I still need to do a bit more work on this to ensure that the number of files resident on any host does not grow too large. We are the first BOINC project to use large persistent data files, so this means some growing pains.
I've got my fingers crossed that we're past the 'core client daily upgrade' point.
Cheers,
Bruce
Director, Einstein@Home
> Any other files can be
)
> Any other files can be deleted (you are welcome to check back with me about
> this first). Another way to tell what's no longer needed is to look in the
> client_state.xml file at the top level. It will contain a bunch of sections.
> Any file which is NOT named in a section of some can be removed.
Bruce, thanks for the reply and the work on the locality scheduler! The files you excluded are the ones I was suspect of. However the client_state.xml still has the old files listed. Here's an example of one section for a file that looks like it is one of the older files (it was only a 3MB download):
------
L1-narrow_169.0_11.0.sft
3168800.000000
0.000000
4e0f39ba696c36ff5bf8b26b4a6a1066
1
http://einstein.phys.uwm.edu/download/L1-narrow_169.0_11.0.sft
------
Just to be safe, is it still okay to delete? Do I need to go into the client_state.xml file and remove the errant sections as well?
EDIT: Oops! It looks like the xml tags were stripped. The info is still there.
Bruce, thank you for the
)
Bruce,
thank you for the update. I think it is worth a minute or two to give us updates on your progress. I appreciate the work you are putting into Einstein and Boinc. Good luck with this part of the project and the rest of the projects that come up.
Jim
> For what it's worth, I've put a lot of work into improving the BOINC
> (locality) scheduler so that hosts get work primarily for the files that they
> already have resident on them, and files that are no longer needed get
> deleted. I still need to do a bit more work on this to ensure that the number
> of files resident on any host does not grow too large. We are the first BOINC
> project to use large persistent data files, so this means some growing pains.
>
> I've got my fingers crossed that we're past the 'core client daily upgrade'
> point.
>
> Cheers,
> Bruce
>
>
Jim
> > > Any other files can be
)
>
> > Any other files can be deleted (you are welcome to check back with me
> about
> > this first). Another way to tell what's no longer needed is to look in
> the
> > client_state.xml file at the top level. It will contain a bunch of
> sections.
> > Any file which is NOT named in a section of some can be removed.
>
> Bruce, thanks for the reply and the work on the locality scheduler! The files
> you excluded are the ones I was suspect of. However the client_state.xml still
> has the old files listed. Here's an example of one section for a file that
> looks like it is one of the older files (it was only a 3MB download):
>
> ------
> L1-narrow_169.0_11.0.sft
> 3168800.000000
> 0.000000
> 4e0f39ba696c36ff5bf8b26b4a6a1066
> 1
>
> http://einstein.phys.uwm.edu/download/L1-narrow_169.0_11.0.sft
> ------
>
> Just to be safe, is it still okay to delete? Do I need to go into the
> client_state.xml file and remove the errant sections as well?
I think it's safe to delete all L1-narrow* and H1-narrow* files. Since no WU depend upon these, the fact that they are listed in client_state.xml shouldn't matter. If you do notice some error message related to these files, then yes, you can also remove the references to them from client_state.xml
Bruce
Director, Einstein@Home
To Bruce and the rest of the
)
To Bruce and the rest of the Einstin@Home Contributors:
Thanks!!! Your responses and behind-the-scene work to keep things running is all very much appreciated!