Excessive Disk Usage

YOU are in full control of defining how much disk-space should be consumed by BOINC as a whole, in the BOINC preferences.

As far as I know, the "Resource Share" that you define for a project is not just used to distribute CPU time in a fair way among the projects, it should also play a role in distributing the disk space and prevent some projects form "stealing" space from other projects.

The individual project is unaware of the space consumed by the other projects. It just gets the info "hey, you can you so and so much disk space" by the BOINC client and will use space up to this limit. There is no way to "play nice" on other projects for the individual projects, because BOINC, not the projects, are supposed to be the broker of resources (CPU and disk space).

CU
HBE

1.3GB just for E@H seems unusually large but you need to consider the reasons why that much space is being used. At the moment, there are three separate subprojects going on, each of which will be using a significant amount of disk space that can't really be avoided (or even shared for that matter). Those projects are S5R6, S5GCE and ABP2. Each subproject needs separate executables, separate data files and separate tasks.

The first two subprojects use locality scheduling that requires a significant number of large data files to exist on each host before any actual task can be downloaded and started. You cannot do a single task without incurring a significant disk space penalty. If you complete a task and expect to do any more in the future, you can't recover disk space in any meaningful way by deleting files since a request for a single new task will immediately occupy all that disk space once again with the added penalty of new data downloads that could have been avoided.

The ABP2 subproject doesn't use locality scheduling so, with it, your main disk usage comes from the executables and the tasks themselves. Each task consists of four 2MB data files which will be deleted when the result is safely returned. So if you don't keep a large cache of work, you wont use extra disk space for ABP2 tasks.

There is nothing the project can effectively do to make the disk space penalty much smaller. Unfortunately there has to be a transition from the previous S5R6 project to the new S5GCE replacement subproject and during this transition, for a variety of complicated reasons, the disk space used may very well be quite a bit more than double what you would otherwise expect it to be if you were in the middle of a single long run. So all I can do is repeat that if the project is consuming more disk space than you are prepared to allow, perhaps this project is not suited to your requirements. That is not meant as a put down or as a means of driving anybody away. It's just an unfortunate fact of life.

A forum moderator (without any other added 'qualification') is an ordinary volunteer, just like yourself, but with the added responsibility of trying to answer questions and handle matters of decorum when necessary. We also have the ability to communicate with project staff if needed. Whilst we do tend to get advance information when something major is about to change and we can refer specific matters to a 'higher authority', it is expected that we should be resourceful enough to handle most queries and events without 'bothering' the Devs, who are perpetually busy anyway. In this specific instance, a complaint about excessive disk usage is something we should be capable of resolving ourselves.

I apologise unreservedly if a poor choice of words has given anyone the impression that their contribution is not valued and fully appreciated by the project staff. But let's get one thing perfectly clear. The project doesn't 'owe' you, or me, or anybody else for that matter and we shouldn't expect any 'reward' (other than personal satisfaction) for 'doing them a favour'. This notion that we are doing them a favour and that gives us rights beyond the right to choose to participate or not is a notion that is quite incorrect, in my opinion.

Why do people VOLUNTEER the use of their hardware and agree to pay for the running costs? I would hope that the answer is something along the lines that supporting the discovery of new knowledge through scientific endeavour is something worthy of your donation. What rights do you have if you are not happy with the way the project is using your volunteered resources? Well you certainly have the right not to participate. You don't have a right to require the project to change how it operates.

Every other volunteer is doing exactly the same.

Bikeman has given you an answer to your points. I would like to give you my own thoughts as well.

There is a substantial difference between the BOINC framework and all the project apps that use BOINC to share what you are prepared to volunteer in the way of your resources. It is an oxymoron to suggest that one project can decide 'not to play nice' with another. Individual projects are totally unaware of any other projects and what their particular requirements might be. Apart from yourself, the only 'judge, jury and executioner' is BOINC itself so if there are problems in supporting competing projects which are not able to share resources in an equitable manner, there are actually only two things to blame and the projects themselves are not one of them.

The obvious first point of blame is BOINC. Clashes between projects that BOINC doesn't handle well can only be resolved by the BOINC Devs implementing new features and controls into BOINC. This must be quite an arduous and thankless task as the number of BOINC projects and their competing requirements continues to spiral upwards. Contributing on the BOINC alpha mailing list would be a way to have your personal opinions about 'projects that aren't playing nice' shared with people who could actually do something about it.

The other thing to blame is actually the volunteer. I trust you will take these comments as my personal and honest thoughts on the subject and not try to construe them as any sort of attack on anyone, least of all yourself. It really is the responsibility of the volunteer to think through how much they are prepared to donate and to choose their projects carefully. That really does involve a lot of reading and a lot of question asking. It doesn't involve blaming the project for not showing the proper respect.

Take Milkyway as an example seeing as you mention it. It limits your cache of work to 6 tasks per cpu at any one time. It used to have a deadline of three days. The deadline is now 8 days. It supports both CPUs and GPUs where a given task might take several hours on even a modern CPU but only a minute or two on a modern ATI GPU. There were specific and credible reasons given by the project staff for both the 6 tasks and 3 days limits. Many people have complained long and loud about these limits and I clearly remember the complaint about MW hogging the resources by always being in high priority mode. In most cases, this complaint was largely due to the participant being unwilling to compromise on cache setting and/or unable to appreciate that setting a large cache for a project with a 3 day deadline would confuse the hell out of BOINC.

I've been crunching MW tasks for over a year now, initially on CPUs that were too slow for E@H but more recently on ATI GPUs that E@H cannot use. Since ATI support was added to BOINC, I've been able to configure BOINC so that both my projects can coexist quite happily and I never see high priority mode or examples of one project hogging resources. I've always been able to find a way to keep things on an even keel by using the various preferences that BOINC provides. This also includes 'micromanaging' in a relatively minor way to allow MW to share with Collatz as a backup project for the ATI GPUs whilst still allowing a decent cache of work for E@H running on the CPUs. I expect that the micromanaging will disappear when the BOINC Devs get a better handle on how to properly manage GPU support. I don't see any logic in blaming any project for BOINC's limitations.

I'm sure the project values your long term support from way back in 2005 and I, for one, hope you will continue that support.

Since you linked to your i7, I had a browse through your tasks list and I can see why you would have a high disk usage. I don't think you have any current S5R6 resends left to crunch now but you did have several that are still visible that were crunched over the last week or so. Each of those resends were for a different frequency range so you would have a complete data subset for each single task. Unfortunately, this is what can happen when we are cleaning up the dregs of a long running project that is coming to a close. The scheduler will be advising your client to delete the large number of data files for the different frequency bands when it is sure that they will be no longer required. This is what has to happen with locality scheduling. If you desperately needed to reclaim the disk space immediately, you can take manual action by editing your state file to clean out the references to the old data files and then deleting the old data files themselves. You really need to understand what you are doing for attempting something like this. It's not a trivial exercise and I certainly wouldn't recommend it in normal circumstances.

The safer course of action is to understand the reasons for this temporary excessive disk usage and to make a decision that perhaps you can cut the project a little slack and increase your disk usage limits sufficiently to tide you over the temporary situation. Also, unfortunately it's not going to be quite as temporary as it normally would be. If you check the server status link on the front page, you will notice that the GCE run will only last for another month. At this point there will be another of these pesky transitions with the same set of problems all over again.

While true, I think erinm thinks that since the preferences in the global preferences/computing preferences are the same on all his projects, that all his projects must know about the other projects.

They don't. These preferences are for the managing program, BOINC.
And then if you look at the Disk and memory settings, they are for all of BOINC, for all projects running under BOINC.

These settings work together with the Network usage settings of "Computer is connected to the Internet about every X days" and "Maintain enough work for an additional Y days". Especially with the latter one. If set high, this will be on all the projects you are attached to on the same venue (and if not overridden by local preferences).

I'm wondering why if you have an i7-965 with Hyper Threading on (so 8 CPUs), you 'constrain' BOINC by giving it only 1.5GB total disk space to work with. I can't imagine that you have spent high money on such a CPU, but cheated on everything else and only installed a 20GB drive in the machine. There must be something bigger in there, else you can't even install Vista Ultimate x64 in a reasonable manner.

So what are your settings for the following?
Use at most A GB disk space
Leave at least B GB disk space free
Use at most C% of total disk space

Computer is connected to the Internet about every D days
Maintain enough work for an additional E days

Thanks for looking in to this. I have enough free disk space so I will leave things as they are until the devs have come up with a permanent solution.