Request to increase task limit per core

Jeroen
Jeroen
Joined: 25 Nov 05
Posts: 379
Credit: 738,881,678
RAC: 1,204
Topic 196400

Hello,

I have request. I am running a 6-core processor on one of my hosts and have the max CPU usage set to 59% currently. Normally I run 50% as to avoid slow down from Hyperthreading and also due to summer heat which drives up CPU temps. HT drags down performance of the GPU tasks. I am hitting the quota limit of 280 tasks (7x40) in a day and 240 tasks yesterday when I was set to 50% usage (6x40). My question is if it would be possible to have the per core limit of 40 tasks increased to 50 tasks? This way my host can run up to 300 tasks (6x50) in a day with 50% max CPU usage for avoiding HT performance loss and extra heat. Thanks.

Bikeman (Heinz-Bernd Eggenstein)
Bikeman (Heinz-...
Moderator
Joined: 28 Aug 06
Posts: 3,516
Credit: 470,854,650
RAC: 54,304

Request to increase task limit per core

Quote:

Hello,

I have request. I am running a 6-core processor on one of my hosts and have the max CPU usage set to 59% currently. Normally I run 50% as to avoid slow down from Hyperthreading and also due to summer heat which drives up CPU temps. HT drags down performance of the GPU tasks. I am hitting the quota limit of 280 tasks (7x40) in a day and 240 tasks yesterday when I was set to 50% usage (6x40). My question is if it would be possible to have the per core limit of 40 tasks increased to 50 tasks? This way my host can run up to 300 tasks (6x50) in a day with 50% max CPU usage for avoiding HT performance loss and extra heat. Thanks.

Yes, I also see this happening on a less extreme system of mine, a Core2 Dual with 2 GPUs and one core set aside to drive the ATI/AMD card. The resulting 40 tasks per day is limiting.

However, the real fix would probably to include the GPUs in the calculation, not just bumping up the limit of jobs per core per day which does have some justification in the absence of GPUs.

The workaround I have until this is fixed is to have a little cc_config.xml file which fools BOINC into assuming the host has more CPUs (say twice as many) than it really has. I tell boinc to re-read this file on-the-fly (with boinccmd command line interface, see boinccmd --help) , ask boinc to update the project a few times to fetch work (using boinccmd as well), and then revert to a cc_config file that has the true number of CPUs (and re-load it again with cc_config.xml).

But yes, this is only a workaround and we need to address the root cause.

Cheers
HBE

archae86
archae86
Joined: 6 Dec 05
Posts: 3,021
Credit: 4,987,386,720
RAC: 3,040,227

I also find the daily

I also find the daily download limit based only on allocated CPU cores to be an awkward limitation on my GPU-employing systems. My plea posted here a couple of months ago may perhaps be added to yours now.

Jeroen
Jeroen
Joined: 25 Nov 05
Posts: 379
Credit: 738,881,678
RAC: 1,204

RE: Yes, I also see this

Quote:


Yes, I also see this happening on a less extreme system of mine, a Core2 Dual with 2 GPUs and one core set aside to drive the ATI/AMD card. The resulting 40 tasks per day is limiting.

However, the real fix would probably to include the GPUs in the calculation, not just bumping up the limit of jobs per core per day which does have some justification in the absence of GPUs.

The workaround I have until this is fixed is to have a little cc_config.xml file which fools BOINC into assuming the host has more CPUs (say twice as many) than it really has. I tell boinc to re-read this file on-the-fly (with boinccmd command line interface, see boinccmd --help) , ask boinc to update the project a few times to fetch work (using boinccmd as well), and then revert to a cc_config file that has the true number of CPUs (and re-load it again with cc_config.xml).

But yes, this is only a workaround and we need to address the root cause.

Cheers
HBE

Thank you for the suggestion with the cc_config.xml. I will give this configuration change a try and also see if I can script something for automation.

Jeroen

Bikeman (Heinz-Bernd Eggenstein)
Bikeman (Heinz-...
Moderator
Joined: 28 Aug 06
Posts: 3,516
Credit: 470,854,650
RAC: 54,304

Hi! Bernd has just changed

Hi!

Bernd has just changed the server config to allow more work being distributed to our GPU crunching volunteers. The new settings are

Max tasks per day = 32 * (number of usable CPU cores) + 160 * (number of reported GPUs) , where the number of cores is limited to max 64 and the max # of GPUs is 8.

The previous formula was:

Max tasks per day = 40 * (number of usable CPU cores) , where the number of cores was limited to max 16.

We'll see how this works out, let us know if there is still a problem with GPUs running dry because of a daily quota issue.

All this is a compromise between allowing clients to download as many WUs as they request (which could be, for a (say) 7 day cache and 3 top notch graphics cards ca 1500 jobs on the initial day!!!) and protecting the server infrastructure against clients that accidentally try to fetch work in an endless loop.

Cheers
HB

Jeroen
Jeroen
Joined: 25 Nov 05
Posts: 379
Credit: 738,881,678
RAC: 1,204

Hello, That is great news.

Hello,

That is great news. Thank you and to Bernd for updating the server to allow for a greater number of tasks per GPU. I had temporarily disabled CPU tasks to help prevent GPUs from running out of tasks but have been wanting to get the CPU projects up and running again. With this change, I have re-enabled running Gravitational-wave and Gamma-ray searches. I am glad to have those projects running.

Thanks again.

Jeroen

Bernd Machenschalk
Bernd Machenschalk
Moderator
Administrator
Joined: 15 Oct 04
Posts: 4,031
Credit: 218,094,505
RAC: 49,928

Has anyone observed any

Has anyone observed any problems regarding the new quota setting?

BM

BM

Jeroen
Jeroen
Joined: 25 Nov 05
Posts: 379
Credit: 738,881,678
RAC: 1,204

I have not seen any problems

I have not seen any problems and my host is no longer running into a daily task limit.

archae86
archae86
Joined: 6 Dec 05
Posts: 3,021
Credit: 4,987,386,720
RAC: 3,040,227

No new problems here, and it

No new problems here, and it solved a problem, in that I was able to reduce the CPU jobs running to one while still being able to get an adequate supply of GPU work. This suits my summertime power reduction scheme, as the power productivity of incremental CPU work is far lower than that of incremental GPU work on my hosts. I also throttle the hosts using Fred's tool, cutting back power consumption more when the room is hotter than when cooler. I should be back up to full production by early October.

Bikeman (Heinz-Bernd Eggenstein)
Bikeman (Heinz-...
Moderator
Joined: 28 Aug 06
Posts: 3,516
Credit: 470,854,650
RAC: 54,304

Thanks for the feedback guys.

Thanks for the feedback guys. The suspected download throughput problem (moved to a separate thread) was also resolved (was actually a user defined throttling), so everything looks fine now in the high performance GPU camp :-)

Happy crunching
HB

joe areeda
joe areeda
Joined: 13 Dec 10
Posts: 285
Credit: 320,378,898
RAC: 0

RE: Has anyone observed any

Quote:

Has anyone observed any problems regarding the new quota setting?

BM


I have 2 systems that I think are running into this limit:

http://einsteinathome.org/host/5771544

and

http://einsteinathome.org/host/5860433

At least they seem to be productive on GPU tasks but neither have any running at certain times.

How can I tell if the reason is this limit?

Joe

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.