Managing the "typical" weekend outage
17 Jan 2021 16:04:46 UTC
Topic 224527
(moderation:
I have two systems as of the moment that typically runs out of gamma-ray gpu tasks to process during the "regular" weekend upload outage.
How much more downloads should I set and how high a fake cpu should I set to keep enough gpu tasks available to crunch while I am waiting for the upload log jam to clear.
Currently, both are set at store at least 0.1 days and store at least an additional 0.1
These systems process a GR task about every 16 minutes. The first system has 1 gpu and a 32 thread cpu. The 2nd has 4 gpus and a 16 thread cpu.
Any guidance?
I know we have a hard limit on the total # of gpu tasks unless we fake a higher # of cpus. So what are some ideas on what I should set everything for?
Tom M
A Proud member of the O.F.A. (Old Farts Association). Be well, do good work, and keep in touch.® (Garrison Keillor) I want some more patience. RIGHT NOW!
The advice to run super-low
)
The advice to run super-low cache settings is apt when a person is operating in a way which gives rise to big fluctuations.
1. running more than one task type (say GW GPU plus GR GPU)
2. changing system configuration (say going from 1X to 2X, or adding cards, or substracting cards)
GR here, in particular, has very stable execution times, so one can configure a cache setting to give, for example, a couple of days of work and expect not to have wild excursions.
So when (if) you stabilize those systems as to configuration and commit them to running GR GPU only, then I think you might reasonably creep up your cache settings until you are really getting about 2 days in stock. This may nor may not occur at a setting of 2 days. So, adjust and observe.
An obstacle for super-productive systems, which may hit you, but does not limit most, is that there is a hard-wired limit that causes your BOINC not to request additional work if the tasks already in stock exceed 1000. If this is less than 2 days for you, you'll see a bit of odd behavior as it goes above and back down to that threshold.
You may not need to fake extra CPU cores at all. Doing that is a way to get more tasks per day, but the allocations when applied to machines like yours are probably plenty generous.
Now, people with more than one VII card and few real cores do have a good reason to push up their daily quota.
My solution: As soon as
)
My solution:
As soon as the Sunday uploading stops, I have time to go for an extended and excellent walk in the snow covered nature at the back of house up the hills, past fields and forests!
Especially since I don't see any GPUs or tasks in that area.
Great for my nerves and my unimportant and trivial PCs are happy to be able to RELAX.
Have a nice week!
The resource-dependent daily
)
The resource-dependent daily task quota allows 32 per available CPU and 256 per available GPU.
Falsified higher CPU counts do increase quota up to some maximum, which I don't currently have handy but think may be 64 CPUs.
Restriction of the number of CPUs BOINC is allowed to assume it can use by the "Use at most" setting in Computing preferences cuts the number proportionally. I don't know how quota responds to limitation by other available means.
Since each GPU adds 256 tasks/day to the quota, then only GPU/task type combinations for which the GPU on average completes more than one task every 338 seconds need help to sustain daily nutrition. 570/580 cards don't get close enough to be any problem.
But VII cards do, and also some of the other highly capable types. But many people running those cards run high core-count CPUs, so the 32/CPU term gets them enough work anyway.
when the GR upload problems
)
when the GR upload problems start, just flip over to GW which doesnt have the problem.
_________________________________________________________________________
Basically, to me the question
)
Basically, to me the question rather is: why are no steps being taken server-side to avoid this upload-jam every weekend?
Erich56 wrote: Basically, to
)
You're looking at it backwards from the admins. The problem is that uploads + weekly server maintenance (IIRC a backup is the big thing adding extra IO); overloads the servers. Pausing uploads during the maintenance window is the free way to address the problem. The other option involves spending a €lots (probably upwards of €10k, and wouldn't surprise if me several times that) for a bigger server. It's possible that they'll size a new server big enough for the load at the next planned hardware refresh; shelling out that kind of money for an unplanned upgrade when there's a work around isn't going to happen in 99% of cases.
It looks like we are able to
)
It looks like we are able to send the finished work in again.
San-Fernando-Valley
)
Superb. TY.
A Proud member of the O.F.A. (Old Farts Association). Be well, do good work, and keep in touch.® (Garrison Keillor) I want some more patience. RIGHT NOW!
MAGIC Quantum Mechanic
)
It did some on each machine and then got jammed up into "backoff" land again.
Tom M
A Proud member of the O.F.A. (Old Farts Association). Be well, do good work, and keep in touch.® (Garrison Keillor) I want some more patience. RIGHT NOW!
archae86 wrote:Since each GPU
)
338 / 60 = 5.633 minutes.
My 5700's run north of 6 minutes on a single task per GPU.
And even though one of the machines is crunching significant GW CPU tasks, those are leftovers. Right now both machines are on GR only GPU profiles.
It looks like I can safely switch to 2 days and 0.25 addons. I have done so. Now all I have to do is wait for the backlog to clear and then see what next weekend brings.
Tom M
A Proud member of the O.F.A. (Old Farts Association). Be well, do good work, and keep in touch.® (Garrison Keillor) I want some more patience. RIGHT NOW!