Pausing BRP4 conditionally

Bernd Machenschalk
Bernd Machenschalk
Moderator
Administrator
Joined: 15 Oct 04
Posts: 4,276
Credit: 245,609,635
RAC: 10,558
Topic 196511

The disks of the server that ships BRP4 data are at their (IOPS) limits. The first sign of this is that the file deleter can't keep up deleting all the old data files, visible at the server status page as "Workunits waiting for file deletion". As a result, these old data files pile up, causing further slowdown.

In order to prevent the disk from running full, we disabled sending out BRP4 tasks for a while, until the old files have been purged. I expect this to take a few hours.

BM

BM

Bernd Machenschalk
Bernd Machenschalk
Moderator
Administrator
Joined: 15 Oct 04
Posts: 4,276
Credit: 245,609,635
RAC: 10,558

Pausing BRP4 conditionally

Sending out BRP4 work again.

BM

BM

Logforme
Logforme
Joined: 13 Aug 10
Posts: 332
Credit: 1,714,373,961
RAC: 0

Not been able to download new

Not been able to download new tasks for a few hours now. Something wrong on server side or am I just unlucky getting a download slot?

diederiks
diederiks
Joined: 30 Jul 05
Posts: 3
Credit: 4,062,713
RAC: 0

Unsure of the cause but i'm

Unsure of the cause but i'm seeing some download problems. Of all the WU data files a WU has, i have been able to download more then 60% of them but not all files of 1 WU.

Update: It's working like a charm now!

Bikeman (Heinz-Bernd Eggenstein)
Bikeman (Heinz-...
Moderator
Joined: 28 Aug 06
Posts: 3,522
Credit: 692,138,935
RAC: 21,843

It's quite possible that we

It's quite possible that we will see some download and work distribution problems in the next few days, as the server is more or less at its limits. Please bear with us while we add additional resources to the project infrastructure.

Cheers
HB

Bernd Machenschalk
Bernd Machenschalk
Moderator
Administrator
Joined: 15 Oct 04
Posts: 4,276
Credit: 245,609,635
RAC: 10,558

We are working on setting up

We are working on setting up a new server to take over half of the load of the BRP4 downloads. However we won't have this in operation until early next week. To get a bit of relief, I disabled the CPU BRP4 Apps for now. This should keep the GPUs busy, and the project currently has enough other work for CPUs.

BM

BM

Bernd Machenschalk
Bernd Machenschalk
Moderator
Administrator
Joined: 15 Oct 04
Posts: 4,276
Credit: 245,609,635
RAC: 10,558

The new server is up and

The new server is up and apparently running well.

We re-enabled sending BRP4 CPU tasks.

BM

BM

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.