Fulfill O3 GPU quorums not validating

archae86
archae86
Joined: 6 Dec 05
Posts: 3,145
Credit: 7,047,034,931
RAC: 1,576,897
Topic 230493

I run exclusively GW tasks of the GPU O3 flavor recently.  This week I've had a sudden big surge in number of tasks pending.  On reviewing some today I found it common to see WUs for which both tasks had been returned, but both were currently reported with status "Completed, waiting for validation"  The server status page does not report validator suspended or otherwise not running, but does report a large number of O3AS pending tasks.

Here are links for a few such WUs:

https://einsteinathome.org/workunit/771901252
https://einsteinathome.org/workunit/771916542
https://einsteinathome.org/workunit/771916652
https://einsteinathome.org/workunit/771916754
https://einsteinathome.org/workunit/771945775
https://einsteinathome.org/workunit/771887645

These were as of 5:14 UTC December 14, 2023

 

Richard M
Richard M
Joined: 11 Nov 04
Posts: 78
Credit: 237,143,266
RAC: 1,047,269

I also have noticed a growing

I also have noticed a growing number of pending tasks that have a minimum quorum of two that have not been validated. 

 

Ian&Steve C.
Ian&Steve C.
Joined: 19 Jan 20
Posts: 3,700
Credit: 34,374,816,958
RAC: 38,634,907

Server Status Page O3AS -

Server Status Page

O3AS - 55,000+ waiting for validation.

something stuck project-side.

_________________________________________________________________________

Bernd Machenschalk
Bernd Machenschalk
Moderator
Administrator
Joined: 15 Oct 04
Posts: 4,273
Credit: 245,140,948
RAC: 13,979

The filesystem on the project

The filesystem on the project server (einstein3) is doing some "scrubbing" that slows down the validator. It's not completely stuck, but slowed down (current delay 1-2d). The scrubbing should be finished in <20h, after that, the validator should be able to catch up. I also moved the second search (BRP7) to another server, which should also help to reduce I/O load.

BM

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.