Strange invalidating of GPU tasks - Linux versus Windows

Gary has given you a good answer.  Let me expand on a couple of points.

Since we as a community ALL experience invalid result determinations on our GPU work at an appreciable rate, it is completely unreasonable for an individual user with a half dozen such units to expect personalized analysis from the project.

I think there is a big difference between CPU and GPU work here at Einstein in this matter.  For CPU work, on a properly setup machine not running error-inducing clock rates, one can expect to see zero invalid (and zero error) determinations in many weeks of operation (so thousands of units).  But for GPU work, I don't think we see invalid rates much below a few tenths of a percent on large samples.

I don't think the real source of this is well known, or at least publicized.  Some people imagine GPUs just make mistakes now and then.  Another possibility is that the methods in use produce slight differences depending on the time-slicing of GPU usage during processing of particular work.  Who knows, there could even be an unintended dependence on an uninitialized variable.

That is not to say that all invalid findings are spurious.  The fleet average is above the irreducible base rate because some participant systems at times operate with substantial real error rates.  I suspect the most common reason is operation above the highest clock rates for which the system can currently generate consistently correct results.  But that can arise not because the user twisted up the clock rate knob, but because the chip temperature rose through fan failure, thermal paste degradation, room ambient temperature rise ...  Some users also push up the clocks too high (I have...), although those users may be more likely than most to notice.  Some cards can't quite consistently get the right answer at stock clocks, and need to be underclocked a little to work well.

Just to give a personal example, at this moment a look at my account tasks summary for Einstein shows All tasks at 3661.  But 1060 of these are shown as In Progress so can't be considered success or failure.  That leaves about 2600.  Of those 12 show as Invalid.  Crudely, that suggest I have about a .5% invalid rate.  Crude, because not all the 2600 have actually had a quorum filled and been compared, but more importantly because the retention time in the various categories is not the same.  

As part of regular health and sanity checking, I click on the invalid link for my whole account, then click on the sorting link over the Sent column twice.  I want to see if one of my three machines seems recently to have started a much higher problem rate than usual.  As the "no problem" rate varies quite a bit, it takes some time to get a feel for what is different enough to be a concern.

In summary--yes it is a real problem, and specifically it makes it much more difficult for users to assess the health of their systems than would be best.  I don't think we can usefully drive for improvement by offering examples of a handful of such results on a specific system.

If you have invalid rates far above the background noise level (which I'll guess is in the order of 0.1% to 1%) then a moderate trial time you can tell something.  But the variability in the base rate is enough that people are just kidding themselves if they draw conclusions on the difference between zero invalids in fifty trials and two in fifty trials.

I spent many weeks inching up the memory and core clocks on two Nvidia Pascal series cards on my main system just recently.  As I approached the cliff of doom in tiny increments, pausing a day or more at each, I never reached a point at which I had a detectable elevation of invalid rates.  Instead I got abnormal early termination with a reported error, or, less commonly, an actual system black screen, blue screen, or unrequested reboot.  Other cards in other systems probably have other behaviors.