Gamma ray GPU tasks hanging?

Diplomat
Diplomat
Joined: 7 Aug 15
Posts: 8
Credit: 521801394
RAC: 187526

Ubuntu 20.04 NVIDIA 2070

Ubuntu 20.04 NVIDIA 2070 Super 460 driver

GPU Used up to 99% GPU memory only 1%

New tasks take forever to finish, aborted the fist one at around 1h45m

Now reading this thread allowed the new one to keep running, 2+ hrs and still going.

Ian&Steve C.
Ian&Steve C.
Joined: 19 Jan 20
Posts: 3682
Credit: 33844445845
RAC: 36698635

alanb1951 wrote:I've got a

alanb1951 wrote:

I've got a GTX 1660 Ti on a Ryzen 3700X running Ubuntu 18.04 (kernel 5.3) with version 440.10 drivers, and it was failing these - hope that's old enough for a useful data point!

That machine is, of course, on NNT and having an E@H holiday at present (running GW doesn't seem to go well with CPDN and some WCG CPU stuff, so I don't switch over...)

Cheers - Al.

Thanks for that. I didn't necessarily think it would work, but nice to know for sure.

_________________________________________________________________________

Loaded
Loaded
Joined: 4 Jan 18
Posts: 2
Credit: 576041583
RAC: 15032

I'm having the same

I'm having the same problems.  My Windows 10 machine with a GTX 1080 is processing the tasks normally.  My other Windows machine with a RTX 2080 usually takes less than 8 minutes but is now unable to finish a single Gamma Ray GPU task.  

The Ancient Mariner
The Ancient Mariner
Joined: 13 Apr 05
Posts: 9
Credit: 690004238
RAC: 397872

Hey all, I'm having the same

Hey all, I'm having the same massive slowdown on my FGRP crunching. Ubuntu 20.04 with Nvidia GeForce GTX 1660Ti.  Was originally cruising along at 15 minutes per task, and am now clocking in at 6 hours per task.  GPU is 100% utilized according to the Nvidia control panel, but hardly working (running at 50 degrees celsius, when normally it would be in the 60's). CPU time for the FGRP GPU tasks is incredibly high (like, higher than the equivalent FGRPSSE CPU tasks, which appear to be running fine in parallel with the slow GPU tasks).

Two important notes:

1) My other machine is running a GeForce GT 1030 and is not having the same slowdown problem with the L00 datasets.

2) My 1660Ti machine is running the FGRPopenclTV-nvidia app, while my GT1030 machine is running FGRPopencl1k-nvidia.

I don't see a way to specify which app to use for the FGRP crunching, and I don't know why the 1660Ti machine got all 'TV' apps while the 1030 machine got all '1k' apps.  What is the difference between those apps?

That's all for now.

alanb1951
alanb1951
Joined: 28 Nov 16
Posts: 18
Credit: 642144145
RAC: 425264

The Ancient Mariner wrote: I

The Ancient Mariner wrote:

I don't see a way to specify which app to use for the FGRP crunching, and I don't know why the 1660Ti machine got all 'TV' apps while the 1030 machine got all '1k' apps.  What is the difference between those apps?

For information...  The TV app is for cards with chipsets newer than Pascal - TV = Turing and Volta! - and your 1660Ti has a Turing chipset!  So your machines are getting the right applications for their respective GPUs (but these new jobs simply don't work on Turing, Volta and Ampere at present.)

Cheers - Al.

Tom M
Tom M
Joined: 2 Feb 06
Posts: 5589
Credit: 7676022617
RAC: 1863884

I had restarted my Gtx 1660

I had restarted my Gtx 1660 Super under Windows 10 to see if I would have the same problems.  I do.

Aborted all Nvidia tasks after I had "one" task run 52,000+ wall clock/CPU seconds and then error out.

I have a couple of P106-90 (aka: gtx 1060 3Gb) mining cards that have been munching along under linux.

Tom M

 

A Proud member of the O.F.A.  (Old Farts Association).  Be well, do good work, and keep in touch.® (Garrison Keillor)

The Ancient Mariner
The Ancient Mariner
Joined: 13 Apr 05
Posts: 9
Credit: 690004238
RAC: 397872

@alanb1951 that makes perfect

@alanb1951 that makes perfect sense, thanks for explaining that TV = Turing/Volta.  Obscure knowledge I never thought I'd have!

Bernd Machenschalk
Bernd Machenschalk
Moderator
Administrator
Joined: 15 Oct 04
Posts: 4267
Credit: 244933143
RAC: 16332

It took us a while to

It took us a while to reproduce the problem. Apparently not all new cards show the same behavior with all new WUs. But we finally found a combination where it occurs. I'm afraid that fixing that will require a lot of debugging.

BM

Ian&Steve C.
Ian&Steve C.
Joined: 19 Jan 20
Posts: 3682
Credit: 33844445845
RAC: 36698635

Bernd Machenschalk wrote: It

Bernd Machenschalk wrote:

It took us a while to reproduce the problem. Apparently not all new cards show the same behavior with all new WUs. But we finally found a combination where it occurs. I'm afraid that fixing that will require a lot of debugging.

Thanks for the feedback Bernd!

_________________________________________________________________________

San-Fernando-Valley
San-Fernando-Valley
Joined: 16 Mar 16
Posts: 260
Credit: 6916021637
RAC: 20217632

Bernd Machenschalk wrote: It

Bernd Machenschalk wrote:

It took us a while to reproduce the problem. Apparently not all new cards show the same behavior with all new WUs. But we finally found a combination where it occurs. I'm afraid that fixing that will require a lot of debugging.

Thanks for informing us that you are working on it.

 

I wonder why you had "problems" to reproduce the situation.

ALL of my GPU fail with ALL tasks aka WUs.

Have a nice week - stay save !!

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.