anybody seen this CUDA error before?
20 Mar 2012 21:54:25 UTC
Topic 196246
(moderation:
i've started getting a type of error that's new to me on my GTX 460s. i chose this particular failed task b/c only ran for ~2000 seconds and consequently has the shortest output file of all the failed tasks:
Quote:
Stderr output
6.12.43
An attempt was made to reference a token that does not exist. (0x3f0) - exit code 1008 (0x3f0)
Activated exception handling...
[10:30:21][456][INFO ] Starting data processing...
[10:30:21][456][INFO ] CUDA global memory status (initial GPU state, including context):
------> Used in total: 535 MB (489 MB free / 1024 MB total) -> Used by this application (assuming a single GPU task): 0 MB
[10:30:21][456][INFO ] Using CUDA device #1 "GeForce GTX 460" (336 CUDA cores / 1209.60 GFLOPS)
[10:30:21][456][INFO ] Version of installed CUDA driver: 4010
[10:30:21][456][INFO ] Version of CUDA driver API used: 3020
[10:30:21][456][INFO ] Checkpoint file unavailable: status.cpt (No such file or directory).
------> Starting from scratch...
[10:30:21][456][INFO ] Header contents:
------> Original WAPP file: ./p2030.20111116.G36.41+01.16.C.b1s0g0.00000_DM369.60
------> Sample time in microseconds: 65.4762
------> Observation time in seconds: 274.62705
------> Time stamp (MJD): 55881.815264416669
------> Number of samples/record: 0
------> Center freq in MHz: 1214.289551
------> Channel band in MHz: 0.33605957
------> Number of channels/record: 960
------> Nifs: 1
------> RA (J2000): 185333.067402
------> DEC (J2000): 34349.0175018
------> Galactic l: 0
------> Galactic b: 0
------> Name: G36.41+01.16.C
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 4194304
------> Trial dispersion measure: 369.6 cm^-3 pc
------> Scale factor: 0.00720348
[10:30:24][456][INFO ] Seed for random number generator is 0.
[10:30:25][456][INFO ] Derived global search parameters:
------> f_A probability = 0.08
------> single bin prob(P_noise > P_thr) = 1.32531e-008
------> thr1 = 18.139
------> thr2 = 21.241
------> thr4 = 26.2686
------> thr8 = 34.6478
------> thr16 = 48.9581
[10:30:25][456][INFO ] CUDA global memory status (GPU setup complete):
------> Used in total: 736 MB (288 MB free / 1024 MB total) -> Used by this application (assuming a single GPU task): 201 MB
[10:31:22][456][INFO ] Checkpoint committed!
[10:32:22][456][INFO ] Checkpoint committed!
[10:33:23][456][INFO ] Checkpoint committed!
[10:34:23][456][INFO ] Checkpoint committed!
[10:35:23][456][INFO ] Checkpoint committed!
[10:36:24][456][INFO ] Checkpoint committed!
[10:37:24][456][INFO ] Checkpoint committed!
[10:38:25][456][INFO ] Checkpoint committed!
[10:39:25][456][INFO ] Checkpoint committed!
[10:40:26][456][INFO ] Checkpoint committed!
[10:40:57][456][INFO ] Data processing finished successfully!
[10:40:57][456][INFO ] Starting data processing...
[10:40:57][456][INFO ] CUDA global memory status (initial GPU state, including context):
------> Used in total: 564 MB (460 MB free / 1024 MB total) -> Used by this application (assuming a single GPU task): 29 MB
[10:40:57][456][INFO ] Using CUDA device #1 "GeForce GTX 460" (336 CUDA cores / 1209.60 GFLOPS)
[10:40:57][456][INFO ] Version of installed CUDA driver: 4010
[10:40:57][456][INFO ] Version of CUDA driver API used: 3020
[10:40:57][456][INFO ] Checkpoint file unavailable: status.cpt (No such file or directory).
------> Starting from scratch...
[10:40:57][456][INFO ] Header contents:
------> Original WAPP file: ./p2030.20111116.G36.41+01.16.C.b1s0g0.00000_DM369.90
------> Sample time in microseconds: 65.4762
------> Observation time in seconds: 274.62705
------> Time stamp (MJD): 55881.815264410565
------> Number of samples/record: 0
------> Center freq in MHz: 1214.289551
------> Channel band in MHz: 0.33605957
------> Number of channels/record: 960
------> Nifs: 1
------> RA (J2000): 185333.067402
------> DEC (J2000): 34349.0175018
------> Galactic l: 0
------> Galactic b: 0
------> Name: G36.41+01.16.C
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 4194304
------> Trial dispersion measure: 369.9 cm^-3 pc
------> Scale factor: 0.00720348
[10:40:59][456][INFO ] Seed for random number generator is 0.
[10:41:00][456][INFO ] Derived global search parameters:
------> f_A probability = 0.08
------> single bin prob(P_noise > P_thr) = 1.32531e-008
------> thr1 = 18.139
------> thr2 = 21.241
------> thr4 = 26.2686
------> thr8 = 34.6478
------> thr16 = 48.9581
[10:41:00][456][INFO ] CUDA global memory status (GPU setup complete):
------> Used in total: 766 MB (258 MB free / 1024 MB total) -> Used by this application (assuming a single GPU task): 231 MB
[10:41:26][456][INFO ] Checkpoint committed!
[10:42:26][456][INFO ] Checkpoint committed!
[10:43:27][456][INFO ] Checkpoint committed!
[10:44:27][456][INFO ] Checkpoint committed!
[10:45:28][456][INFO ] Checkpoint committed!
[10:46:28][456][INFO ] Checkpoint committed!
[10:47:29][456][INFO ] Checkpoint committed!
[10:48:29][456][INFO ] Checkpoint committed!
[10:49:30][456][INFO ] Checkpoint committed!
[10:50:30][456][INFO ] Checkpoint committed!
[10:51:30][456][INFO ] Checkpoint committed!
[10:51:42][456][INFO ] Data processing finished successfully!
[10:51:42][456][INFO ] Starting data processing...
[10:51:42][456][INFO ] CUDA global memory status (initial GPU state, including context):
------> Used in total: 535 MB (489 MB free / 1024 MB total) -> Used by this application (assuming a single GPU task): 0 MB
[10:51:42][456][INFO ] Using CUDA device #1 "GeForce GTX 460" (336 CUDA cores / 1209.60 GFLOPS)
[10:51:42][456][INFO ] Version of installed CUDA driver: 4010
[10:51:42][456][INFO ] Version of CUDA driver API used: 3020
[10:51:42][456][INFO ] Checkpoint file unavailable: status.cpt (No such file or directory).
------> Starting from scratch...
[10:51:42][456][INFO ] Header contents:
------> Original WAPP file: ./p2030.20111116.G36.41+01.16.C.b1s0g0.00000_DM370.20
------> Sample time in microseconds: 65.4762
------> Observation time in seconds: 274.62705
------> Time stamp (MJD): 55881.815264404468
------> Number of samples/record: 0
------> Center freq in MHz: 1214.289551
------> Channel band in MHz: 0.33605957
------> Number of channels/record: 960
------> Nifs: 1
------> RA (J2000): 185333.067402
------> DEC (J2000): 34349.0175018
------> Galactic l: 0
------> Galactic b: 0
------> Name: G36.41+01.16.C
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 4194304
------> Trial dispersion measure: 370.2 cm^-3 pc
------> Scale factor: 0.00719656
[10:51:44][456][INFO ] Seed for random number generator is 0.
[10:51:45][456][INFO ] Derived global search parameters:
------> f_A probability = 0.08
------> single bin prob(P_noise > P_thr) = 1.32531e-008
------> thr1 = 18.139
------> thr2 = 21.241
------> thr4 = 26.2686
------> thr8 = 34.6478
------> thr16 = 48.9581
[10:51:45][456][INFO ] CUDA global memory status (GPU setup complete):
------> Used in total: 768 MB (256 MB free / 1024 MB total) -> Used by this application (assuming a single GPU task): 233 MB
[10:52:31][456][INFO ] Checkpoint committed!
[10:53:31][456][INFO ] Checkpoint committed!
[10:54:32][456][INFO ] Checkpoint committed!
[10:55:32][456][INFO ] Checkpoint committed!
[10:56:32][456][INFO ] Checkpoint committed!
[10:57:33][456][INFO ] Checkpoint committed!
[10:58:33][456][INFO ] Checkpoint committed!
[10:59:34][456][INFO ] Checkpoint committed!
[11:00:34][456][INFO ] Checkpoint committed!
[11:01:34][456][INFO ] Checkpoint committed!
[11:02:35][456][INFO ] Checkpoint committed!
[11:02:47][456][INFO ] Data processing finished successfully!
[11:02:47][456][INFO ] Starting data processing...
[11:02:47][456][INFO ] CUDA global memory status (initial GPU state, including context):
------> Used in total: 553 MB (471 MB free / 1024 MB total) -> Used by this application (assuming a single GPU task): 18 MB
[11:02:47][456][INFO ] Using CUDA device #1 "GeForce GTX 460" (336 CUDA cores / 1209.60 GFLOPS)
[11:02:47][456][INFO ] Version of installed CUDA driver: 4010
[11:02:47][456][INFO ] Version of CUDA driver API used: 3020
[11:02:47][456][INFO ] Checkpoint file unavailable: status.cpt (No such file or directory).
------> Starting from scratch...
[11:02:47][456][INFO ] Header contents:
------> Original WAPP file: ./p2030.20111116.G36.41+01.16.C.b1s0g0.00000_DM370.50
------> Sample time in microseconds: 65.4762
------> Observation time in seconds: 274.62705
------> Time stamp (MJD): 55881.815264398363
------> Number of samples/record: 0
------> Center freq in MHz: 1214.289551
------> Channel band in MHz: 0.33605957
------> Number of channels/record: 960
------> Nifs: 1
------> RA (J2000): 185333.067402
------> DEC (J2000): 34349.0175018
------> Galactic l: 0
------> Galactic b: 0
------> Name: G36.41+01.16.C
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 4194304
------> Trial dispersion measure: 370.5 cm^-3 pc
------> Scale factor: 0.00719656
[11:02:49][456][INFO ] Seed for random number generator is 0.
[11:02:50][456][INFO ] Derived global search parameters:
------> f_A probability = 0.08
------> single bin prob(P_noise > P_thr) = 1.32531e-008
------> thr1 = 18.139
------> thr2 = 21.241
------> thr4 = 26.2686
------> thr8 = 34.6478
------> thr16 = 48.9581
[11:02:50][456][INFO ] CUDA global memory status (GPU setup complete):
------> Used in total: 738 MB (286 MB free / 1024 MB total) -> Used by this application (assuming a single GPU task): 203 MB
[11:03:35][456][INFO ] Checkpoint committed!
[11:04:34][456][ERROR] Error during CUDA device->host time series mean transfer (error: 700)
[11:04:34][456][ERROR] Demodulation failed (error: 1008)!
11:04:34 (456): called boinc_finish]]>
the platform is WinXP Pro SP3 x32, AMD X6 1090T CPU, 2 x GTX 460 (each run 3 BRP tasks simultaneously). i have been experiencing GPU throttling lately, which may have something to do with it...though i don't know why, as both cards are always kept under 60°C, even at full load, and neither are overvolted or overclocked. are there any other reasons why a GPU might throttle back? is anyone familiar with this particular error?
TIA,
Eric
anybody seen this CUDA error before?
)
i didn't get a chance to post the BOINC event log at around the time the above error occurred. however, another of the same error just occurred a few minutes ago (
An attempt was made to reference a token that does not exist. (0x3f0) - exit code 1008 (0x3f0)
), and here's the associated BOINC event log at the time of the error:
it does not appear as though this one was cause by a video driver reset b/c neither of my GTX 460s have throttled back to 405mhz, and both are still running at their stock clocks...not that i'm certain that the other errors were caused by a video driver reset, but it seemed like it could have been a possibility before, since i kept discovering that one of my GPUs was throttled down by the time i discovered the errors themselves.
some additional information
)
some additional information that i should have included in my first post:
BOINC v6.12.43
nVidia 285.58 drivers
hmm...112 views and nobody
)
hmm...112 views and nobody has seen this error before, eh? they seemed to have been occurring less and less, and then all of the sudden i got 3 of them today alone.