Task 1710594483

Name: Ter5_1_dns_cfbf00110_segment_6_dms_200_40000_148_7650000_0
Workunit ID: 862352437
Created: 8 Jan 2025 6:07:49 UTC
Sent: 8 Jan 2025 12:58:06 UTC
Report deadline: 22 Jan 2025 12:58:06 UTC
Received: 20 Jan 2025 0:22:02 UTC
Server state: Over
Outcome: Success
Client state: Done
Exit status: 0 (0x00000000)
Computer: 12843866
Run time (sec): 16,995.83
CPU time (sec): 411.28
Peak working set size (MB): 482.36
Peak swap size (MB): 766.64
Peak disk usage (MB): 75.39
Validation state: Valid
Granted credit: 3,333
Application: Binary Radio Pulsar Search (MeerKAT) v0.12 (BRP7-cuda55)
windows_x86_64
Stderr output

<core_client_version>8.0.2</core_client_version>
<![CDATA[
<stderr_txt>
----> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 16777216
------> Trial dispersion measure: 244.8 cm^-3 pc
------> Scale factor: 1.01616
[09:50:27][9092][INFO ] Seed for random number generator is -1031807597.
[09:50:31][9092][ERROR] Couldn't allocate 7706856 bytes of CUDA HS summing memory (error: 2)!
[09:50:31][9092][ERROR] Demodulation failed (error: 1006)!
[09:50:31][9092][WARN ] Sorry, at the moment your system doesn't have enough free CPU/GPU memory to run this task!
------> Returning control to BOINC, delaying next attempt for at least 15 minutes...
------> If this problem persists you should consider aborting this task...
Activated exception handling...
[10:05:35][18500][INFO ] Starting data processing...
[10:05:35][18500][INFO ] CUDA device ID: 0
[10:05:35][18500][INFO ] CUDA global memory status (initial GPU state, including context):
------> Used in total: 185 MB (840 MB free / 1025 MB total) -> Used by this application (assuming a single GPU task): 0 MB
[10:05:35][18500][INFO ] Using CUDA device #0 "Quadro K600" (192 CUDA cores / 336.19 GFLOPS)
[10:05:35][18500][INFO ] Version of installed CUDA driver: 9010
[10:05:35][18500][INFO ] Version of CUDA driver API used: 3020
[10:06:09][18500][INFO ] Number of generated templates to be used: 50000
[10:06:09][18500][INFO ] Continuing work on ../../projects/einstein.phys.uwm.edu/Ter5_1_dns_cfbf00110_segment_6_dms_200_148.binary at template no. 8947
[10:06:09][18500][INFO ] Header contents:
------> Original WAPP file: /atlas/data/TRAPUM_GC/Terzan5/epoch1/30min_segments/dedispersed_files/cfbf00110/Ter5_1_dns_cfbf00110_segment_6_dms_200_DM244.80
------> Sample time in microseconds: 153.121
------> Observation time in seconds: 2568.9524
------> Time stamp (MJD): 59097.776208975287
------> Number of samples/record: 0
------> Center freq in MHz: 857.5673828
------> Channel band in MHz: 3.34375
------> Number of channels/record: 256
------> Nifs: 1
------> RA (J2000): 174801.97
------> DEC (J2000): -244716.1
------> Galactic l: 0
------> Galactic b: 0
------> Name: J1748-2446N
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 16777216
------> Trial dispersion measure: 244.8 cm^-3 pc
------> Scale factor: 1.01616
[10:06:09][18500][INFO ] Seed for random number generator is -1031807597.
[10:06:14][18500][ERROR] Couldn't allocate 7706856 bytes of CUDA HS summing memory (error: 2)!
[10:06:14][18500][ERROR] Demodulation failed (error: 1006)!
[10:06:14][18500][WARN ] Sorry, at the moment your system doesn't have enough free CPU/GPU memory to run this task!
------> Returning control to BOINC, delaying next attempt for at least 15 minutes...
------> If this problem persists you should consider aborting this task...
Activated exception handling...
[10:21:15][1976][INFO ] Starting data processing...
[10:21:15][1976][INFO ] CUDA device ID: 0
[10:21:15][1976][INFO ] CUDA global memory status (initial GPU state, including context):
------> Used in total: 185 MB (840 MB free / 1025 MB total) -> Used by this application (assuming a single GPU task): 0 MB
[10:21:15][1976][INFO ] Using CUDA device #0 "Quadro K600" (192 CUDA cores / 336.19 GFLOPS)
[10:21:15][1976][INFO ] Version of installed CUDA driver: 9010
[10:21:15][1976][INFO ] Version of CUDA driver API used: 3020
[10:21:49][1976][INFO ] Number of generated templates to be used: 50000
[10:21:49][1976][INFO ] Continuing work on ../../projects/einstein.phys.uwm.edu/Ter5_1_dns_cfbf00110_segment_6_dms_200_148.binary at template no. 8947
[10:21:49][1976][INFO ] Header contents:
------> Original WAPP file: /atlas/data/TRAPUM_GC/Terzan5/epoch1/30min_segments/dedispersed_files/cfbf00110/Ter5_1_dns_cfbf00110_segment_6_dms_200_DM244.80
------> Sample time in microseconds: 153.121
------> Observation time in seconds: 2568.9524
------> Time stamp (MJD): 59097.776208975287
------> Number of samples/record: 0
------> Center freq in MHz: 857.5673828
------> Channel band in MHz: 3.34375
------> Number of channels/record: 256
------> Nifs: 1
------> RA (J2000): 174801.97
------> DEC (J2000): -244716.1
------> Galactic l: 0
------> Galactic b: 0
------> Name: J1748-2446N
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 16777216
------> Trial dispersion measure: 244.8 cm^-3 pc
------> Scale factor: 1.01616
[10:21:49][1976][INFO ] Seed for random number generator is -1031807597.
[10:21:54][1976][ERROR] Couldn't allocate 7706856 bytes of CUDA HS summing memory (error: 2)!
[10:21:54][1976][ERROR] Demodulation failed (error: 1006)!
[10:21:54][1976][WARN ] Sorry, at the moment your system doesn't have enough free CPU/GPU memory to run this task!
------> Returning control to BOINC, delaying next attempt for at least 15 minutes...
------> If this problem persists you should consider aborting this task...
Activated exception handling...
[10:22:15][14100][INFO ] Starting data processing...
[10:22:15][14100][INFO ] CUDA device ID: 0
[10:22:15][14100][INFO ] CUDA global memory status (initial GPU state, including context):
------> Used in total: 185 MB (840 MB free / 1025 MB total) -> Used by this application (assuming a single GPU task): 0 MB
[10:22:15][14100][INFO ] Using CUDA device #0 "Quadro K600" (192 CUDA cores / 336.19 GFLOPS)
[10:22:15][14100][INFO ] Version of installed CUDA driver: 9010
[10:22:15][14100][INFO ] Version of CUDA driver API used: 3020
Activated exception handling...
[10:36:58][19584][INFO ] Starting data processing...
[10:36:58][19584][INFO ] CUDA device ID: 0
[10:36:58][19584][INFO ] CUDA global memory status (initial GPU state, including context):
------> Used in total: 185 MB (840 MB free / 1025 MB total) -> Used by this application (assuming a single GPU task): 0 MB
[10:36:58][19584][INFO ] Using CUDA device #0 "Quadro K600" (192 CUDA cores / 336.19 GFLOPS)
[10:36:58][19584][INFO ] Version of installed CUDA driver: 9010
[10:36:58][19584][INFO ] Version of CUDA driver API used: 3020
[10:37:32][19584][INFO ] Number of generated templates to be used: 50000
[10:37:32][19584][INFO ] Continuing work on ../../projects/einstein.phys.uwm.edu/Ter5_1_dns_cfbf00110_segment_6_dms_200_148.binary at template no. 8947
[10:37:32][19584][INFO ] Header contents:
------> Original WAPP file: /atlas/data/TRAPUM_GC/Terzan5/epoch1/30min_segments/dedispersed_files/cfbf00110/Ter5_1_dns_cfbf00110_segment_6_dms_200_DM244.80
------> Sample time in microseconds: 153.121
------> Observation time in seconds: 2568.9524
------> Time stamp (MJD): 59097.776208975287
------> Number of samples/record: 0
------> Center freq in MHz: 857.5673828
------> Channel band in MHz: 3.34375
------> Number of channels/record: 256
------> Nifs: 1
------> RA (J2000): 174801.97
------> DEC (J2000): -244716.1
------> Galactic l: 0
------> Galactic b: 0
------> Name: J1748-2446N
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 16777216
------> Trial dispersion measure: 244.8 cm^-3 pc
------> Scale factor: 1.01616
[10:37:32][19584][INFO ] Seed for random number generator is -1031807597.
[10:37:37][19584][ERROR] Couldn't allocate 7706856 bytes of CUDA HS summing memory (error: 2)!
[10:37:37][19584][ERROR] Demodulation failed (error: 1006)!
[10:37:37][19584][WARN ] Sorry, at the moment your system doesn't have enough free CPU/GPU memory to run this task!
------> Returning control to BOINC, delaying next attempt for at least 15 minutes...
------> If this problem persists you should consider aborting this task...
Activated exception handling...
[10:52:40][11012][INFO ] Starting data processing...
[10:52:40][11012][INFO ] CUDA device ID: 0
[10:52:40][11012][INFO ] CUDA global memory status (initial GPU state, including context):
------> Used in total: 185 MB (840 MB free / 1025 MB total) -> Used by this application (assuming a single GPU task): 0 MB
[10:52:40][11012][INFO ] Using CUDA device #0 "Quadro K600" (192 CUDA cores / 336.19 GFLOPS)
[10:52:40][11012][INFO ] Version of installed CUDA driver: 9010
[10:52:40][11012][INFO ] Version of CUDA driver API used: 3020
[10:53:14][11012][INFO ] Number of generated templates to be used: 50000
[10:53:14][11012][INFO ] Continuing work on ../../projects/einstein.phys.uwm.edu/Ter5_1_dns_cfbf00110_segment_6_dms_200_148.binary at template no. 8947
[10:53:14][11012][INFO ] Header contents:
------> Original WAPP file: /atlas/data/TRAPUM_GC/Terzan5/epoch1/30min_segments/dedispersed_files/cfbf00110/Ter5_1_dns_cfbf00110_segment_6_dms_200_DM244.80
------> Sample time in microseconds: 153.121
------> Observation time in seconds: 2568.9524
------> Time stamp (MJD): 59097.776208975287
------> Number of samples/record: 0
------> Center freq in MHz: 857.5673828
------> Channel band in MHz: 3.34375
------> Number of channels/record: 256
------> Nifs: 1
------> RA (J2000): 174801.97
------> DEC (J2000): -244716.1
------> Galactic l: 0
------> Galactic b: 0
------> Name: J1748-2446N
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 16777216
------> Trial dispersion measure: 244.8 cm^-3 pc
------> Scale factor: 1.01616
[10:53:14][11012][INFO ] Seed for random number generator is -1031807597.
[10:53:19][11012][ERROR] Couldn't allocate 7706856 bytes of CUDA HS summing memory (error: 2)!
[10:53:19][11012][ERROR] Demodulation failed (error: 1006)!
[10:53:19][11012][WARN ] Sorry, at the moment your system doesn't have enough free CPU/GPU memory to run this task!
------> Returning control to BOINC, delaying next attempt for at least 15 minutes...
------> If this problem persists you should consider aborting this task...
Activated exception handling...
[11:08:22][9660][INFO ] Starting data processing...
[11:08:22][9660][INFO ] CUDA device ID: 0
[11:08:22][9660][INFO ] CUDA global memory status (initial GPU state, including context):
------> Used in total: 185 MB (840 MB free / 1025 MB total) -> Used by this application (assuming a single GPU task): 0 MB
[11:08:22][9660][INFO ] Using CUDA device #0 "Quadro K600" (192 CUDA cores / 336.19 GFLOPS)
[11:08:22][9660][INFO ] Version of installed CUDA driver: 9010
[11:08:22][9660][INFO ] Version of CUDA driver API used: 3020
[11:08:57][9660][INFO ] Number of generated templates to be used: 50000
[11:08:57][9660][INFO ] Continuing work on ../../projects/einstein.phys.uwm.edu/Ter5_1_dns_cfbf00110_segment_6_dms_200_148.binary at template no. 8947
[11:08:57][9660][INFO ] Header contents:
------> Original WAPP file: /atlas/data/TRAPUM_GC/Terzan5/epoch1/30min_segments/dedispersed_files/cfbf00110/Ter5_1_dns_cfbf00110_segment_6_dms_200_DM244.80
------> Sample time in microseconds: 153.121
------> Observation time in seconds: 2568.9524
------> Time stamp (MJD): 59097.776208975287
------> Number of samples/record: 0
------> Center freq in MHz: 857.5673828
------> Channel band in MHz: 3.34375
------> Number of channels/record: 256
------> Nifs: 1
------> RA (J2000): 174801.97
------> DEC (J2000): -244716.1
------> Galactic l: 0
------> Galactic b: 0
------> Name: J1748-2446N
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 16777216
------> Trial dispersion measure: 244.8 cm^-3 pc
------> Scale factor: 1.01616
[11:08:57][9660][INFO ] Seed for random number generator is -1031807597.
[11:09:01][9660][ERROR] Couldn't allocate 7706856 bytes of CUDA HS summing memory (error: 2)!
[11:09:01][9660][ERROR] Demodulation failed (error: 1006)!
[11:09:01][9660][WARN ] Sorry, at the moment your system doesn't have enough free CPU/GPU memory to run this task!
------> Returning control to BOINC, delaying next attempt for at least 15 minutes...
------> If this problem persists you should consider aborting this task...
Activated exception handling...
[11:24:05][12428][INFO ] Starting data processing...
[11:24:06][12428][INFO ] CUDA device ID: 0
[11:24:06][12428][INFO ] CUDA global memory status (initial GPU state, including context):
------> Used in total: 185 MB (840 MB free / 1025 MB total) -> Used by this application (assuming a single GPU task): 0 MB
[11:24:06][12428][INFO ] Using CUDA device #0 "Quadro K600" (192 CUDA cores / 336.19 GFLOPS)
[11:24:06][12428][INFO ] Version of installed CUDA driver: 9010
[11:24:06][12428][INFO ] Version of CUDA driver API used: 3020
[11:24:40][12428][INFO ] Number of generated templates to be used: 50000
[11:24:40][12428][INFO ] Continuing work on ../../projects/einstein.phys.uwm.edu/Ter5_1_dns_cfbf00110_segment_6_dms_200_148.binary at template no. 8947
[11:24:40][12428][INFO ] Header contents:
------> Original WAPP file: /atlas/data/TRAPUM_GC/Terzan5/epoch1/30min_segments/dedispersed_files/cfbf00110/Ter5_1_dns_cfbf00110_segment_6_dms_200_DM244.80
------> Sample time in microseconds: 153.121
------> Observation time in seconds: 2568.9524
------> Time stamp (MJD): 59097.776208975287
------> Number of samples/record: 0
------> Center freq in MHz: 857.5673828
------> Channel band in MHz: 3.34375
------> Number of channels/record: 256
------> Nifs: 1
------> RA (J2000): 174801.97
------> DEC (J2000): -244716.1
------> Galactic l: 0
------> Galactic b: 0
------> Name: J1748-2446N
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 16777216
------> Trial dispersion measure: 244.8 cm^-3 pc
------> Scale factor: 1.01616
[11:24:40][12428][INFO ] Seed for random number generator is -1031807597.
[11:24:44][12428][ERROR] Couldn't allocate 7706856 bytes of CUDA HS summing memory (error: 2)!
[11:24:44][12428][ERROR] Demodulation failed (error: 1006)!
[11:24:44][12428][WARN ] Sorry, at the moment your system doesn't have enough free CPU/GPU memory to run this task!
------> Returning control to BOINC, delaying next attempt for at least 15 minutes...
------> If this problem persists you should consider aborting this task...
Activated exception handling...
[11:39:58][5196][INFO ] Starting data processing...
[11:39:58][5196][INFO ] CUDA device ID: 0
[11:39:58][5196][INFO ] CUDA global memory status (initial GPU state, including context):
------> Used in total: 185 MB (840 MB free / 1025 MB total) -> Used by this application (assuming a single GPU task): 0 MB
[11:39:58][5196][INFO ] Using CUDA device #0 "Quadro K600" (192 CUDA cores / 336.19 GFLOPS)
[11:39:58][5196][INFO ] Version of installed CUDA driver: 9010
[11:39:58][5196][INFO ] Version of CUDA driver API used: 3020
[11:40:12][5196][INFO ] Number of generated templates to be used: 50000
[11:40:12][5196][INFO ] Continuing work on ../../projects/einstein.phys.uwm.edu/Ter5_1_dns_cfbf00110_segment_6_dms_200_148.binary at template no. 8947
[11:40:12][5196][INFO ] Header contents:
------> Original WAPP file: /atlas/data/TRAPUM_GC/Terzan5/epoch1/30min_segments/dedispersed_files/cfbf00110/Ter5_1_dns_cfbf00110_segment_6_dms_200_DM244.80
------> Sample time in microseconds: 153.121
------> Observation time in seconds: 2568.9524
------> Time stamp (MJD): 59097.776208975287
------> Number of samples/record: 0
------> Center freq in MHz: 857.5673828
------> Channel band in MHz: 3.34375
------> Number of channels/record: 256
------> Nifs: 1
------> RA (J2000): 174801.97
------> DEC (J2000): -244716.1
------> Galactic l: 0
------> Galactic b: 0
------> Name: J1748-2446N
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 16777216
------> Trial dispersion measure: 244.8 cm^-3 pc
------> Scale factor: 1.01616
[11:40:12][5196][INFO ] Seed for random number generator is -1031807597.
[11:40:17][5196][ERROR] Couldn't allocate 7706856 bytes of CUDA HS summing memory (error: 2)!
[11:40:17][5196][ERROR] Demodulation failed (error: 1006)!
[11:40:17][5196][WARN ] Sorry, at the moment your system doesn't have enough free CPU/GPU memory to run this task!
------> Returning control to BOINC, delaying next attempt for at least 15 minutes...
------> If this problem persists you should consider aborting this task...
Activated exception handling...
[11:57:21][20856][INFO ] Starting data processing...
[11:57:21][20856][INFO ] CUDA device ID: 0
[11:57:21][20856][INFO ] CUDA global memory status (initial GPU state, including context):
------> Used in total: 185 MB (840 MB free / 1025 MB total) -> Used by this application (assuming a single GPU task): 0 MB
[11:57:21][20856][INFO ] Using CUDA device #0 "Quadro K600" (192 CUDA cores / 336.19 GFLOPS)
[11:57:21][20856][INFO ] Version of installed CUDA driver: 9010
[11:57:21][20856][INFO ] Version of CUDA driver API used: 3020
[11:57:35][20856][INFO ] Number of generated templates to be used: 50000
[11:57:35][20856][INFO ] Continuing work on ../../projects/einstein.phys.uwm.edu/Ter5_1_dns_cfbf00110_segment_6_dms_200_148.binary at template no. 8947
[11:57:35][20856][INFO ] Header contents:
------> Original WAPP file: /atlas/data/TRAPUM_GC/Terzan5/epoch1/30min_segments/dedispersed_files/cfbf00110/Ter5_1_dns_cfbf00110_segment_6_dms_200_DM244.80
------> Sample time in microseconds: 153.121
------> Observation time in seconds: 2568.9524
------> Time stamp (MJD): 59097.776208975287
------> Number of samples/record: 0
------> Center freq in MHz: 857.5673828
------> Channel band in MHz: 3.34375
------> Number of channels/record: 256
------> Nifs: 1
------> RA (J2000): 174801.97
------> DEC (J2000): -244716.1
------> Galactic l: 0
------> Galactic b: 0
------> Name: J1748-2446N
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 16777216
------> Trial dispersion measure: 244.8 cm^-3 pc
------> Scale factor: 1.01616
[11:57:35][20856][INFO ] Seed for random number generator is -1031807597.
[11:57:40][20856][ERROR] Couldn't allocate 7706856 bytes of CUDA HS summing memory (error: 2)!
[11:57:40][20856][ERROR] Demodulation failed (error: 1006)!
[11:57:40][20856][WARN ] Sorry, at the moment your system doesn't have enough free CPU/GPU memory to run this task!
------> Returning control to BOINC, delaying next attempt for at least 15 minutes...
------> If this problem persists you should consider aborting this task...
Activated exception handling...
[12:32:04][20268][INFO ] Starting data processing...
[12:32:04][20268][INFO ] CUDA device ID: 0
[12:32:04][20268][INFO ] CUDA global memory status (initial GPU state, including context):
------> Used in total: 185 MB (840 MB free / 1025 MB total) -> Used by this application (assuming a single GPU task): 0 MB
[12:32:04][20268][INFO ] Using CUDA device #0 "Quadro K600" (192 CUDA cores / 336.19 GFLOPS)
[12:32:04][20268][INFO ] Version of installed CUDA driver: 9010
[12:32:04][20268][INFO ] Version of CUDA driver API used: 3020
[12:32:19][20268][INFO ] Number of generated templates to be used: 50000
[12:32:19][20268][INFO ] Continuing work on ../../projects/einstein.phys.uwm.edu/Ter5_1_dns_cfbf00110_segment_6_dms_200_148.binary at template no. 8947
[12:32:19][20268][INFO ] Header contents:
------> Original WAPP file: /atlas/data/TRAPUM_GC/Terzan5/epoch1/30min_segments/dedispersed_files/cfbf00110/Ter5_1_dns_cfbf00110_segment_6_dms_200_DM244.80
------> Sample time in microseconds: 153.121
------> Observation time in seconds: 2568.9524
------> Time stamp (MJD): 59097.776208975287
------> Number of samples/record: 0
------> Center freq in MHz: 857.5673828
------> Channel band in MHz: 3.34375
------> Number of channels/record: 256
------> Nifs: 1
------> RA (J2000): 174801.97
------> DEC (J2000): -244716.1
------> Galactic l: 0
------> Galactic b: 0
------> Name: J1748-2446N
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 16777216
------> Trial dispersion measure: 244.8 cm^-3 pc
------> Scale factor: 1.01616
[12:32:19][20268][INFO ] Seed for random number generator is -1031807597.
[12:32:23][20268][INFO ] Derived global search parameters:
------> f_A probability = 0.04
------> single bin prob(P_noise > P_thr) = 3.24424e-009
------> thr1 = 19.5464
------> thr2 = 22.7124
------> thr4 = 27.844
------> thr8 = 36.3869
------> thr16 = 50.9469
[12:32:23][20268][INFO ] CUDA global memory status (GPU setup complete):
------> Used in total: 597 MB (428 MB free / 1025 MB total) -> Used by this application (assuming a single GPU task): 412 MB
[12:33:05][20268][INFO ] Checkpoint committed!
[12:34:05][20268][INFO ] Checkpoint committed!
[12:35:05][20268][INFO ] Checkpoint committed!
[12:36:06][20268][INFO ] Checkpoint committed!
[12:37:06][20268][INFO ] Checkpoint committed!
[12:38:06][20268][INFO ] Checkpoint committed!
[12:39:06][20268][INFO ] Checkpoint committed!
[12:40:06][20268][INFO ] Checkpoint committed!
[12:41:06][20268][INFO ] Checkpoint committed!
[12:42:07][20268][INFO ] Checkpoint committed!
[12:43:07][20268][INFO ] Checkpoint committed!
[12:44:08][20268][INFO ] Checkpoint committed!
[12:45:08][20268][INFO ] Checkpoint committed!
[12:46:08][20268][INFO ] Checkpoint committed!
[12:47:08][20268][INFO ] Checkpoint committed!
[12:48:08][20268][INFO ] Checkpoint committed!
[12:49:09][20268][INFO ] Checkpoint committed!
[12:50:09][20268][INFO ] Checkpoint committed!
[12:51:09][20268][INFO ] Checkpoint committed!
[12:52:10][20268][INFO ] Checkpoint committed!
[12:53:10][20268][INFO ] Checkpoint committed!
[12:54:10][20268][INFO ] Checkpoint committed!
[12:55:10][20268][INFO ] Checkpoint committed!
[12:56:11][20268][INFO ] Checkpoint committed!
[12:57:11][20268][INFO ] Checkpoint committed!
[12:58:11][20268][INFO ] Checkpoint committed!
[12:59:11][20268][INFO ] Checkpoint committed!
[13:00:12][20268][INFO ] Checkpoint committed!
[13:01:12][20268][INFO ] Checkpoint committed!
[13:02:12][20268][INFO ] Checkpoint committed!
[13:03:12][20268][INFO ] Checkpoint committed!
[13:04:12][20268][INFO ] Checkpoint committed!
[13:05:13][20268][INFO ] Checkpoint committed!
[13:06:13][20268][INFO ] Checkpoint committed!
[13:07:13][20268][INFO ] Checkpoint committed!
[13:08:13][20268][INFO ] Checkpoint committed!
[13:09:14][20268][INFO ] Checkpoint committed!
[13:10:14][20268][INFO ] Checkpoint committed!
[13:11:14][20268][INFO ] Checkpoint committed!
[13:12:14][20268][INFO ] Checkpoint committed!
[13:13:15][20268][INFO ] Checkpoint committed!
[13:14:15][20268][INFO ] Checkpoint committed!
[13:15:15][20268][INFO ] Checkpoint committed!
[13:16:15][20268][INFO ] Checkpoint committed!
[13:17:16][20268][INFO ] Checkpoint committed!
[13:18:16][20268][INFO ] Checkpoint committed!
[13:19:16][20268][INFO ] Checkpoint committed!
[13:20:16][20268][INFO ] Checkpoint committed!
[13:21:16][20268][INFO ] Checkpoint committed!
[13:22:17][20268][INFO ] Checkpoint committed!
[13:23:17][20268][INFO ] Checkpoint committed!
[13:24:17][20268][INFO ] Checkpoint committed!
[13:25:18][20268][INFO ] Checkpoint committed!
[13:26:18][20268][INFO ] Checkpoint committed!
[13:27:18][20268][INFO ] Checkpoint committed!
[13:28:18][20268][INFO ] Checkpoint committed!
[13:29:18][20268][INFO ] Checkpoint committed!
[13:30:19][20268][INFO ] Checkpoint committed!
[13:31:19][20268][INFO ] Checkpoint committed!
[13:32:19][20268][INFO ] Checkpoint committed!
[13:33:19][20268][INFO ] Checkpoint committed!
[13:34:20][20268][INFO ] Checkpoint committed!
[13:35:20][20268][INFO ] Checkpoint committed!
[13:36:20][20268][INFO ] Checkpoint committed!
[13:37:21][20268][INFO ] Checkpoint committed!
[13:38:21][20268][INFO ] Checkpoint committed!
Activated exception handling...
[13:50:19][9484][INFO ] Starting data processing...
[13:50:19][9484][INFO ] CUDA device ID: 0
[13:50:19][9484][INFO ] CUDA global memory status (initial GPU state, including context):
------> Used in total: 185 MB (840 MB free / 1025 MB total) -> Used by this application (assuming a single GPU task): 0 MB
[13:50:19][9484][INFO ] Using CUDA device #0 "Quadro K600" (192 CUDA cores / 336.19 GFLOPS)
[13:50:19][9484][INFO ] Version of installed CUDA driver: 9010
[13:50:19][9484][INFO ] Version of CUDA driver API used: 3020
[13:50:33][9484][INFO ] Number of generated templates to be used: 50000
[13:50:33][9484][INFO ] Continuing work on ../../projects/einstein.phys.uwm.edu/Ter5_1_dns_cfbf00110_segment_6_dms_200_148.binary at template no. 22332
[13:50:33][9484][INFO ] Header contents:
------> Original WAPP file: /atlas/data/TRAPUM_GC/Terzan5/epoch1/30min_segments/dedispersed_files/cfbf00110/Ter5_1_dns_cfbf00110_segment_6_dms_200_DM244.80
------> Sample time in microseconds: 153.121
------> Observation time in seconds: 2568.9524
------> Time stamp (MJD): 59097.776208975287
------> Number of samples/record: 0
------> Center freq in MHz: 857.5673828
------> Channel band in MHz: 3.34375
------> Number of channels/record: 256
------> Nifs: 1
------> RA (J2000): 174801.97
------> DEC (J2000): -244716.1
------> Galactic l: 0
------> Galactic b: 0
------> Name: J1748-2446N
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 16777216
------> Trial dispersion measure: 244.8 cm^-3 pc
------> Scale factor: 1.01616
[13:50:34][9484][INFO ] Seed for random number generator is -1031807597.
[13:50:38][9484][INFO ] Derived global search parameters:
------> f_A probability = 0.04
------> single bin prob(P_noise > P_thr) = 3.24424e-009
------> thr1 = 19.5464
------> thr2 = 22.7124
------> thr4 = 27.844
------> thr8 = 36.3869
------> thr16 = 50.9469
[13:50:38][9484][INFO ] CUDA global memory status (GPU setup complete):
------> Used in total: 1025 MB (0 MB free / 1025 MB total) -> Used by this application (assuming a single GPU task): 840 MB
[13:50:38][9484][ERROR] Couldn't allocate 37640 bytes of CUDA HS summing memory (error: 2)!
[13:50:38][9484][ERROR] Demodulation failed (error: 1006)!
[13:50:38][9484][WARN ] Sorry, at the moment your system doesn't have enough free CPU/GPU memory to run this task!
------> Returning control to BOINC, delaying next attempt for at least 15 minutes...
------> If this problem persists you should consider aborting this task...
Activated exception handling...
[14:06:05][17156][INFO ] Starting data processing...
[14:06:05][17156][INFO ] CUDA device ID: 0
[14:06:05][17156][INFO ] CUDA global memory status (initial GPU state, including context):
------> Used in total: 185 MB (840 MB free / 1025 MB total) -> Used by this application (assuming a single GPU task): 0 MB
[14:06:05][17156][INFO ] Using CUDA device #0 "Quadro K600" (192 CUDA cores / 336.19 GFLOPS)
[14:06:05][17156][INFO ] Version of installed CUDA driver: 9010
[14:06:05][17156][INFO ] Version of CUDA driver API used: 3020
[14:06:40][17156][INFO ] Number of generated templates to be used: 50000
[14:06:40][17156][INFO ] Continuing work on ../../projects/einstein.phys.uwm.edu/Ter5_1_dns_cfbf00110_segment_6_dms_200_148.binary at template no. 22332
[14:06:40][17156][INFO ] Header contents:
------> Original WAPP file: /atlas/data/TRAPUM_GC/Terzan5/epoch1/30min_segments/dedispersed_files/cfbf00110/Ter5_1_dns_cfbf00110_segment_6_dms_200_DM244.80
------> Sample time in microseconds: 153.121
------> Observation time in seconds: 2568.9524
------> Time stamp (MJD): 59097.776208975287
------> Number of samples/record: 0
------> Center freq in MHz: 857.5673828
------> Channel band in MHz: 3.34375
------> Number of channels/record: 256
------> Nifs: 1
------> RA (J2000): 174801.97
------> DEC (J2000): -244716.1
------> Galactic l: 0
------> Galactic b: 0
------> Name: J1748-2446N
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 16777216
------> Trial dispersion measure: 244.8 cm^-3 pc
------> Scale factor: 1.01616
[14:06:40][17156][INFO ] Seed for random number generator is -1031807597.
[14:06:45][17156][INFO ] Derived global search parameters:
------> f_A probability = 0.04
------> single bin prob(P_noise > P_thr) = 3.24424e-009
------> thr1 = 19.5464
------> thr2 = 22.7124
------> thr4 = 27.844
------> thr8 = 36.3869
------> thr16 = 50.9469
[14:06:45][17156][INFO ] CUDA global memory status (GPU setup complete):
------> Used in total: 597 MB (428 MB free / 1025 MB total) -> Used by this application (assuming a single GPU task): 412 MB
[14:07:06][17156][INFO ] Checkpoint committed!
[14:08:06][17156][INFO ] Checkpoint committed!
[14:09:06][17156][INFO ] Checkpoint committed!
[14:10:06][17156][INFO ] Checkpoint committed!
[14:11:06][17156][INFO ] Checkpoint committed!
[14:12:07][17156][INFO ] Checkpoint committed!
[14:13:07][17156][INFO ] Checkpoint committed!
[14:14:07][17156][INFO ] Checkpoint committed!
[14:15:07][17156][INFO ] Checkpoint committed!
[14:16:07][17156][INFO ] Checkpoint committed!
[14:17:08][17156][INFO ] Checkpoint committed!
[14:18:08][17156][INFO ] Checkpoint committed!
[14:19:08][17156][INFO ] Checkpoint committed!
[14:20:09][17156][INFO ] Checkpoint committed!
[14:21:09][17156][INFO ] Checkpoint committed!
[14:22:09][17156][INFO ] Checkpoint committed!
[14:23:09][17156][INFO ] Checkpoint committed!
[14:24:09][17156][INFO ] Checkpoint committed!
[14:25:10][17156][INFO ] Checkpoint committed!
[14:26:10][17156][INFO ] Checkpoint committed!
[14:27:10][17156][INFO ] Checkpoint committed!
[14:28:11][17156][INFO ] Checkpoint committed!
[14:29:11][17156][INFO ] Checkpoint committed!
[14:30:11][17156][INFO ] Checkpoint committed!
[14:31:11][17156][INFO ] Checkpoint committed!
[14:32:12][17156][INFO ] Checkpoint committed!
[14:33:12][17156][INFO ] Checkpoint committed!
[14:34:12][17156][INFO ] Checkpoint committed!
[14:35:12][17156][INFO ] Checkpoint committed!
[14:36:12][17156][INFO ] Checkpoint committed!
[14:37:13][17156][INFO ] Checkpoint committed!
[14:38:13][17156][INFO ] Checkpoint committed!
[14:39:13][17156][INFO ] Checkpoint committed!
[14:40:14][17156][INFO ] Checkpoint committed!
[14:41:14][17156][INFO ] Checkpoint committed!
[14:42:14][17156][INFO ] Checkpoint committed!
[14:43:14][17156][INFO ] Checkpoint committed!
[14:44:15][17156][INFO ] Checkpoint committed!
Activated exception handling...
[14:47:34][17824][INFO ] Starting data processing...
[14:47:34][17824][INFO ] CUDA device ID: 0
[14:47:34][17824][INFO ] CUDA global memory status (initial GPU state, including context):
------> Used in total: 185 MB (840 MB free / 1025 MB total) -> Used by this application (assuming a single GPU task): 0 MB
[14:47:34][17824][INFO ] Using CUDA device #0 "Quadro K600" (192 CUDA cores / 336.19 GFLOPS)
[14:47:34][17824][INFO ] Version of installed CUDA driver: 9010
[14:47:34][17824][INFO ] Version of CUDA driver API used: 3020
[14:47:49][17824][INFO ] Number of generated templates to be used: 50000
[14:47:49][17824][INFO ] Continuing work on ../../projects/einstein.phys.uwm.edu/Ter5_1_dns_cfbf00110_segment_6_dms_200_148.binary at template no. 30881
[14:47:49][17824][INFO ] Header contents:
------> Original WAPP file: /atlas/data/TRAPUM_GC/Terzan5/epoch1/30min_segments/dedispersed_files/cfbf00110/Ter5_1_dns_cfbf00110_segment_6_dms_200_DM244.80
------> Sample time in microseconds: 153.121
------> Observation time in seconds: 2568.9524
------> Time stamp (MJD): 59097.776208975287
------> Number of samples/record: 0
------> Center freq in MHz: 857.5673828
------> Channel band in MHz: 3.34375
------> Number of channels/record: 256
------> Nifs: 1
------> RA (J2000): 174801.97
------> DEC (J2000): -244716.1
------> Galactic l: 0
------> Galactic b: 0
------> Name: J1748-2446N
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 16777216
------> Trial dispersion measure: 244.8 cm^-3 pc
------> Scale factor: 1.01616
[14:47:49][17824][INFO ] Seed for random number generator is -1031807597.
[14:47:53][17824][INFO ] Derived global search parameters:
------> f_A probability = 0.04
------> single bin prob(P_noise > P_thr) = 3.24424e-009
------> thr1 = 19.5464
------> thr2 = 22.7124
------> thr4 = 27.844
------> thr8 = 36.3869
------> thr16 = 50.9469
[14:47:53][17824][INFO ] CUDA global memory status (GPU setup complete):
------> Used in total: 597 MB (428 MB free / 1025 MB total) -> Used by this application (assuming a single GPU task): 412 MB
[14:48:34][17824][INFO ] Checkpoint committed!
[14:49:34][17824][INFO ] Checkpoint committed!
[14:50:35][17824][INFO ] Checkpoint committed!
[14:51:35][17824][INFO ] Checkpoint committed!
[14:52:35][17824][INFO ] Checkpoint committed!
[14:53:36][17824][INFO ] Checkpoint committed!
[14:54:36][17824][INFO ] Checkpoint committed!
[14:55:36][17824][INFO ] Checkpoint committed!
[14:56:36][17824][INFO ] Checkpoint committed!
[14:57:36][17824][INFO ] Checkpoint committed!
[14:58:37][17824][INFO ] Checkpoint committed!
[14:59:37][17824][INFO ] Checkpoint committed!
[15:00:37][17824][INFO ] Checkpoint committed!
[15:01:38][17824][INFO ] Checkpoint committed!
[15:02:38][17824][INFO ] Checkpoint committed!
[15:03:38][17824][INFO ] Checkpoint committed!
[15:04:38][17824][INFO ] Checkpoint committed!
[15:05:38][17824][INFO ] Checkpoint committed!
[15:06:39][17824][INFO ] Checkpoint committed!
[15:07:39][17824][INFO ] Checkpoint committed!
[15:08:40][17824][INFO ] Checkpoint committed!
[15:09:40][17824][INFO ] Checkpoint committed!
[15:10:40][17824][INFO ] Checkpoint committed!
[15:11:40][17824][INFO ] Checkpoint committed!
[15:12:40][17824][INFO ] Checkpoint committed!
[15:13:41][17824][INFO ] Checkpoint committed!
[15:14:41][17824][INFO ] Checkpoint committed!
[15:15:41][17824][INFO ] Checkpoint committed!
[15:16:41][17824][INFO ] Checkpoint committed!
[15:17:42][17824][INFO ] Checkpoint committed!
[15:18:42][17824][INFO ] Checkpoint committed!
[15:19:42][17824][INFO ] Checkpoint committed!
[15:20:43][17824][INFO ] Checkpoint committed!
[15:21:43][17824][INFO ] Checkpoint committed!
[15:22:43][17824][INFO ] Checkpoint committed!
[15:23:43][17824][INFO ] Checkpoint committed!
[15:24:44][17824][INFO ] Checkpoint committed!
[15:25:44][17824][INFO ] Checkpoint committed!
[15:26:44][17824][INFO ] Checkpoint committed!
[15:27:44][17824][INFO ] Checkpoint committed!
[15:28:45][17824][INFO ] Checkpoint committed!
[15:29:45][17824][INFO ] Checkpoint committed!
[15:30:46][17824][INFO ] Checkpoint committed!
[15:31:46][17824][INFO ] Checkpoint committed!
[15:32:46][17824][INFO ] Checkpoint committed!
[15:33:46][17824][INFO ] Checkpoint committed!
[15:34:46][17824][INFO ] Checkpoint committed!
[15:35:47][17824][INFO ] Checkpoint committed!
[15:36:47][17824][INFO ] Checkpoint committed!
[15:37:47][17824][INFO ] Checkpoint committed!
[15:38:47][17824][INFO ] Checkpoint committed!
[15:39:48][17824][INFO ] Checkpoint committed!
[15:40:48][17824][INFO ] Checkpoint committed!
[15:41:49][17824][INFO ] Checkpoint committed!
[15:42:49][17824][INFO ] Checkpoint committed!
[15:43:49][17824][INFO ] Checkpoint committed!
[15:44:49][17824][INFO ] Checkpoint committed!
[15:45:49][17824][INFO ] Checkpoint committed!
[15:46:50][17824][INFO ] Checkpoint committed!
[15:47:50][17824][INFO ] Checkpoint committed!
[15:48:51][17824][INFO ] Checkpoint committed!
[15:49:51][17824][INFO ] Checkpoint committed!
[15:50:51][17824][INFO ] Checkpoint committed!
[15:51:51][17824][INFO ] Checkpoint committed!
[15:52:51][17824][INFO ] Checkpoint committed!
[15:53:52][17824][INFO ] Checkpoint committed!
[15:54:52][17824][INFO ] Checkpoint committed!
[15:55:52][17824][INFO ] Checkpoint committed!
[15:56:53][17824][INFO ] Checkpoint committed!
[15:57:53][17824][INFO ] Checkpoint committed!
[15:58:53][17824][INFO ] Checkpoint committed!
[15:59:53][17824][INFO ] Checkpoint committed!
[16:00:54][17824][INFO ] Checkpoint committed!
[16:01:54][17824][INFO ] Checkpoint committed!
[16:02:54][17824][INFO ] Checkpoint committed!
[16:03:55][17824][INFO ] Checkpoint committed!
[16:04:55][17824][INFO ] Checkpoint committed!
[16:05:55][17824][INFO ] Checkpoint committed!
[16:06:55][17824][INFO ] Checkpoint committed!
[16:07:56][17824][INFO ] Checkpoint committed!
[16:08:56][17824][INFO ] Checkpoint committed!
Activated exception handling...
[16:12:47][3780][INFO ] Starting data processing...
[16:12:47][3780][INFO ] CUDA device ID: 0
[16:12:47][3780][INFO ] CUDA global memory status (initial GPU state, including context):
------> Used in total: 185 MB (840 MB free / 1025 MB total) -> Used by this application (assuming a single GPU task): 0 MB
[16:12:47][3780][INFO ] Using CUDA device #0 "Quadro K600" (192 CUDA cores / 336.19 GFLOPS)
[16:12:47][3780][INFO ] Version of installed CUDA driver: 9010
[16:12:47][3780][INFO ] Version of CUDA driver API used: 3020
[16:13:02][3780][INFO ] Number of generated templates to be used: 50000
[16:13:02][3780][INFO ] Continuing work on ../../projects/einstein.phys.uwm.edu/Ter5_1_dns_cfbf00110_segment_6_dms_200_148.binary at template no. 40915
[16:13:02][3780][INFO ] Header contents:
------> Original WAPP file: /atlas/data/TRAPUM_GC/Terzan5/epoch1/30min_segments/dedispersed_files/cfbf00110/Ter5_1_dns_cfbf00110_segment_6_dms_200_DM244.80
------> Sample time in microseconds: 153.121
------> Observation time in seconds: 2568.9524
------> Time stamp (MJD): 59097.776208975287
------> Number of samples/record: 0
------> Center freq in MHz: 857.5673828
------> Channel band in MHz: 3.34375
------> Number of channels/record: 256
------> Nifs: 1
------> RA (J2000): 174801.97
------> DEC (J2000): -244716.1
------> Galactic l: 0
------> Galactic b: 0
------> Name: J1748-2446N
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 16777216
------> Trial dispersion measure: 244.8 cm^-3 pc
------> Scale factor: 1.01616
[16:13:02][3780][INFO ] Seed for random number generator is -1031807597.
[16:13:06][3780][INFO ] Derived global search parameters:
------> f_A probability = 0.04
------> single bin prob(P_noise > P_thr) = 3.24424e-009
------> thr1 = 19.5464
------> thr2 = 22.7124
------> thr4 = 27.844
------> thr8 = 36.3869
------> thr16 = 50.9469
[16:13:06][3780][INFO ] CUDA global memory status (GPU setup complete):
------> Used in total: 938 MB (87 MB free / 1025 MB total) -> Used by this application (assuming a single GPU task): 753 MB
[16:13:47][3780][INFO ] Checkpoint committed!
[16:14:48][3780][INFO ] Checkpoint committed!
[16:15:48][3780][INFO ] Checkpoint committed!
[16:16:48][3780][INFO ] Checkpoint committed!
[16:17:48][3780][INFO ] Checkpoint committed!
[16:18:49][3780][INFO ] Checkpoint committed!
[16:19:49][3780][INFO ] Checkpoint committed!
[16:20:49][3780][INFO ] Checkpoint committed!
[16:21:49][3780][INFO ] Checkpoint committed!
[16:22:50][3780][INFO ] Checkpoint committed!
[16:23:50][3780][INFO ] Checkpoint committed!
[16:24:50][3780][INFO ] Checkpoint committed!
[16:25:51][3780][INFO ] Checkpoint committed!
[16:26:50][3780][INFO ] Checkpoint committed!
[16:27:51][3780][INFO ] Checkpoint committed!
[16:28:51][3780][INFO ] Checkpoint committed!
[16:29:51][3780][INFO ] Checkpoint committed!
[16:30:52][3780][INFO ] Checkpoint committed!
[16:31:52][3780][INFO ] Checkpoint committed!
[16:32:53][3780][INFO ] Checkpoint committed!
[16:33:52][3780][INFO ] Checkpoint committed!
[16:34:53][3780][INFO ] Checkpoint committed!
[16:35:53][3780][INFO ] Checkpoint committed!
[16:36:53][3780][INFO ] Checkpoint committed!
[16:37:42][3780][ERROR] Couldn't allocate 37640 bytes of CUDA HS summing memory (error: 2)!
[16:37:42][3780][ERROR] Demodulation failed (error: 1006)!
[16:37:42][3780][WARN ] Sorry, at the moment your system doesn't have enough free CPU/GPU memory to run this task!
------> Returning control to BOINC, delaying next attempt for at least 15 minutes...
------> If this problem persists you should consider aborting this task...
Activated exception handling...
[16:52:56][16300][INFO ] Starting data processing...
[16:52:56][16300][INFO ] CUDA device ID: 0
[16:52:57][16300][INFO ] CUDA global memory status (initial GPU state, including context):
------> Used in total: 185 MB (840 MB free / 1025 MB total) -> Used by this application (assuming a single GPU task): 0 MB
[16:52:57][16300][INFO ] Using CUDA device #0 "Quadro K600" (192 CUDA cores / 336.19 GFLOPS)
[16:52:57][16300][INFO ] Version of installed CUDA driver: 9010
[16:52:57][16300][INFO ] Version of CUDA driver API used: 3020
[16:53:30][16300][INFO ] Number of generated templates to be used: 50000
[16:53:30][16300][INFO ] Continuing work on ../../projects/einstein.phys.uwm.edu/Ter5_1_dns_cfbf00110_segment_6_dms_200_148.binary at template no. 43870
[16:53:30][16300][INFO ] Header contents:
------> Original WAPP file: /atlas/data/TRAPUM_GC/Terzan5/epoch1/30min_segments/dedispersed_files/cfbf00110/Ter5_1_dns_cfbf00110_segment_6_dms_200_DM244.80
------> Sample time in microseconds: 153.121
------> Observation time in seconds: 2568.9524
------> Time stamp (MJD): 59097.776208975287
------> Number of samples/record: 0
------> Center freq in MHz: 857.5673828
------> Channel band in MHz: 3.34375
------> Number of channels/record: 256
------> Nifs: 1
------> RA (J2000): 174801.97
------> DEC (J2000): -244716.1
------> Galactic l: 0
------> Galactic b: 0
------> Name: J1748-2446N
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 16777216
------> Trial dispersion measure: 244.8 cm^-3 pc
------> Scale factor: 1.01616
[16:53:30][16300][INFO ] Seed for random number generator is -1031807597.
[16:53:36][16300][INFO ] Derived global search parameters:
------> f_A probability = 0.04
------> single bin prob(P_noise > P_thr) = 3.24424e-009
------> thr1 = 19.5464
------> thr2 = 22.7124
------> thr4 = 27.844
------> thr8 = 36.3869
------> thr16 = 50.9469
[16:53:36][16300][INFO ] CUDA global memory status (GPU setup complete):
------> Used in total: 981 MB (44 MB free / 1025 MB total) -> Used by this application (assuming a single GPU task): 796 MB
[16:53:57][16300][INFO ] Checkpoint committed!
[16:54:27][16300][ERROR] Couldn't allocate 37640 bytes of CUDA HS summing memory (error: 2)!
[16:54:27][16300][ERROR] Demodulation failed (error: 1006)!
[16:54:27][16300][WARN ] Sorry, at the moment your system doesn't have enough free CPU/GPU memory to run this task!
------> Returning control to BOINC, delaying next attempt for at least 15 minutes...
------> If this problem persists you should consider aborting this task...
Activated exception handling...
[17:09:51][10904][INFO ] Starting data processing...
[17:09:52][10904][INFO ] CUDA device ID: 0
[17:09:52][10904][INFO ] CUDA global memory status (initial GPU state, including context):
------> Used in total: 185 MB (840 MB free / 1025 MB total) -> Used by this application (assuming a single GPU task): 0 MB
[17:09:52][10904][INFO ] Using CUDA device #0 "Quadro K600" (192 CUDA cores / 336.19 GFLOPS)
[17:09:52][10904][INFO ] Version of installed CUDA driver: 9010
[17:09:52][10904][INFO ] Version of CUDA driver API used: 3020
[17:10:05][10904][INFO ] Number of generated templates to be used: 50000
[17:10:05][10904][INFO ] Continuing work on ../../projects/einstein.phys.uwm.edu/Ter5_1_dns_cfbf00110_segment_6_dms_200_148.binary at template no. 43908
[17:10:05][10904][INFO ] Header contents:
------> Original WAPP file: /atlas/data/TRAPUM_GC/Terzan5/epoch1/30min_segments/dedispersed_files/cfbf00110/Ter5_1_dns_cfbf00110_segment_6_dms_200_DM244.80
------> Sample time in microseconds: 153.121
------> Observation time in seconds: 2568.9524
------> Time stamp (MJD): 59097.776208975287
------> Number of samples/record: 0
------> Center freq in MHz: 857.5673828
------> Channel band in MHz: 3.34375
------> Number of channels/record: 256
------> Nifs: 1
------> RA (J2000): 174801.97
------> DEC (J2000): -244716.1
------> Galactic l: 0
------> Galactic b: 0
------> Name: J1748-2446N
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 16777216
------> Trial dispersion measure: 244.8 cm^-3 pc
------> Scale factor: 1.01616
[17:10:05][10904][INFO ] Seed for random number generator is -1031807597.
[17:10:12][10904][INFO ] Derived global search parameters:
------> f_A probability = 0.04
------> single bin prob(P_noise > P_thr) = 3.24424e-009
------> thr1 = 19.5464
------> thr2 = 22.7124
------> thr4 = 27.844
------> thr8 = 36.3869
------> thr16 = 50.9469
[17:10:12][10904][INFO ] CUDA global memory status (GPU setup complete):
------> Used in total: 990 MB (35 MB free / 1025 MB total) -> Used by this application (assuming a single GPU task): 805 MB
[17:10:16][10904][ERROR] Couldn't allocate 37640 bytes of CUDA HS summing memory (error: 2)!
[17:10:16][10904][ERROR] Demodulation failed (error: 1006)!
[17:10:16][10904][WARN ] Sorry, at the moment your system doesn't have enough free CPU/GPU memory to run this task!
------> Returning control to BOINC, delaying next attempt for at least 15 minutes...
------> If this problem persists you should consider aborting this task...
Activated exception handling...
[17:25:33][5656][INFO ] Starting data processing...
[17:25:33][5656][INFO ] CUDA device ID: 0
[17:25:34][5656][INFO ] CUDA global memory status (initial GPU state, including context):
------> Used in total: 185 MB (840 MB free / 1025 MB total) -> Used by this application (assuming a single GPU task): 0 MB
[17:25:34][5656][INFO ] Using CUDA device #0 "Quadro K600" (192 CUDA cores / 336.19 GFLOPS)
[17:25:34][5656][INFO ] Version of installed CUDA driver: 9010
[17:25:34][5656][INFO ] Version of CUDA driver API used: 3020
[17:25:46][5656][INFO ] Number of generated templates to be used: 50000
[17:25:46][5656][INFO ] Continuing work on ../../projects/einstein.phys.uwm.edu/Ter5_1_dns_cfbf00110_segment_6_dms_200_148.binary at template no. 43908
[17:25:46][5656][INFO ] Header contents:
------> Original WAPP file: /atlas/data/TRAPUM_GC/Terzan5/epoch1/30min_segments/dedispersed_files/cfbf00110/Ter5_1_dns_cfbf00110_segment_6_dms_200_DM244.80
------> Sample time in microseconds: 153.121
------> Observation time in seconds: 2568.9524
------> Time stamp (MJD): 59097.776208975287
------> Number of samples/record: 0
------> Center freq in MHz: 857.5673828
------> Channel band in MHz: 3.34375
------> Number of channels/record: 256
------> Nifs: 1
------> RA (J2000): 174801.97
------> DEC (J2000): -244716.1
------> Galactic l: 0
------> Galactic b: 0
------> Name: J1748-2446N
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 16777216
------> Trial dispersion measure: 244.8 cm^-3 pc
------> Scale factor: 1.01616
[17:25:47][5656][INFO ] Seed for random number generator is -1031807597.
[17:25:53][5656][INFO ] Derived global search parameters:
------> f_A probability = 0.04
------> single bin prob(P_noise > P_thr) = 3.24424e-009
------> thr1 = 19.5464
------> thr2 = 22.7124
------> thr4 = 27.844
------> thr8 = 36.3869
------> thr16 = 50.9469
[17:25:53][5656][INFO ] CUDA global memory status (GPU setup complete):
------> Used in total: 1004 MB (21 MB free / 1025 MB total) -> Used by this application (assuming a single GPU task): 819 MB
[17:25:59][5656][ERROR] Couldn't allocate 37640 bytes of CUDA HS summing memory (error: 2)!
[17:25:59][5656][ERROR] Demodulation failed (error: 1006)!
[17:25:59][5656][WARN ] Sorry, at the moment your system doesn't have enough free CPU/GPU memory to run this task!
------> Returning control to BOINC, delaying next attempt for at least 15 minutes...
------> If this problem persists you should consider aborting this task...
Activated exception handling...
[17:41:25][9660][INFO ] Starting data processing...
[17:41:26][9660][INFO ] CUDA device ID: 0
[17:41:26][9660][INFO ] CUDA global memory status (initial GPU state, including context):
------> Used in total: 185 MB (840 MB free / 1025 MB total) -> Used by this application (assuming a single GPU task): 0 MB
[17:41:26][9660][INFO ] Using CUDA device #0 "Quadro K600" (192 CUDA cores / 336.19 GFLOPS)
[17:41:26][9660][INFO ] Version of installed CUDA driver: 9010
[17:41:26][9660][INFO ] Version of CUDA driver API used: 3020
[17:41:40][9660][INFO ] Number of generated templates to be used: 50000
[17:41:40][9660][INFO ] Continuing work on ../../projects/einstein.phys.uwm.edu/Ter5_1_dns_cfbf00110_segment_6_dms_200_148.binary at template no. 43908
[17:41:40][9660][INFO ] Header contents:
------> Original WAPP file: /atlas/data/TRAPUM_GC/Terzan5/epoch1/30min_segments/dedispersed_files/cfbf00110/Ter5_1_dns_cfbf00110_segment_6_dms_200_DM244.80
------> Sample time in microseconds: 153.121
------> Observation time in seconds: 2568.9524
------> Time stamp (MJD): 59097.776208975287
------> Number of samples/record: 0
------> Center freq in MHz: 857.5673828
------> Channel band in MHz: 3.34375
------> Number of channels/record: 256
------> Nifs: 1
------> RA (J2000): 174801.97
------> DEC (J2000): -244716.1
------> Galactic l: 0
------> Galactic b: 0
------> Name: J1748-2446N
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 16777216
------> Trial dispersion measure: 244.8 cm^-3 pc
------> Scale factor: 1.01616
[17:41:41][9660][INFO ] Seed for random number generator is -1031807597.
[17:41:45][9660][ERROR] Couldn't allocate 7706856 bytes of CUDA HS summing memory (error: 2)!
[17:41:45][9660][ERROR] Demodulation failed (error: 1006)!
[17:41:45][9660][WARN ] Sorry, at the moment your system doesn't have enough free CPU/GPU memory to run this task!
------> Returning control to BOINC, delaying next attempt for at least 15 minutes...
------> If this problem persists you should consider aborting this task...
Activated exception handling...
[17:56:51][15544][INFO ] Starting data processing...
[17:56:51][15544][INFO ] CUDA device ID: 0
[17:56:51][15544][INFO ] CUDA global memory status (initial GPU state, including context):
------> Used in total: 185 MB (840 MB free / 1025 MB total) -> Used by this application (assuming a single GPU task): 0 MB
[17:56:51][15544][INFO ] Using CUDA device #0 "Quadro K600" (192 CUDA cores / 336.19 GFLOPS)
[17:56:51][15544][INFO ] Version of installed CUDA driver: 9010
[17:56:51][15544][INFO ] Version of CUDA driver API used: 3020
[17:57:24][15544][INFO ] Number of generated templates to be used: 50000
[17:57:24][15544][INFO ] Continuing work on ../../projects/einstein.phys.uwm.edu/Ter5_1_dns_cfbf00110_segment_6_dms_200_148.binary at template no. 43908
[17:57:24][15544][INFO ] Header contents:
------> Original WAPP file: /atlas/data/TRAPUM_GC/Terzan5/epoch1/30min_segments/dedispersed_files/cfbf00110/Ter5_1_dns_cfbf00110_segment_6_dms_200_DM244.80
------> Sample time in microseconds: 153.121
------> Observation time in seconds: 2568.9524
------> Time stamp (MJD): 59097.776208975287
------> Number of samples/record: 0
------> Center freq in MHz: 857.5673828
------> Channel band in MHz: 3.34375
------> Number of channels/record: 256
------> Nifs: 1
------> RA (J2000): 174801.97
------> DEC (J2000): -244716.1
------> Galactic l: 0
------> Galactic b: 0
------> Name: J1748-2446N
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 16777216
------> Trial dispersion measure: 244.8 cm^-3 pc
------> Scale factor: 1.01616
[17:57:25][15544][INFO ] Seed for random number generator is -1031807597.
[17:57:31][15544][ERROR] Couldn't allocate 7706856 bytes of CUDA HS summing memory (error: 2)!
[17:57:31][15544][ERROR] Demodulation failed (error: 1006)!
[17:57:31][15544][WARN ] Sorry, at the moment your system doesn't have enough free CPU/GPU memory to run this task!
------> Returning control to BOINC, delaying next attempt for at least 15 minutes...
------> If this problem persists you should consider aborting this task...
Activated exception handling...
[18:14:45][9484][INFO ] Starting data processing...
[18:14:45][9484][INFO ] CUDA device ID: 0
[18:14:45][9484][INFO ] CUDA global memory status (initial GPU state, including context):
------> Used in total: 185 MB (840 MB free / 1025 MB total) -> Used by this application (assuming a single GPU task): 0 MB
[18:14:45][9484][INFO ] Using CUDA device #0 "Quadro K600" (192 CUDA cores / 336.19 GFLOPS)
[18:14:45][9484][INFO ] Version of installed CUDA driver: 9010
[18:14:45][9484][INFO ] Version of CUDA driver API used: 3020
[18:15:00][9484][INFO ] Number of generated templates to be used: 50000
[18:15:00][9484][INFO ] Continuing work on ../../projects/einstein.phys.uwm.edu/Ter5_1_dns_cfbf00110_segment_6_dms_200_148.binary at template no. 43908
[18:15:00][9484][INFO ] Header contents:
------> Original WAPP file: /atlas/data/TRAPUM_GC/Terzan5/epoch1/30min_segments/dedispersed_files/cfbf00110/Ter5_1_dns_cfbf00110_segment_6_dms_200_DM244.80
------> Sample time in microseconds: 153.121
------> Observation time in seconds: 2568.9524
------> Time stamp (MJD): 59097.776208975287
------> Number of samples/record: 0
------> Center freq in MHz: 857.5673828
------> Channel band in MHz: 3.34375
------> Number of channels/record: 256
------> Nifs: 1
------> RA (J2000): 174801.97
------> DEC (J2000): -244716.1
------> Galactic l: 0
------> Galactic b: 0
------> Name: J1748-2446N
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 16777216
------> Trial dispersion measure: 244.8 cm^-3 pc
------> Scale factor: 1.01616
[18:15:00][9484][INFO ] Seed for random number generator is -1031807597.
[18:15:05][9484][ERROR] Couldn't allocate 7706856 bytes of CUDA HS summing memory (error: 2)!
[18:15:05][9484][ERROR] Demodulation failed (error: 1006)!
[18:15:05][9484][WARN ] Sorry, at the moment your system doesn't have enough free CPU/GPU memory to run this task!
------> Returning control to BOINC, delaying next attempt for at least 15 minutes...
------> If this problem persists you should consider aborting this task...
Activated exception handling...
[18:41:24][5184][INFO ] Starting data processing...
[18:41:24][5184][INFO ] CUDA device ID: 0
[18:41:24][5184][INFO ] CUDA global memory status (initial GPU state, including context):
------> Used in total: 185 MB (840 MB free / 1025 MB total) -> Used by this application (assuming a single GPU task): 0 MB
[18:41:24][5184][INFO ] Using CUDA device #0 "Quadro K600" (192 CUDA cores / 336.19 GFLOPS)
[18:41:24][5184][INFO ] Version of installed CUDA driver: 9010
[18:41:24][5184][INFO ] Version of CUDA driver API used: 3020
[18:41:39][5184][INFO ] Number of generated templates to be used: 50000
[18:41:39][5184][INFO ] Continuing work on ../../projects/einstein.phys.uwm.edu/Ter5_1_dns_cfbf00110_segment_6_dms_200_148.binary at template no. 43908
[18:41:39][5184][INFO ] Header contents:
------> Original WAPP file: /atlas/data/TRAPUM_GC/Terzan5/epoch1/30min_segments/dedispersed_files/cfbf00110/Ter5_1_dns_cfbf00110_segment_6_dms_200_DM244.80
------> Sample time in microseconds: 153.121
------> Observation time in seconds: 2568.9524
------> Time stamp (MJD): 59097.776208975287
------> Number of samples/record: 0
------> Center freq in MHz: 857.5673828
------> Channel band in MHz: 3.34375
------> Number of channels/record: 256
------> Nifs: 1
------> RA (J2000): 174801.97
------> DEC (J2000): -244716.1
------> Galactic l: 0
------> Galactic b: 0
------> Name: J1748-2446N
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 16777216
------> Trial dispersion measure: 244.8 cm^-3 pc
------> Scale factor: 1.01616
[18:41:39][5184][INFO ] Seed for random number generator is -1031807597.
[18:41:44][5184][INFO ] Derived global search parameters:
------> f_A probability = 0.04
------> single bin prob(P_noise > P_thr) = 3.24424e-009
------> thr1 = 19.5464
------> thr2 = 22.7124
------> thr4 = 27.844
------> thr8 = 36.3869
------> thr16 = 50.9469
[18:41:44][5184][INFO ] CUDA global memory status (GPU setup complete):
------> Used in total: 597 MB (428 MB free / 1025 MB total) -> Used by this application (assuming a single GPU task): 412 MB
[18:42:24][5184][INFO ] Checkpoint committed!
[18:43:25][5184][INFO ] Checkpoint committed!
Activated exception handling...
[18:54:27][17432][INFO ] Starting data processing...
[18:54:27][17432][INFO ] CUDA device ID: 0
[18:54:27][17432][INFO ] CUDA global memory status (initial GPU state, including context):
------> Used in total: 185 MB (840 MB free / 1025 MB total) -> Used by this application (assuming a single GPU task): 0 MB
[18:54:27][17432][INFO ] Using CUDA device #0 "Quadro K600" (192 CUDA cores / 336.19 GFLOPS)
[18:54:27][17432][INFO ] Version of installed CUDA driver: 9010
[18:54:27][17432][INFO ] Version of CUDA driver API used: 3020
[18:54:42][17432][INFO ] Number of generated templates to be used: 50000
[18:54:42][17432][INFO ] Continuing work on ../../projects/einstein.phys.uwm.edu/Ter5_1_dns_cfbf00110_segment_6_dms_200_148.binary at template no. 44146
[18:54:42][17432][INFO ] Header contents:
------> Original WAPP file: /atlas/data/TRAPUM_GC/Terzan5/epoch1/30min_segments/dedispersed_files/cfbf00110/Ter5_1_dns_cfbf00110_segment_6_dms_200_DM244.80
------> Sample time in microseconds: 153.121
------> Observation time in seconds: 2568.9524
------> Time stamp (MJD): 59097.776208975287
------> Number of samples/record: 0
------> Center freq in MHz: 857.5673828
------> Channel band in MHz: 3.34375
------> Number of channels/record: 256
------> Nifs: 1
------> RA (J2000): 174801.97
------> DEC (J2000): -244716.1
------> Galactic l: 0
------> Galactic b: 0
------> Name: J1748-2446N
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 16777216
------> Trial dispersion measure: 244.8 cm^-3 pc
------> Scale factor: 1.01616
[18:54:42][17432][INFO ] Seed for random number generator is -1031807597.
[18:54:47][17432][INFO ] Derived global search parameters:
------> f_A probability = 0.04
------> single bin prob(P_noise > P_thr) = 3.24424e-009
------> thr1 = 19.5464
------> thr2 = 22.7124
------> thr4 = 27.844
------> thr8 = 36.3869
------> thr16 = 50.9469
[18:54:47][17432][INFO ] CUDA global memory status (GPU setup complete):
------> Used in total: 597 MB (428 MB free / 1025 MB total) -> Used by this application (assuming a single GPU task): 412 MB
[18:55:27][17432][INFO ] Checkpoint committed!
[18:56:27][17432][INFO ] Checkpoint committed!
[18:57:28][17432][INFO ] Checkpoint committed!
[18:58:28][17432][INFO ] Checkpoint committed!
[18:59:28][17432][INFO ] Checkpoint committed!
[19:00:29][17432][INFO ] Checkpoint committed!
[19:01:29][17432][INFO ] Checkpoint committed!
[19:02:29][17432][INFO ] Checkpoint committed!
[19:03:29][17432][INFO ] Checkpoint committed!
[19:04:30][17432][INFO ] Checkpoint committed!
[19:05:30][17432][INFO ] Checkpoint committed!
[19:06:30][17432][INFO ] Checkpoint committed!
[19:07:30][17432][INFO ] Checkpoint committed!
[19:08:31][17432][INFO ] Checkpoint committed!
[19:09:31][17432][INFO ] Checkpoint committed!
[19:10:31][17432][INFO ] Checkpoint committed!
[19:11:31][17432][INFO ] Checkpoint committed!
[19:12:32][17432][INFO ] Checkpoint committed!
[19:13:32][17432][INFO ] Checkpoint committed!
[19:14:32][17432][INFO ] Checkpoint committed!
[19:15:32][17432][INFO ] Checkpoint committed!
[19:16:33][17432][INFO ] Checkpoint committed!
[19:17:33][17432][INFO ] Checkpoint committed!
[19:18:33][17432][INFO ] Checkpoint committed!
[19:19:33][17432][INFO ] Checkpoint committed!
[19:20:34][17432][INFO ] Checkpoint committed!
[19:21:34][17432][INFO ] Checkpoint committed!
[19:21:50][17432][INFO ] Statistics: count dirty SumSpec pages 34583 (not checkpointed), Page Size 1024, fundamental_idx_hi-window_2: 1926214
[19:21:50][17432][INFO ] Data processing finished successfully!
19:21:50 (17432): called boinc_finish(0)

</stderr_txt>
]]>