Task 1731976778

Name: h1_0185.80_O3aLC01Cl1In0__O3ASBu_186.00Hz_2430_0
Workunit ID: 875220513
Created: 18 Feb 2025 21:20:56 UTC
Sent: 18 Feb 2025 21:20:57 UTC
Report deadline: 25 Feb 2025 21:20:57 UTC
Received: 20 Feb 2025 6:15:08 UTC
Server state: Over
Outcome: Success
Client state: Done
Exit status: 0 (0x00000000)
Computer: 13063321
Run time (sec): 4,868.49
CPU time (sec): 4,819.97
Peak working set size (MB): 1228.09
Peak swap size (MB): 44868.26
Peak disk usage (MB): 60.85
Validation state: Valid
Granted credit: 20,000
Application: All-Sky Gravitational Wave search on O3 v1.16 (GW-cuda-3)
x86_64-pc-linux-gnu
Stderr output

<core_client_version>7.20.5</core_client_version>
<![CDATA[
<stderr_txt>
putenv 'LAL_DEBUG_LEVEL=3'
2025-02-19 21:56:46.7516 (19266) [normal]: This program is published under the GNU General Public License, version 2
2025-02-19 21:56:46.7516 (19266) [normal]: For details see http://einstein.phys.uwm.edu/license.php
2025-02-19 21:56:46.7516 (19266) [normal]: This Einstein@home App was built at: Mar  7 2024 14:38:26

2025-02-19 21:56:46.7516 (19266) [normal]: Start of BOINC application '../../projects/einstein.phys.uwm.edu/einstein_O3AS_1.16_x86_64-pc-linux-gnu__GW-cuda-3'.
2025-02-19 21:56:46.7524 (19266) [debug]: Flags: LAL_DEBUG, OPTIMIZE, HS_OPTIMIZATION, GC_SSE2_OPT, CUDA, X64, SSE, SSE2, GNUC X86 GNUX86
2025-02-19 21:56:46.7524 (19266) [debug]: glibc version/release: 2.36/stable
2025-02-19 21:56:46.752431 - mytime()
2025-02-19 21:56:46.7525 (19266) [debug]: Set up communication with graphics process.
2025-02-19 21:56:46.7646 (19266) [normal]: Parsed user input successfully
Code-version: %% LAL: 7.1.4.1 (UNCLEAN d065dd0b1daca1f30b01d44a0f807e06f37b4958)
%% LALPulsar: 3.1.0.1 (UNCLEAN d065dd0b1daca1f30b01d44a0f807e06f37b4958)
%% LALApps: 7.3.0.1 (UNCLEAN d065dd0b1daca1f30b01d44a0f807e06f37b4958)

2025-02-19 21:56:46.7647 (19266) [normal]: Initialise compartments with freqWidth = 0.05 and candidates per compartment = 10000.
2025-02-19 21:56:46.7649 (19266) [debug]: Opening temp output file '../../projects/einstein.phys.uwm.edu/h1_0185.80_O3aLC01Cl1In0__O3ASBu_186.00Hz_2430_0_0.tmp' for writing ...
2025-02-19 21:56:47.0035 (19266) [normal]: Reading input data ... 
2025-02-19 21:56:47.0035 (19266) [normal]: Loading SFTs matching '../../projects/einstein.phys.uwm.edu/h1_0185.80_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0185.80_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0186.00_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0186.00_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0186.20_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0186.20_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0186.40_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0186.40_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0186.60_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0186.60_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0186.80_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0186.80_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0187.00_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0187.00_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0187.20_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0187.20_O3aLC01Cl1In0' into catalog ...2025-02-19 21:56:47.2197 (19266) [normal]: done.
2025-02-19 21:56:47.2197 (19266) [normal]: Validating SFTs (detectors: H1, L1, ) ... success.
2025-02-19 21:56:49.7962 (19266) [normal]: Search FstatMethod used: 'ResampGPU'
2025-02-19 21:56:49.7962 (19266) [normal]: Recalc FstatMethod used: 'DemodGPU'
2025-02-19 21:56:49.7963 (19266) [normal]: GPU Device used for Search/Recalc and/or semi coherent step: 'NVIDIA GeForce GTX 980'
2025-02-19 21:56:49.7963 (19266) [normal]: GPU Backend used for Search/Recalc and/or semi coherent step: 'CUDA'
2025-02-19 21:56:49.7963 (19266) [normal]: GPU version is used for the semi-coherent step!
2025-02-19 21:57:06.5517 (19266) [normal]: Number of segments: 18, total number of SFTs in segments: 5748
2025-02-19 21:57:06.5556 (19266) [normal]: Finished reading input data.
2025-02-19 21:57:06.5556 (19266) [normal]: GPS reference time = 1246070525.0000 ,  GPS data mid time = 1246070525.0000
2025-02-19 21:57:06.5556 (19266) [normal]: dFreqStack = 1.000000e-06, df1dot = 7.500000e-11, df2dot = 0.000000e+00, df3dot = 0.000000e+00
2025-02-19 21:57:06.5556 (19266) [normal]: Setup, N = 18, T = 878400 s, Tobs = 15809012 s, gammaRefine = 250, gamma2Refine = 1098, gamma3Refine = 1

DEPRECATION WARNING: program has invoked obsolete function InitDopplerSkyScan(). Please see XLALInitDopplerSkyScan() for information about a replacement.
2025-02-19 21:57:18.9907 (19266) [normal]: INFO: No checkpoint checkpoint.cpt found - starting from scratch
2025-02-19 21:57:19.0395 (19266) [normal]: Cpt:0,  total:9000,  sky:1/225,  f1dot:1/40

0.2025-02-19 21:57:19.0401 (19266) [normal]: CG:18010710 FG:1000000  f1dotmin_fg:-2.73735059761e-09 df1dot_fg:2.988047808765e-13 f2dotmin_fg:0 df2dot_fg:0 f3dotmin_fg:0 df3dot_fg:1
.......................................
1........c
................................
2........................................
3......................................c
..
4........................................
5........................................
6.............................c
...........
7........................................
8........................................
9...................c
.....................
10........................................
11........................................
12..........c
..............................
13........................................
14........................................
15.c
.......................................
16........................................
17.................................c
.......
18........................................
19........................................
20.........................c
...............
21........................................
22........................................
23.................c
.......................
24........................................
25........................................
26..........c
..............................
27........................................
28........................................
29....c
....................................
30........................................
31.....................................c
...
32........................................
33........................................
34...............................c
.........
35........................................
36........................................
37........................c
................
38........................................
39........................................
40..................c
......................
41........................................
42........................................
43.............c
...........................
44........................................
45........................................
46.......c
.................................
47...............................
2025-02-19 22:14:22.282304
-- signal handler called: signal 15
XLAL Error - XLALGPUMarkStreamsAsFree (/home/boinc/lalsuite/EinsteinAtHome/source/lalsuite/lalpulsar/lib/GPUUtils/GPUUtils.c:1961): GPU is not initialized!
XLAL Error - XLALGPUMarkStreamsAsFree (/home/boinc/lalsuite/EinsteinAtHome/source/lalsuite/lalpulsar/lib/GPUUtils/GPUUtils.c:1961): Generic failure
XLAL Error - XLALGPUWait (/home/boinc/lalsuite/EinsteinAtHome/source/lalsuite/lalpulsar/lib/GPUUtils/GPUUtils.c:476): Check failed: XLALGPUMarkStreamsAsFree(((void *)0)) == XLAL_SUCCESS
XLAL Error - XLALGPUWait (/home/boinc/lalsuite/EinsteinAtHome/source/lalsuite/lalpulsar/lib/GPUUtils/GPUUtils.c:476): Internal function call failed: Generic failure
XLAL Error - XLALSemiCohStep_GPU (/home/boinc/lalsuite/EinsteinAtHome/source/lalsuite/lalapps/src/pulsar/GCT/HierarchSearchGCT.c:4677): SemiCohStep
XLAL Error - XLALSemiCohStep_GPU (/home/boinc/lalsuite/EinsteinAtHome/source/lalsuite/lalapps/src/pulsar/GCT/HierarchSearchGCT.c:4677): Internal function call failed: Generic failure
XLAL Error - MAIN (/home/boinc/lalsuite/EinsteinAtHome/source/lalsuite/lalapps/src/pulsar/GCT/HierarchSearchGCT.c:2308): Check failed: XLALSemiCohStep_GPU( &coarsegrid, &finegrid, nStacks, &usefulParams, NSegmentsInv, uvar->SortToplist, compartment, compartment2, compartment3) == XLAL_SUCCESS
XLAL Error - MAIN (/home/boinc/lalsuite/EinsteinAtHome/source/lalsuite/lalapps/src/pulsar/GCT/HierarchSearchGCT.c:2308): Internal function call failed: Generic failure
2025-02-19 22:14:22.2829 (19266) [CRITICAL]: ERROR: MAIN() returned with error '-1'
Code-version: %% LAL: 7.1.4.1 (UNCLEAN d065dd0b1daca1f30b01d44a0f807e06f37b4958)
%% LALPulsar: 3.1.0.1 (UNCLEAN d065dd0b1daca1f30b01d44a0f807e06f37b4958)
%% LALApps: 7.3.0.1 (UNCLEAN d065dd0b1daca1f30b01d44a0f807e06f37b4958)

FPU status flags: 
2025-02-19 22:14:22.2832 (19266) [debug]: worker done. return(-1) to caller
2025-02-19 22:14:22.2832 (19266) [normal]: done. calling boinc_finish(-1).
22:14:22 (19266): called boinc_finish
XLALCUDARelease: Total malloc from <TYPE>VectorGPU is greater than 0: 2077409693XLALCUDARelease: Total malloc from COMPLEX8FFTPlanGPU is greater than 0: 671088648 stack frames obtained for this thread:
Use gdb command: 'info line *0xADDRESS' to print corresponding line numbers.
../../projects/einstein.phys.uwm.edu/einstein_O3AS_1.16_x86_64-pc-linux-gnu__GW-cuda-3[0x4301f7]
/lib/x86_64-linux-gnu/libc.so.6(+0x3c050)[0x7f694edc6050]
/lib/x86_64-linux-gnu/libc.so.6(poll+0x4f)[0x7f694ee861df]
/lib/x86_64-linux-gnu/libcuda.so.1(+0x2baa2f)[0x7f694d2baa2f]
/lib/x86_64-linux-gnu/libcuda.so.1(+0x37d79f)[0x7f694d37d79f]
/lib/x86_64-linux-gnu/libcuda.so.1(+0x2b595f)[0x7f694d2b595f]
/lib/x86_64-linux-gnu/libc.so.6(+0x891c4)[0x7f694ee131c4]
/lib/x86_64-linux-gnu/libc.so.6(+0x10985c)[0x7f694ee9385c]

End of stacktrace
Stack trace of LAL functions in worker thread:
InitDopplerSkyScan at /home/boinc/lalsuite/EinsteinAtHome/source/lalsuite/lalpulsar/lib/DopplerScan.c:291
At lowest level status code = 0: NO LAL ERROR REGISTERED
putenv 'LAL_DEBUG_LEVEL=3'
2025-02-20 00:05:35.9939 (1296) [normal]: This program is published under the GNU General Public License, version 2
2025-02-20 00:05:35.9939 (1296) [normal]: For details see http://einstein.phys.uwm.edu/license.php
2025-02-20 00:05:35.9939 (1296) [normal]: This Einstein@home App was built at: Mar  7 2024 14:38:26

2025-02-20 00:05:35.9939 (1296) [normal]: Start of BOINC application '../../projects/einstein.phys.uwm.edu/einstein_O3AS_1.16_x86_64-pc-linux-gnu__GW-cuda-3'.
2025-02-20 00:05:35.9975 (1296) [debug]: Flags: LAL_DEBUG, OPTIMIZE, HS_OPTIMIZATION, GC_SSE2_OPT, CUDA, X64, SSE, SSE2, GNUC X86 GNUX86
2025-02-20 00:05:35.9976 (1296) [debug]: glibc version/release: 2.36/stable
2025-02-20 00:05:35.997609 - mytime()
2025-02-20 00:05:35.9978 (1296) [debug]: Set up communication with graphics process.
2025-02-20 00:05:36.0104 (1296) [normal]: Parsed user input successfully
Code-version: %% LAL: 7.1.4.1 (UNCLEAN d065dd0b1daca1f30b01d44a0f807e06f37b4958)
%% LALPulsar: 3.1.0.1 (UNCLEAN d065dd0b1daca1f30b01d44a0f807e06f37b4958)
%% LALApps: 7.3.0.1 (UNCLEAN d065dd0b1daca1f30b01d44a0f807e06f37b4958)

2025-02-20 00:05:36.0104 (1296) [normal]: Initialise compartments with freqWidth = 0.05 and candidates per compartment = 10000.
2025-02-20 00:05:36.0106 (1296) [debug]: Opening temp output file '../../projects/einstein.phys.uwm.edu/h1_0185.80_O3aLC01Cl1In0__O3ASBu_186.00Hz_2430_0_0.tmp' for writing ...
2025-02-20 00:05:36.2407 (1296) [normal]: Reading input data ... 
2025-02-20 00:05:36.2408 (1296) [normal]: Loading SFTs matching '../../projects/einstein.phys.uwm.edu/h1_0185.80_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0185.80_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0186.00_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0186.00_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0186.20_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0186.20_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0186.40_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0186.40_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0186.60_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0186.60_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0186.80_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0186.80_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0187.00_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0187.00_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0187.20_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0187.20_O3aLC01Cl1In0' into catalog ...2025-02-20 00:05:36.4470 (1296) [normal]: done.
2025-02-20 00:05:36.4471 (1296) [normal]: Validating SFTs (detectors: H1, L1, ) ... success.
2025-02-20 00:05:38.9339 (1296) [normal]: Search FstatMethod used: 'ResampGPU'
2025-02-20 00:05:38.9339 (1296) [normal]: Recalc FstatMethod used: 'DemodGPU'
2025-02-20 00:05:38.9339 (1296) [normal]: GPU Device used for Search/Recalc and/or semi coherent step: 'NVIDIA GeForce GTX 980'
2025-02-20 00:05:38.9339 (1296) [normal]: GPU Backend used for Search/Recalc and/or semi coherent step: 'CUDA'
2025-02-20 00:05:38.9339 (1296) [normal]: GPU version is used for the semi-coherent step!
2025-02-20 00:05:54.6451 (1296) [normal]: Number of segments: 18, total number of SFTs in segments: 5748
2025-02-20 00:05:54.6493 (1296) [normal]: Finished reading input data.
2025-02-20 00:05:54.6494 (1296) [normal]: GPS reference time = 1246070525.0000 ,  GPS data mid time = 1246070525.0000
2025-02-20 00:05:54.6494 (1296) [normal]: dFreqStack = 1.000000e-06, df1dot = 7.500000e-11, df2dot = 0.000000e+00, df3dot = 0.000000e+00
2025-02-20 00:05:54.6494 (1296) [normal]: Setup, N = 18, T = 878400 s, Tobs = 15809012 s, gammaRefine = 250, gamma2Refine = 1098, gamma3Refine = 1

DEPRECATION WARNING: program has invoked obsolete function InitDopplerSkyScan(). Please see XLALInitDopplerSkyScan() for information about a replacement.
2025-02-20 00:06:07.2921 (1296) [normal]: Successfully read checkpoint:1847
2025-02-20 00:06:07.3648 (1296) [normal]: Cpt:1847,  total:9000,  sky:47/225,  f1dot:8/40

46.2025-02-20 00:06:07.3653 (1296) [normal]: CG:18010710 FG:1000000  f1dotmin_fg:-2.21235059761e-09 df1dot_fg:2.988047808765e-13 f2dotmin_fg:0 df2dot_fg:0 f3dotmin_fg:0 df3dot_fg:1
................................
47.....................c
...................
48........................................
49........................................
50...............c
.........................
51........................................
52........................................
53........c
................................
54........................................
55........................................
56.c
.......................................
57........................................
58..................................c
......
59........................................
60........................................
61...........................c
.............
62........................................
63........................................
64....................c
....................
65........................................
66........................................
67..............c
..........................
68........................................
69........................................
70........c
................................
71........................................
72........................................
73..c
......................................
74........................................
75......................................c
..
76........................................
77........................................
78................................c
........
79........................................
80........................................
81...........................c
.............
82........................................
83........................................
84.....................c
...................
85........................................
86........................................
87................c
........................
88........................................
89........................................
90...........c
.............................
91........................................
92........................................
93......c
..................................
94........................................
95........................................
96..c
......................................
97........................................
98....................................c
....
99........................................
100........................................
101...............................c
.........
102........................................
103........................................
104.........................c
...............
105........................................
106........................................
107....................c
....................
108........................................
109........................................
110..............c
..........................
111........................................
112........................................
113.........c
...............................
114........................................
115........................................
116....c
....................................
117........................................
118.......................................c
.
119........................................
120........................................
121...................................c
.....
122........................................
123........................................
124..............................c
..........
125........................................
126........................................
127.........................c
...............
128........................................
129........................................
130....................c
....................
131........................................
132........................................
133..............c
..........................
134........................................
135........................................
136........c
................................
137........................................
138........................................
139..c
......................................
140........................................
141.....................................c
...
142........................................
143........................................
144.................................c
.......
145........................................
146........................................
147.............................c
...........
148........................................
149........................................
150........................c
................
151........................................
152........................................
153....................c
....................
154........................................
155........................................
156...............c
.........................
157........................................
158........................................
159.........c
...............................
160........................................
161........................................
162.....c
...................................
163........................................
164........................................c

165........................................
166........................................
167....................................c
....
168........................................
169........................................
170..............................c
..........
171........................................
172........................................
173.........................c
...............
174........................................
175........................................
176....................c
....................
177........................................
178........................................
179................c
........................
180........................................
181........................................
182............c
............................
183........................................
184........................................
185.......c
.................................
186........................................
187........................................
188..c
......................................
189........................................
190....................................c
....
191........................................
192........................................
193..............................c
..........
194........................................
195........................................
196........................c
................
197........................................
198........................................
199..................c
......................
200........................................
201........................................
202...........c
.............................
203........................................
204........................................
205......c
..................................
206........................................
207........................................
208.c
.......................................
209........................................
210.....................................c
...
211........................................
212........................................
213...............................c
.........
214........................................
215........................................
216.........................c
...............
217........................................
218........................................
219....................c
....................
220........................................
221........................................
222...............c
.........................
223........................................
224........................................
2025-02-20 01:09:03.0064 (1296) [normal]: Finished main analysis.
2025-02-20 01:09:03.0064 (1296) [normal]: Recalculating statistics for the final toplist(s)...
2025-02-20 01:09:06.6839 (1296) [normal]: Recalculation step: 20000/200000
2025-02-20 01:09:10.2029 (1296) [normal]: Recalculation step: 40000/200000
2025-02-20 01:09:13.7218 (1296) [normal]: Recalculation step: 60000/200000
2025-02-20 01:09:17.2128 (1296) [normal]: Recalculation step: 80000/200000
2025-02-20 01:09:20.7226 (1296) [normal]: Recalculation step: 100000/200000
2025-02-20 01:09:24.2467 (1296) [normal]: Recalculation step: 120000/200000
2025-02-20 01:09:27.7800 (1296) [normal]: Recalculation step: 140000/200000
2025-02-20 01:09:31.3270 (1296) [normal]: Recalculation step: 160000/200000
2025-02-20 01:09:34.8426 (1296) [normal]: Recalculation step: 180000/200000
2025-02-20 01:09:38.3493 (1296) [normal]: Recalculation step: 200000/200000
2025-02-20 01:09:38.5575 (1296) [normal]: Finished recalculating toplist statistics.
2025-02-20 01:09:38.5576 (1296) [normal]: Finished in 3842.55 s with peak RAM usage: 1399.9 MB on CPU 'AMD Ryzen 7 2700X Eight-Core Processor         ', peak VRAM usage: 2045.2 MB on GPU Device: 'NVIDIA GeForce GTX 980' with backend: 'CUDA'.
2025-02-20 01:09:38.5898 (1296) [debug]: Writing output ... Closing temp output file '../../projects/einstein.phys.uwm.edu/h1_0185.80_O3aLC01Cl1In0__O3ASBu_186.00Hz_2430_0_0.tmp' ... renaming temp output file '../../projects/einstein.phys.uwm.edu/h1_0185.80_O3aLC01Cl1In0__O3ASBu_186.00Hz_2430_0_0.tmp' to '../../projects/einstein.phys.uwm.edu/h1_0185.80_O3aLC01Cl1In0__O3ASBu_186.00Hz_2430_0_0' ... done.
2025-02-20 01:09:38.8368 (1296) [normal]: Restarted from checkpoint 1847

DEPRECATION WARNING: program has invoked obsolete function FreeDopplerSkyScan(). Please see XLALDestroyDopplerSkyScan() for information about a replacement.
Code-version: %% LAL: 7.1.4.1 (UNCLEAN d065dd0b1daca1f30b01d44a0f807e06f37b4958)
%% LALPulsar: 3.1.0.1 (UNCLEAN d065dd0b1daca1f30b01d44a0f807e06f37b4958)
%% LALApps: 7.3.0.1 (UNCLEAN d065dd0b1daca1f30b01d44a0f807e06f37b4958)

FPU status flags: 
2025-02-20 01:09:38.9452 (1296) [debug]: worker done. return(0) to caller
2025-02-20 01:09:38.9453 (1296) [normal]: done. calling boinc_finish(0).
01:09:38 (1296): called boinc_finish

</stderr_txt>
]]>