Task 1735205518

Stderr output

<core_client_version>8.0.2</core_client_version>
<![CDATA[
<stderr_txt>
putenv 'LAL_DEBUG_LEVEL=3'
2025-02-25 22:01:43.3078 (12587) [normal]: This program is published under the GNU General Public License, version 2
2025-02-25 22:01:43.3079 (12587) [normal]: For details see http://einstein.phys.uwm.edu/license.php
2025-02-25 22:01:43.3079 (12587) [normal]: This Einstein@home App was built at: Nov  9 2023 13:12:46

2025-02-25 22:01:43.3079 (12587) [normal]: Start of BOINC application '../../projects/einstein.phys.uwm.edu/einstein_O3AS_1.16_x86_64-pc-linux-gnu__GW-opencl-ati-3'.
[DEBUG} GPU type: 2
[DEBUG} got GPU info from BOINC
[DEBUG} got VendorID 4098
2025-02-25 22:01:43.5126 (12587) [debug]: Flags: LAL_DEBUG, OPTIMIZE, HS_OPTIMIZATION, GC_SSE2_OPT, X64, SSE, SSE2, GNUC X86 GNUX86
2025-02-25 22:01:43.5127 (12587) [debug]: glibc version/release: 2.39/stable
2025-02-25 22:01:43.512723 - mytime()
2025-02-25 22:01:43.5131 (12587) [debug]: Set up communication with graphics process.
2025-02-25 22:01:43.5136 (12587) [normal]: Parsed user input successfully
Code-version: %% LAL: 7.1.4.1 (CLEAN )
%% LALPulsar: 3.1.0.1 (CLEAN )
%% LALApps: 7.3.0.1 (CLEAN )

2025-02-25 22:01:43.5136 (12587) [normal]: Initialise compartments with freqWidth = 0.05 and candidates per compartment = 10000.
2025-02-25 22:01:43.5140 (12587) [debug]: Opening temp output file '../../projects/einstein.phys.uwm.edu/h1_0190.80_O3aLC01Cl1In0__O3ASBu_191.00Hz_70749_0_0.tmp' for writing ...
2025-02-25 22:01:43.8063 (12587) [normal]: Reading input data ... 
2025-02-25 22:01:43.8064 (12587) [normal]: Loading SFTs matching '../../projects/einstein.phys.uwm.edu/h1_0190.80_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0190.80_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0191.00_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0191.00_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0191.20_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0191.20_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0191.40_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0191.40_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0191.60_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0191.60_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0191.80_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0191.80_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0192.00_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0192.00_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0192.20_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0192.20_O3aLC01Cl1In0' into catalog ...2025-02-25 22:01:44.1154 (12587) [normal]: done.
2025-02-25 22:01:44.1155 (12587) [normal]: Validating SFTs (detectors: H1, L1, ) ... success.
2025-02-25 22:01:50.8310 (12587) [normal]: Search FstatMethod used: 'ResampGPU'
2025-02-25 22:01:50.8311 (12587) [normal]: Recalc FstatMethod used: 'DemodSSE'
2025-02-25 22:01:50.8311 (12587) [normal]: GPU Device used for Search/Recalc and/or semi coherent step: 'gfx1010:xnack- ( Platform: AMD Accelerated Parallel Processing )'
2025-02-25 22:01:50.8311 (12587) [normal]: GPU Backend used for Search/Recalc and/or semi coherent step: 'OpenCL'
2025-02-25 22:01:50.8312 (12587) [normal]: GPU version is used for the semi-coherent step!
2025-02-25 22:02:14.0984 (12587) [normal]: Number of segments: 18, total number of SFTs in segments: 5748
2025-02-25 22:02:14.1195 (12587) [normal]: Finished reading input data.
2025-02-25 22:02:14.1196 (12587) [normal]: GPS reference time = 1246070525.0000 ,  GPS data mid time = 1246070525.0000
2025-02-25 22:02:14.1198 (12587) [normal]: dFreqStack = 1.000000e-06, df1dot = 7.500000e-11, df2dot = 0.000000e+00, df3dot = 0.000000e+00
2025-02-25 22:02:14.1198 (12587) [normal]: Setup, N = 18, T = 878400 s, Tobs = 15809012 s, gammaRefine = 250, gamma2Refine = 1098, gamma3Refine = 1

DEPRECATION WARNING: program has invoked obsolete function InitDopplerSkyScan(). Please see XLALInitDopplerSkyScan() for information about a replacement.
2025-02-25 22:02:29.4682 (12587) [normal]: INFO: No checkpoint checkpoint.cpt found - starting from scratch
2025-02-25 22:02:29.4683 (12587) [normal]: Cpt:0,  total:9000,  sky:1/225,  f1dot:1/40

0.2025-02-25 22:02:29.4719 (12587) [normal]: CG:18010710 FG:1000000  f1dotmin_fg:-2.73735059761e-09 df1dot_fg:2.988047808765e-13 f2dotmin_fg:0 df2dot_fg:0 f3dotmin_fg:0 df3dot_fg:1
.......................................
1.............c
...........................
2........................................
3........................................
4...............................c
.........
5........................................
6........................................
7........................................
8............c
............................
9........................................
10........................................
11.....................................c
...
12........................................
13........................................
14........................................
15............................c
............
16........................................
17........................................
18........................................
19.......................c
.................
20........................................
21........................................
22........................................
23...................c
.....................
24........................................
25........................................
26........................................
27...............c
.........................
28........................................
29........................................
30........................................
31...............c
.........................
32........................................
33........................................
34........................................
35..............c
..........................
36........................................
37........................................
38........................................
39..............c
..........................
40........................................
41........................................
42........................................
43..............c
..........................
44........................................
45........................................
46........................................
47............c
............................
48........................................
49........................................
50........................................
51................c
........................
52........................................
53........................................
54........................................
55....................c
....................
56........................................
57........................................HW Exception by GPU node-1 (Agent handle: 0xe1b2160) reason :GPU Hang

2025-02-25 22:25:21.15025
-- signal handler called: signal 6

2025-02-25 22:57:10.34096
-- signal handler called: signal 15
19 stack frames obtained for this thread:
Use gdb command: 'info line *0xADDRESS' to print corresponding line numbers.
../../projects/einstein.phys.uwm.edu/einstein_O3AS_1.16_x86_64-pc-linux-gnu__GW-opencl-ati-3[0x430327]
/lib/x86_64-linux-gnu/libc.so.6(+0x45330)[0x70bbb6445330]
/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0xb0)[0x70bbb649eac0]
/opt/rocm-6.3.0/lib/libamdocl64.so(+0xcf8b3)[0x70bbb62fa8b3]
/opt/rocm-6.3.0/lib/libamdocl64.so(+0xdcd98)[0x70bbb6307d98]
/opt/rocm-6.3.0/lib/libamdocl64.so(+0xee872)[0x70bbb6319872]
/opt/rocm-6.3.0/lib/libamdocl64.so(clReleaseCommandQueue+0xb6)[0x70bbb62a9376]
../../projects/einstein.phys.uwm.edu/einstein_O3AS_1.16_x86_64-pc-linux-gnu__GW-opencl-ati-3[0x5b53c5]
../../projects/einstein.phys.uwm.edu/einstein_O3AS_1.16_x86_64-pc-linux-gnu__GW-opencl-ati-3[0x5a255c]
../../projects/einstein.phys.uwm.edu/einstein_O3AS_1.16_x86_64-pc-linux-gnu__GW-opencl-ati-3[0x4302f5]
/lib/x86_64-linux-gnu/libc.so.6(+0x45330)[0x70bbb6445330]
/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x11c)[0x70bbb649eb2c]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x1e)[0x70bbb644527e]
/lib/x86_64-linux-gnu/libc.so.6(abort+0xdf)[0x70bbb64288ff]
/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1(+0x2a61b)[0x70bbad22a61b]
/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1(+0x93969)[0x70bbad293969]
/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1(+0x33e4b)[0x70bbad233e4b]
/lib/x86_64-linux-gnu/libc.so.6(+0x9caa4)[0x70bbb649caa4]
/lib/x86_64-linux-gnu/libc.so.6(+0x129c3c)[0x70bbb6529c3c]

End of stacktrace
Stack trace of LAL functions in worker thread:
InitDopplerSkyScan at /home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/LIBC215/TARGET/linux-x86_64/EinsteinAtHome/source/lalsuite/lalpulsar/lib/DopplerScan.c:291
At lowest level status code = 0: NO LAL ERROR REGISTERED
Warning:  Program terminating, but clFFT resources not freed.
Please consider explicitly calling clfftTeardown( ).
putenv 'LAL_DEBUG_LEVEL=3'
2025-02-25 22:57:48.4325 (2988) [normal]: This program is published under the GNU General Public License, version 2
2025-02-25 22:57:48.4326 (2988) [normal]: For details see http://einstein.phys.uwm.edu/license.php
2025-02-25 22:57:48.4326 (2988) [normal]: This Einstein@home App was built at: Nov  9 2023 13:12:46

2025-02-25 22:57:48.4326 (2988) [normal]: Start of BOINC application '../../projects/einstein.phys.uwm.edu/einstein_O3AS_1.16_x86_64-pc-linux-gnu__GW-opencl-ati-3'.
[DEBUG} GPU type: 2
[DEBUG} got GPU info from BOINC
[DEBUG} got VendorID 4098
2025-02-25 22:57:48.4883 (2988) [debug]: Flags: LAL_DEBUG, OPTIMIZE, HS_OPTIMIZATION, GC_SSE2_OPT, X64, SSE, SSE2, GNUC X86 GNUX86
2025-02-25 22:57:48.4883 (2988) [debug]: glibc version/release: 2.39/stable
2025-02-25 22:57:48.488380 - mytime()
2025-02-25 22:57:48.4885 (2988) [debug]: Set up communication with graphics process.
2025-02-25 22:57:48.4908 (2988) [normal]: Parsed user input successfully
Code-version: %% LAL: 7.1.4.1 (CLEAN )
%% LALPulsar: 3.1.0.1 (CLEAN )
%% LALApps: 7.3.0.1 (CLEAN )

2025-02-25 22:57:48.4909 (2988) [normal]: Initialise compartments with freqWidth = 0.05 and candidates per compartment = 10000.
2025-02-25 22:57:48.4910 (2988) [debug]: Opening temp output file '../../projects/einstein.phys.uwm.edu/h1_0190.80_O3aLC01Cl1In0__O3ASBu_191.00Hz_70749_0_0.tmp' for writing ...
2025-02-25 22:57:48.7145 (2988) [normal]: Reading input data ... 
2025-02-25 22:57:48.7145 (2988) [normal]: Loading SFTs matching '../../projects/einstein.phys.uwm.edu/h1_0190.80_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0190.80_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0191.00_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0191.00_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0191.20_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0191.20_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0191.40_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0191.40_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0191.60_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0191.60_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0191.80_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0191.80_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0192.00_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0192.00_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0192.20_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0192.20_O3aLC01Cl1In0' into catalog ...2025-02-25 22:57:49.7314 (2988) [normal]: done.
2025-02-25 22:57:49.7314 (2988) [normal]: Validating SFTs (detectors: H1, L1, ) ... success.
2025-02-25 22:57:56.5295 (2988) [normal]: Search FstatMethod used: 'ResampGPU'
2025-02-25 22:57:56.5296 (2988) [normal]: Recalc FstatMethod used: 'DemodSSE'
2025-02-25 22:57:56.5296 (2988) [normal]: GPU Device used for Search/Recalc and/or semi coherent step: 'gfx1010:xnack- ( Platform: AMD Accelerated Parallel Processing )'
2025-02-25 22:57:56.5296 (2988) [normal]: GPU Backend used for Search/Recalc and/or semi coherent step: 'OpenCL'
2025-02-25 22:57:56.5296 (2988) [normal]: GPU version is used for the semi-coherent step!
2025-02-25 22:58:23.1980 (2988) [normal]: Number of segments: 18, total number of SFTs in segments: 5748
2025-02-25 22:58:23.2399 (2988) [normal]: Finished reading input data.
2025-02-25 22:58:23.2401 (2988) [normal]: GPS reference time = 1246070525.0000 ,  GPS data mid time = 1246070525.0000
2025-02-25 22:58:23.2402 (2988) [normal]: dFreqStack = 1.000000e-06, df1dot = 7.500000e-11, df2dot = 0.000000e+00, df3dot = 0.000000e+00
2025-02-25 22:58:23.2403 (2988) [normal]: Setup, N = 18, T = 878400 s, Tobs = 15809012 s, gammaRefine = 250, gamma2Refine = 1098, gamma3Refine = 1

DEPRECATION WARNING: program has invoked obsolete function InitDopplerSkyScan(). Please see XLALInitDopplerSkyScan() for information about a replacement.
2025-02-25 22:58:38.7894 (2988) [normal]: Successfully read checkpoint:2220
2025-02-25 22:58:38.8336 (2988) [normal]: Cpt:2220,  total:9000,  sky:56/225,  f1dot:21/40

55.2025-02-25 22:58:38.8350 (2988) [normal]: CG:18010710 FG:1000000  f1dotmin_fg:-1.23735059761e-09 df1dot_fg:2.988047808765e-13 f2dotmin_fg:0 df2dot_fg:0 f3dotmin_fg:0 df3dot_fg:1
...................
56........................................
57............c
............................
58........................................
59........................................
60........................................
61...................c
.....................
62........................................
63........................................
64........................................
65.............................c
...........
66........................................
67........................................
68........................................
69.......................................c
.
70........................................
71........................................
72........................................
73........................................
74.........c
...............................
75........................................
76........................................
77........................................
78...................c
.....................
79........................................
80........................................
81........................................
82.................................c
.......
83........................................
84........................................
85........................................
86........................................
87....c
....................................
88........................................
89........................................
90........................................
91...............c
.........................
92........................................
93........................................
94........................................
95............................c
............
96........................................
97........................................
98........................................
99........................................c

100........................................
101........................................
102........................................
103........................................
104............c
............................
105........................................
106........................................
107........................................
108.......................c
.................
109........................................
110........................................
111........................................
112.....................................c
...
113........................................
114........................................
115........................................
116........................................
117..........c
..............................
118........................................
119........................................
120........................................
121.......................c
.................
122........................................
123........................................
124........................................
125....................................c
....
126........................................
127........................................
128........................................
129........................................
130.........c
...............................
131........................................
132........................................
133........................................
134.......................c
.................
135........................................
136........................................
137........................................
138....................................c
....
139........................................
140........................................
141........................................
142........................................
143.............c
...........................
144........................................
145........................................
146........................................
147............................c
............
148........................................
149........................................
150........................................
151........................................
152.c
.......................................
153........................................
154........................................
155........................................
156..................c
......................
157........................................
158........................................
159........................................
160.................................c
.......
161........................................
162........................................
163........................................
164........................................
165......c
..................................
166........................................
167........................................
168........................................
169....................c
....................
170........................................
171........................................
172........................................
173.....................................c
...
174........................................
175........................................
176........................................
177........................................
178...........c
.............................
179........................................
180........................................
181........................................
182..............................c
..........
183........................................
184........................................
185........................................
186........................................
187.....c
...................................
188........................................
189........................................
190........................................
191.....................c
...................
192........................................
193........................................
194........................................
195...................................c
.....
196........................................
197........................................
198........................................
199........................................
200........c
................................
201........................................
202........................................
203........................................
204.......................c
.................
205........................................
206........................................
207........................................
208......................................c
..
209........................................
210........................................
211........................................
212........................................
213.............c
...........................
214........................................
215........................................
216........................................
217..............................c
..........
218........................................
219........................................
220........................................
221........................................
222.....c
...................................
223........................................
224........................................
2025-02-25 23:57:23.2954 (2988) [normal]: Finished main analysis.
2025-02-25 23:57:23.2954 (2988) [normal]: Recalculating statistics for the final toplist(s)...
2025-02-25 23:59:11.5065 (2988) [normal]: Finished recalculating toplist statistics.
2025-02-25 23:59:11.5065 (2988) [normal]: Finished in 0.00 s with peak RAM usage: 1467.1 MB on CPU 'AMD Ryzen 9 5900X 12-Core Processor            ', peak VRAM usage: 1838.0 MB on GPU Device: 'gfx1010:xnack- ( Platform: AMD Accelerated Parallel Processing )' with backend: 'OpenCL'.
2025-02-25 23:59:11.6818 (2988) [debug]: Writing output ... Closing temp output file '../../projects/einstein.phys.uwm.edu/h1_0190.80_O3aLC01Cl1In0__O3ASBu_191.00Hz_70749_0_0.tmp' ... renaming temp output file '../../projects/einstein.phys.uwm.edu/h1_0190.80_O3aLC01Cl1In0__O3ASBu_191.00Hz_70749_0_0.tmp' to '../../projects/einstein.phys.uwm.edu/h1_0190.80_O3aLC01Cl1In0__O3ASBu_191.00Hz_70749_0_0' ... done.
2025-02-25 23:59:11.9182 (2988) [normal]: Restarted from checkpoint 2220

DEPRECATION WARNING: program has invoked obsolete function FreeDopplerSkyScan(). Please see XLALDestroyDopplerSkyScan() for information about a replacement.
Code-version: %% LAL: 7.1.4.1 (CLEAN )
%% LALPulsar: 3.1.0.1 (CLEAN )
%% LALApps: 7.3.0.1 (CLEAN )

FPU status flags:  COND_1 PRECISION
2025-02-25 23:59:12.0169 (2988) [debug]: worker done. return(0) to caller
2025-02-25 23:59:12.0169 (2988) [normal]: done. calling boinc_finish(0).
23:59:12 (2988): called boinc_finish

</stderr_txt>
]]>