Computation Error on GW Search

DollarD
DollarD
Joined: 6 Jun 05
Posts: 2
Credit: 73617963
RAC: 0
Topic 223862

Since about 2 weeks ago I'm getting computation errors on all my tasks:

https://einsteinathome.org/task/1023968851

<core_client_version>7.16.11</core_client_version>
<![CDATA[
<message>
(unknown error) - exit code 1024 (0x400)</message>
<stderr_txt>
putenv 'LAL_DEBUG_LEVEL=3'
2020-10-30 15:53:49.2773 (600) [normal]: This program is published under the GNU General Public License, version 2
2020-10-30 15:53:49.2803 (600) [normal]: For details see http://einstein.phys.uwm.edu/license.php
2020-10-30 15:53:49.2813 (600) [normal]: This Einstein@home App was built at: Dec 19 2019 12:14:49

2020-10-30 15:53:49.2823 (600) [normal]: Start of BOINC application 'projects/einstein.phys.uwm.edu/einstein_O2MDF_2.07_windows_x86_64__GW-opencl-nvidia.exe'.
Activated exception handling...
[DEBUG} GPU type: 1
[DEBUG} got GPU info from BOINC
[DEBUG} got VendorID 4318
2020-10-30 15:53:49.3273 (600) [debug]: BSGL output files
2020-10-30 15:53:49.3343 (600) [debug]: Flags: LAL_DEBUG, OPTIMIZE, HS_OPTIMIZATION, GC_SSE2_OPT, X64, SSE, SSE2, GNUC X86 GNUX86
2020-10-30 15:53:49.3883 (600) [debug]: Set up communication with graphics process.

DEPRECATION WARNING: program has invoked obsolete function XLALGetVersionString(). Please see XLALVCSInfoString() for information about a replacement.
Code-version: %% LAL: 6.19.2.1 (CLEAN 98bbe72a728eb25935e9195dafae691335dabf8c)
%% LALPulsar: 1.17.1.1 (CLEAN 98bbe72a728eb25935e9195dafae691335dabf8c)
%% LALApps: 6.23.0.1 (CLEAN 98bbe72a728eb25935e9195dafae691335dabf8c)

2020-10-30 15:53:49.9503 (600) [normal]: Reading input data ... 2020-10-30 15:54:39.6981 (600) [normal]: Search FstatMethod used: 'ResampOpenCL'
2020-10-30 15:54:39.6981 (600) [normal]: Recalc FstatMethod used: 'DemodSSE'
2020-10-30 15:54:39.6991 (600) [normal]: OpenCL Device used for Search/Recalc and/or semi coherent step: 'GeForce GTX 960 (Platform: NVIDIA CUDA, global memory: 2048 MiB)'
2020-10-30 15:54:39.7001 (600) [normal]: OpenCL version is used for the semi-coherent step!
2020-10-30 15:54:58.7380 (600) [normal]: Number of segments: 12, total number of SFTs in segments: 10192
done.
% --- GPS reference time = 1177858472.0000 , GPS data mid time = 1177858472.0000
2020-10-30 15:54:58.7750 (600) [normal]: dFreqStack = 2.251046e-007, df1dot = 5.685400e-013, df2dot = 4.020648e-019, df3dot = 0.000000e+000
% --- Setup, N = 12, T = 1296000 s, Tobs = 19750204 s, gammaRefine = 9, gamma2Refine = 23, gamma3Refine = 1

DEPRECATION WARNING: program has invoked obsolete function InitDopplerSkyScan(). Please see XLALInitDopplerSkyScan() for information about a replacement.
2020-10-30 15:54:58.7850 (600) [normal]: INFO: No checkpoint checkpoint.cpt found - starting from scratch
% --- Cpt:0, total:25, sky:1/1, f1dot:1/25

0.% --- CG:2669916 FG:222119 f1dotmin_fg:-3.753928118444e-008 df1dot_fg:6.317111111111e-014 f2dotmin_fg:-1.922918608696e-019 df2dot_fg:1.748107826087e-020 f3dotmin_fg:0 df3dot_fg:1
XLAL Error - XLALComputeECLFFT_OpenCL (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lalpulsar/src/ComputeFstat_Resamp_OpenCL.c:1248): Processing FFT failed: CL_MEM_OBJECT_ALLOCATION_FAILURE
XLAL Error - XLALComputeECLFFT_OpenCL (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lalpulsar/src/ComputeFstat_Resamp_OpenCL.c:1248): Internal function call failed
XLAL Error - XLALComputeFaFb_Resamp_OpenCL (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lalpulsar/src/ComputeFstat_Resamp_OpenCL.c:654): Check failed: (*fftfuncs->computefft_func) ( fftfuncs->fftplan, ws->TS_FFT, ((void *)0) ) == XLAL_SUCCESS
XLAL Error - XLALComputeFaFb_Resamp_OpenCL (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lalpulsar/src/ComputeFstat_Resamp_OpenCL.c:654): Internal function call failed
XLAL Error - XLALComputeFstatResamp_OpenCL (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lalpulsar/src/ComputeFstat_Resamp_OpenCL.c:441): Check failed: XLALComputeFaFb_Resamp_OpenCL ( resamp, ws, thisPoint, common->dFreq, numFreqBins, TimeSeriesX_SRC_a, TimeSeriesX_SRC_b ) == XLAL_SUCCESS
XLAL Error - XLALComputeFstatResamp_OpenCL (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lalpulsar/src/ComputeFstat_Resamp_OpenCL.c:441): Internal function call failed
XLAL Error - XLALComputeFstat (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lalpulsar/src/ComputeFstat.c:875): Check failed: (input->method_funcs.compute_func) ( *Fstats, common, input->method_data ) == XLAL_SUCCESS
XLAL Error - XLALComputeFstat (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lalpulsar/src/ComputeFstat.c:875): Internal function call failed
MAIN: XLALComputeFstat() failed with errno=1024
2020-10-30 15:54:59.5965 (600) [CRITICAL]: ERROR: MAIN() returned with error '1024'
2020-10-30 15:54:59.5965 (600) [debug]: resultfile '../../projects/einstein.phys.uwm.edu/h1_0415.25_O2C02Cl4In0__O2MDFS2_Spotlight_415.75Hz_2017_1_0' (len 96), current config file: 0
Code-version: %% LAL: 6.19.2.1 (CLEAN 98bbe72a728eb25935e9195dafae691335dabf8c)
%% LALPulsar: 1.17.1.1 (CLEAN 98bbe72a728eb25935e9195dafae691335dabf8c)
%% LALApps: 6.23.0.1 (CLEAN 98bbe72a728eb25935e9195dafae691335dabf8c)

FPU status flags: PRECISION
2020-10-30 15:54:59.6005 (600) [debug]: worker done. return(1024) to caller
2020-10-30 15:54:59.6015 (600) [normal]: done. calling boinc_finish(1024).
15:54:59 (600): called boinc_finish

</stderr_txt>
]]>

 

Any ideas?

 

Richie
Richie
Joined: 7 Mar 14
Posts: 656
Credit: 1702989778
RAC: 0

DollarD wrote:2020-10-30

DollarD wrote:
2020-10-30 15:54:39.6991 (600) [normal]: OpenCL Device used for Search/Recalc and/or semi coherent step: 'GeForce GTX 960 (Platform: NVIDIA CUDA, global memory: 2048 MiB)'

Those GW GPU tasks need more memory than what your GTX 960 with 2GB has. Currently a fail safe solution with 2GB GPUs would be deselecting 'Gravitational Wave search O2 Multi-Directional GPU' app on the project preferences. Your GPU is able to run tasks from 'Gamma-ray pulsar binary search #1 on GPUs' though.

DollarD
DollarD
Joined: 6 Jun 05
Posts: 2
Credit: 73617963
RAC: 0

Thanks! I'll do so in the

Thanks! I'll do so in the meantime

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.