tasks starting with h1_0903.80_O3aC01Cl1In0 error out

Greg_BE
Greg_BE
Joined: 15 Aug 08
Posts: 90
Credit: 104,796,076
RAC: 27,827
Topic 229772

This is just one of 45 tasks that blew up: https://einsteinathome.org/task/1494807644

DEPRECATION WARNING: program has invoked obsolete function InitDopplerSkyScan(). Please see XLALInitDopplerSkyScan() for information about a replacement.
2023-07-08 04:33:48.3169 (5656) [normal]: INFO: No checkpoint checkpoint.cpt found - starting from scratch
2023-07-08 04:33:48.3179 (5656) [normal]: Cpt:0,  total:3000,  sky:1/150,  f1dot:1/20

0.2023-07-08 04:33:48.3179 (5656) [normal]: CG:36521827 FG:500000 f1dotmin_fg:-2.773529411765e-009 df1dot_fg:2.941176470588e-012 f2dotmin_fg:0 df2dot_fg:0 f3dotmin_fg:0 df3dot_fg:1
XLAL Error - XLALOpenCLExecuteKernel (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lalpulsar/lib/GPUUtils/OpenCLUtils.c:652): Enqueue OpenCL kernel failed with OpenCL error: CL_MEM_OBJECT_ALLOCATION_FAILURE
XLAL Error - XLALOpenCLExecuteKernel (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lalpulsar/lib/GPUUtils/OpenCLUtils.c:652): Generic failure
XLAL Error - XLALLoopOverCoarseGridFrequencyBins_OpenCL (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lalapps/src/pulsar/GCT/HierarchSearchGCT_OpenCL.c:65): Check failed: XLALOpenCLExecuteKernel ( &(GCTOpenCLKernels.kernel_XLALLoopOverCoarseGridFrequencyBins), &numFreqBins, 1 ) == XLAL_SUCCESS
XLAL Error - XLALLoopOverCoarseGridFrequencyBins_OpenCL (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lalapps/src/pulsar/GCT/HierarchSearchGCT_OpenCL.c:65): Internal function call failed: Generic failure
XLAL Error - XLALLoopOverCoarseGridFrequencyBins_GPU (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lalapps/src/pulsar/GCT/HierarchSearchGCT.c:4495): Check failed: (*funcs->LoopOverCoarseGridFrequencyBins) (Fstat_resTwoFPerDet, Fstat_resTwoF , freq_event, myf0, dFreqStack, f1dot_event, A1, B1, u1start, u1winInv, ifreq, k, numFreqBins, numDetectors, freqlength, coarsegrid->TwoF_GPU->data, TwoFX, Fstat_detectorNames->data, coarsegrid->length, detectorIDs_GPU, computeBSGL) == XLAL_SUCCESS
XLAL Error - XLALLoopOverCoarseGridFrequencyBins_GPU (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lalapps/src/pulsar/GCT/HierarchSearchGCT.c:4495): Internal function call failed: Generic failure
XLAL Error - MAIN (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lalapps/src/pulsar/GCT/HierarchSearchGCT.c:2060): Check failed: XLALLoopOverCoarseGridFrequencyBins_GPU(Fstat_res, freq_event, myf0, dFreqStack, f1dot_event, A1, B1, u1start, u1winInv, &coarsegrid, ifreq, k, Fstat_detectorNames, detectorIDs_GPU, uvar->computeBSGL, usefulParams.gct_gpu_funcs ) == XLAL_SUCCESS
XLAL Error - MAIN (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lalapps/src/pulsar/GCT/HierarchSearchGCT.c:2060): Internal function call failed: Generic failure
2023-07-08 04:33:48.4780 (5656) [CRITICAL]: ERROR: MAIN() returned with error '1'
Code-version: %% LAL: 7.1.4.1 (CLEAN 0f21cd857dfe0a6e5457bff62dcdb43abc9d5cfa)
%% LALPulsar: 3.1.0.1 (CLEAN 0f21cd857dfe0a6e5457bff62dcdb43abc9d5cfa)
%% LALApps: 7.3.0.1 (CLEAN 0f21cd857dfe0a6e5457bff62dcdb43abc9d5cfa)


Ian&Steve C.
Ian&Steve C.
Joined: 19 Jan 20
Posts: 3,914
Credit: 43,924,399,309
RAC: 63,511,240

Are you trying to run 2 or

Are you trying to run 2 or three at a time? 
 

your errors stem from a “CL_MEM_OBJECT_ALLOCATION” which usually happens when you run out of VRAM 

_________________________________________________________________________

Greg_BE
Greg_BE
Joined: 15 Aug 08
Posts: 90
Credit: 104,796,076
RAC: 27,827

But...I see that the person

But...I see that the person before me completed it ok.

They have:

Coprocessors: NVIDIA NVIDIA GeForce RTX 3060 (12287MB) driver: 531.61

Operating system: Microsoft Windows 11 Professional x64 Edition, (10.00.22621.00)

I run Win10 with a GTX1080 or it might have hit my 1050.

Wingman just got it...so have to wait.

Ian&Steve C.
Ian&Steve C.
Joined: 19 Jan 20
Posts: 3,914
Credit: 43,924,399,309
RAC: 63,511,240

Are you running 2 or more at

Are you running 2 or more at a time? 

_________________________________________________________________________

Ian&Steve C.
Ian&Steve C.
Joined: 19 Jan 20
Posts: 3,914
Credit: 43,924,399,309
RAC: 63,511,240

You can see in the error log

You can see in the error log that all your error tasks tried to run on your 1050Ti. You have 3 successful runs on your 1080. 
 

if you are running 2x per GPU. Change to 1x. 
 

if you are already running 1x, then you will need to exclude the 1050ti from running the Gravitational Wave application by adding the appropriate exclude arguments to your cc_config.xml file. 

_________________________________________________________________________

Greg_BE
Greg_BE
Joined: 15 Aug 08
Posts: 90
Credit: 104,796,076
RAC: 27,827

Put in a cc_config directing

Put in a cc_config directing BOINC to use 1 cpu.

I think that's how its supposed to work, been a long time since I have had to do that.

 

I thought there was something in app_config that was similar, but I was looking at it really late last night and it didn't make sense.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.