tasks starting with h1_0903.80_O3aC01Cl1In0 error out

Forums › Cruncher's Corner

Greg_BE
Greg_BE
Joined: 15 Aug 08
Posts: 90
Credit: 106642159
RAC: 22507
8 Jul 2023 16:42:23 UTC
Topic 229772
(moderation:
    )

    This is just one of 45 tasks that blew up: https://einsteinathome.org/task/1494807644

    DEPRECATION WARNING: program has invoked obsolete function InitDopplerSkyScan(). Please see XLALInitDopplerSkyScan() for information about a replacement.
2023-07-08 04:33:48.3169 (5656) [normal]: INFO: No checkpoint checkpoint.cpt found - starting from scratch
2023-07-08 04:33:48.3179 (5656) [normal]: Cpt:0,  total:3000,  sky:1/150,  f1dot:1/20

    0.2023-07-08 04:33:48.3179 (5656) [normal]: CG:36521827 FG:500000 f1dotmin_fg:-2.773529411765e-009 df1dot_fg:2.941176470588e-012 f2dotmin_fg:0 df2dot_fg:0 f3dotmin_fg:0 df3dot_fg:1
    XLAL Error - XLALOpenCLExecuteKernel (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lalpulsar/lib/GPUUtils/OpenCLUtils.c:652): Enqueue OpenCL kernel failed with OpenCL error: CL_MEM_OBJECT_ALLOCATION_FAILURE
    XLAL Error - XLALOpenCLExecuteKernel (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lalpulsar/lib/GPUUtils/OpenCLUtils.c:652): Generic failure
    XLAL Error - XLALLoopOverCoarseGridFrequencyBins_OpenCL (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lalapps/src/pulsar/GCT/HierarchSearchGCT_OpenCL.c:65): Check failed: XLALOpenCLExecuteKernel ( &(GCTOpenCLKernels.kernel_XLALLoopOverCoarseGridFrequencyBins), &numFreqBins, 1 ) == XLAL_SUCCESS
    XLAL Error - XLALLoopOverCoarseGridFrequencyBins_OpenCL (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lalapps/src/pulsar/GCT/HierarchSearchGCT_OpenCL.c:65): Internal function call failed: Generic failure
    XLAL Error - XLALLoopOverCoarseGridFrequencyBins_GPU (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lalapps/src/pulsar/GCT/HierarchSearchGCT.c:4495): Check failed: (*funcs->LoopOverCoarseGridFrequencyBins) (Fstat_resTwoFPerDet, Fstat_resTwoF , freq_event, myf0, dFreqStack, f1dot_event, A1, B1, u1start, u1winInv, ifreq, k, numFreqBins, numDetectors, freqlength, coarsegrid->TwoF_GPU->data, TwoFX, Fstat_detectorNames->data, coarsegrid->length, detectorIDs_GPU, computeBSGL) == XLAL_SUCCESS
    XLAL Error - XLALLoopOverCoarseGridFrequencyBins_GPU (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lalapps/src/pulsar/GCT/HierarchSearchGCT.c:4495): Internal function call failed: Generic failure
    XLAL Error - MAIN (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lalapps/src/pulsar/GCT/HierarchSearchGCT.c:2060): Check failed: XLALLoopOverCoarseGridFrequencyBins_GPU(Fstat_res, freq_event, myf0, dFreqStack, f1dot_event, A1, B1, u1start, u1winInv, &coarsegrid, ifreq, k, Fstat_detectorNames, detectorIDs_GPU, uvar->computeBSGL, usefulParams.gct_gpu_funcs ) == XLAL_SUCCESS
    XLAL Error - MAIN (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lalapps/src/pulsar/GCT/HierarchSearchGCT.c:2060): Internal function call failed: Generic failure
    2023-07-08 04:33:48.4780 (5656) [CRITICAL]: ERROR: MAIN() returned with error '1'
    Code-version: %% LAL: 7.1.4.1 (CLEAN 0f21cd857dfe0a6e5457bff62dcdb43abc9d5cfa)
    %% LALPulsar: 3.1.0.1 (CLEAN 0f21cd857dfe0a6e5457bff62dcdb43abc9d5cfa)
    %% LALApps: 7.3.0.1 (CLEAN 0f21cd857dfe0a6e5457bff62dcdb43abc9d5cfa)


    Ian&Steve C.
    Ian&Steve C.
    Joined: 19 Jan 20
    Posts: 4015
    Credit: 47626808380
    RAC: 43921845

    Are you trying to run 2 or

    8 Jul 2023 16:45:03 UTC
    Message 214588
    (moderation:
      )

      Are you trying to run 2 or three at a time? 
       

      your errors stem from a “CL_MEM_OBJECT_ALLOCATION” which usually happens when you run out of VRAM 

      _________________________________________________________________________

      Greg_BE
      Greg_BE
      Joined: 15 Aug 08
      Posts: 90
      Credit: 106642159
      RAC: 22507

      But...I see that the person

      8 Jul 2023 16:46:09 UTC
      Message 214589
      (moderation:
        )

        But...I see that the person before me completed it ok.

        They have:

        Coprocessors: NVIDIA NVIDIA GeForce RTX 3060 (12287MB) driver: 531.61

        Operating system: Microsoft Windows 11 Professional x64 Edition, (10.00.22621.00)

        I run Win10 with a GTX1080 or it might have hit my 1050.

        Wingman just got it...so have to wait.

        Ian&Steve C.
        Ian&Steve C.
        Joined: 19 Jan 20
        Posts: 4015
        Credit: 47626808380
        RAC: 43921845

        Are you running 2 or more at

        8 Jul 2023 16:49:01 UTC
        Message 214590
        (moderation:
          )

          Are you running 2 or more at a time? 

          _________________________________________________________________________

          Ian&Steve C.
          Ian&Steve C.
          Joined: 19 Jan 20
          Posts: 4015
          Credit: 47626808380
          RAC: 43921845

          You can see in the error log

          8 Jul 2023 16:55:16 UTC
          Message 214591
          (moderation:
            )

            You can see in the error log that all your error tasks tried to run on your 1050Ti. You have 3 successful runs on your 1080. 
             

            if you are running 2x per GPU. Change to 1x. 
             

            if you are already running 1x, then you will need to exclude the 1050ti from running the Gravitational Wave application by adding the appropriate exclude arguments to your cc_config.xml file. 

            _________________________________________________________________________

            Greg_BE
            Greg_BE
            Joined: 15 Aug 08
            Posts: 90
            Credit: 106642159
            RAC: 22507

            Put in a cc_config directing

            9 Jul 2023 8:52:14 UTC
            Message 214616
            (moderation:
              )

              Put in a cc_config directing BOINC to use 1 cpu.

              I think that's how its supposed to work, been a long time since I have had to do that.

               

              I thought there was something in app_config that was similar, but I was looking at it really late last night and it didn't make sense.

              Comment viewing options

              Select your preferred way to display the comments and click "Save settings" to activate your changes.

              Forums › Cruncher's Corner