Computation Error: Output files absent

Forums › Problems and Bug Reports

Dr_Mabuse
Dr_Mabuse
Joined: 9 May 05
Posts: 11
Credit: 9389045
RAC: 26
17 Oct 2020 12:43:48 UTC
Topic 223735
(moderation:
    )

    Hello experts

    I got again this kiind of error:

    17.10.2020 14:25:32 | Einstein@Home | Computation for task h1_0383.45_O2C02Cl4In0__O2MDFS2_Spotlight_383.95Hz_1585_1 finished
    17.10.2020 14:25:32 | Einstein@Home | Output file h1_0383.45_O2C02Cl4In0__O2MDFS2_Spotlight_383.95Hz_1585_1_0 for task h1_0383.45_O2C02Cl4In0__O2MDFS2_Spotlight_383.95Hz_1585_1 absent
    17.10.2020 14:25:32 | Einstein@Home | Output file h1_0383.45_O2C02Cl4In0__O2MDFS2_Spotlight_383.95Hz_1585_1_1 for task h1_0383.45_O2C02Cl4In0__O2MDFS2_Spotlight_383.95Hz_1585_1 absent
    17.10.2020 14:25:32 | Einstein@Home | Output file h1_0383.45_O2C02Cl4In0__O2MDFS2_Spotlight_383.95Hz_1585_1_2 for task h1_0383.45_O2C02Cl4In0__O2MDFS2_Spotlight_383.95Hz_1585_1 absent
    17.10.2020 14:25:45 | Einstein@Home | Starting task h1_0383.45_O2C02Cl4In0__O2MDFS2_Spotlight_383.95Hz_1592_2
    17.10.2020 14:25:46 | Einstein@Home | Started upload of h1_0383.45_O2C02Cl4In0__O2MDFS2_Spotlight_383.95Hz_1585_1_3
    17.10.2020 14:25:48 | Einstein@Home | Finished upload of h1_0383.45_O2C02Cl4In0__O2MDFS2_Spotlight_383.95Hz_1585_1_3
    17.10.2020 14:25:55 |  | Suspending computation - CPU is busy
    17.10.2020 14:26:05 |  | Resuming computation
    17.10.2020 14:26:46 | Einstein@Home | Project requested delay of 60 seconds
    17.10.2020 14:31:25 | Einstein@Home | Computation for task h1_0383.45_O2C02Cl4In0__O2MDFS2_Spotlight_383.95Hz_1592_2 finished
    17.10.2020 14:31:25 | Einstein@Home | Output file h1_0383.45_O2C02Cl4In0__O2MDFS2_Spotlight_383.95Hz_1592_2_0 for task h1_0383.45_O2C02Cl4In0__O2MDFS2_Spotlight_383.95Hz_1592_2 absent
    17.10.2020 14:31:25 | Einstein@Home | Output file h1_0383.45_O2C02Cl4In0__O2MDFS2_Spotlight_383.95Hz_1592_2_1 for task h1_0383.45_O2C02Cl4In0__O2MDFS2_Spotlight_383.95Hz_1592_2 absent
    17.10.2020 14:31:25 | Einstein@Home | Output file h1_0383.45_O2C02Cl4In0__O2MDFS2_Spotlight_383.95Hz_1592_2_2 for task h1_0383.45_O2C02Cl4In0__O2MDFS2_Spotlight_383.95Hz_1592_2 absent
    17.10.2020 14:31:47 | Einstein@Home | Starting task h1_0383.45_O2C02Cl4In0__O2MDFS2_Spotlight_383.95Hz_1591_2
    17.10.2020 14:31:49 | Einstein@Home | Started upload of h1_0383.45_O2C02Cl4In0__O2MDFS2_Spotlight_383.95Hz_1592_2_3
    17.10.2020 14:31:51 | Einstein@Home | Finished upload of h1_0383.45_O2C02Cl4In0__O2MDFS2_Spotlight_383.95Hz_1592_2_3
    17.10.2020 14:33:10 | Einstein@Home | Project requested delay of 60 seconds
    17.10.2020 14:35:46 | Einstein@Home | Computation for task h1_0383.45_O2C02Cl4In0__O2MDFS2_Spotlight_383.95Hz_1591_2 finished
    17.10.2020 14:35:46 | Einstein@Home | Output file h1_0383.45_O2C02Cl4In0__O2MDFS2_Spotlight_383.95Hz_1591_2_0 for task h1_0383.45_O2C02Cl4In0__O2MDFS2_Spotlight_383.95Hz_1591_2 absent
    17.10.2020 14:35:46 | Einstein@Home | Output file h1_0383.45_O2C02Cl4In0__O2MDFS2_Spotlight_383.95Hz_1591_2_1 for task h1_0383.45_O2C02Cl4In0__O2MDFS2_Spotlight_383.95Hz_1591_2 absent
    17.10.2020 14:35:46 | Einstein@Home | Output file h1_0383.45_O2C02Cl4In0__O2MDFS2_Spotlight_383.95Hz_1591_2_2 for task h1_0383.45_O2C02Cl4In0__O2MDFS2_Spotlight_383.95Hz_1591_2 absent

    What can I do to prvent this error ?

    Please help

    thanks

    Jochen

     

    Harri Liljeroos
    Harri Liljeroos
    Joined: 10 Dec 05
    Posts: 4311
    Credit: 3187984872
    RAC: 1976114

    That is not the error, that

    17 Oct 2020 12:56:07 UTC
    Message 180495
    (moderation:
      )

      That is not the error, that is the result of the error. Because there was an error the result files are absent. You have to look at the stderr output for the actual error. That can be found here on the web site on your account page where your tasks are listed. 

      Richie
      Richie
      Joined: 7 Mar 14
      Posts: 656
      Credit: 1702989778
      RAC: 0

      Hi! It is because of

      17 Oct 2020 13:01:31 UTC
      Message 180496
      (moderation:
        )

        Hi!

        It is because of "OpenCL Device used for Search/Recalc and/or semi coherent step: 'GeForce GTX 1050 (Platform: NVIDIA CUDA, global memory: 2048 MiB)". All those tasks that failed had so called DF of 0.50 . They required more memory than what your 2GB GPU is able to provide. Some GW GPU tasks with lower DF could run with a 2GB card, but a user isn't able to choose what kind of tasks the project server will send, considering the DF.

        It would be a fail safe solution to run only "Gamma-ray pulsar binary search #1 on GPUs" with a 2GB card.

        Paul Vleugels
        Paul Vleugels
        Joined: 1 Mar 05
        Posts: 12
        Credit: 33200970
        RAC: 24489

        Hello Harri, Last 2 weeks

        21 Oct 2020 16:01:45 UTC
        Message 180588
        (moderation:
          )

          Hello Harri,

          Last 2 weeks I'm also facing the same problem mentioned in this ticket.

          Many WU's were cancelled after +/- 2 min CPU time. Based on your comment I had a look to the stderr output, but there're a lot of errors being mentioned (like unknown with exit code 1024) that I've no clue what they mean or how to correct this. An insider / developer will understand for sure, but I'm not.

          This is the stderr of one of the failing WU's, can you please have a look?

          Task 1015155880

          Naam: h1_0397.30_O2C02Cl4In0__O2MDFS2_Spotlight_397.80Hz_1937_0

          Werkeenheid ID: 493618356

          Aangemaakt: 7 Oct 2020 9:11:55 UTC

          Verzonden: 12 Oct 2020 6:35:52 UTC

          Rapporteren voor: 19 Oct 2020 6:35:52 UTC

          Ontvangen: 12 Oct 2020 15:38:52 UTC

          Server status: Over

          Uitkomst: Computation error

          Client status: Compute error

          Afsluit status: 1024 (0x00000400) Unknown error code

          Computer: 12535157

          Run time (sec): 107.04

          CPU time (sec): 102.80

          Peak working set size (MB): 373.27

          Peak swap size (MB): 2075.45

          Peak disk usage (MB): 0.02

          Validatie status: Invalid

          Toegekend punten: 0

          Applicatie: Gravitational Wave search O2 Multi-Directional GPU v2.07 (GW-opencl-nvidia)
          windows_x86_64

          Stderr output

          <core_client_version>7.16.11</core_client_version>
<![CDATA[
<message>
(unknown error) - exit code 1024 (0x400)</message>
<stderr_txt>
putenv 'LAL_DEBUG_LEVEL=3'
2020-10-12 17:32:16.1058 (63356) [normal]: This program is published under the GNU General Public License, version 2
2020-10-12 17:32:16.1058 (63356) [normal]: For details see http://einstein.phys.uwm.edu/license.php
2020-10-12 17:32:16.1138 (63356) [normal]: This Einstein@home App was built at: Dec 19 2019 12:14:49

          2020-10-12 17:32:16.1159 (63356) [normal]: Start of BOINC application 'projects/einstein.phys.uwm.edu/einstein_O2MDF_2.07_windows_x86_64__GW-opencl-nvidia.exe'.
          Activated exception handling...
          [DEBUG} GPU type: 1
          [DEBUG} got GPU info from BOINC
          [DEBUG} got VendorID 4318
          2020-10-12 17:32:16.4763 (63356) [debug]: BSGL output files
          2020-10-12 17:32:16.4883 (63356) [debug]: Flags: LAL_DEBUG, OPTIMIZE, HS_OPTIMIZATION, GC_SSE2_OPT, X64, SSE, SSE2, GNUC X86 GNUX86
          2020-10-12 17:32:16.4883 (63356) [debug]: Set up communication with graphics process.

          DEPRECATION WARNING: program has invoked obsolete function XLALGetVersionString(). Please see XLALVCSInfoString() for information about a replacement.
          Code-version: %% LAL: 6.19.2.1 (CLEAN 98bbe72a728eb25935e9195dafae691335dabf8c)
          %% LALPulsar: 1.17.1.1 (CLEAN 98bbe72a728eb25935e9195dafae691335dabf8c)
          %% LALApps: 6.23.0.1 (CLEAN 98bbe72a728eb25935e9195dafae691335dabf8c)

          2020-10-12 17:32:17.5658 (63356) [normal]: Reading input data ... 2020-10-12 17:33:32.9681 (63356) [normal]: Search FstatMethod used: 'ResampOpenCL'
          2020-10-12 17:33:32.9681 (63356) [normal]: Recalc FstatMethod used: 'DemodSSE'
          2020-10-12 17:33:32.9701 (63356) [normal]: OpenCL Device used for Search/Recalc and/or semi coherent step: 'GeForce GTX 950M (Platform: NVIDIA CUDA, global memory: 2048 MiB)'
          2020-10-12 17:33:32.9701 (63356) [normal]: OpenCL version is used for the semi-coherent step!
          2020-10-12 17:33:59.8248 (63356) [normal]: Number of segments: 12, total number of SFTs in segments: 10192
          done.
          % --- GPS reference time = 1177858472.0000 , GPS data mid time = 1177858472.0000
          2020-10-12 17:33:59.8896 (63356) [normal]: dFreqStack = 2.251046e-007, df1dot = 5.685400e-013, df2dot = 4.020648e-019, df3dot = 0.000000e+000
          % --- Setup, N = 12, T = 1296000 s, Tobs = 19750204 s, gammaRefine = 9, gamma2Refine = 23, gamma3Refine = 1

          DEPRECATION WARNING: program has invoked obsolete function InitDopplerSkyScan(). Please see XLALInitDopplerSkyScan() for information about a replacement.
          2020-10-12 17:33:59.9066 (63356) [normal]: INFO: No checkpoint checkpoint.cpt found - starting from scratch
          % --- Cpt:0, total:26, sky:1/1, f1dot:1/26

          0.% --- CG:2669916 FG:222119 f1dotmin_fg:-3.753135252444e-008 df1dot_fg:6.317111111111e-014 f2dotmin_fg:-1.922918608696e-019 df2dot_fg:1.748107826087e-020 f3dotmin_fg:0 df3dot_fg:1
          XLAL Error - XLALComputeECLFFT_OpenCL (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lalpulsar/src/ComputeFstat_Resamp_OpenCL.c:1248): Processing FFT failed: CL_MEM_OBJECT_ALLOCATION_FAILURE
          XLAL Error - XLALComputeECLFFT_OpenCL (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lalpulsar/src/ComputeFstat_Resamp_OpenCL.c:1248): Internal function call failed
          XLAL Error - XLALComputeFaFb_Resamp_OpenCL (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lalpulsar/src/ComputeFstat_Resamp_OpenCL.c:654): Check failed: (*fftfuncs->computefft_func) ( fftfuncs->fftplan, ws->TS_FFT, ((void *)0) ) == XLAL_SUCCESS
          XLAL Error - XLALComputeFaFb_Resamp_OpenCL (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lalpulsar/src/ComputeFstat_Resamp_OpenCL.c:654): Internal function call failed
          XLAL Error - XLALComputeFstatResamp_OpenCL (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lalpulsar/src/ComputeFstat_Resamp_OpenCL.c:441): Check failed: XLALComputeFaFb_Resamp_OpenCL ( resamp, ws, thisPoint, common->dFreq, numFreqBins, TimeSeriesX_SRC_a, TimeSeriesX_SRC_b ) == XLAL_SUCCESS
          XLAL Error - XLALComputeFstatResamp_OpenCL (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lalpulsar/src/ComputeFstat_Resamp_OpenCL.c:441): Internal function call failed
          XLAL Error - XLALComputeFstat (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lalpulsar/src/ComputeFstat.c:875): Check failed: (input->method_funcs.compute_func) ( *Fstats, common, input->method_data ) == XLAL_SUCCESS
          XLAL Error - XLALComputeFstat (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lalpulsar/src/ComputeFstat.c:875): Internal function call failed
          MAIN: XLALComputeFstat() failed with errno=1024
          2020-10-12 17:34:00.7417 (63356) [CRITICAL]: ERROR: MAIN() returned with error '1024'
          2020-10-12 17:34:00.7438 (63356) [debug]: resultfile '../../projects/einstein.phys.uwm.edu/h1_0397.30_O2C02Cl4In0__O2MDFS2_Spotlight_397.80Hz_1937_0_0' (len 96), current config file: 0
          Code-version: %% LAL: 6.19.2.1 (CLEAN 98bbe72a728eb25935e9195dafae691335dabf8c)
          %% LALPulsar: 1.17.1.1 (CLEAN 98bbe72a728eb25935e9195dafae691335dabf8c)
          %% LALApps: 6.23.0.1 (CLEAN 98bbe72a728eb25935e9195dafae691335dabf8c)

          FPU status flags: COND_0 PRECISION
          2020-10-12 17:34:00.7438 (63356) [debug]: worker done. return(1024) to caller
          2020-10-12 17:34:00.7438 (63356) [normal]: done. calling boinc_finish(1024).
          17:34:00 (63356): called boinc_finish

          </stderr_txt>
          ]]>





          Richie
          Richie
          Joined: 7 Mar 14
          Posts: 656
          Credit: 1702989778
          RAC: 0

          Paul Vleugels wrote:Many WU's

          21 Oct 2020 17:17:51 UTC
          Message 180589
          (moderation:
            )

            Paul Vleugels wrote:

            Many WU's were cancelled after +/- 2 min CPU time. Based on your comment I had a look to the stderr output, but there're a lot of errors being mentioned (like unknown with exit code 1024) that I've no clue what they mean or how to correct this.

            Naam: h1_0397.30_O2C02Cl4In0__O2MDFS2_Spotlight_397.80Hz_1937_0

            Applicatie: Gravitational Wave search O2 Multi-Directional GPU v2.07 (GW-opencl-nvidia)
            windows_x86_64

            Stderr output

            2020-10-12 17:33:32.9701 (63356) [normal]: OpenCL Device used for Search/Recalc and/or semi coherent step: 'GeForce GTX 950M (Platform: NVIDIA CUDA, global memory: 2048 MiB)'

            Hi Paul !

            Your computer has "NVIDIA GeForce GTX 950M (2048MB) driver: 388.00".

            All the GW GPU tasks that failed had a DF of 0.50 (task ID's had xxxx.30 ... xxx.80 or xxxx.35 ... xxx..85 in their names, 85 - 35 or 80 - 30 = DF of 0.50). Those GW GPU tasks with high DF value will require more than 2GB of GPU memory. That's why they have crashed early in the beginning.

            Some GW GPU tasks with lower DF could run with a 2GB card, but a user isn't able to choose what kind of GW GPU tasks the project server will send, considering the DF.

            Currently it would be a fail safe solution to run only "Gamma-ray pulsar binary search #1 on GPUs" with a 2GB GPU.

            Also, the Nvidia GPU driver 388.00 on your computer is a three years old version. That could be a major problem for any kind of succesfull crunching with Nvidia GPU. I would definitely suggest you to visit Nvidia driver site and upgrade that driver to the latest version even if you didn't run Boinc.

            Paul Vleugels
            Paul Vleugels
            Joined: 1 Mar 05
            Posts: 12
            Credit: 33200970
            RAC: 24489

            Hello Richie, Thanks for

            21 Oct 2020 19:24:14 UTC
            Message 180592 in response to message 180589
            (moderation:
              )

              Hello Richie,

              Thanks for your quick feedback !

              Looking to my Speccy output I can see that my Nvidia GeForce GTX 950 M driver is current'ly '23.21.13.8800'. I wonder where and how you've see '388.00' (maybe your 388 is a sub string within 23.21.13.8800'?

              Finding a driver update at the Nvidia side is not easy (can't find the 950 M at all), but I'll continue anyway.

              Second question: how and where can I set I prefer to process 'Gamma-ray pulsar binary search #1 on GPUs' WU's as you suggest?

              San-Fernando-Valley
              San-Fernando-Valley
              Joined: 16 Mar 16
              Posts: 382
              Credit: 9898423455
              RAC: 24166937

              DRIVER: https://www.nvidia

              21 Oct 2020 19:35:56 UTC
              Message 180593 in response to message 180592
              (moderation:
                )

                DRIVER:

                https://www.nvidia.com/en-us/drivers/results/165685/

                 

                San-Fernando-Valley
                San-Fernando-Valley
                Joined: 16 Mar 16
                Posts: 382
                Credit: 9898423455
                RAC: 24166937

                Gamma Ray:    

                21 Oct 2020 19:39:00 UTC
                Message 180594 in response to message 180592
                (moderation:
                  )

                  Gamma Ray:

                   

                    ACCOUNT

                       PREFERENCE

                          PROJECT

                   

                  make a check for Gamma Ray

                  and don'T forget to "save" change at the bottom

                   

                  Gary Roberts
                  Gary Roberts
                  Moderator
                  Joined: 9 Feb 05
                  Posts: 5872
                  Credit: 117236248758
                  RAC: 36156136

                  Paul Vleugels wrote:... I had

                  21 Oct 2020 20:12:30 UTC
                  Message 180595 in response to message 180588
                  (moderation:
                    )

                    Paul Vleugels wrote:
                    ... I had a look to the stderr output, but there're a lot of errors being mentioned (like unknown with exit code 1024) that I've no clue what they mean or how to correct this. An insider / developer will understand for sure, but I'm not.

                    None of the people likely to reply here are "insiders" - we're all just volunteers like yourself.

                    The trick is not to look at error codes but to look (earlier in the output) for keywords.  In your case, the key information is contained in the line that says:

                    Processing FFT failed: CL_MEM_OBJECT_ALLOCATION_FAILURE

                    The app uses a process referred to as a "Fast Fourier Transform" (FFT) to process a very large number of data points, all of which requires a lot of memory.  The problem is that your GPU doesn't have enough memory to store the data - a memory allocation failure.

                    At earlier stages for this particular search, memory requirements were less so perhaps your card may just have had enough memory earlier on.  As the run progresses, memory requirements are likely to increase even further so even if some tasks are not failing now, the situation is likely to get worse with time.  As a general comment for anyone with a 2GB card, you should avoid the GW search and select the gamma-ray pulsar search instead.  2GB will be fine there.

                    Cheers,
                    Gary.

                    Richie
                    Richie
                    Joined: 7 Mar 14
                    Posts: 656
                    Credit: 1702989778
                    RAC: 0

                    Quote:Paul Vleugels wrote:I

                    21 Oct 2020 20:26:25 UTC
                    Message 180596 in response to message 180592
                    (moderation:
                      )
                      Quote:

                      Paul Vleugels wrote:
                      I wonder where and how you've see '388.00'

                      From the line "Coprosessors:" at the information page of your host... https://einsteinathome.org/host/12535157

                      That's what Boinc currently knows about it.

                      Quote:
                      (maybe your 388 is a sub string within 23.21.13.8800'?

                      Yes, that sub string would tell the version also.

                      Quote:
                      Finding a driver update at the Nvidia side is not easy (can't find the 950 M at all), but I'll continue anyway.

                      San-Fernando-Valley shot with precision already. Well, here would be just an alternative page to start searching for Nvidia drivers if in a need some day: https://www.nvidia.com/Download/Find.aspx?lang=en-us

                      Product Type: GeForce , Product Series: GeForce 900M Series (Notebook) , Product: GeForce GTX 950M , Operating System: Windows 10 64-bit

                      I'm not sure if that very old driver was yet of type 'DCH' but you could try an installation with that type at first.

                      Alternatively, you could first download and install GPU-Z . It would show "DCH" included at the "Driver Version" information if the currently installed driver is a DCH version. If it doesn't show "DCH" then you should choose the 'Standard' version.

                      Paul Vleugels
                      Paul Vleugels
                      Joined: 1 Mar 05
                      Posts: 12
                      Credit: 33200970
                      RAC: 24489

                      Thanks everybody for all your

                      22 Oct 2020 17:21:51 UTC
                      Message 180613 in response to message 180596
                      (moderation:
                        )

                        Thanks everybody for all your advices and tips. Will process it and get back to you soon, enjoy your weekend !

                        Comment viewing options

                        Select your preferred way to display the comments and click "Save settings" to activate your changes.

                        Forums › Problems and Bug Reports