Multi GPU not consistantly applied per BOINC's request

Forums › Problems and Bug Reports

Toby Broom
Toby Broom
Joined: 2 Sep 23
Posts: 13
Credit: 302764792
RAC: 731062
6 Nov 2023 19:07:54 UTC
Topic 230322
(moderation:
    )

    Hello,

    I found a bug with the schedulling of work, you don't send the work to the correct GPU consistently, it seems like maybe if 2 WU start close together they both go to GPU 0.

     

    image

    image

     

    mikey
    mikey
    Joined: 22 Jan 05
    Posts: 12770
    Credit: 1852122604
    RAC: 867510

    Do you have <use_all_gpus> in

    6 Nov 2023 21:47:21 UTC
    Message 218904
    (moderation:
      )

      Do you have <use_all_gpus> in the cc_config.xml file?

      Keith Myers
      Keith Myers
      Joined: 11 Feb 11
      Posts: 5013
      Credit: 18905270237
      RAC: 6737980

      Most assuredly does.  The

      7 Nov 2023 0:13:24 UTC
      Message 218911 in response to message 218904
      (moderation:
        )

        Most assuredly does.  The problem is that the Manager shows each task on a different gpu.

        But nvidia-smi shows that actually, BOTH tasks are only on gpu#0

        Should not be possible unless he is setup for 0.5 gpu usage.

        But he is NOT as shown by the Manager listing of 1.0 gpu usage in the Tasks list.

        Something screwy is going on.

         

        Ian&Steve C.
        Ian&Steve C.
        Joined: 19 Jan 20
        Posts: 4041
        Credit: 47941064060
        RAC: 35656070

        ive seen it happen in cases

        7 Nov 2023 0:40:14 UTC
        Message 218912 in response to message 218911
        (moderation:
          )

          ive seen it happen in cases where the app had a bug that put the task on device 0 no matter what BOINC says. i think the SRbase GPU acts like this. and some early beta apps from GPUGRID Python did the same thing.

          this new GW app (new version to reduce VRAM use) was recently released. and it's possible that they made a mistake like that here.

          _________________________________________________________________________

          Bernd Machenschalk
          Bernd Machenschalk
          Moderator
          Administrator
          Joined: 15 Oct 04
          Posts: 4330
          Credit: 251504923
          RAC: 36723

          No, we didn't change anything

          7 Nov 2023 15:12:56 UTC
          Message 218944
          (moderation:
            )

            No, we didn't change anything regarding the device assignment code there. All we changed is some handling / renaming / boinc_resolving of the result files.

            Our app stops when it's not getting proper OpenCL device info from the BOINC Client, it doesn't use an arbitrary device then.

            BM

            Ian&Steve C.
            Ian&Steve C.
            Joined: 19 Jan 20
            Posts: 4041
            Credit: 47941064060
            RAC: 35656070

            thanks for the confirmation

            9 Nov 2023 12:16:41 UTC
            Message 218946
            (moderation:
              )

              thanks for the confirmation Bernd.

              must be something getting confused on the client side then. but I couldn't really guess what would cause this to happen. maybe something wonky with the drivers? worth a try to purge the drivers and re-install, including the opencl libs.

              _________________________________________________________________________

              Richard Haselgrove
              Richard Haselgrove
              Joined: 10 Dec 05
              Posts: 2143
              Credit: 2977404341
              RAC: 781987

              There was a change, many

              7 Nov 2023 17:22:04 UTC
              Message 218953
              (moderation:
                )

                There was a change, many years ago, in the way the device assignment was passed from the BOINC client to the science application. In the early years of GPU crunching, the device number was passed on the command line: in later versions, it was passed in an xml file.

                Symptoms like the app always running on device 0, although the client thinks it's given an instruction to run on device 1 (and reported as such to the Manager), can be the result of a mis-match in the intended signalling mode between the client and the science app. The signal should be given by the API version number in the <app_version> control block passed when the task is assigned, but problems arise if, for example, the API version number is left out of an app_info.xml file (i.e., when using the anonymous platform mechanism)

                Toby Broom
                Toby Broom
                Joined: 2 Sep 23
                Posts: 13
                Credit: 302764792
                RAC: 731062

                I don't have any GPU settings

                7 Nov 2023 21:07:35 UTC
                Message 218964
                (moderation:
                  )

                  I don't have any GPU settings in cc_config, why would I need as I said it sometimes works sometime not and as you can see BOINC knows there is 2 GPU else it wouldn't have 0 & 1 in the task list.

                  I can try it to see if it makes a difference?

                  As I said its seems like if there is 2 WUs started close togther is does not wait to find out what GPU is used so it like richard said maybe?

                   

                  <app_version>
                      <app_name>einstein_O3AS</app_name>
                      <version_num>106</version_num>
                      <api_version>7.3.0</api_version>
                  <file_ref>
                      <file_name>einstein_O3AS_1.06_x86_64-pc-linux-gnu__GW-opencl-nvidia-2</file_name>
                      <main_program/>
                  </file_ref>
                  <file_ref>
                      <file_name>O3ASHF1b_0.config</file_name>
                  </file_ref>
                  <file_ref>
                      <file_name>O3ASHF1b_1.config</file_name>
                  </file_ref>
                      <platform>x86_64-pc-linux-gnu</platform>
                      <plan_class>GW-opencl-nvidia-2</plan_class>
                      <avg_ncpus>0.900000</avg_ncpus>
                      <max_ncpus>1.000000</max_ncpus>
                      <flops>59598533140.034119</flops>
                      <coproc>
                          <type>CUDA</type>
                          <count>1.000000</count>
                      </coproc>
                      <gpu_ram>2621440000.000000</gpu_ram>
                  </app_version>

                  Bernd Machenschalk
                  Bernd Machenschalk
                  Moderator
                  Administrator
                  Joined: 15 Oct 04
                  Posts: 4330
                  Credit: 251504923
                  RAC: 36723

                  The reason why this was

                  7 Nov 2023 21:07:00 UTC
                  Message 218965 in response to message 218953
                  (moderation:
                    )

                    There was a change, many years ago, in the way the device assignment was passed from the BOINC client to the science application.

                    The reason why this was changed was OpenCL. For simple, single device type interfaces like CUDA, CAL or even Metal there exists a constant device numbering, and passing this number on the command line was enough to uniquely identify a device. For OpenCL things aren't that simple. You can ask for list of devices, but the number of devices and even the ordering in that list (actually two nested ones) depends on many things and is not even guaranteed to be the same between two such scans. A device is uniquely identified by PlatformID and DeviceID, which are, in fact, pointers to complex data structures. These are difficult to pass to the app on the command line, so these are passed in the "init data" (init_data.xml in the slot directory) and read out by a BOINC API call.

                    When our app doesn't find that information (i.e. the API call returns nothing useful), you'll see a message

                    [ERROR] Couldn't get OpenCL device from BOINC (-1)!

                    in stderr, and the app should terminate (with the somewhat confusing error message "Generic Backend not implemented yet.", though).

                    If you peek into init_data.xml of the tasks you should find that the opencl_device_id is the same for both tasks, thus it's likely the client which does something wrong. Not sure, though, how this comes about.

                    BM

                    Toby Broom
                    Toby Broom
                    Joined: 2 Sep 23
                    Posts: 13
                    Credit: 302764792
                    RAC: 731062

                    I removed everything, so

                    7 Nov 2023 21:18:39 UTC
                    Message 218966
                    (moderation:
                      )

                      I removed everything, so clinfo had no data, then reinstalled. I check tomorrow

                      Toby Broom
                      Toby Broom
                      Joined: 2 Sep 23
                      Posts: 13
                      Credit: 302764792
                      RAC: 731062

                      putenv

                      7 Nov 2023 22:17:47 UTC
                      Message 218968
                      (moderation:
                        )

                        Good one

                        putenv 'LAL_DEBUG_LEVEL=3'
                        2023-11-07 17:48:30.4038 (519072) [normal]: This program is published under the GNU General Public License, version 2
                        2023-11-07 17:48:30.4038 (519072) [normal]: For details see http://einstein.phys.uwm.edu/license.php
                        2023-11-07 17:48:30.4038 (519072) [normal]: This Einstein@home App was built at: Oct 25 2023 15:42:58

                        2023-11-07 17:48:30.4038 (519072) [normal]: Start of BOINC application '../../projects/einstein.phys.uwm.edu/einstein_O3AS_1.06_x86_64-pc-linux-gnu__GW-opencl-nvidia-2'.
                        [DEBUG} GPU type: 1
                        [DEBUG} got GPU info from BOINC
                        [DEBUG} got VendorID 4318
                        2023-11-07 17:48:30.4104 (519072) [debug]: Flags: LAL_DEBUG, OPTIMIZE, HS_OPTIMIZATION, GC_SSE2_OPT, X64, SSE, SSE2, GNUC X86 GNUX86
                        2023-11-07 17:48:30.4104 (519072) [debug]: glibc version/release: 2.38/stable
                        2023-11-07 17:48:30.410485 - mytime()
                        2023-11-07 17:48:30.4105 (519072) [debug]: Set up communication with graphics process.

                        einstein_O3AS_1.06_x86_64-pc-linux-gnu__GW-opencl-nvidia-2: option `--printCand1' is DEPRECATED: Print 1st stage candidates (ignored, no longer used)

                        2023-11-07 17:48:30.4109 (519072) [normal]: Parsed user input successfully
                        Code-version: %% LAL: 7.1.4.1 (CLEAN )
                        %% LALPulsar: 3.1.0.1 (CLEAN )
                        %% LALApps: 7.3.0.1 (CLEAN )

                        2023-11-07 17:48:30.4109 (519072) [normal]: Initialise compartments with freqWidth = 0.05 and candidates per compartment = 12500.
                        2023-11-07 17:48:30.4111 (519072) [debug]: Opening temp output file '../../projects/einstein.phys.uwm.edu/h1_0977.80_O3aC01Cl1In0__O3ASHF1b_978.00Hz_13469_0_0.tmp' for writing ...
                        2023-11-07 17:48:30.5186 (519072) [normal]: Reading input data ... 
                        2023-11-07 17:48:30.5186 (519072) [normal]: Loading SFTs matching '../../projects/einstein.phys.uwm.edu/h1_0977.80_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0977.80_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0978.00_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0978.00_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0978.20_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0978.20_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0978.40_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0978.40_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0978.60_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0978.60_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0978.80_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0978.80_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0979.00_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0979.00_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0979.20_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0979.20_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0979.40_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0979.40_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0979.60_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0979.60_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0979.80_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0979.80_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0980.00_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0980.00_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0980.20_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0980.20_O3aC01Cl1In0' into catalog ...2023-11-07 17:48:30.8660 (519072) [normal]: done.
                        2023-11-07 17:48:30.8660 (519072) [normal]: Validating SFTs (detectors: H1, L1, ) ... success.
                        2023-11-07 17:48:33.2243 (519072) [normal]: Search FstatMethod used: 'ResampGPU'
                        2023-11-07 17:48:33.2243 (519072) [normal]: Recalc FstatMethod used: 'DemodSSE'
                        2023-11-07 17:48:33.2243 (519072) [normal]: GPU Device used for Search/Recalc and/or semi coherent step: 'NVIDIA RTX A5000 ( Platform: NVIDIA CUDA )'
                        2023-11-07 17:48:33.2243 (519072) [normal]: GPU Backend used for Search/Recalc and/or semi coherent step: 'OpenCL'
                        2023-11-07 17:48:33.2243 (519072) [normal]: GPU version is used for the semi-coherent step!
                        2023-11-07 17:48:41.4350 (519072) [normal]: Number of segments: 73, total number of SFTs in segments: 11650
                        2023-11-07 17:48:41.4454 (519072) [normal]: Finished reading input data.
                        2023-11-07 17:48:41.4454 (519072) [normal]: GPS reference time = 1246070525.0000 ,  GPS data mid time = 1246070525.0000
                        2023-11-07 17:48:41.4454 (519072) [normal]: dFreqStack = 4.000000e-06, df1dot = 1.500000e-10, df2dot = 0.000000e+00, df3dot = 0.000000e+00
                        2023-11-07 17:48:41.4454 (519072) [normal]: Setup, N = 73, T = 216000 s, Tobs = 15809012 s, gammaRefine = 50, gamma2Refine = 18653, gamma3Refine = 1

                        DEPRECATION WARNING: program has invoked obsolete function InitDopplerSkyScan(). Please see XLALInitDopplerSkyScan() for information about a replacement.
                        2023-11-07 17:48:42.7365 (519072) [normal]: INFO: No checkpoint checkpoint.cpt found - starting from scratch
                        2023-11-07 17:48:42.7416 (519072) [normal]: Cpt:0,  total:3000,  sky:1/150,  f1dot:1/20

                        0.2023-11-07 17:48:42.7417 (519072) [normal]: CG:18271827 FG:250000  f1dotmin_fg:-2.773529411765e-09 df1dot_fg:2.941176470588e-12 f2dotmin_fg:0 df2dot_fg:0 f3dotmin_fg:0 df3dot_fg:1

                        2023-11-07 17:51:56.0686 (519072) [normal]: Finished main analysis.
                        2023-11-07 17:51:56.0687 (519072) [normal]: Recalculating statistics for the final toplist(s)...
                        2023-11-07 17:53:25.0463 (519072) [normal]: Finished recalculating toplist statistics.
                        2023-11-07 17:53:25.0464 (519072) [normal]: Finished in 294.63 s with peak RAM usage: 655.0 MB on CPU 'AMD Ryzen Threadripper PRO 5955WX 16-Cores     ', peak VRAM usage: 1648.6 MB on GPU Device: 'NVIDIA RTX A5000 ( Platform: NVIDIA CUDA )' with backend: 'OpenCL'.
                        2023-11-07 17:53:25.0670 (519072) [debug]: Writing output ... Closing temp output file '../../projects/einstein.phys.uwm.edu/h1_0977.80_O3aC01Cl1In0__O3ASHF1b_978.00Hz_13469_0_0.tmp' ... renaming temp output file '../../projects/einstein.phys.uwm.edu/h1_0977.80_O3aC01Cl1In0__O3ASHF1b_978.00Hz_13469_0_0.tmp' to '../../projects/einstein.phys.uwm.edu/h1_0977.80_O3aC01Cl1In0__O3ASHF1b_978.00Hz_13469_0_0' ... done.

                        DEPRECATION WARNING: program has invoked obsolete function FreeDopplerSkyScan(). Please see XLALDestroyDopplerSkyScan() for information about a replacement.

                        einstein_O3AS_1.06_x86_64-pc-linux-gnu__GW-opencl-nvidia-2: option `--printCand1' is DEPRECATED: Print 1st stage candidates (ignored, no longer used)

                        2023-11-07 17:53:25.2881 (519072) [normal]: Parsed user input successfully
                        Code-version: %% LAL: 7.1.4.1 (CLEAN )
                        %% LALPulsar: 3.1.0.1 (CLEAN )
                        %% LALApps: 7.3.0.1 (CLEAN )

                        2023-11-07 17:53:25.2881 (519072) [normal]: Initialise compartments with freqWidth = 0.05 and candidates per compartment = 12500.
                        2023-11-07 17:53:25.2882 (519072) [debug]: Opening temp output file '../../projects/einstein.phys.uwm.edu/h1_0977.80_O3aC01Cl1In0__O3ASHF1b_978.00Hz_13469_0_1.tmp' for writing ...
                        2023-11-07 17:53:25.3810 (519072) [normal]: Reading input data ... 
                        2023-11-07 17:53:25.3810 (519072) [normal]: Loading SFTs matching '../../projects/einstein.phys.uwm.edu/h1_0977.80_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0977.80_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0978.00_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0978.00_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0978.20_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0978.20_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0978.40_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0978.40_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0978.60_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0978.60_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0978.80_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0978.80_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0979.00_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0979.00_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0979.20_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0979.20_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0979.40_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0979.40_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0979.60_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0979.60_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0979.80_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0979.80_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0980.00_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0980.00_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0980.20_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0980.20_O3aC01Cl1In0' into catalog ...2023-11-07 17:53:25.7256 (519072) [normal]: done.
                        2023-11-07 17:53:25.7256 (519072) [normal]: Validating SFTs (detectors: H1, L1, ) ... success.
                        2023-11-07 17:53:28.0741 (519072) [normal]: Search FstatMethod used: 'ResampGPU'
                        2023-11-07 17:53:28.0741 (519072) [normal]: Recalc FstatMethod used: 'DemodSSE'
                        2023-11-07 17:53:28.0742 (519072) [normal]: GPU Device used for Search/Recalc and/or semi coherent step: 'NVIDIA RTX A5000 ( Platform: NVIDIA CUDA )'
                        2023-11-07 17:53:28.0742 (519072) [normal]: GPU Backend used for Search/Recalc and/or semi coherent step: 'OpenCL'
                        2023-11-07 17:53:28.0742 (519072) [normal]: GPU version is used for the semi-coherent step!
                        2023-11-07 17:53:36.6496 (519072) [normal]: Number of segments: 73, total number of SFTs in segments: 11650
                        2023-11-07 17:53:36.6591 (519072) [normal]: Finished reading input data.
                        2023-11-07 17:53:36.6591 (519072) [normal]: GPS reference time = 1246070525.0000 ,  GPS data mid time = 1246070525.0000
                        2023-11-07 17:53:36.6592 (519072) [normal]: dFreqStack = 4.000000e-06, df1dot = 1.500000e-10, df2dot = 0.000000e+00, df3dot = 0.000000e+00
                        2023-11-07 17:53:36.6592 (519072) [normal]: Setup, N = 73, T = 216000 s, Tobs = 15809012 s, gammaRefine = 50, gamma2Refine = 18653, gamma3Refine = 1
                        2023-11-07 17:53:37.9286 (519072) [normal]: INFO: No checkpoint checkpoint.cpt found - starting from scratch
                        2023-11-07 17:53:37.9287 (519072) [normal]: Cpt:0,  total:3000,  sky:1/150,  f1dot:1/20

                        0.2023-11-07 17:53:37.9287 (519072) [normal]: CG:18271827 FG:250000  f1dotmin_fg:-2.773529411765e-09 df1dot_fg:2.941176470588e-12 f2dotmin_fg:0 df2dot_fg:0 f3dotmin_fg:0 df3dot_fg:1
                         

                        <gpu_type>NVIDIA</gpu_type>
                        <gpu_device_num>0</gpu_device_num>
                        <gpu_opencl_dev_index>0</gpu_opencl_dev_index>
                        <gpu_usage>1.000000</gpu_usage>

                         

                        then

                        <gpu_type>NVIDIA</gpu_type>
                        <gpu_device_num>1</gpu_device_num>
                        <gpu_opencl_dev_index>1</gpu_opencl_dev_index>

                         

                        putenv 'LAL_DEBUG_LEVEL=3'
                        2023-11-07 17:48:09.8722 (519030) [normal]: This program is published under the GNU General Public License, version 2
                        2023-11-07 17:48:09.8723 (519030) [normal]: For details see http://einstein.phys.uwm.edu/license.php
                        2023-11-07 17:48:09.8723 (519030) [normal]: This Einstein@home App was built at: Oct 25 2023 15:42:58

                        2023-11-07 17:48:09.8723 (519030) [normal]: Start of BOINC application '../../projects/einstein.phys.uwm.edu/einstein_O3AS_1.06_x86_64-pc-linux-gnu__GW-opencl-nvidia-2'.
                        [DEBUG} GPU type: 1
                        [DEBUG} got GPU info from BOINC
                        [DEBUG} got VendorID 4318
                        2023-11-07 17:48:09.8798 (519030) [debug]: Flags: LAL_DEBUG, OPTIMIZE, HS_OPTIMIZATION, GC_SSE2_OPT, X64, SSE, SSE2, GNUC X86 GNUX86
                        2023-11-07 17:48:09.8798 (519030) [debug]: glibc version/release: 2.38/stable
                        2023-11-07 17:48:09.879879 - mytime()
                        2023-11-07 17:48:09.8799 (519030) [debug]: Set up communication with graphics process.

                        einstein_O3AS_1.06_x86_64-pc-linux-gnu__GW-opencl-nvidia-2: option `--printCand1' is DEPRECATED: Print 1st stage candidates (ignored, no longer used)

                        2023-11-07 17:48:09.8803 (519030) [normal]: Parsed user input successfully
                        Code-version: %% LAL: 7.1.4.1 (CLEAN )
                        %% LALPulsar: 3.1.0.1 (CLEAN )
                        %% LALApps: 7.3.0.1 (CLEAN )

                        2023-11-07 17:48:09.8803 (519030) [normal]: Initialise compartments with freqWidth = 0.05 and candidates per compartment = 12500.
                        2023-11-07 17:48:09.8805 (519030) [debug]: Opening temp output file '../../projects/einstein.phys.uwm.edu/h1_0977.80_O3aC01Cl1In0__O3ASHF1b_978.00Hz_13470_0_0.tmp' for writing ...
                        2023-11-07 17:48:09.9904 (519030) [normal]: Reading input data ... 
                        2023-11-07 17:48:09.9904 (519030) [normal]: Loading SFTs matching '../../projects/einstein.phys.uwm.edu/h1_0977.80_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0977.80_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0978.00_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0978.00_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0978.20_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0978.20_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0978.40_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0978.40_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0978.60_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0978.60_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0978.80_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0978.80_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0979.00_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0979.00_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0979.20_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0979.20_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0979.40_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0979.40_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0979.60_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0979.60_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0979.80_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0979.80_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0980.00_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0980.00_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0980.20_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0980.20_O3aC01Cl1In0' into catalog ...2023-11-07 17:48:10.3430 (519030) [normal]: done.
                        2023-11-07 17:48:10.3430 (519030) [normal]: Validating SFTs (detectors: H1, L1, ) ... success.
                        2023-11-07 17:48:12.7106 (519030) [normal]: Search FstatMethod used: 'ResampGPU'
                        2023-11-07 17:48:12.7106 (519030) [normal]: Recalc FstatMethod used: 'DemodSSE'
                        2023-11-07 17:48:12.7106 (519030) [normal]: GPU Device used for Search/Recalc and/or semi coherent step: 'NVIDIA RTX A5000 ( Platform: NVIDIA CUDA )'
                        2023-11-07 17:48:12.7106 (519030) [normal]: GPU Backend used for Search/Recalc and/or semi coherent step: 'OpenCL'
                        2023-11-07 17:48:12.7106 (519030) [normal]: GPU version is used for the semi-coherent step!
                        2023-11-07 17:48:20.9708 (519030) [normal]: Number of segments: 73, total number of SFTs in segments: 11650
                        2023-11-07 17:48:20.9813 (519030) [normal]: Finished reading input data.
                        2023-11-07 17:48:20.9813 (519030) [normal]: GPS reference time = 1246070525.0000 ,  GPS data mid time = 1246070525.0000
                        2023-11-07 17:48:20.9814 (519030) [normal]: dFreqStack = 4.000000e-06, df1dot = 1.500000e-10, df2dot = 0.000000e+00, df3dot = 0.000000e+00
                        2023-11-07 17:48:20.9814 (519030) [normal]: Setup, N = 73, T = 216000 s, Tobs = 15809012 s, gammaRefine = 50, gamma2Refine = 18653, gamma3Refine = 1

                        DEPRECATION WARNING: program has invoked obsolete function InitDopplerSkyScan(). Please see XLALInitDopplerSkyScan() for information about a replacement.
                        2023-11-07 17:48:22.2756 (519030) [normal]: INFO: No checkpoint checkpoint.cpt found - starting from scratch
                        2023-11-07 17:48:22.2807 (519030) [normal]: Cpt:0,  total:3000,  sky:1/150,  f1dot:1/20

                        0.2023-11-07 17:48:22.2807 (519030) [normal]: CG:18271827 FG:250000  f1dotmin_fg:-2.773529411765e-09 df1dot_fg:2.941176470588e-12 f2dotmin_fg:0 df2dot_fg:0 f3dotmin_fg:0 df3dot_fg:1

                        2023-11-07 17:51:36.3235 (519030) [normal]: Finished main analysis.
                        2023-11-07 17:51:36.3236 (519030) [normal]: Recalculating statistics for the final toplist(s)...
                        2023-11-07 17:53:06.2033 (519030) [normal]: Finished recalculating toplist statistics.
                        2023-11-07 17:53:06.2034 (519030) [normal]: Finished in 296.32 s with peak RAM usage: 655.1 MB on CPU 'AMD Ryzen Threadripper PRO 5955WX 16-Cores     ', peak VRAM usage: 1648.6 MB on GPU Device: 'NVIDIA RTX A5000 ( Platform: NVIDIA CUDA )' with backend: 'OpenCL'.
                        2023-11-07 17:53:06.2256 (519030) [debug]: Writing output ... Closing temp output file '../../projects/einstein.phys.uwm.edu/h1_0977.80_O3aC01Cl1In0__O3ASHF1b_978.00Hz_13470_0_0.tmp' ... renaming temp output file '../../projects/einstein.phys.uwm.edu/h1_0977.80_O3aC01Cl1In0__O3ASHF1b_978.00Hz_13470_0_0.tmp' to '../../projects/einstein.phys.uwm.edu/h1_0977.80_O3aC01Cl1In0__O3ASHF1b_978.00Hz_13470_0_0' ... done.

                        DEPRECATION WARNING: program has invoked obsolete function FreeDopplerSkyScan(). Please see XLALDestroyDopplerSkyScan() for information about a replacement.

                        einstein_O3AS_1.06_x86_64-pc-linux-gnu__GW-opencl-nvidia-2: option `--printCand1' is DEPRECATED: Print 1st stage candidates (ignored, no longer used)

                        2023-11-07 17:53:06.5397 (519030) [normal]: Parsed user input successfully
                        Code-version: %% LAL: 7.1.4.1 (CLEAN )
                        %% LALPulsar: 3.1.0.1 (CLEAN )
                        %% LALApps: 7.3.0.1 (CLEAN )

                        2023-11-07 17:53:06.5397 (519030) [normal]: Initialise compartments with freqWidth = 0.05 and candidates per compartment = 12500.
                        2023-11-07 17:53:06.5397 (519030) [debug]: Opening temp output file '../../projects/einstein.phys.uwm.edu/h1_0977.80_O3aC01Cl1In0__O3ASHF1b_978.00Hz_13470_0_1.tmp' for writing ...
                        2023-11-07 17:53:06.6329 (519030) [normal]: Reading input data ... 
                        2023-11-07 17:53:06.6329 (519030) [normal]: Loading SFTs matching '../../projects/einstein.phys.uwm.edu/h1_0977.80_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0977.80_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0978.00_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0978.00_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0978.20_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0978.20_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0978.40_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0978.40_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0978.60_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0978.60_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0978.80_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0978.80_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0979.00_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0979.00_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0979.20_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0979.20_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0979.40_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0979.40_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0979.60_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0979.60_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0979.80_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0979.80_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0980.00_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0980.00_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0980.20_O3aC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0980.20_O3aC01Cl1In0' into catalog ...2023-11-07 17:53:06.9793 (519030) [normal]: done.
                        2023-11-07 17:53:06.9793 (519030) [normal]: Validating SFTs (detectors: H1, L1, ) ... success.
                        2023-11-07 17:53:09.3543 (519030) [normal]: Search FstatMethod used: 'ResampGPU'
                        2023-11-07 17:53:09.3543 (519030) [normal]: Recalc FstatMethod used: 'DemodSSE'
                        2023-11-07 17:53:09.3544 (519030) [normal]: GPU Device used for Search/Recalc and/or semi coherent step: 'NVIDIA RTX A5000 ( Platform: NVIDIA CUDA )'
                        2023-11-07 17:53:09.3544 (519030) [normal]: GPU Backend used for Search/Recalc and/or semi coherent step: 'OpenCL'
                        2023-11-07 17:53:09.3544 (519030) [normal]: GPU version is used for the semi-coherent step!
                        2023-11-07 17:53:17.9694 (519030) [normal]: Number of segments: 73, total number of SFTs in segments: 11650
                        2023-11-07 17:53:17.9789 (519030) [normal]: Finished reading input data.
                        2023-11-07 17:53:17.9789 (519030) [normal]: GPS reference time = 1246070525.0000 ,  GPS data mid time = 1246070525.0000
                        2023-11-07 17:53:17.9789 (519030) [normal]: dFreqStack = 4.000000e-06, df1dot = 1.500000e-10, df2dot = 0.000000e+00, df3dot = 0.000000e+00
                        2023-11-07 17:53:17.9789 (519030) [normal]: Setup, N = 73, T = 216000 s, Tobs = 15809012 s, gammaRefine = 50, gamma2Refine = 18653, gamma3Refine = 1
                        2023-11-07 17:53:19.2555 (519030) [normal]: INFO: No checkpoint checkpoint.cpt found - starting from scratch
                        2023-11-07 17:53:19.2555 (519030) [normal]: Cpt:0,  total:3000,  sky:1/150,  f1dot:1/20

                        0.2023-11-07 17:53:19.2556 (519030) [normal]: CG:18271827 FG:250000  f1dotmin_fg:-2.773529411765e-09 df1dot_fg:2.941176470588e-12 f2dotmin_fg:0 df2dot_fg:0 f3dotmin_fg:0 df3dot_fg:1
                         

                         

                         

                        Comment viewing options

                        Select your preferred way to display the comments and click "Save settings" to activate your changes.

                        Forums › Problems and Bug Reports