O3 All-Sky fails on OpenCL NVIDIA Linux
18 Nov 2021 8:28:20 UTC
Topic 226429
(moderation:
Having a lot of instant failures on start of Gravitational Wave search O3 All-Sky #1 v1.01 (GW-opencl-nvidia)
x86_64-pc-linux-gnu tasks with computation error - probably because of parameter --device used instead of --GPUDevice, as GPU tasks from other projects seem to work fine.
See https://einsteinathome.org/task/1192511451, for example:
TASK 1192511451
Name:h1_0361.80_O3aC01Cl1In0__O3AS1_362.00Hz_6523_0
Workunit ID:587629420
Created:17 Nov 2021 7:29:23 UTC
Sent:17 Nov 2021 7:29:24 UTC
Report deadline:24 Nov 2021 7:29:24 UTC
Received:17 Nov 2021 8:45:10 UTC
Server state:Over
Outcome:Computation error
Client state:Compute error
Exit status:1 (0x00000001) Unknown error code
Computer:12858660
Run time (sec):5.53
CPU time (sec):0.01
Peak working set size (MB):0
Peak swap size (MB):0
Peak disk usage (MB):0.02
Validation state:Invalid
Granted credit:0
Application:Gravitational Wave search O3 All-Sky #1 v1.01 (GW-opencl-nvidia)
x86_64-pc-linux-gnu
Stderr output
<core_client_version>7.9.3</core_client_version> <![CDATA[ <message> process exited with code 1 (0x1, -255)</message> <stderr_txt> putenv 'LAL_DEBUG_LEVEL=3' 2021-11-17 09:30:08.4021 (4779) [normal]: This program is published under the GNU General Public License, version 2 2021-11-17 09:30:08.4022 (4779) [normal]: For details see http://einstein.phys.uwm.edu/license.php 2021-11-17 09:30:08.4022 (4779) [normal]: This Einstein@home App was built at: Aug 5 2021 17:20:502021-11-17 09:30:08.4022 (4779) [normal]: Start of BOINC application '../../projects/einstein.phys.uwm.edu/einstein_O3AS_1.01_x86_64-pc-linux-gnu__GW-opencl-nvidia'.
[DEBUG} GPU type: 1
[ERROR] Couldn't get OpenCL device from BOINC (-1)!
2021-11-17 09:30:08.4448 (4779) [debug]: Flags: LAL_DEBUG, OPTIMIZE, HS_OPTIMIZATION, GC_SSE2_OPT, X64, SSE, SSE2, GNUC X86 GNUX86
2021-11-17 09:30:08.4448 (4779) [debug]: glibc version/release: 2.27/stable
2021-11-17 09:30:08.444835 - mytime()
2021-11-17 09:30:08.4450 (4779) [debug]: Set up communication with graphics process.einstein_O3AS_1.01_x86_64-pc-linux-gnu__GW-opencl-nvidia: unrecognized option `--device'
Usage: einstein_O3AS_1.01_x86_64-pc-linux-gnu__GW-opencl-nvidia [-h|--help] [-v|--version] [@<config-file>] [--log] [--semiCohToplist] [--DataFiles1] [--IFOs] [--skyRegion] [--numSkyPartitions] [--partitionIndex] [--skyGridFile] [--dAlpha] [--dDelta] [-f|--Freq] [--dFreq] [-b|--FreqBand] [--f1dot] [--df1dot] [--f1dotBand] [--f2dot] [--df2dot] [--f2dotBand] [--f3dot] [--df3dot] [--f3dotBand] [--peakThrF] [-m|--mismatch1] [--gridType1] [--metricType1] [-g|--gammaRefine] [-G|--gamma2Refine] [-o|--fnameout] [--fnameChkPoint] [-n|--nCand1] [--printCand1] [--refTime] [--ephemEarth] [--ephemSun] [--minStartTime1] [--maxStartTime1] [--printFstat1] [--assumeSqrtSX] [--nStacksMax] [-T|--tStack] [--segmentList] [--recalcToplistStats] [--loudestSegOutput] [--writeLeanerOutput] [--tlCompartments] [--computeBSGL] [--Fstar0sc] [--oLGX] [--getMaxFperSeg] [--SortToplist] [--FstatMethod] [--FstatMethodRecalc] [--injectionSources] [--injectSqrtSX] [--timestampsFiles] [--Tsft] [--useGPUSemiCoh] [--GPUDevice]
2021-11-17 09:30:08.4462 (4779) [CRITICAL]: ERROR: MAIN() returned with error '1'
DEPRECATION WARNING: program has invoked obsolete function XLALGetVersionString(). Please see XLALVCSInfoString() for information about a replacement.
Code-version: %% LAL: 6.21.0.1 (CLEAN 8d0838c264f9ff9adc8c3cdbfa17b5154eaa2994)
%% LALPulsar: 1.18.2.1 (CLEAN 8d0838c264f9ff9adc8c3cdbfa17b5154eaa2994)
%% LALApps: 6.25.1.1 (CLEAN 8d0838c264f9ff9adc8c3cdbfa17b5154eaa2994)FPU status flags:
2021-11-17 09:30:08.4468 (4779) [debug]: worker done. return(1) to caller
2021-11-17 09:30:08.4468 (4779) [normal]: done. calling boinc_finish(1).
09:30:08 (4779): called boinc_finish</stderr_txt>
]]>
Are you using
)
Are you using an <exclude_gpu> in an app_config.xml file?
Sounds like you haven't used the correct syntax for the statement. You need to specify the gpu type and use the BOINC enumeration of your device in the statement. You can read how to construct the statement here.
Client configuration
No, I don't have a
)
No, I don't have a configuration on GPU restriction, not app-wise, nor project-wise.
I decided to give up on O3 tasks completely for now as I have no computer with more than 2GB video RAM anyway
first, it seems you might not
)
first, it seems you might not have the openCL drivers installed since your stderr output indicates that it cannot find an openCL device. often times with Nvidia drivers on Ubuntu, if you installed the drivers via some PPA or repository, openCL components are not included, however they are included if you do the nvidia .run installer.
install them with:
sudo apt install ocl-icd-libopencl1second, 2GB will likely not be enough for GW tasks anyway. you will fail tasks again for not enough memory. but with the opencl drivers properly installed, you will at least be able to run the gamma ray tasks.
_________________________________________________________________________
Thank you for the advice! I
)
Thank you for the advice! I already have the openCL components installed though:
So I guess that the reason of fail may really be in the wrong option name the task is being invoked with:
The current batch of these
)
The current batch of these tasks has lower RAM requirements. So I'm running a test system with an older 2GB Nvidia GPU with no problems. Differences I noticed are that it uses glibc version/release: 2.31 which is deployed with Ubuntu 20 but is backwards compatible and Boinc 7.16 instead of 7.9. :-)
It may sound strange, but it
)
It may sound strange, but it seems that the problem is gone - it might be some initial video card misconfiguration fixed after system restart, - I have a hypothesis that it was NVIDIA using Power Saving Mode PRIME profile by default (so the calls were sent to integrated Intel graphic card instead of NVIDIA one.
At now I have O3 tasks on the same system going just fine with valid outcome now, see https://einsteinathome.org/task/1193782597, for example:
TASK 1193782597
Name:h1_0361.80_O3aC01Cl1In0__O3AS1_362.00Hz_5001_1
Workunit ID:588263137
Created:20 Nov 2021 9:57:32 UTC
Sent:20 Nov 2021 11:02:37 UTC
Report deadline:27 Nov 2021 11:02:37 UTC
Received:20 Nov 2021 17:49:30 UTC
Server state:Over
Outcome:Success
Client state:Done
Exit status:0 (0x00000000)
Computer:12858660
Run time (sec):3,304.21
CPU time (sec):3,336.09
Peak working set size (MB):251.11
Peak swap size (MB):26472.66
Peak disk usage (MB):4.6
Validation state:Valid
Granted credit:1,000
Application:Gravitational Wave search O3 All-Sky #1 v1.01 (GW-opencl-nvidia)
x86_64-pc-linux-gnu
Stderr output
The question now is why I still receive O3 tasks if I turned them off in the project preferences here, but I think it's just a matter of time.
Have a good weekend!