1. set a really short queue depth until you see how your choices and your hardware play together, and completion time estimates stabilize. Try 0.1 days.
2. Consider only enabling one particular type of Einstein task to run on a given system--then alter that choice to see what works best for you.
Thank you for the Advice.
Tom M
A Proud member of the O.F.A. (Old Farts Association). Be well, do good work, and keep in touch.® (Garrison Keillor) I want some more patience. RIGHT NOW!
I'm starting to setup E@h as backup on most hosts so already encountered some issue:
2020-03-28 15:41:18.3343 (3664) [normal]: SSE2 version is used for the semi-coherent step!
XLALError - XLALMallocLong in /home/jenkins/workspace/workspace/EaH-GW-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lal/src/tools/Sequence_source.c:50XLAL Error - XLALMallocLong (/home/jenkins/workspace/workspace/EaH-GW-Testing/SLAVE/MinGW6.3/TARGET/windows-x64/EinsteinAtHome/source/lalsuite/lal/src/std/LALMalloc.c:75): Memory allocation error
Few questions arise:
1) Obviously it's known that this search/app memory hungry one. So why hard failure instead of allocation error handling with sleep or temporary exit at last?
For that particular task 13 minutes of work simply lost.
2) I installed latest BOINC client available on that host. So why hard failure instead of "waiting for memory" ?
And more general question - could someone point to short info (preferably a table with info) regarding what resources (memory/HDD) consumes what kind of modern Einstein's apps.
With lack of error-resistance inside app itself it would make easier to properly configure host to avoid hard situations for app.
P.S. Actually third question. Why memory allocation not done on very beginning of task? That could fail task w/o CPU wasting at least. But it seems unavailable amount of memory this type of task request steadely after 13 minutes of CPU work.
Is it some surprise for algorithm that it needs more memory to not to allocate it at very beginning??
And for me that's always been the quandry get the best card for project a which is not the best card for project b or get the best card for the money and settle for the results being great at project a and medicore at project b, but still getting credits at 2, or more, projects instead of just the few that support one brand or the other. Personally over time I am able to buy both types of cards but since they are aquired over years they are not all the same quality they once were so project a may get a less than ideal gpu put on it because project d has a much better gpu on it.
Same issue we had on SETI too, where 2 app showed different performance on different hardware. And my observations are the same. If OpenCL involved (and MW adds another consideration - double precesion, that AMD handles better too) better to use AMD card and save NV card for CUDA directly.
I have been getting "waiting for memory" messages on my Linux 7.14.2 setup. I have ordered more memory.
It is not clear to me if that error is a gpu-based one or the fact that it is mentioning SSE2 means it is a cpu based one.
Simply limiting the maximum # of tasks has gotten rid of my "waiting on memory" messages (when I wanted it to).
If this is a gpu issue, you might try backgrading the Nvidia driver version to something low enough that it wasn't having the previous troubles we were having on S@H. In theory the newest windows versions of the Nvidia driver fixed that bug.
So maybe, maybe not. Might try an "older" than 4.35? version.
HTH,
Tom M
A Proud member of the O.F.A. (Old Farts Association). Be well, do good work, and keep in touch.® (Garrison Keillor) I want some more patience. RIGHT NOW!
I just got a bunch of computation errors on an intel box using Nvidia gpus.
<core_client_version>7.14.2</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
putenv 'LAL_DEBUG_LEVEL=3'
2020-03-28 11:07:48.9584 (8948) [normal]: This program is published under the GNU General Public License, version 2
2020-03-28 11:07:48.9584 (8948) [normal]: For details see http://einstein.phys.uwm.edu/license.php
2020-03-28 11:07:48.9584 (8948) [normal]: This Einstein@home App was built at: Dec 19 2019 13:18:43
2020-03-28 11:07:48.9584 (8948) [normal]: Start of BOINC application '../../projects/einstein.phys.uwm.edu/einstein_O2MDF_2.07_x86_64-pc-linux-gnu__GW-opencl-nvidia'.
[DEBUG} GPU type: 1
[ERROR] Couldn't get OpenCL device from BOINC (-1)!
2020-03-28 11:07:48.9938 (8948) [debug]: BSGL output files
2020-03-28 11:07:48.9941 (8948) [debug]: Flags: LAL_DEBUG, OPTIMIZE, HS_OPTIMIZATION, GC_SSE2_OPT, X64, SSE, SSE2, GNUC X86 GNUX86
2020-03-28 11:07:48.9941 (8948) [debug]: glibc version/release: 2.27/stable
2020-03-28 11:07:48.994175 - mytime()
2020-03-28 11:07:48.9942 (8948) [debug]: Set up communication with graphics process.
DEPRECATION WARNING: program has invoked obsolete function XLALGetVersionString(). Please see XLALVCSInfoString() for information about a replacement.
Code-version: %% LAL: 6.19.2.1 (CLEAN 98bbe72a728eb25935e9195dafae691335dabf8c)
%% LALPulsar: 1.17.1.1 (CLEAN 98bbe72a728eb25935e9195dafae691335dabf8c)
%% LALApps: 6.23.0.1 (CLEAN 98bbe72a728eb25935e9195dafae691335dabf8c)
XLAL Error - XLALSetCLDeviceByNumber (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/LIBC215/TARGET/linux-x86_64/EinsteinAtHome/source/lalsuite/lalpulsar/src/OpenCLutils.c:1107): There are only 6 devices available!
XLAL Error - XLALSetCLDeviceByNumber (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/LIBC215/TARGET/linux-x86_64/EinsteinAtHome/source/lalsuite/lalpulsar/src/OpenCLutils.c:1107): Invalid argument
XLAL Error - MAIN (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/LIBC215/TARGET/linux-x86_64/EinsteinAtHome/source/lalsuite/lalapps/src/pulsar/GCT/HierarchSearchGCT.c:845): Check failed: XLALSetCLDeviceByNumber ( uvar_Device ) == XLAL_SUCCESS
XLAL Error - MAIN (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/LIBC215/TARGET/linux-x86_64/EinsteinAtHome/source/lalsuite/lalapps/src/pulsar/GCT/HierarchSearchGCT.c:845): Internal function call failed: Invalid argument
2020-03-28 11:07:49.0172 (8948) [CRITICAL]: ERROR: MAIN() returned with error '1'
2020-03-28 11:07:49.0172 (8948) [debug]: resultfile '../../projects/einstein.phys.uwm.edu/h1_1285.40_O2C02Cl4In0__O2MDFV2f_VelaJr1_1285.80Hz_95_4_0' (len 94), current config file: 0
Code-version: %% LAL: 6.19.2.1 (CLEAN 98bbe72a728eb25935e9195dafae691335dabf8c)
%% LALPulsar: 1.17.1.1 (CLEAN 98bbe72a728eb25935e9195dafae691335dabf8c)
%% LALApps: 6.23.0.1 (CLEAN 98bbe72a728eb25935e9195dafae691335dabf8c)
FPU status flags:
2020-03-28 11:07:49.0175 (8948) [debug]: worker done. return(1) to caller
2020-03-28 11:07:49.0175 (8948) [normal]: done. calling boinc_finish(1).
11:07:49 (8948): called boinc_finish
</stderr_txt>
]]>
A Proud member of the O.F.A. (Old Farts Association). Be well, do good work, and keep in touch.® (Garrison Keillor) I want some more patience. RIGHT NOW!
GPU#2 was the middle card that was cheek to butt with the other cards and was't cooling very well since it was clogged up with dust and hair after not being blown out in over 2 months.
De-dusted with the leaf blower fixed things right up.
Took out a hundred or so Einstein and GPUGrid tasks each before I caught the error.
I just got a bunch of computation errors on an intel box using Nvidia gpus.
Hi Tom, I've selected a small excerpt from what you posted and have posted the same bit from one of the successful tasks from one of my hosts. I don't use nvidia stuff at all so I'm no use for any info about drivers but it looks like your problem is that BOINC isn't detecting the OpenCL capabilities of your GPU for some reason.
2020-03-28 11:07:48.9584 (8948) [normal]: Start of BOINC application '../../projects/einstein.phys.uwm.edu/einstein_O2MDF_2.07_x86_64-pc-linux-gnu__GW-opencl-nvidia'.
[DEBUG} GPU type: 1
[ERROR] Couldn't get OpenCL device from BOINC (-1)!
2020-03-28 11:07:48.9938 (8948) [debug]: BSGL output files
2020-03-28 11:07:48.9941 (8948) [debug]: Flags: LAL_DEBUG, OPTIMIZE, HS_OPTIMIZATION, GC_SSE2_OPT, X64, SSE, SSE2, GNUC X86 GNUX86
2020-03-28 11:07:48.9941 (8948) [debug]: glibc version/release: 2.27/stable
2020-03-28 11:07:48.994175 - mytime()
2020-03-28 11:07:48.9942 (8948) [debug]: Set up communication with graphics process.
The following is from one of my tasks. I've left lines at the start so you can see things like the client version as well.
<core_client_version>7.15.0</core_client_version>
<stderr_txt>
putenv 'LAL_DEBUG_LEVEL=3'
2020-03-29 07:25:01.3145 (12134) [normal]: This program is published under the GNU General Public License, version 2
2020-03-29 07:25:01.3146 (12134) [normal]: For details see http://einstein.phys.uwm.edu/license.php
2020-03-29 07:25:01.3146 (12134) [normal]: This Einstein@home App was built at: Feb 21 2020 08:57:16
2020-03-29 07:25:01.3146 (12134) [normal]: Start of BOINC application '../../projects/einstein.phys.uwm.edu/einstein_O2MDF_2.08_x86_64-pc-linux-gnu__GW-opencl-ati-Beta'.
[DEBUG} GPU type: 2
[DEBUG} got GPU info from BOINC
[DEBUG} got VendorID 4098
2020-03-29 07:25:01.3514 (12134) [debug]: BSGL output files
2020-03-29 07:25:01.3517 (12134) [debug]: Flags: LAL_DEBUG, OPTIMIZE, HS_OPTIMIZATION, GC_SSE2_OPT, X64, SSE, SSE2, GNUC X86 GNUX86
2020-03-29 07:25:01.3517 (12134) [debug]: glibc version/release: 2.30/stable
2020-03-29 07:25:01.351720 - mytime()
2020-03-29 07:25:01.3518 (12134) [debug]: Set up communication with graphics process.
Are you sure the OpenCL capabilities of your GPU are properly detected in the client startup messages?
Question: what system requirements, in particular disk space and RAM does the Android application have? I tried to check at WUprop, but the information there seem to consider just the required disk space for the slots directories and not the entire project, for Gravitational Wave it shows for example 4-5MB on Windows, after downloading my 3rd WU however the entire Einstein folder is already over half GB. Not an issue on my laptop, but I also have this piece of cr... hardware: https://setiathome.berkeley.edu/show_host_detail.php?hostid=8282562
Are you sure the OpenCL capabilities of your GPU are properly detected in the client startup messages?
That is a great question. I have disabled that gpu task and am trying to run it only as a cpu task for now.
I have now had that error on both of my boxes. Both are Linux, both run Nvidia only, one is Amd, one is intel. I will confirm that I have installed the OpenCL that I have to install manually.
Tom
A Proud member of the O.F.A. (Old Farts Association). Be well, do good work, and keep in touch.® (Garrison Keillor) I want some more patience. RIGHT NOW!
Are you sure the OpenCL capabilities of your GPU are properly detected in the client startup messages?
That is a great question. I have disabled that gpu task and am trying to run it only as a cpu task for now.
I have now had that error on both of my boxes. Both are Linux, both run Nvidia only, one is Amd, one is intel. I will confirm that I have installed the OpenCL that I have to install manually.
Tom
For my AMD box which also saw errors this is the startup I just copied.
Sun 29 Mar 2020 10:47:42 AM CDT | | Starting BOINC client version 7.16.5 for x86_64-pc-linux-gnu
Sun 29 Mar 2020 10:47:42 AM CDT | | log flags: file_xfer, sched_ops, task, sched_op_debug
Sun 29 Mar 2020 10:47:42 AM CDT | | Libraries: libcurl/7.58.0 GnuTLS/3.5.18 zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3
Sun 29 Mar 2020 10:47:42 AM CDT | | Data directory: /home/tlgalenson/Desktop/BOINC
Sun 29 Mar 2020 10:47:43 AM CDT | | CUDA: NVIDIA GPU 0: GeForce RTX 2060 SUPER (driver version 440.64, CUDA version 10.2, compute capability 7.5, 4096MB, 3970MB available, 7311 GFLOPS peak)
Sun 29 Mar 2020 10:47:43 AM CDT | | CUDA: NVIDIA GPU 1: GeForce GTX 1660 SUPER (driver version 440.64, CUDA version 10.2, compute capability 7.5, 4096MB, 3974MB available, 5153 GFLOPS peak)
Sun 29 Mar 2020 10:47:43 AM CDT | | OpenCL: NVIDIA GPU 0: GeForce RTX 2060 SUPER (driver version 440.64, device version OpenCL 1.2 CUDA, 7982MB, 3970MB available, 7311 GFLOPS peak)
Sun 29 Mar 2020 10:47:43 AM CDT | | OpenCL: NVIDIA GPU 1: GeForce GTX 1660 SUPER (driver version 440.64, device version OpenCL 1.2 CUDA, 5942MB, 3974MB available, 5153 GFLOPS peak)
Sun 29 Mar 2020 10:47:43 AM CDT | SETI@home | Found app_info.xml; using anonymous platform
Sun 29 Mar 2020 10:47:43 AM CDT | | libc: Ubuntu GLIBC 2.27-3ubuntu1 version 2.27
Sun 29 Mar 2020 10:47:43 AM CDT | | Host name: moonshot3
Sun 29 Mar 2020 10:47:43 AM CDT | | Processor: 32 AuthenticAMD AMD Ryzen 9 3950X 16-Core Processor [Family 23 Model 113 Stepping 0]
Sun 29 Mar 2020 10:47:43 AM CDT | | Processor features: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate sme ssbd mba sev ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif umip rdpid overflow_recov succor smca
Sun 29 Mar 2020 10:47:43 AM CDT | | OS: Linux Ubuntu: Ubuntu 18.04.4 LTS [5.3.0-42-generic|libc 2.27 (Ubuntu GLIBC 2.27-3ubuntu1)]
Sun 29 Mar 2020 10:47:43 AM CDT | | Memory: 15.59 GB physical, 2.00 GB virtual
Sun 29 Mar 2020 10:47:43 AM CDT | | Disk: 218.57 GB total, 186.91 GB free
Sun 29 Mar 2020 10:47:43 AM CDT | | Local time is UTC -5 hours
Sun 29 Mar 2020 10:47:43 AM CDT | Rosetta@home | Found app_config.xml
Sun 29 Mar 2020 10:47:43 AM CDT | SETI@home | Found app_config.xml
Sun 29 Mar 2020 10:47:43 AM CDT | World Community Grid | Found app_config.xml
Sun 29 Mar 2020 10:47:43 AM CDT | Rosetta@home | Max 4 concurrent jobs
Sun 29 Mar 2020 10:47:43 AM CDT | SETI@home | Max 7 concurrent jobs
Sun 29 Mar 2020 10:47:43 AM CDT | World Community Grid | Max 18 concurrent jobs
Sun 29 Mar 2020 10:47:43 AM CDT | | Config: use all coprocessors
Sun 29 Mar 2020 10:47:43 AM CDT | Einstein@Home | General prefs: from Einstein@Home (last modified ---)
Sun 29 Mar 2020 10:47:43 AM CDT | Einstein@Home | Computer location: home
Sun 29 Mar 2020 10:47:43 AM CDT | | General prefs: using separate prefs for home
Sun 29 Mar 2020 12:11:48 PM EDT | | Starting BOINC client version 7.14.2 for x86_64-pc-linux-gnu
Sun 29 Mar 2020 12:11:48 PM EDT | | log flags: file_xfer, sched_ops, task, sched_op_debug
Sun 29 Mar 2020 12:11:48 PM EDT | | Libraries: libcurl/7.58.0 GnuTLS/3.5.18 zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3
Sun 29 Mar 2020 12:11:48 PM EDT | | Data directory: /home/tlgalenson/Desktop/BOINC
Sun 29 Mar 2020 12:11:50 PM EDT | | CUDA: NVIDIA GPU 0: GeForce GTX 1660 SUPER (driver version 440.64, CUDA version 10.2, compute capability 7.5, 4096MB, 3974MB available, 5153 GFLOPS peak)
Sun 29 Mar 2020 12:11:50 PM EDT | | CUDA: NVIDIA GPU 1: GeForce GTX 1660 SUPER (driver version 440.64, CUDA version 10.2, compute capability 7.5, 4096MB, 3974MB available, 5153 GFLOPS peak)
Sun 29 Mar 2020 12:11:50 PM EDT | | CUDA: NVIDIA GPU 2: P102-100 (driver version 440.64, CUDA version 10.2, compute capability 6.1, 4096MB, 3964MB available, 10771 GFLOPS peak)
Sun 29 Mar 2020 12:11:50 PM EDT | | CUDA: NVIDIA GPU 3: P102-100 (driver version 440.64, CUDA version 10.2, compute capability 6.1, 4096MB, 3964MB available, 10771 GFLOPS peak)
Sun 29 Mar 2020 12:11:50 PM EDT | | CUDA: NVIDIA GPU 4: P102-100 (driver version 440.64, CUDA version 10.2, compute capability 6.1, 4096MB, 3964MB available, 10771 GFLOPS peak)
Sun 29 Mar 2020 12:11:50 PM EDT | | CUDA: NVIDIA GPU 5: P102-100 (driver version 440.64, CUDA version 10.2, compute capability 6.1, 4096MB, 3964MB available, 10771 GFLOPS peak)
Sun 29 Mar 2020 12:11:50 PM EDT | | CUDA: NVIDIA GPU 6: P102-100 (driver version 440.64, CUDA version 10.2, compute capability 6.1, 4096MB, 3964MB available, 10771 GFLOPS peak)
Sun 29 Mar 2020 12:11:50 PM EDT | | OpenCL: NVIDIA GPU 0: GeForce GTX 1660 SUPER (driver version 440.64, device version OpenCL 1.2 CUDA, 5942MB, 3974MB available, 5153 GFLOPS peak)
Sun 29 Mar 2020 12:11:50 PM EDT | | OpenCL: NVIDIA GPU 1: GeForce GTX 1660 SUPER (driver version 440.64, device version OpenCL 1.2 CUDA, 5945MB, 3974MB available, 5153 GFLOPS peak)
Sun 29 Mar 2020 12:11:50 PM EDT | | OpenCL: NVIDIA GPU 2: P102-100 (driver version 440.64, device version OpenCL 1.2 CUDA, 5060MB, 3964MB available, 10771 GFLOPS peak)
Sun 29 Mar 2020 12:11:50 PM EDT | | OpenCL: NVIDIA GPU 3: P102-100 (driver version 440.64, device version OpenCL 1.2 CUDA, 5060MB, 3964MB available, 10771 GFLOPS peak)
Sun 29 Mar 2020 12:11:50 PM EDT | | OpenCL: NVIDIA GPU 4: P102-100 (driver version 440.64, device version OpenCL 1.2 CUDA, 5060MB, 3964MB available, 10771 GFLOPS peak)
Sun 29 Mar 2020 12:11:50 PM EDT | | OpenCL: NVIDIA GPU 5: P102-100 (driver version 440.64, device version OpenCL 1.2 CUDA, 5060MB, 3964MB available, 10771 GFLOPS peak)
Sun 29 Mar 2020 12:11:50 PM EDT | | OpenCL: NVIDIA GPU 6: P102-100 (driver version 440.64, device version OpenCL 1.2 CUDA, 5060MB, 3964MB available, 10771 GFLOPS peak)
Sun 29 Mar 2020 12:11:50 PM EDT | SETI@home | Found app_info.xml; using anonymous platform
Sun 29 Mar 2020 12:11:51 PM EDT | | [libc detection] gathered: 2.27, Ubuntu GLIBC 2.27-3ubuntu1
Sun 29 Mar 2020 12:11:51 PM EDT | | Host name: moonshot4
Sun 29 Mar 2020 12:11:51 PM EDT | | Processor: 16 GenuineIntel Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz [Family 6 Model 158 Stepping 13]
Sun 29 Mar 2020 12:11:51 PM EDT | | Processor features: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d arch_capabilities
Sun 29 Mar 2020 12:11:51 PM EDT | | OS: Linux Ubuntu: Ubuntu 18.04.3 LTS [5.3.0-42-generic|libc 2.27 (Ubuntu GLIBC 2.27-3ubuntu1)]
Sun 29 Mar 2020 12:11:51 PM EDT | | Memory: 15.58 GB physical, 2.00 GB virtual
Sun 29 Mar 2020 12:11:51 PM EDT | | Disk: 915.40 GB total, 854.03 GB free
Sun 29 Mar 2020 12:11:51 PM EDT | | Local time is UTC -4 hours
Sun 29 Mar 2020 12:11:51 PM EDT | SETI@home | Found app_config.xml
Sun 29 Mar 2020 12:11:51 PM EDT | World Community Grid | Found app_config.xml
Sun 29 Mar 2020 12:11:51 PM EDT | | Config: use all coprocessors
Sun 29 Mar 2020 12:11:51 PM EDT | Einstein@Home | URL http://einstein.phys.uwm.edu/; Computer ID 12806167; resource share 100
A Proud member of the O.F.A. (Old Farts Association). Be well, do good work, and keep in touch.® (Garrison Keillor) I want some more patience. RIGHT NOW!
Quote: Brief version: 1.
)
Thank you for the Advice.
Tom M
A Proud member of the O.F.A. (Old Farts Association). Be well, do good work, and keep in touch.® (Garrison Keillor) I want some more patience. RIGHT NOW!
I'm starting to setup E@h as
)
I'm starting to setup E@h as backup on most hosts so already encountered some issue:
mikey написал:Keith Myers
)
Same issue we had on SETI too, where 2 app showed different performance on different hardware. And my observations are the same. If OpenCL involved (and MW adds another consideration - double precesion, that AMD handles better too) better to use AMD card and save NV card for CUDA directly.
I have been getting "waiting
)
I have been getting "waiting for memory" messages on my Linux 7.14.2 setup. I have ordered more memory.
It is not clear to me if that error is a gpu-based one or the fact that it is mentioning SSE2 means it is a cpu based one.
Simply limiting the maximum # of tasks has gotten rid of my "waiting on memory" messages (when I wanted it to).
If this is a gpu issue, you might try backgrading the Nvidia driver version to something low enough that it wasn't having the previous troubles we were having on S@H. In theory the newest windows versions of the Nvidia driver fixed that bug.
So maybe, maybe not. Might try an "older" than 4.35? version.
HTH,
Tom M
A Proud member of the O.F.A. (Old Farts Association). Be well, do good work, and keep in touch.® (Garrison Keillor) I want some more patience. RIGHT NOW!
I just got a bunch of
)
I just got a bunch of computation errors on an intel box using Nvidia gpus.
A Proud member of the O.F.A. (Old Farts Association). Be well, do good work, and keep in touch.® (Garrison Keillor) I want some more patience. RIGHT NOW!
Those are like the error I
)
Those are like the error I got when one of my GTX 1070 Ti's fell off the bus in my https://einsteinathome.org/host/12600970/tasks/0/54 host.
GPU#2 was the middle card that was cheek to butt with the other cards and was't cooling very well since it was clogged up with dust and hair after not being blown out in over 2 months.
De-dusted with the leaf blower fixed things right up.
Took out a hundred or so Einstein and GPUGrid tasks each before I caught the error.
Tom Miller wrote:I just got a
)
Hi Tom, I've selected a small excerpt from what you posted and have posted the same bit from one of the successful tasks from one of my hosts. I don't use nvidia stuff at all so I'm no use for any info about drivers but it looks like your problem is that BOINC isn't detecting the OpenCL capabilities of your GPU for some reason.
2020-03-28 11:07:48.9584 (8948) [normal]: Start of BOINC application '../../projects/einstein.phys.uwm.edu/einstein_O2MDF_2.07_x86_64-pc-linux-gnu__GW-opencl-nvidia'. [DEBUG} GPU type: 1 [ERROR] Couldn't get OpenCL device from BOINC (-1)! 2020-03-28 11:07:48.9938 (8948) [debug]: BSGL output files 2020-03-28 11:07:48.9941 (8948) [debug]: Flags: LAL_DEBUG, OPTIMIZE, HS_OPTIMIZATION, GC_SSE2_OPT, X64, SSE, SSE2, GNUC X86 GNUX86 2020-03-28 11:07:48.9941 (8948) [debug]: glibc version/release: 2.27/stable 2020-03-28 11:07:48.994175 - mytime() 2020-03-28 11:07:48.9942 (8948) [debug]: Set up communication with graphics process.The following is from one of my tasks. I've left lines at the start so you can see things like the client version as well.
<core_client_version>7.15.0</core_client_version> <stderr_txt> putenv 'LAL_DEBUG_LEVEL=3' 2020-03-29 07:25:01.3145 (12134) [normal]: This program is published under the GNU General Public License, version 2 2020-03-29 07:25:01.3146 (12134) [normal]: For details see http://einstein.phys.uwm.edu/license.php 2020-03-29 07:25:01.3146 (12134) [normal]: This Einstein@home App was built at: Feb 21 2020 08:57:16 2020-03-29 07:25:01.3146 (12134) [normal]: Start of BOINC application '../../projects/einstein.phys.uwm.edu/einstein_O2MDF_2.08_x86_64-pc-linux-gnu__GW-opencl-ati-Beta'. [DEBUG} GPU type: 2 [DEBUG} got GPU info from BOINC [DEBUG} got VendorID 4098 2020-03-29 07:25:01.3514 (12134) [debug]: BSGL output files 2020-03-29 07:25:01.3517 (12134) [debug]: Flags: LAL_DEBUG, OPTIMIZE, HS_OPTIMIZATION, GC_SSE2_OPT, X64, SSE, SSE2, GNUC X86 GNUX86 2020-03-29 07:25:01.3517 (12134) [debug]: glibc version/release: 2.30/stable 2020-03-29 07:25:01.351720 - mytime() 2020-03-29 07:25:01.3518 (12134) [debug]: Set up communication with graphics process.Are you sure the OpenCL capabilities of your GPU are properly detected in the client startup messages?
Cheers,
Gary.
Next SETI orphan
)
Next SETI orphan here...
Question: what system requirements, in particular disk space and RAM does the Android application have? I tried to check at WUprop, but the information there seem to consider just the required disk space for the slots directories and not the entire project, for Gravitational Wave it shows for example 4-5MB on Windows, after downloading my 3rd WU however the entire Einstein folder is already over half GB. Not an issue on my laptop, but I also have this piece of cr... hardware: https://setiathome.berkeley.edu/show_host_detail.php?hostid=8282562
.
Quote: Are you sure the
)
That is a great question. I have disabled that gpu task and am trying to run it only as a cpu task for now.
I have now had that error on both of my boxes. Both are Linux, both run Nvidia only, one is Amd, one is intel. I will confirm that I have installed the OpenCL that I have to install manually.
Tom
A Proud member of the O.F.A. (Old Farts Association). Be well, do good work, and keep in touch.® (Garrison Keillor) I want some more patience. RIGHT NOW!
Tom Miller wrote:Quote:Are
)
For my AMD box which also saw errors this is the startup I just copied.
Sun 29 Mar 2020 10:47:42 AM CDT | | Starting BOINC client version 7.16.5 for x86_64-pc-linux-gnu
Sun 29 Mar 2020 10:47:42 AM CDT | | log flags: file_xfer, sched_ops, task, sched_op_debug
Sun 29 Mar 2020 10:47:42 AM CDT | | Libraries: libcurl/7.58.0 GnuTLS/3.5.18 zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3
Sun 29 Mar 2020 10:47:42 AM CDT | | Data directory: /home/tlgalenson/Desktop/BOINC
Sun 29 Mar 2020 10:47:43 AM CDT | | CUDA: NVIDIA GPU 0: GeForce RTX 2060 SUPER (driver version 440.64, CUDA version 10.2, compute capability 7.5, 4096MB, 3970MB available, 7311 GFLOPS peak)
Sun 29 Mar 2020 10:47:43 AM CDT | | CUDA: NVIDIA GPU 1: GeForce GTX 1660 SUPER (driver version 440.64, CUDA version 10.2, compute capability 7.5, 4096MB, 3974MB available, 5153 GFLOPS peak)
Sun 29 Mar 2020 10:47:43 AM CDT | | OpenCL: NVIDIA GPU 0: GeForce RTX 2060 SUPER (driver version 440.64, device version OpenCL 1.2 CUDA, 7982MB, 3970MB available, 7311 GFLOPS peak)
Sun 29 Mar 2020 10:47:43 AM CDT | | OpenCL: NVIDIA GPU 1: GeForce GTX 1660 SUPER (driver version 440.64, device version OpenCL 1.2 CUDA, 5942MB, 3974MB available, 5153 GFLOPS peak)
Sun 29 Mar 2020 10:47:43 AM CDT | SETI@home | Found app_info.xml; using anonymous platform
Sun 29 Mar 2020 10:47:43 AM CDT | | libc: Ubuntu GLIBC 2.27-3ubuntu1 version 2.27
Sun 29 Mar 2020 10:47:43 AM CDT | | Host name: moonshot3
Sun 29 Mar 2020 10:47:43 AM CDT | | Processor: 32 AuthenticAMD AMD Ryzen 9 3950X 16-Core Processor [Family 23 Model 113 Stepping 0]
Sun 29 Mar 2020 10:47:43 AM CDT | | Processor features: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate sme ssbd mba sev ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif umip rdpid overflow_recov succor smca
Sun 29 Mar 2020 10:47:43 AM CDT | | OS: Linux Ubuntu: Ubuntu 18.04.4 LTS [5.3.0-42-generic|libc 2.27 (Ubuntu GLIBC 2.27-3ubuntu1)]
Sun 29 Mar 2020 10:47:43 AM CDT | | Memory: 15.59 GB physical, 2.00 GB virtual
Sun 29 Mar 2020 10:47:43 AM CDT | | Disk: 218.57 GB total, 186.91 GB free
Sun 29 Mar 2020 10:47:43 AM CDT | | Local time is UTC -5 hours
Sun 29 Mar 2020 10:47:43 AM CDT | Rosetta@home | Found app_config.xml
Sun 29 Mar 2020 10:47:43 AM CDT | SETI@home | Found app_config.xml
Sun 29 Mar 2020 10:47:43 AM CDT | World Community Grid | Found app_config.xml
Sun 29 Mar 2020 10:47:43 AM CDT | Rosetta@home | Max 4 concurrent jobs
Sun 29 Mar 2020 10:47:43 AM CDT | SETI@home | Max 7 concurrent jobs
Sun 29 Mar 2020 10:47:43 AM CDT | World Community Grid | Max 18 concurrent jobs
Sun 29 Mar 2020 10:47:43 AM CDT | | Config: use all coprocessors
Sun 29 Mar 2020 10:47:43 AM CDT | Einstein@Home | General prefs: from Einstein@Home (last modified ---)
Sun 29 Mar 2020 10:47:43 AM CDT | Einstein@Home | Computer location: home
Sun 29 Mar 2020 10:47:43 AM CDT | | General prefs: using separate prefs for home
========================================================
For my Intel box;
Sun 29 Mar 2020 12:11:48 PM EDT | | Starting BOINC client version 7.14.2 for x86_64-pc-linux-gnu
Sun 29 Mar 2020 12:11:48 PM EDT | | log flags: file_xfer, sched_ops, task, sched_op_debug
Sun 29 Mar 2020 12:11:48 PM EDT | | Libraries: libcurl/7.58.0 GnuTLS/3.5.18 zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3
Sun 29 Mar 2020 12:11:48 PM EDT | | Data directory: /home/tlgalenson/Desktop/BOINC
Sun 29 Mar 2020 12:11:50 PM EDT | | CUDA: NVIDIA GPU 0: GeForce GTX 1660 SUPER (driver version 440.64, CUDA version 10.2, compute capability 7.5, 4096MB, 3974MB available, 5153 GFLOPS peak)
Sun 29 Mar 2020 12:11:50 PM EDT | | CUDA: NVIDIA GPU 1: GeForce GTX 1660 SUPER (driver version 440.64, CUDA version 10.2, compute capability 7.5, 4096MB, 3974MB available, 5153 GFLOPS peak)
Sun 29 Mar 2020 12:11:50 PM EDT | | CUDA: NVIDIA GPU 2: P102-100 (driver version 440.64, CUDA version 10.2, compute capability 6.1, 4096MB, 3964MB available, 10771 GFLOPS peak)
Sun 29 Mar 2020 12:11:50 PM EDT | | CUDA: NVIDIA GPU 3: P102-100 (driver version 440.64, CUDA version 10.2, compute capability 6.1, 4096MB, 3964MB available, 10771 GFLOPS peak)
Sun 29 Mar 2020 12:11:50 PM EDT | | CUDA: NVIDIA GPU 4: P102-100 (driver version 440.64, CUDA version 10.2, compute capability 6.1, 4096MB, 3964MB available, 10771 GFLOPS peak)
Sun 29 Mar 2020 12:11:50 PM EDT | | CUDA: NVIDIA GPU 5: P102-100 (driver version 440.64, CUDA version 10.2, compute capability 6.1, 4096MB, 3964MB available, 10771 GFLOPS peak)
Sun 29 Mar 2020 12:11:50 PM EDT | | CUDA: NVIDIA GPU 6: P102-100 (driver version 440.64, CUDA version 10.2, compute capability 6.1, 4096MB, 3964MB available, 10771 GFLOPS peak)
Sun 29 Mar 2020 12:11:50 PM EDT | | OpenCL: NVIDIA GPU 0: GeForce GTX 1660 SUPER (driver version 440.64, device version OpenCL 1.2 CUDA, 5942MB, 3974MB available, 5153 GFLOPS peak)
Sun 29 Mar 2020 12:11:50 PM EDT | | OpenCL: NVIDIA GPU 1: GeForce GTX 1660 SUPER (driver version 440.64, device version OpenCL 1.2 CUDA, 5945MB, 3974MB available, 5153 GFLOPS peak)
Sun 29 Mar 2020 12:11:50 PM EDT | | OpenCL: NVIDIA GPU 2: P102-100 (driver version 440.64, device version OpenCL 1.2 CUDA, 5060MB, 3964MB available, 10771 GFLOPS peak)
Sun 29 Mar 2020 12:11:50 PM EDT | | OpenCL: NVIDIA GPU 3: P102-100 (driver version 440.64, device version OpenCL 1.2 CUDA, 5060MB, 3964MB available, 10771 GFLOPS peak)
Sun 29 Mar 2020 12:11:50 PM EDT | | OpenCL: NVIDIA GPU 4: P102-100 (driver version 440.64, device version OpenCL 1.2 CUDA, 5060MB, 3964MB available, 10771 GFLOPS peak)
Sun 29 Mar 2020 12:11:50 PM EDT | | OpenCL: NVIDIA GPU 5: P102-100 (driver version 440.64, device version OpenCL 1.2 CUDA, 5060MB, 3964MB available, 10771 GFLOPS peak)
Sun 29 Mar 2020 12:11:50 PM EDT | | OpenCL: NVIDIA GPU 6: P102-100 (driver version 440.64, device version OpenCL 1.2 CUDA, 5060MB, 3964MB available, 10771 GFLOPS peak)
Sun 29 Mar 2020 12:11:50 PM EDT | SETI@home | Found app_info.xml; using anonymous platform
Sun 29 Mar 2020 12:11:51 PM EDT | | [libc detection] gathered: 2.27, Ubuntu GLIBC 2.27-3ubuntu1
Sun 29 Mar 2020 12:11:51 PM EDT | | Host name: moonshot4
Sun 29 Mar 2020 12:11:51 PM EDT | | Processor: 16 GenuineIntel Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz [Family 6 Model 158 Stepping 13]
Sun 29 Mar 2020 12:11:51 PM EDT | | Processor features: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d arch_capabilities
Sun 29 Mar 2020 12:11:51 PM EDT | | OS: Linux Ubuntu: Ubuntu 18.04.3 LTS [5.3.0-42-generic|libc 2.27 (Ubuntu GLIBC 2.27-3ubuntu1)]
Sun 29 Mar 2020 12:11:51 PM EDT | | Memory: 15.58 GB physical, 2.00 GB virtual
Sun 29 Mar 2020 12:11:51 PM EDT | | Disk: 915.40 GB total, 854.03 GB free
Sun 29 Mar 2020 12:11:51 PM EDT | | Local time is UTC -4 hours
Sun 29 Mar 2020 12:11:51 PM EDT | SETI@home | Found app_config.xml
Sun 29 Mar 2020 12:11:51 PM EDT | World Community Grid | Found app_config.xml
Sun 29 Mar 2020 12:11:51 PM EDT | | Config: use all coprocessors
Sun 29 Mar 2020 12:11:51 PM EDT | Einstein@Home | URL http://einstein.phys.uwm.edu/; Computer ID 12806167; resource share 100
A Proud member of the O.F.A. (Old Farts Association). Be well, do good work, and keep in touch.® (Garrison Keillor) I want some more patience. RIGHT NOW!