Both tasks on GPU0, GPU1 will not run tasks
11 Aug 2019 18:46:55 UTC
Topic 219367
(moderation:
Somehow I cannot get the 2nd GPU to make computations..
My GPUs: Both cards are GTX1070, one short one long.
App: FGRPopenCL (gamma ray pulsar binary search)
Even when Boinc sais it is running task 1 on DEV0 and task 2 on DEV1, they are actually both running on the same GPU0!!
Since today I had got this weird error in event viewer (after numerous crashes attempting to run 2 tasks on the same gpu), I reinstalled the drivers and that seems fixed:
Faulting application name: GPUQuery_External.exe, version: 0.0.0.0, time stamp: 0x5d135423
Faulting module name: igdrcl64.dll, version: 20.19.15.4531, time stamp: 0x57ed260c
Exception code: 0xc0000005
Fault offset: 0x000000000007a74a
Faulting process id: 0x1ddc
Faulting application start time: 0x01d550689396cb3d
Faulting application path: C:\Users\Massimo\AppData\Local\Temp\GPUQuery_External.exe
Faulting module path: C:\Windows\SYSTEM32\igdrcl64.dll
Report Id: 460553f9-7133-42ad-a5af-da1fd7cd0755
Faulting package full name:
Faulting package-relative application ID:
Yet, I get the same behavior as before, only GPU0 will do the job..
This is the start up log. Notice that each OpenCL is shown twice !! What does that mean ??
OpenCL: NVIDIA GPU 0:
OpenCL: NVIDIA GPU 0:
OpenCL: NVIDIA GPU 1:
OpenCL: NVIDIA GPU 1:
Thanks in advance for any illuminating advice :)
08/11/19 20:03:34 | | Starting BOINC client version 7.14.2 for windows_x86_64
08/11/19 20:03:34 | | log flags: file_xfer, sched_ops, task
08/11/19 20:03:34 | | Libraries: libcurl/7.47.1 OpenSSL/1.0.2g zlib/1.2.8
08/11/19 20:03:34 | | Data directory: C:\ProgramData\BOINC
08/11/19 20:03:34 | | Running under account Massimo
08/11/19 20:03:34 | | CUDA: NVIDIA GPU 0: GeForce GTX 1070 (driver version 431.60, CUDA version 10.1, compute capability 6.1, 4096MB, 3556MB available, 6609 GFLOPS peak)
08/11/19 20:03:34 | | CUDA: NVIDIA GPU 1: GeForce GTX 1070 (driver version 431.60, CUDA version 10.1, compute capability 6.1, 4096MB, 3556MB available, 6852 GFLOPS peak)
08/11/19 20:03:34 | | OpenCL: NVIDIA GPU 0: GeForce GTX 1070 (driver version 431.60, device version OpenCL 1.2 CUDA, 8192MB, 3556MB available, 6609 GFLOPS peak)
08/11/19 20:03:34 | | OpenCL: NVIDIA GPU 0: GeForce GTX 1070 (driver version 431.60, device version OpenCL 1.2 CUDA, 8192MB, 3556MB available, 6609 GFLOPS peak)
08/11/19 20:03:34 | | OpenCL: NVIDIA GPU 1: GeForce GTX 1070 (driver version 431.60, device version OpenCL 1.2 CUDA, 8192MB, 3556MB available, 6852 GFLOPS peak)
08/11/19 20:03:34 | | OpenCL: NVIDIA GPU 1: GeForce GTX 1070 (driver version 431.60, device version OpenCL 1.2 CUDA, 8192MB, 3556MB available, 6852 GFLOPS peak)
08/11/19 20:03:34 | | Host name: DESKTOP-2HGB23B
08/11/19 20:03:34 | | Processor: 16 AuthenticAMD AMD Ryzen 7 2700X Eight-Core Processor [Family 23 Model 8 Stepping 2]
08/11/19 20:03:34 | | Processor features: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 htt pni ssse3 fma cx16 sse4_1 sse4_2 movebe popcnt aes f16c rdrandsyscall nx lm avx avx2 svm sse4a osvw skinit wdt tce topx page1gb rdtscp fsgsbase bmi1 smep
08/11/19 20:03:34 | | OS: Microsoft Windows 10: Professional x64 Edition, (10.00.17763.00)
08/11/19 20:03:34 | | Memory: 15.95 GB physical, 63.95 GB virtual
08/11/19 20:03:34 | | Disk: 465.05 GB total, 326.21 GB free
08/11/19 20:03:34 | | Local time is UTC +2 hours
08/11/19 20:03:34 | | No WSL found.
08/11/19 20:03:34 | Einstein@Home | Found app_config.xml
08/11/19 20:03:34 | | Config: use HTTP 1.0
08/11/19 20:03:34 | | Config: report completed tasks immediately
08/11/19 20:03:34 | | Config: use all coprocessors
08/11/19 20:03:34 | Einstein@Home | URL http://einstein.phys.uwm.edu/; Computer ID 12765495; resource share 100
08/11/19 20:03:34 | GPUGRID | URL http://www.gpugrid.net/; Computer ID 498139; resource share 500
08/11/19 20:03:34 | World Community Grid | URL http://www.worldcommunitygrid.org/; Computer ID 5237756; resource share 100
08/11/19 20:03:34 | World Community Grid | General prefs: from World Community Grid (last modified 23-Aug-2017 18:34:32)
08/11/19 20:03:34 | World Community Grid | Computer location: home
08/11/19 20:03:34 | | General prefs: using separate prefs for home
08/11/19 20:03:34 | | Reading preferences override file
08/11/19 20:03:34 | | Preferences:
08/11/19 20:03:34 | | max memory usage when active: 5716.23 MB
08/11/19 20:03:34 | | max memory usage when idle: 5716.23 MB
08/11/19 20:03:34 | | max disk usage: 100.00 GB
08/11/19 20:03:34 | | max CPUs used: 5
08/11/19 20:03:34 | | (to change preferences, visit a project web site or select Preferences in the Manager)
08/11/19 20:03:34 | | Setting up project and slot directories
08/11/19 20:03:34 | | Checking active tasks
08/11/19 20:03:34 | | Using account manager BOINCstatsBAM!
08/11/19 20:03:34 | | Setting up GUI RPC socket
08/11/19 20:03:34 | | Checking presence of 546 project files
08/11/19 20:03:34 | | Suspending computation - initial delay
08/11/19 20:16:00 | | Re-reading cc_config.xml
08/11/19 20:16:00 | | Config: use HTTP 1.0
08/11/19 20:16:00 | | Config: report completed tasks immediately
08/11/19 20:16:00 | | Config: use all coprocessors
08/11/19 20:16:00 | | log flags: file_xfer, sched_ops, task
08/11/19 20:16:00 | Einstein@Home | Found app_config.xml
08/11/19 20:16:04 | Einstein@Home | update requested by user
08/11/19 20:16:05 | Einstein@Home | Sending scheduler request: Requested by user.
08/11/19 20:16:05 | Einstein@Home | Not requesting tasks: don't need (CPU: not highest priority project; NVIDIA GPU: job cache full)
08/11/19 20:16:06 | Einstein@Home | Scheduler request completed
08/11/19 20:16:06 | Einstein@Home | General prefs: from Einstein@Home (last modified ---)
08/11/19 20:16:06 | Einstein@Home | Computer location: home
In app_config: whatever number I put in gpu_usage, cpu_usage, it doesn't help.
<app_config>
<app>
<name>hsgamma_FGRPB1G</name>
<max_concurrent>4</max_concurrent>
<gpu_versions>
<gpu_usage>0.5</gpu_usage>
<cpu_usage>0.5</cpu_usage>
</gpu_versions>
</app>
</app_config>
<cc_config>
<options>
<use_all_gpus>1</use_all_gpus>
<report_results_immediately>1</report_results_immediately>
<client_version_check_url>http://www.worldcommunitygrid.org/download.php?xml=1</client_version_check_url>
<client_download_url>http://www.worldcommunitygrid.org/download.php</client_download_url>
<network_test_url>http://www.ibm.com/</network_test_url>
<start_delay>30</start_delay>
<http_1_0>1</http_1_0>
</options>
</cc_config>
maximus73 wrote:Somehow I
)
This is too new of a driver for Einstein, drop back to driver: 430.86
This is controlled on the website at Einstein...go to account, preferences, home in your case, and then there is a some boxes near the bottom that have 1.0 in them, change them 0.5 to run 2 wu's at a time. The amount of cpu usage is hardcoded and Einstein always reserves a cpu core for the gpu to use.
Thanks Mikey for the
)
Thanks Mikey for the suggestion, unfortunately it did not help.
I did change that nr (GPU utilization factor for FGRP in preferences/project/other settings/) to 0.5,saved, updated project.
Now it still behaves the same:
<gpu_usage>0.5</gpu_usage> --> both tasks Dev0
<gpu_usage>1.0</gpu_usage> --> 1 tasks Dev0 + 1 task Dev1 --> Boinc lies! both on Dev0...
Boinc tries to send them to both GPU's but they both end up in the same one, as if there was an address clash or the like...
maximus73 wrote:...I did
)
None of that will have any beneficial effect because these project settings are ignored when you are using an app_config.xml file. You need to realise that once you set up app_config.xml, you can't just go back to project settings. Even if you were to delete the app_config.xml file it would make no difference since the contents of that file have been added to the state file (client_state.xml) and will NOT be removed if you delete app_config.xml. Now that you have added that file, you need to make the changes there and then make sure BOINC sees those changes. I suspect you may not be forcing BOINC to re-read the config files every time you make a change. Clicking the update button does not make BOINC re-read the config files. There is a special option for doing that. It's called 'reread config files'. Have you found that option?
Going back to your first message, the first part shows something called GPUQuery_External.exe that is crashing. I don't use Windows so have no knowledge of this app or its purpose. I don't believe it's anything to do with BOINC. Perhaps you might describe its purpose and explain why the fact that it crashes has anything to do with how BOINC chooses to allocate GPU tasks to individual GPUs?
You also seem to have extra options in cc_config.xml. Are those some sort of WCG requirement? Some of those options seem to be specific to WCG. I've never looked at WCG so don't know if they're really needed or not. The report_results_immediately option is not a good idea. If you have a pair of fast GPUs churning out results quickly, it's quite anti-social to force the project's bandwidth to be hammered for every single result. It's much more project friendly to allow several to accumulate and all be reported automatically by BOINC in a single exchange.
The event log you posted shows that you support 3 separate projects - WCG, GPUGRID and Einstein. Without knowing the full range of different applications you are trying to run and the resources they need, it's impossible to know what effect other things may be having on how BOINC is dealing with Einstein GPU tasks.
Your CPU has 8 physical cores and 16 threads. For nvidia GPUs you are going to sabotage efficient performance if you don't allow sufficient CPU resources to be available to support each GPU task. Your posted app_config.xml file shows you are trying to run a total of 4 concurrent GPU tasks (gpu_usage=0.5) but you are not forcing BOINC to make sure there is sufficient CPU support available. cpu_usage=0.5 is absolutely not enough. Do you restrict CPU usage in any other way - eg by setting the % of cores that BOINC is allowed to use to something much less than 100%? If not, then exactly how many CPU tasks (from any project) are running when you are trying to get GPU tasks running on both your GPUs?
Perhaps you might like to tell us which project is most important to you (if any) and what mix of tasks/applications and the numbers thereof that you would like to have running concurrently?
Cheers,
Gary.
Gary, thanks for the prompt
)
Gary, thanks for the prompt reply. Unofrtunately you barely touch my point, and mostly go around it. But thanks anyways.
I do not have much time, so I will be short and to the point.
1) I do aplpy read config from BOINC/options/read config files/, every single time, else I wouldn't see any change in nr of tasks being run. duh!
2) I did put 1 CPU instead of 0.5 no change here too.
3) With GPUgrid I manage to happily crunch 2 tasks on the 2 GPU's. UNfortunately these last days no tasks from GPUgrid, and I turned back to Einstein.
4) With Einstein since a while getting both GPU in use is impossible, no matter what you do with app_config and cc_config.
5) What puzzles me is that even when boinc is sayng it is using "2 devices for the 2 running tasks" it is only using "one device with both running on it", you can check with any tool that checks the ACTUAL gpu usage! How is that on earth possible ? And why BOINC thinks I have 4 openCL-GPU (2 for dev0 and 2 for dev1), is that what makes all go bezerk, which causes dev0-dev1 mixed up somewhere in the chain boinc 2 hardware?
Point 5), is the last and most important point (the others can be ignored please). It is also the title of the post, and it has not been addressed neither by you nor by Mikey.
About GPUQuery_External.exe I checked that is trying to find about some Intel internal GPU, which is absent, and I finally even disabled from the windows registry (IntelOpenCL32.dll key).
Anyhow, I guess this story is costing me more confusion than I wished for, so that made me sound annoyed. I hope now I was clear enough, once and for all. Thanks guys for your patience.
maximus73 wrote:This is the
)
It means that your problem lies in the GPU driver stack(s), not in BOINC.
Use Display Driver Uninstaller to remove all vestiges of what looks like a duplicated OpenCL installation, then install the manufacturer's "last known good" driver, and leave well alone after that.
maximus73 wrote:Somehow I
)
I would not use AMD drivers for NVIDIA GPUs ....
Try following (probably nothing new to you...):
1) save your app_config and cc_config changes somewhere else, so you won't " lose" them.
2) wait till all WUs have finished and then do a "Reset project" for EINSTEIN.
3) re-download a fresh copy of the NVIDIA GRD driver for GEFORCE GTX 1070 for WIN10 (431.60 works fine!).
4) then do a re-install of the NVIDIA driver -- BUT be sure to
a) make a checkmark in the selection box for "Custom (Advanced)" installation, AND
b) then make a checkmark at the bottom for "Perform a clean installation".
5) then try it BEFORE re-applying any "config" files.
I'm sure you will now say "I have done all this already ...", but I just wanted to make this reminder of the "standard procedure" for installing NVIDIA drivers.
Always works fine for us.
We do have multiple GPUs on more than one rig running OK at first try after installing them.
Thanks guys, to all of you
)
Thanks guys, to all of you :)!, Richard and San Fernando Valley.
Richard, great point! I will indeed try some clean uninstall starting from safe-mode using some Display Driver Uninstaller. I had tried IoBit uninstaller but it did not wipe the Nvidia driver well. In fact, after uninstalling, if I go to Nvidia driver check it can still detect via Java the driver version. Moreover the screen still remains in hi-res so it does not uninstall properly I fear.
San Fernando Valley, thanks for your list, I did not do those things in that same order, so that is a helpful guide.
Will let you know how this goes, after work when I have time to try this out :)
mikey wrote: Quote:In
)
The website settings are separate and an attempt to do the same thing as the app_config file and can double up. 0.5 in each will try to run 4 tasks afaik.
mmonnin wrote:The website
)
That should not be the case, app_config.xml should and in my experience will override the website settings.
It should also be pointed out that once one has used an app_config.xml settings are stored in client_config.xml that can only be undone by a project reset (recommended way) or a manual edit of the file (not recommended!) if one want's to revert to using the website settings.