I haven't heard your approach mentioned anywhere before, so, I'm very curious.
It's not documented elsewhere that I've seen, although I haven't really been looking. After some preliminary reading about the differences between the free and proprietary versions of AMDGPU and after reading through the install script for one of the 'supported' Linux -PRO versions and noting the --compute option (I think that's what it was called) I formed the opinion that it might be possible to extract from the full -PRO package, just those components that --compute pointed to. As I recall, there was a small list of components listed under this option. I'm not a programmer so what I tried was all just an experiment to see what would happen.
Please realise that this was months ago and I wasn't really expecting it to work. But it did - straight away. So I set up a file share and wrote a script to do the install from that file share and didn't bother making detailed notes of every single step or thought that I went through along the way. I had a working final procedure and that was all that really mattered. The details are foggy but I'll answer as best I can. I wanted to use the proprietary components if possible because I had seen evidence that the open source versions of OpenCL were not as good.
Paul_134 wrote:
What do you mean you extracted the OpenCL components from the -PRO driver? I assume you mean you found some shared libraries in the -PRO tarball that you identify just by name as OpenCL components? Do you mean specific components? All possible OpenCL related files?
4 specific questions, so 4 specific answers.
I ran tar -xvzf on the tarball and then I removed (extracted) what I wanted from the tree that was created.
No. I didn't try to second guess based on name. I was guided by what the --compute option was pointing to.
Yes - see 2.
Yes. Because I've no real knowledge or experience with these things, I didn't try to exclude anything by drawing conclusions as to whether or not it was really needed. I didn't really expect it to work because I imagined there would be clashes with the open source driver components.
Paul_134 wrote:
What do you mean by "without Mesa?" You don't have any Mesa packages installed on your machine?
No, I don't. I use proprietary drivers (fglrx) on all machines that aren't able to run AMDGPU. I use AMDGPU plus proprietary OpenCL components on the remainder. I'm very happy with the compute performance in all cases.
Paul_134 wrote:
That's sounds non-default, to me, but I'm not sure. So, are you suggesting I remove Mesa? Does it conflict in some way?
I'm not suggesting anything. I'm merely passing on some details of a procedure that works for me. What you do with that information is entirely up to you. I've never needed to install Mesa so have zero first hand experience. I do read and have noted some reports of problems when using it. I'm quite averse to changing something that is working well. If it ain't broke, don't fix it :-). If something isn't working, I'll doggedly try to find a workaround.
Paul_134 wrote:
What do you mean "where they can be found by BOINC?" I think there are some shared libraries in the BOINC directory and the project directories. Can I put the OpenCL "components" there? Or do I need to install them system wide and ldconfig?
My distro doesn't have BOINC in its repo. They used to have a long time ago but it got removed because of the frequent releases of what they regarded as alpha quality software. I think that nobody was prepared to put the effort into packaging it frequently. I had gotten into the habit of installing exactly the version I wanted direct from Berkeley so I'd never used the repo version even when they did have it. I continue to use 7.2.42 because it seems to work without issues, and does everything that I need. I grabbed the source for 7.6.33 and built it on a single machine. It runs fine but doesn't give me anything I'm missing so have not deployed it across the fleet. I built it because I thought I might need it for running post-fglrx AMD drivers. That turned out not to be the case so I'm still using 7.2.42 for all AMD GPUs I now have, bar that one.
When I first started using AMD GPUs after my distro of choice had developed a fglrx package, BOINC could not use the GPU's because it said the GPU could not do OpenCL (which of course was wrong). The problem turned out to be that BOINC was looking for the OpenCL libs in /usr/lib64. My distro was installing the libs in /usr/lib64/fglrx-current/. I would just get into the habit of creating symbolic links in /usr/lib64/ pointing to the installed libs. That's what I meant by "where they can be found by BOINC". After creating the symbolic links I would just run ldconfig -X to rebuild the cache and everything would start working.
Here is the script I use to install the OpenCL components for the open source AMDGPU driver used with RX 460 and Rx 560 cards of mine.
#!/bin/bash
#
# Script to move all AMDGPU-PRO OpenCL components into place.
# This is needed for new installs/upgrades where there is an RX 4xx GPU.
# The components came from the amdgpu-pro-16.60-379184.tar.xz package.
# Run this script as root after the amdgpu FOSS driver is in use.
#
# If any files / locations change, amend these definitions accordingly.
#
BASE_DIR=/media/Elements/STORAGE/PCLOS_Stuff/AMDGPU-PRO
OPT_DIR=$BASE_DIR/opt/amdgpu-pro
ETC_DIR=$BASE_DIR/etc
LIB_DIR=$BASE_DIR/lib
OCL_DIR=/opt/amdgpu-pro/lib64
#
# Before attempting to copy, check that directories needed do exist.
#
if [ ! -d $BASE_DIR ]; then
echo "The directory $BASE_DIR cant be found. Is Elements mounted?"
exit 0
fi
if [ ! -d /opt ]; then
echo -n "Creating the base install directory /opt ... "
mkdir /opt
echo Done.
fi
if [ ! -d $OPT_DIR ]; then
echo "Elements is mounted but the directory $OPT_DIR cant be found."
exit 0
else
echo -n "Installing stuff to /opt/amdgpu-pro ... "
cp -rp $OPT_DIR /opt
echo Done.
fi
if [ ! -d $ETC_DIR ]; then
echo "Config components for /etc cant be found."
echo "You may need to install these manually."
else
echo -n "Installing stuff to /etc/OpenCL/vendors ... "
cp -rp $ETC_DIR/OpenCL /etc
echo Done.
fi
if [ ! -d $LIB_DIR ]; then
echo "udev rules for /lib/udev cant be found."
echo "You may need to install these manually."
else
echo -n "Installing stuff to /lib/udev ... "
cp -rp $LIB_DIR/udev /lib
echo Done.
fi
echo -n "Creating symbolic links to the installed libs to keep BOINC happy ... "
ln -s $OCL_DIR/libamdocl64.so $OCL_DIR/libOpenCL.so.1
ln -s $OCL_DIR/libOpenCL.so.1 $OCL_DIR/libOpenCL.so
ldconfig -X
echo Done.
echo
echo "Except if noted above, OpenCL libs were copied successfully to /opt ..."
echo
Paul_134 wrote:
Below are some start up lines from BOINC. These look fine to me, meaning, when everything was working, the start up was very similar, only the versions are different. I crunched that way for a couple years before this problem started. BOINC detects the GPU just fine and, as you can see from my previous posts, requests GPU work and starts GPU applications for all projects. It's the apps that crash.
Yes, the only thing that would bother me is the two references to missing libs. I guess you must have an nvidia GPU that you're not using so that's OK but I don't know about the second one.
For reference here are three sets of startup messages from three of my hosts. The first is for a machine running two R7 370 GPUs using an old kernel and the fglrx driver. The second is for the machine on which I built and am running BOINC 7.6.33 with a RX 460 GPU using AMDGPU + OpenCL from AMDGPU-PRO. The third is an old machine I resurrected from retirement (it had a motherboard with a burnt choke which had damaged 4 adjacent capacitors - now fully repaired and working like a champ) and am using to test an RX 560 GPU. I'm particularly pleased with that last one because it didn't seem to want to work reliably until I figured out that the RX 460 (no PCIe power connector) that had been in it was trying to draw more from the PCIe slot that the board was prepared to give. Having replaced the 460 with a 560 which HAS a PCIe power connector, everything seems rock solid :-).
Starting BOINC client version 7.2.42 for x86_64-pc-linux-gnu
log flags: file_xfer, sched_ops, task
Libraries: libcurl/7.36.0 OpenSSL/1.0.1g zlib/1.2.8 libidn/1.28 libssh2/1.3.0 librtmp/2.3
Running as a daemon
Data directory: /home/gary/BOINC
OpenCL: AMD/ATI GPU 0: Pitcairn (driver version 1348.5 (VM), device version OpenCL 1.2
AMD-APP (1348.5), 1944MB, 1944MB available, 1306 GFLOPS peak)
OpenCL: AMD/ATI GPU 1: Pitcairn (driver version 1348.5 (VM), device version OpenCL 1.2
AMD-APP (1348.5), 1947MB, 1947MB available, 1306 GFLOPS peak)
OpenCL CPU: Intel(R) Pentium(R) CPU G3260 @ 3.30GHz (OpenCL driver vendor: AMD, Inc.,
driver version 1348.5 (sse2), device version OpenCL 1.2 AMD-APP (1348.5))
Host name: xxxxxx
Processor: 2 GenuineIntel Intel(R) Pentium(R) CPU G3260 @ 3.30GHz [Family 6 Model 60 Step 3]
Processor features: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 ..
OS: Linux: 3.12.16-pclos3
Memory: 7.77 GB physical, 2.00 GB virtual
Disk: 27.70 GB total, 23.33 GB free
============
Starting BOINC client version 7.6.33 for x86_64-pc-linux-gnu
log flags: file_xfer, sched_ops, task
Libraries: libcurl/7.54.0 OpenSSL/1.0.2l zlib/1.2.10 libssh2/1.6.0 librtmp/2.3
Running as a daemon
Data directory: /home/gary/BOINC
OpenCL: AMD/ATI GPU 0: AMD Radeon (TM) RX 460 Graphics (driver version 2264.10, device version
OpenCL 1.2 AMD-APP (2264.10), 2017MB, 2017MB available, 2193 GFLOPS peak)
Host name: xxxxxx
Processor: 2 GenuineIntel Intel(R) Pentium(R) CPU G3260 @ 3.30GHz [Family 6 Model 60 Step 3]
Processor features: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 ..
OS: Linux: 4.10.16-pclos1
Memory: 7.73 GB physical, 2.00 GB virtual
Disk: 14.64 GB total, 13.44 GB free
============
Starting BOINC client version 7.2.42 for x86_64-pc-linux-gnu
log flags: file_xfer, sched_ops, task
Libraries: libcurl/7.55.0 OpenSSL/1.0.2l zlib/1.2.10 libssh2/1.8.0 librtmp/2.3
Running as a daemon
Data directory: /home/gary/BOINC
OpenCL: AMD/ATI GPU 0: Baffin (driver version 2264.10, device version OpenCL 1.2 AMD-APP
(2264.10), 1935MB, 1935MB available, 1357 GFLOPS peak)
Host name: xxxxxx
Processor: 4 GenuineIntel Intel(R) Core(TM)2 Quad CPU Q6600-[at]-2.40GHz [Family 6 Model 15 Step 11]
Processor features: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 ..
OS: Linux: 4.12.8-pclos1
Memory: 1.95 GB physical, 1024.00 MB virtual
Disk: 10.35 GB total, 9.48 GB free
One of the main detection differences is that 7.6.33 sees the GPU by its model name (RX 460) whilst the much older 7.2.42 sees the Volcanic Islands name of Baffin. All three see the OpenCL capability as 1.2 whilst your Mesa driver lists it as 1.1. I don't know if that has any significance as far as running the current EAH apps is concerned.
which includes the -PRO driver. But, you didn't extract any of those files from the RPMs, right? From your script, it doesn't look like you did anything with those files, but I need to ask. You didn't do anything with the -dkms or libdrm- packages, right?
Well, it turns out I cannot even install the RPMs for the OpenCL components as they require the AMDGPU-PRO core RPM. So, if I want to follow your path, I'll have to break open the RPMs and install the libraries, globally, as you did. I really don't want to do that, but let me know if you follow what I'm saying so I know that I'm not missing something.
Yes, that's exactly the list of packages pointed to by the --compute option in the full install script.
Paul_134 wrote:
But, you didn't extract any of those files from the RPMs, right?
Boy, now you're really asking stuff which I can't reliably remember. Despite previous statements that I didn't pick and choose, I seem to think that I may have unpacked each specified RPM and made some decisions. I vaguely remember thinking that I should populate the /opt/amdgpu-pro/ tree with only the most likely to be required stuff - the minimalist approach - and to add more later if anything still seemed to be missing.
Having thought some more, I think the best course is to just give you the complete list of files that I ended up with on the file share. You should be able to work out what came from where. As everything just worked from the outset, this should probably be sufficient. There may well be stuff not needed at all. That doesn't seem to matter. When everything just worked - and kept working, task after task - I lost interest in going back to see what unnecessary stuff there was that I might be able to remove. Using the definitions in the script, the file paths are (under $BASE_DIR) as listed below. These are the install paths as well. I have included sizes of shared libs and indicated any symbolic links.
So that's it, just 14 full file paths relative to '/' on a given machine. At the time I first tried this, I was too stunned that it actually worked to go back and try an even more minimalist list. With the benefit of hindsight and by being prodded to think again, I guess I should just leave out all the drm stuff and try it again.
From experience of where BOINC expected to find the OpenCL libs for the fglrx driver (and the specific name to be used) I created two extra symbolic links in /opt/amdgpu-pro/lib64/. From memory, the links were to the larger of the shared libs with the intention of trying the other if the first choice didn't work. I wasn't precisely sure of what BOINC would be looking for so I created the two most likely (from memory). I never did get around to working out precisely which name BOINC expects.
Please ask again if there's still stuff that's not clear. Are you intending to ditch Mesa/LLVM and see if you can get it working with the AMDGPU FOSS driver plus OpenCL libs from -PRO?
... let me know if you follow what I'm saying so I know that I'm not missing something.
Yes, I know exactly what you're saying. It took me quite a while (other competing responsibilities) to research and compose a reply to your previous message, so I'm sorry I didn't get back to you sooner. Since the machines I'm talking about do nothing but crunch, I have no reservations about installing bits and pieces globally if that's what it takes to permit crunching. I don't have to worry about other things the machine might be used for.
In doing the preparation and the thinking, I'm wondering if it might work (for me anyway since a basic install of my distro uses the AMDGPU FOSS driver and all BOINC seems to need to do is find a libOpenCL.so) if I just take the single file libamdocl64.so and put it straight in /usr/lib64 and create the libOpenCL.so symbolic link there too. Since I really don't understand the concept of ICDs (?installable client drivers? or whatever it stands for) I would also put the .icd file in /etc since perhaps it's needed, I guess. I'll probably end up with more RX nnn GPUs so I might try some experiments along those lines with the next one.
Wow. Thank you so much for taking the time to help me, again. Such a well thought out and complete answer. Thank you.
That makes is very clear. I'm still impressed. It makes a lot of sense that that might work. It's annoying that you have to pull them out of the RPMs, but I totally get it.
Thanks for the list, too. I might check over the RPM manifest and see if I can get what you have by just forcing some of the opencl RPMs to install without their requires. That might work.
Actually, I think ICD is the best thing since sliced bread. I remember when libOpenCL.so was just overwritten by the Catalyst drivers; that was one of the worst things about it! That made it pointless to use without installing everything from the proprietary drivers since, at least with the provided uninstaller, you could restore the old one safely, if you were lucky. I'm thrilled that they finally have ICD working. That's how this is all supposed to work!
I still don't understand why the OSS driver path is so broken, though.
Again, thanks for your help and guidance. I was able to install the opencl RPMs, including the icd stuff, without installing the actual PRO driver. I assumed that doing this would change clinfo output, but it didn't. Well, the AMD clinfo output looks very different, but the data reported is the same, form which I deduce all the capabilities are the same.
Platform Name: Clover
Number of devices: 1
Device Type: CL_DEVICE_TYPE_GPU
Vendor ID: 1002h
Max compute units: 36
Max work items dimensions: 3
Max work items[0]: 256
Max work items[1]: 256
Max work items[2]: 256
Max work group size: 256
Preferred vector width char: 16
Preferred vector width short: 8
Preferred vector width int: 4
Preferred vector width long: 2
Preferred vector width float: 4
Preferred vector width double: 2
Native vector width char: 16
Native vector width short: 8
Native vector width int: 4
Native vector width long: 2
Native vector width float: 4
Native vector width double: 2
Max clock frequency: 1288Mhz
Address bits: 64
Max memory allocation: 6011663974
Image support: No
Max size of kernel argument: 1024
Alignment (bits) of base address: 1024
Minimum alignment (bytes) for any datatype: 128
Single precision floating point capability
Denorms: No
Quiet NaNs: Yes
Round to nearest even: Yes
Round to zero: No
Round to +ve and infinity: No
IEEE754-2008 fused multiply-add: No
Cache type: None
Cache line size: 0
Cache size: 0
Global memory size: 8588091392
Constant buffer size: 2147483647
Max number of constant args: 16
Local memory type: Scratchpad
Local memory size: 32768
Kernel Preferred work group size multiple: 64
Error correction support: 0
Unified memory for Host and Device: 1
Profiling timer resolution: 0
Device endianess: Little
Available: Yes
Compiler available: Yes
Execution capabilities:
Execute OpenCL kernels: Yes
Execute native function: No
Queue on Host properties:
Out-of-Order: No
Profiling : Yes
Platform ID: 0x7f2bfd60d5e0
Name: AMD POLARIS10 (DRM 3.15.0 / 4.12.13-300.fc26.x86_64, LLVM 4.0.0)
Vendor: AMD
Device OpenCL C version: OpenCL C 1.1
Driver version: 17.1.7
Profile: FULL_PROFILE
Version: OpenCL 1.1 Mesa 17.1.7
Extensions: cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_byte_addressable_store cl_khr_fp64
Platform Name: Portable Computing Language
Number of devices: 1
Device Type: Vendor ID: 1022h
Max compute units: 16
Max work items dimensions: 3
Max work items[0]: 4096
Max work items[1]: 4096
Max work items[2]: 4096
Max work group size: 4096
Preferred vector width char: 16
Preferred vector width short: 8
Preferred vector width int: 4
Preferred vector width long: 2
Preferred vector width float: 4
Preferred vector width double: 2
Native vector width char: 16
Native vector width short: 8
Native vector width int: 4
Native vector width long: 2
Native vector width float: 4
Native vector width double: 2
Max clock frequency: 3850Mhz
Address bits: 64
Max memory allocation: 18966855680
Image support: Yes
Max number of images read arguments: 128
Max number of images write arguments: 128
Max image 2D width: 32768
Max image 2D height: 32768
Max image 3D width: 2048
Max image 3D height: 2048
Max image 3D depth: 2048
Max samplers within kernel: 16
Max size of kernel argument: 1024
Alignment (bits) of base address: 1024
Minimum alignment (bytes) for any datatype: 128
Single precision floating point capability
Denorms: No
Quiet NaNs: Yes
Round to nearest even: Yes
Round to zero: No
Round to +ve and infinity: No
IEEE754-2008 fused multiply-add: No
Cache type: Read/Write
Cache line size: 64
Cache size: 32768
Global memory size: 18966855680
Constant buffer size: 18966855680
Max number of constant args: 8
Local memory type: Global
Local memory size: 18966855680
Max pipe arguments: 16
Max pipe active reservations: 1
Max pipe packet size: 1024
Max global variable size: 0
Max global variable preferred total size: 0
Max read/write image args: 128
Max on device events: 1024
Queue on device max size: 262144
Max on device queues: 1
Queue on device preferred size: 16384
SVM capabilities:
Coarse grain buffer: Yes
Fine grain buffer: Yes
Fine grain system: No
Atomics: Yes
Preferred platform atomic alignment: 0
Preferred global atomic alignment: 0
Preferred local atomic alignment: 0
Kernel Preferred work group size multiple: 8
Error correction support: 0
Unified memory for Host and Device: 1
Profiling timer resolution: 1
Device endianess: Little
Available: Yes
Compiler available: Yes
Execution capabilities:
Execute OpenCL kernels: Yes
Execute native function: Yes
Queue on Host properties:
Out-of-Order: No
Profiling : Yes
Queue on Device properties:
Out-of-Order: Yes
Profiling : Yes
Platform ID: 0x7f2bf4651000
Name: pthread-AMD Ryzen 7 1800X Eight-Core Processor
Vendor: AuthenticAMD
Device OpenCL C version: OpenCL C 2.0
Driver version: 0.14
Profile: FULL_PROFILE
Version: OpenCL 2.0 pocl HSTR: pthread-x86_64-unknown-linux-gnu-haswell
Extensions: cl_khr_byte_addressable_store cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_spir cl_khr_int64 cl_khr_fp64 cl_khr_int64_base_atomics cl_khr_int64_extended_atomics
Hmm, when I add /opt/amdgpu-pro/lib64 to LD_LIBRARY_PATH, both clinfo programs crash. Makes me think your approach may not work for me. AFAIK, this is equivalent to doing what you did, except that not every program is getting the AMD opencl, only clinfo.
Hmm, wait, I don't have the drm or kms libraries installed. clinfo isn't linked to them, but, perhaps I should try to do that, first.
Sure. But first I'll point out something that I've recently remembered that I did back earlier in the year when I first started playing around with an RX 460 and installing the compute components. I remember coming across a comment somewhere about adding /opt/amdgpu-pro/lib64 to the LD_LIBRARY_PATH - so I did just that (but in the script itself that I use to launch BOINC and not more generally) and promptly forgot about it. I didn't really know that what I was doing would actually work, so I wanted to quarantine those libs so only BOINC would know about them. Seemed like a good idea at the time :-). The script contains the lines
if [ -x /home/gary/BOINC/boinc ]
then
( cd /home/gary/BOINC; LD_LIBRARY_PATH=/opt/amdgpu-pro/lib64 ./boinc --daemon )
fi
I remembered this when I just did a 'which clinfo' and it couldn't be found. That triggered a memory that /opt/amdgpu-pro/bin (where clinfo is installed) isn't part of the PATH and /opt/amdgpu-pro/lib64 isn't part of LD_LIBRARY_PATH. Sure enough, to get clinfo to launch I had to use the command
and the output it produced is listed below. I've left out stuff to do with the CPU.
Number of platforms: 1
Platform Profile: FULL_PROFILE
Platform Version: OpenCL 2.0 AMD-APP (2264.10)
Platform Name: AMD Accelerated Parallel Processing
Platform Vendor: Advanced Micro Devices, Inc.
Platform Extensions: cl_khr_icd cl_amd_event_callback cl_amd_offline_devices
Platform Name: AMD Accelerated Parallel Processing
Number of devices: 2
Device Type: CL_DEVICE_TYPE_GPU
Vendor ID: 1002h
Board name: AMD Radeon (TM) RX 460 Graphics
Device Topology: PCI[ B#1, D#0, F#0 ]
Max compute units: 14
Max work items dimensions: 3
Max work items[0]: 256
Max work items[1]: 256
Max work items[2]: 256
Max work group size: 256
Preferred vector width char: 4
Preferred vector width short: 2
Preferred vector width int: 1
Preferred vector width long: 1
Preferred vector width float: 1
Preferred vector width double: 1
Native vector width char: 4
Native vector width short: 2
Native vector width int: 1
Native vector width long: 1
Native vector width float: 1
Native vector width double: 1
Max clock frequency: 1210Mhz
Address bits: 64
Max memory allocation: 681255936
Image support: Yes
Max number of images read arguments: 128
Max number of images write arguments: 8
Max image 2D width: 16384
Max image 2D height: 16384
Max image 3D width: 2048
Max image 3D height: 2048
Max image 3D depth: 2048
Max samplers within kernel: 16
Max size of kernel argument: 1024
Alignment (bits) of base address: 2048
Minimum alignment (bytes) for any datatype: 128
Single precision floating point capability
Denorms: No
Quiet NaNs: Yes
Round to nearest even: Yes
Round to zero: Yes
Round to +ve and infinity: Yes
IEEE754-2008 fused multiply-add: Yes
Cache type: Read/Write
Cache line size: 64
Cache size: 16384
Global memory size: 1158799360
Constant buffer size: 681255936
Max number of constant args: 8
Local memory type: Scratchpad
Local memory size: 32768
Max pipe arguments: 0
Max pipe active reservations: 0
Max pipe packet size: 0
Max global variable size: 0
Max global variable preferred total size: 0
Max read/write image args: 0
Max on device events: 0
Queue on device max size: 0
Max on device queues: 0
Queue on device preferred size: 0
SVM capabilities:
Coarse grain buffer: No
Fine grain buffer: No
Fine grain system: No
Atomics: No
Preferred platform atomic alignment: 0
Preferred global atomic alignment: 0
Preferred local atomic alignment: 0
Kernel Preferred work group size multiple: 64
Error correction support: 0
Unified memory for Host and Device: 0
Profiling timer resolution: 1
Device endianess: Little
Available: Yes
Compiler available: Yes
Execution capabilities:
Execute OpenCL kernels: Yes
Execute native function: No
Queue on Host properties:
Out-of-Order: No
Profiling : Yes
Queue on Device properties:
Out-of-Order: No
Profiling : No
Platform ID: 0x7fb1413ccb18
Name: Baffin
Vendor: Advanced Micro Devices, Inc.
Device OpenCL C version: OpenCL C 1.2
Driver version: 2264.10
Profile: FULL_PROFILE
Version: OpenCL 1.2 AMD-APP (2264.10)
Extensions: cl_khr_fp64 cl_amd_fp64 cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_3d_image_writes cl_khr_byte_addressable_store cl_khr_fp16 cl_khr_gl_sharing cl_amd_device_attribute_query cl_amd_vec3 cl_amd_printf cl_amd_media_ops cl_amd_media_ops2 cl_amd_popcnt cl_khr_image2d_from_buffer cl_khr_spir cl_khr_gl_event
I have purchased more RX 460s and I have experimented with leaving drm and kms stuff out as mentioned previously. If I do that, BOINC says there is no usable GPU. As soon as I put them back in, all is good.
I really don't know enough to give proper advice (just enough to be dangerous :-) ) but it seems to me the difference in the 'Platform Version:' line says it all - Mesa as opposed to AMD-APP. I hope the above means more to you than it does to me :-).
Paul_134 wrote:I haven't
)
It's not documented elsewhere that I've seen, although I haven't really been looking. After some preliminary reading about the differences between the free and proprietary versions of AMDGPU and after reading through the install script for one of the 'supported' Linux -PRO versions and noting the --compute option (I think that's what it was called) I formed the opinion that it might be possible to extract from the full -PRO package, just those components that --compute pointed to. As I recall, there was a small list of components listed under this option. I'm not a programmer so what I tried was all just an experiment to see what would happen.
Please realise that this was months ago and I wasn't really expecting it to work. But it did - straight away. So I set up a file share and wrote a script to do the install from that file share and didn't bother making detailed notes of every single step or thought that I went through along the way. I had a working final procedure and that was all that really mattered. The details are foggy but I'll answer as best I can. I wanted to use the proprietary components if possible because I had seen evidence that the open source versions of OpenCL were not as good.
4 specific questions, so 4 specific answers.
No, I don't. I use proprietary drivers (fglrx) on all machines that aren't able to run AMDGPU. I use AMDGPU plus proprietary OpenCL components on the remainder. I'm very happy with the compute performance in all cases.
I'm not suggesting anything. I'm merely passing on some details of a procedure that works for me. What you do with that information is entirely up to you. I've never needed to install Mesa so have zero first hand experience. I do read and have noted some reports of problems when using it. I'm quite averse to changing something that is working well. If it ain't broke, don't fix it :-). If something isn't working, I'll doggedly try to find a workaround.
My distro doesn't have BOINC in its repo. They used to have a long time ago but it got removed because of the frequent releases of what they regarded as alpha quality software. I think that nobody was prepared to put the effort into packaging it frequently. I had gotten into the habit of installing exactly the version I wanted direct from Berkeley so I'd never used the repo version even when they did have it. I continue to use 7.2.42 because it seems to work without issues, and does everything that I need. I grabbed the source for 7.6.33 and built it on a single machine. It runs fine but doesn't give me anything I'm missing so have not deployed it across the fleet. I built it because I thought I might need it for running post-fglrx AMD drivers. That turned out not to be the case so I'm still using 7.2.42 for all AMD GPUs I now have, bar that one.
When I first started using AMD GPUs after my distro of choice had developed a fglrx package, BOINC could not use the GPU's because it said the GPU could not do OpenCL (which of course was wrong). The problem turned out to be that BOINC was looking for the OpenCL libs in /usr/lib64. My distro was installing the libs in /usr/lib64/fglrx-current/. I would just get into the habit of creating symbolic links in /usr/lib64/ pointing to the installed libs. That's what I meant by "where they can be found by BOINC". After creating the symbolic links I would just run ldconfig -X to rebuild the cache and everything would start working.
Here is the script I use to install the OpenCL components for the open source AMDGPU driver used with RX 460 and Rx 560 cards of mine.
Yes, the only thing that would bother me is the two references to missing libs. I guess you must have an nvidia GPU that you're not using so that's OK but I don't know about the second one.
For reference here are three sets of startup messages from three of my hosts. The first is for a machine running two R7 370 GPUs using an old kernel and the fglrx driver. The second is for the machine on which I built and am running BOINC 7.6.33 with a RX 460 GPU using AMDGPU + OpenCL from AMDGPU-PRO. The third is an old machine I resurrected from retirement (it had a motherboard with a burnt choke which had damaged 4 adjacent capacitors - now fully repaired and working like a champ) and am using to test an RX 560 GPU. I'm particularly pleased with that last one because it didn't seem to want to work reliably until I figured out that the RX 460 (no PCIe power connector) that had been in it was trying to draw more from the PCIe slot that the board was prepared to give. Having replaced the 460 with a 560 which HAS a PCIe power connector, everything seems rock solid :-).
============
============
One of the main detection differences is that 7.6.33 sees the GPU by its model name (RX 460) whilst the much older 7.2.42 sees the Volcanic Islands name of Baffin. All three see the OpenCL capability as 1.2 whilst your Mesa driver lists it as 1.1. I don't know if that has any significance as far as running the current EAH apps is concerned.
Cheers,
Gary.
Wow, thanks for the detailed
)
Wow, thanks for the detailed reply! I'm fascinated; that's a very clever approach.
I'm still confused about one thing, though. The list of packages installed by the AMDGPU-PRO install script is
libopencl-amdgpu-pro-clinfo libopencl-amdgpu-pro-icd amdgpu-pro-dkms libdrm-amdgpu-pro
which includes the -PRO driver. But, you didn't extract any of those files from the RPMs, right? From your script, it doesn't look like you did anything with those files, but I need to ask. You didn't do anything with the -dkms or libdrm- packages, right?
Well, it turns out I cannot
)
Well, it turns out I cannot even install the RPMs for the OpenCL components as they require the AMDGPU-PRO core RPM. So, if I want to follow your path, I'll have to break open the RPMs and install the libraries, globally, as you did. I really don't want to do that, but let me know if you follow what I'm saying so I know that I'm not missing something.
Paul_134 wrote:The list of
)
Yes, that's exactly the list of packages pointed to by the --compute option in the full install script.
Boy, now you're really asking stuff which I can't reliably remember. Despite previous statements that I didn't pick and choose, I seem to think that I may have unpacked each specified RPM and made some decisions. I vaguely remember thinking that I should populate the /opt/amdgpu-pro/ tree with only the most likely to be required stuff - the minimalist approach - and to add more later if anything still seemed to be missing.
Having thought some more, I think the best course is to just give you the complete list of files that I ended up with on the file share. You should be able to work out what came from where. As everything just worked from the outset, this should probably be sufficient. There may well be stuff not needed at all. That doesn't seem to matter. When everything just worked - and kept working, task after task - I lost interest in going back to see what unnecessary stuff there was that I might be able to remove. Using the definitions in the script, the file paths are (under $BASE_DIR) as listed below. These are the install paths as well. I have included sizes of shared libs and indicated any symbolic links.
So that's it, just 14 full file paths relative to '/' on a given machine. At the time I first tried this, I was too stunned that it actually worked to go back and try an even more minimalist list. With the benefit of hindsight and by being prodded to think again, I guess I should just leave out all the drm stuff and try it again.
From experience of where BOINC expected to find the OpenCL libs for the fglrx driver (and the specific name to be used) I created two extra symbolic links in /opt/amdgpu-pro/lib64/. From memory, the links were to the larger of the shared libs with the intention of trying the other if the first choice didn't work. I wasn't precisely sure of what BOINC would be looking for so I created the two most likely (from memory). I never did get around to working out precisely which name BOINC expects.
Please ask again if there's still stuff that's not clear. Are you intending to ditch Mesa/LLVM and see if you can get it working with the AMDGPU FOSS driver plus OpenCL libs from -PRO?
Cheers,
Gary.
Paul_134 wrote:... let me
)
Yes, I know exactly what you're saying. It took me quite a while (other competing responsibilities) to research and compose a reply to your previous message, so I'm sorry I didn't get back to you sooner. Since the machines I'm talking about do nothing but crunch, I have no reservations about installing bits and pieces globally if that's what it takes to permit crunching. I don't have to worry about other things the machine might be used for.
In doing the preparation and the thinking, I'm wondering if it might work (for me anyway since a basic install of my distro uses the AMDGPU FOSS driver and all BOINC seems to need to do is find a libOpenCL.so) if I just take the single file libamdocl64.so and put it straight in /usr/lib64 and create the libOpenCL.so symbolic link there too. Since I really don't understand the concept of ICDs (?installable client drivers? or whatever it stands for) I would also put the .icd file in /etc since perhaps it's needed, I guess. I'll probably end up with more RX nnn GPUs so I might try some experiments along those lines with the next one.
Cheers,
Gary.
Wow. Thank you so much for
)
Wow. Thank you so much for taking the time to help me, again. Such a well thought out and complete answer. Thank you.
That makes is very clear. I'm still impressed. It makes a lot of sense that that might work. It's annoying that you have to pull them out of the RPMs, but I totally get it.
Thanks for the list, too. I might check over the RPM manifest and see if I can get what you have by just forcing some of the opencl RPMs to install without their requires. That might work.
Actually, I think ICD is the best thing since sliced bread. I remember when libOpenCL.so was just overwritten by the Catalyst drivers; that was one of the worst things about it! That made it pointless to use without installing everything from the proprietary drivers since, at least with the provided uninstaller, you could restore the old one safely, if you were lucky. I'm thrilled that they finally have ICD working. That's how this is all supposed to work!
I still don't understand why the OSS driver path is so broken, though.
Hi Gary, Again, thanks
)
Hi Gary,
Again, thanks for your help and guidance. I was able to install the opencl RPMs, including the icd stuff, without installing the actual PRO driver. I assumed that doing this would change clinfo output, but it didn't. Well, the AMD clinfo output looks very different, but the data reported is the same, form which I deduce all the capabilities are the same.
I wonder if you could share your clinfo?
Platform Name: Clover
Number of devices: 1
Device Type: CL_DEVICE_TYPE_GPU
Vendor ID: 1002h
Max compute units: 36
Max work items dimensions: 3
Max work items[0]: 256
Max work items[1]: 256
Max work items[2]: 256
Max work group size: 256
Preferred vector width char: 16
Preferred vector width short: 8
Preferred vector width int: 4
Preferred vector width long: 2
Preferred vector width float: 4
Preferred vector width double: 2
Native vector width char: 16
Native vector width short: 8
Native vector width int: 4
Native vector width long: 2
Native vector width float: 4
Native vector width double: 2
Max clock frequency: 1288Mhz
Address bits: 64
Max memory allocation: 6011663974
Image support: No
Max size of kernel argument: 1024
Alignment (bits) of base address: 1024
Minimum alignment (bytes) for any datatype: 128
Single precision floating point capability
Denorms: No
Quiet NaNs: Yes
Round to nearest even: Yes
Round to zero: No
Round to +ve and infinity: No
IEEE754-2008 fused multiply-add: No
Cache type: None
Cache line size: 0
Cache size: 0
Global memory size: 8588091392
Constant buffer size: 2147483647
Max number of constant args: 16
Local memory type: Scratchpad
Local memory size: 32768
Kernel Preferred work group size multiple: 64
Error correction support: 0
Unified memory for Host and Device: 1
Profiling timer resolution: 0
Device endianess: Little
Available: Yes
Compiler available: Yes
Execution capabilities:
Execute OpenCL kernels: Yes
Execute native function: No
Queue on Host properties:
Out-of-Order: No
Profiling : Yes
Platform ID: 0x7f2bfd60d5e0
Name: AMD POLARIS10 (DRM 3.15.0 / 4.12.13-300.fc26.x86_64, LLVM 4.0.0)
Vendor: AMD
Device OpenCL C version: OpenCL C 1.1
Driver version: 17.1.7
Profile: FULL_PROFILE
Version: OpenCL 1.1 Mesa 17.1.7
Extensions: cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_byte_addressable_store cl_khr_fp64
Platform Name: Portable Computing Language
Number of devices: 1
Device Type: Vendor ID: 1022h
Max compute units: 16
Max work items dimensions: 3
Max work items[0]: 4096
Max work items[1]: 4096
Max work items[2]: 4096
Max work group size: 4096
Preferred vector width char: 16
Preferred vector width short: 8
Preferred vector width int: 4
Preferred vector width long: 2
Preferred vector width float: 4
Preferred vector width double: 2
Native vector width char: 16
Native vector width short: 8
Native vector width int: 4
Native vector width long: 2
Native vector width float: 4
Native vector width double: 2
Max clock frequency: 3850Mhz
Address bits: 64
Max memory allocation: 18966855680
Image support: Yes
Max number of images read arguments: 128
Max number of images write arguments: 128
Max image 2D width: 32768
Max image 2D height: 32768
Max image 3D width: 2048
Max image 3D height: 2048
Max image 3D depth: 2048
Max samplers within kernel: 16
Max size of kernel argument: 1024
Alignment (bits) of base address: 1024
Minimum alignment (bytes) for any datatype: 128
Single precision floating point capability
Denorms: No
Quiet NaNs: Yes
Round to nearest even: Yes
Round to zero: No
Round to +ve and infinity: No
IEEE754-2008 fused multiply-add: No
Cache type: Read/Write
Cache line size: 64
Cache size: 32768
Global memory size: 18966855680
Constant buffer size: 18966855680
Max number of constant args: 8
Local memory type: Global
Local memory size: 18966855680
Max pipe arguments: 16
Max pipe active reservations: 1
Max pipe packet size: 1024
Max global variable size: 0
Max global variable preferred total size: 0
Max read/write image args: 128
Max on device events: 1024
Queue on device max size: 262144
Max on device queues: 1
Queue on device preferred size: 16384
SVM capabilities:
Coarse grain buffer: Yes
Fine grain buffer: Yes
Fine grain system: No
Atomics: Yes
Preferred platform atomic alignment: 0
Preferred global atomic alignment: 0
Preferred local atomic alignment: 0
Kernel Preferred work group size multiple: 8
Error correction support: 0
Unified memory for Host and Device: 1
Profiling timer resolution: 1
Device endianess: Little
Available: Yes
Compiler available: Yes
Execution capabilities:
Execute OpenCL kernels: Yes
Execute native function: Yes
Queue on Host properties:
Out-of-Order: No
Profiling : Yes
Queue on Device properties:
Out-of-Order: Yes
Profiling : Yes
Platform ID: 0x7f2bf4651000
Name: pthread-AMD Ryzen 7 1800X Eight-Core Processor
Vendor: AuthenticAMD
Device OpenCL C version: OpenCL C 2.0
Driver version: 0.14
Profile: FULL_PROFILE
Version: OpenCL 2.0 pocl HSTR: pthread-x86_64-unknown-linux-gnu-haswell
Extensions: cl_khr_byte_addressable_store cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_spir cl_khr_int64 cl_khr_fp64 cl_khr_int64_base_atomics cl_khr_int64_extended_atomics
Hmm, when I add
)
Hmm, when I add /opt/amdgpu-pro/lib64 to LD_LIBRARY_PATH, both clinfo programs crash. Makes me think your approach may not work for me. AFAIK, this is equivalent to doing what you did, except that not every program is getting the AMD opencl, only clinfo.
Hmm, wait, I don't have the drm or kms libraries installed. clinfo isn't linked to them, but, perhaps I should try to do that, first.
Okay, just adding those libs
)
Okay, just adding those libs didn't help.
Paul_134 wrote:I wonder if
)
Sure. But first I'll point out something that I've recently remembered that I did back earlier in the year when I first started playing around with an RX 460 and installing the compute components. I remember coming across a comment somewhere about adding /opt/amdgpu-pro/lib64 to the LD_LIBRARY_PATH - so I did just that (but in the script itself that I use to launch BOINC and not more generally) and promptly forgot about it. I didn't really know that what I was doing would actually work, so I wanted to quarantine those libs so only BOINC would know about them. Seemed like a good idea at the time :-). The script contains the lines
I remembered this when I just did a 'which clinfo' and it couldn't be found. That triggered a memory that /opt/amdgpu-pro/bin (where clinfo is installed) isn't part of the PATH and /opt/amdgpu-pro/lib64 isn't part of LD_LIBRARY_PATH. Sure enough, to get clinfo to launch I had to use the command
and the output it produced is listed below. I've left out stuff to do with the CPU.
I have purchased more RX 460s and I have experimented with leaving drm and kms stuff out as mentioned previously. If I do that, BOINC says there is no usable GPU. As soon as I put them back in, all is good.
I really don't know enough to give proper advice (just enough to be dangerous :-) ) but it seems to me the difference in the 'Platform Version:' line says it all - Mesa as opposed to AMD-APP. I hope the above means more to you than it does to me :-).
Cheers,
Gary.