Can anyone explain this result?
22 Apr 2014 4:19:56 UTC
Topic 197550
(moderation:
http://einsteinathome.org/task/431943848
This task was completed on an integrated AMD HD3000.
Process Explorer showed that the task was using only 2% - 4% of the CPU and 35% - 47% of the GPU. Since the HD3000 aren't opencl capable, how did it run at all?
Can anyone explain this result?
)
What makes you say that the task was computed on an "integrated AMD HD3000"?
The task output (stderr) says 'Using OpenCL device "Cypress"', and the computer description says it's fitted with 'ATI Radeon HD 5800/5900 series (Cypress/Hemlock)' - and the 'AMD Athlon(tm) II X2 250 Processor' didn't have an integrated APU when it was launched in 2009!
Computation on the discrete HD 5xxx card sounds more plausible.
CAL: ATI GPU 0: ATI Radeon HD
)
CAL: ATI GPU 0: ATI Radeon HD 5800/5900 series (Cypress/Hemlock) (CAL version 1.4.1741, 1024MB, 991MB available, 4176 GFLOPS peak)
CAL: ATI GPU 1: ATI Radeon HD 2300/2400/3200/4200 (RV610) (CAL version 1.4.1741, 256MB, 224MB available, 56 GFLOPS peak)
OpenCL: AMD/ATI GPU 0: ATI Radeon HD 5800/5900 series (Cypress/Hemlock) (driver version CAL 1.4.1741 (VM), device version OpenCL 1.2 AMD-APP (938.2), 1024MB, 991MB available, 4176 GFLOPS peak)
OpenCL CPU: AMD Athlon(tm) II X2 250 Processor (OpenCL driver vendor: Advanced Micro Devices, Inc., driver version 2.0 (sse2), device version OpenCL 1.2 AMD-APP (938.2))
Because at the time this task was running Boinc Manager showed that the task was running on device 1. I have since added an exclusion to the cc_config.xml to ensure that it only runs on device 0.
Just posted to see if anyone might have an idea as to how the task managed to run on the HD3000.
RE: CAL: ATI GPU 0: ATI
)
It didn't manage to run on the HD3000, it ran on the HD 5800/5900, Boinc doesn't support natively mixing AMD OpenCL and non-OpenCL GPUs together,
we went through all this at Setiathome a year or so ago (this is before GPU exclusions were available),
Boinc will report its best GPU to a project, the project will send work suitable for the requirements of that GPU, ie OpenCL work here,
Boinc will then allocate work to it's GPUs, two tasks, one for each GPU, But since there is only one OpenCL GPU device (use CLinfo to check) both tasks will end up being run on that.
Claggy
From the Boinc Log, this is
)
From the Boinc Log, this is going to be a long, long post:
16-Apr-2014 17:14:00 [---] Starting BOINC client version 7.3.15 for windows_x86_64
16-Apr-2014 17:14:00 [---] log flags: file_xfer, sched_ops, task, coproc_debug, scrsave_debug, task_debug
16-Apr-2014 17:14:00 [---] Libraries: libcurl/7.33.0 OpenSSL/1.0.1f zlib/1.2.8
16-Apr-2014 17:14:00 [---] Data directory: C:\ProgramData\BOINC
16-Apr-2014 17:14:00 [---] Running under account Darrell
16-Apr-2014 17:14:00 [---] [coproc] launching child process at C:\Program Files\BOINC\boinc.exe
16-Apr-2014 17:14:00 [---] [coproc] relative to directory C:\ProgramData\BOINC
16-Apr-2014 17:14:00 [---] [coproc] with data directory "C:\ProgramData\BOINC"
16-Apr-2014 17:14:00 [---] CAL: ATI GPU 0: ATI Radeon HD 5800/5900 series (Cypress/Hemlock) (CAL version 1.4.1741, 1024MB, 991MB available, 4176 GFLOPS peak)
16-Apr-2014 17:14:00 [---] CAL: ATI GPU 1: ATI Radeon HD 2300/2400/3200/4200 (RV610) (CAL version 1.4.1741, 256MB, 224MB available, 56 GFLOPS peak)
16-Apr-2014 17:14:00 [---] OpenCL: AMD/ATI GPU 0: ATI Radeon HD 5800/5900 series (Cypress/Hemlock) (driver version CAL 1.4.1741 (VM), device version OpenCL 1.2 AMD-APP (938.2), 1024MB, 991MB available, 4176 GFLOPS peak)
16-Apr-2014 17:14:00 [---] OpenCL CPU: AMD Athlon(tm) II X2 250 Processor (OpenCL driver vendor: Advanced Micro Devices, Inc., driver version 2.0 (sse2), device version OpenCL 1.2 AMD-APP (938.2))
16-Apr-2014 17:14:00 [---] No NVIDIA library found
16-Apr-2014 17:14:00 [---] Host name: darrell-II
16-Apr-2014 17:14:00 [---] Processor: 2 AuthenticAMD AMD Athlon(tm) II X2 250 Processor [Family 16 Model 6 Stepping 3]
16-Apr-2014 17:14:00 [---] Processor features: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 htt pni cx16 popcnt syscall nx lm svm sse4a osvw ibs skinit wdt page1gb rdtscp 3dnowext 3dnow
16-Apr-2014 17:14:00 [---] OS: Microsoft Windows 7: Professional x64 Edition, Service Pack 1, (06.01.7601.00)
16-Apr-2014 17:14:00 [---] Memory: 7.75 GB physical, 15.50 GB virtual
16-Apr-2014 17:14:00 [---] Disk: 931.41 GB total, 819.35 GB free
16-Apr-2014 17:14:00 [---] Local time is UTC -5 hours
16-Apr-2014 17:14:00 [---] VirtualBox version: 4.3.10
16-Apr-2014 17:14:00 [Poem@Home] Found app_config.xml
16-Apr-2014 17:14:00 [Poem@Home] Your app_config.xml file refers to an unknown application 'poemcl'. Known applications: None
16-Apr-2014 17:14:00 [Einstein@Home] Found app_config.xml
16-Apr-2014 17:14:00 [Milkyway@Home] Found app_config.xml
16-Apr-2014 17:14:00 [Moo! Wrapper] Found app_config.xml
16-Apr-2014 17:14:00 [SETI@home] Found app_config.xml
16-Apr-2014 17:14:00 [SETI@home Beta Test] Found app_config.xml
16-Apr-2014 17:14:00 [PrimeGrid] Found app_config.xml
16-Apr-2014 17:14:00 [---] Config: report completed tasks immediately
16-Apr-2014 17:14:00 [---] Config: use all coprocessors
16-Apr-2014 17:14:00 [SETI@home] Config: excluded GPU. Type: ATI. App: astropulse_v6. Device: 1
16-Apr-2014 17:14:00 [SETI@home] Config: excluded GPU. Type: ATI. App: setiathome_v7. Device: 1
16-Apr-2014 17:14:00 [SETI@home Beta Test] Config: excluded GPU. Type: ATI. App: setiathome_v7. Device: 1
16-Apr-2014 17:14:00 [SETI@home Beta Test] Config: excluded GPU. Type: ATI. App: astropulse_v6. Device: 1
16-Apr-2014 17:14:00 [Einstein@Home] Config: excluded GPU. Type: ATI. App: einsteinbinary_BRP5. Device: 1
16-Apr-2014 17:14:00 [Einstein@Home] Config: excluded GPU. Type: ATI. App: hsgamma_FGRP3. Device: 1
16-Apr-2014 17:14:00 [PrimeGrid] Config: excluded GPU. Type: ATI. App: genefer. Device: 0
16-Apr-2014 17:14:00 [PrimeGrid] Config: excluded GPU. Type: ATI. App: pps_sr2sieve. Device: 1
16-Apr-2014 17:14:00 [Poem@Home] Config: excluded GPU. Type: ATI. App: poemcl. Device: 1
16-Apr-2014 17:14:00 [Poem@Home] Config: excluded GPU. Type: ATI. App: poempp. Device: 1
16-Apr-2014 17:14:00 [Milkyway@Home] Config: excluded GPU. Type: ATI. App: milkyway_nbody. Device: 1
16-Apr-2014 17:14:00 [Milkyway@Home] Config: excluded GPU. Type: ATI. App: milkyway_separation__modified_fit. Device: 1
16-Apr-2014 17:14:00 [Milkyway@Home] Config: excluded GPU. Type: ATI. App: milkyway. Device: 1
16-Apr-2014 17:14:00 [Collatz Conjecture] Config: excluded GPU. Type: ATI. App: solo_collatz. Device: 0
16-Apr-2014 17:14:00 [Moo! Wrapper] Config: excluded GPU. Type: ATI. App: dnetc. Device: 0
16-Apr-2014 17:14:00 [Poem@Home] A GPU exclusion in your cc_config.xml file refers to an unknown application 'poemcl'. Known applications: None
16-Apr-2014 17:14:00 [Poem@Home] A GPU exclusion in your cc_config.xml file refers to an unknown application 'poempp'. Known applications: None
16-Apr-2014 17:14:00 [---] Version change (7.3.11 -> 7.3.15)
This is the start-up, this computer has been in storage for half a year, Einstein added new apps during that time. Device 1 or instance 1 is the HD3000.
16-Apr-2014 17:14:01 Initialization completed
16-Apr-2014 17:14:01 [---] Running CPU benchmarks
16-Apr-2014 17:14:01 [---] Suspending computation - CPU benchmarks in progress
16-Apr-2014 17:14:33 [---] Benchmark results:
16-Apr-2014 17:14:33 [---] Number of CPUs: 2
16-Apr-2014 17:14:33 [---] 2776 floating point MIPS (Whetstone) per CPU
16-Apr-2014 17:14:33 [---] 7093 integer MIPS (Dhrystone) per CPU
16-Apr-2014 17:14:34 [Milkyway@Home] [coproc] Assigning 0.500000 of ATI free instance 0 to de_modfit_15_3s_130_wrap_1_1396965303_1897902_1
16-Apr-2014 17:14:34 [SETI@home] [coproc] Assigning 0.500000 of ATI instance 0 to ap_09dc08ad_B0_P1_00257_20140416_15148.wu_1
16-Apr-2014 17:14:34 [Einstein@Home] [coproc] Assigning 0.500000 of ATI free instance 1 to p2030.20131122.G177.14+00.50.S.b3s0g0.00000_3792_0
16-Apr-2014 17:14:34 [Einstein@Home] [task] task_state=EXECUTING for p2030.20131122.G177.14+00.50.S.b3s0g0.00000_3792_0 from start
16-Apr-2014 17:14:34 [SETI@home] [task] task_state=EXECUTING for ap_09dc08ad_B0_P1_00257_20140416_15148.wu_1 from start
16-Apr-2014 17:14:34 [malariacontrol.net] [task] task_state=EXECUTING for wu_4271_415_37036_0_1397679909_0 from start
16-Apr-2014 17:14:34 [malariacontrol.net] [task] task_state=EXECUTING for wu_4271_414_37036_0_1397679909_0 from start
16-Apr-2014 17:14:34 [Milkyway@Home] [task] task_state=EXECUTING for de_modfit_15_3s_130_wrap_1_1396965303_1897902_1 from start
16-Apr-2014 17:15:34 [Milkyway@Home] [coproc] ATI instance 0; 0.500000 pending for de_modfit_15_3s_130_wrap_1_1396965303_1897902_1
16-Apr-2014 17:15:34 [SETI@home] [coproc] ATI instance 0; 0.500000 pending for ap_09dc08ad_B0_P1_00257_20140416_15148.wu_1
16-Apr-2014 17:15:34 [Einstein@Home] [coproc] ATI instance 0; 0.500000 pending for p2030.20131122.G177.14+00.50.S.b3s0g0.00000_3792_0
16-Apr-2014 17:15:34 [Milkyway@Home] [coproc] ATI instance 0: confirming 0.500000 instance for de_modfit_15_3s_130_wrap_1_1396965303_1897902_1
16-Apr-2014 17:15:34 [SETI@home] [coproc] ATI instance 0: confirming 0.500000 instance for ap_09dc08ad_B0_P1_00257_20140416_15148.wu_1
16-Apr-2014 17:15:34 [Einstein@Home] [coproc] ATI instance 1: confirming 0.500000 instance for p2030.20131122.G177.14+00.50.S.b3s0g0.00000_3792_0
16-Apr-2014 17:15:34 [Milkyway@Home] [coproc] Insufficient ATI for de_separation_86_DR8_Rev_7_4_002_1396965303_3647857_0: need 0.500000
16-Apr-2014 17:15:34 [Milkyway@Home] [coproc] Insufficient ATI for de_separation_86_DR8_Rev_7_4_002_1396965303_3647856_0: need 0.500000
16-Apr-2014 17:15:34 [Collatz Conjecture] [coproc] Insufficient ATI for solo_collatz_2383757491694540170355_824633720832_0; need 1, available 0
16-Apr-2014 17:15:35 [Milkyway@Home] [task] result de_modfit_15_3s_130_wrap_1_1396965303_1897902_1 checkpointed
16-Apr-2014 17:15:36 [Einstein@Home] [task] result p2030.20131122.G177.14+00.50.S.b3s0g0.00000_3792_0 checkpointed
16-Apr-2014 17:15:41 [malariacontrol.net] [task] result wu_4271_415_37036_0_1397679909_0 checkpointed
16-Apr-2014 17:15:47 [malariacontrol.net] [task] result wu_4271_414_37036_0_1397679909_0 checkpointed
16-Apr-2014 17:16:34 [Milkyway@Home] [coproc] ATI instance 0; 0.500000 pending for de_modfit_15_3s_130_wrap_1_1396965303_1897902_1
16-Apr-2014 17:16:34 [SETI@home] [coproc] ATI instance 0; 0.500000 pending for ap_09dc08ad_B0_P1_00257_20140416_15148.wu_1
16-Apr-2014 17:16:34 [Einstein@Home] [coproc] ATI instance 0; 0.500000 pending for p2030.20131122.G177.14+00.50.S.b3s0g0.00000_3792_0
16-Apr-2014 17:16:34 [Milkyway@Home] [coproc] ATI instance 0: confirming 0.500000 instance for de_modfit_15_3s_130_wrap_1_1396965303_1897902_1
16-Apr-2014 17:16:34 [SETI@home] [coproc] ATI instance 0: confirming 0.500000 instance for ap_09dc08ad_B0_P1_00257_20140416_15148.wu_1
16-Apr-2014 17:16:34 [Einstein@Home] [coproc] ATI instance 1: confirming 0.500000 instance for p2030.20131122.G177.14+00.50.S.b3s0g0.00000_3792_0
16-Apr-2014 17:16:34 [Milkyway@Home] [coproc] Insufficient ATI for de_separation_86_DR8_Rev_7_4_002_1396965303_3647857_0: need 0.500000
16-Apr-2014 17:16:34 [Milkyway@Home] [coproc] Insufficient ATI for de_separation_86_DR8_Rev_7_4_002_1396965303_3647856_0: need 0.500000
16-Apr-2014 17:16:34 [Collatz Conjecture] [coproc] Insufficient ATI for solo_collatz_2383757491694540170355_824633720832_0; need 1, available 0
16-Apr-2014 17:16:39 [Einstein@Home] [task] result p2030.20131122.G177.14+00.50.S.b3s0g0.00000_3792_0 checkpointed
16-Apr-2014 17:16:40 [Milkyway@Home] [task] Process for de_modfit_15_3s_130_wrap_1_1396965303_1897902_1 exited, exit code 0, task state 1
16-Apr-2014 17:16:40 [Milkyway@Home] [task] task_state=EXITED for de_modfit_15_3s_130_wrap_1_1396965303_1897902_1 from handle_exited_app
16-Apr-2014 17:16:40 [Milkyway@Home] Computation for task de_modfit_15_3s_130_wrap_1_1396965303_1897902_1 finished
16-Apr-2014 17:16:40 [Milkyway@Home] [task] result state=FILES_UPLOADING for de_modfit_15_3s_130_wrap_1_1396965303_1897902_1 from CS::app_finished
16-Apr-2014 17:16:40 [Milkyway@Home] [task] result state=FILES_UPLOADED for de_modfit_15_3s_130_wrap_1_1396965303_1897902_1 from CS::update_results
16-Apr-2014 17:16:40 [SETI@home] [coproc] ATI instance 0; 0.500000 pending for ap_09dc08ad_B0_P1_00257_20140416_15148.wu_1
16-Apr-2014 17:16:40 [Einstein@Home] [coproc] ATI instance 0; 0.500000 pending for p2030.20131122.G177.14+00.50.S.b3s0g0.00000_3792_0
16-Apr-2014 17:16:40 [SETI@home] [coproc] ATI instance 0: confirming 0.500000 instance for ap_09dc08ad_B0_P1_00257_20140416_15148.wu_1
16-Apr-2014 17:16:40 [Einstein@Home] [coproc] ATI instance 1: confirming 0.500000 instance for p2030.20131122.G177.14+00.50.S.b3s0g0.00000_3792_0
16-Apr-2014 17:16:40 [Milkyway@Home] [coproc] Assigning 0.500000 of ATI instance 0 to de_separation_86_DR8_Rev_7_4_002_1396965303_3647857_0
16-Apr-2014 17:16:40 [Milkyway@Home] [coproc] Insufficient ATI for de_separation_86_DR8_Rev_7_4_002_1396965303_3647856_0: need 0.500000
16-Apr-2014 17:16:40 [Collatz Conjecture] [coproc] Insufficient ATI for solo_collatz_2383757491694540170355_824633720832_0; need 1, available 0
16-Apr-2014 17:16:40 [Milkyway@Home] [task] task_state=EXECUTING for de_separation_86_DR8_Rev_7_4_002_1396965303_3647857_0 from start
16-Apr-2014 17:16:40 [Milkyway@Home] Starting task de_separation_86_DR8_Rev_7_4_002_1396965303_3647857_0
16-Apr-2014 17:16:43 [Milkyway@Home] Sending scheduler request: To report completed tasks.
16-Apr-2014 17:16:43 [Milkyway@Home] Reporting 1 completed tasks
16-Apr-2014 17:16:43 [Milkyway@Home] Not requesting tasks: don't need (CPU: not highest priority project; AMD/ATI GPU: job cache full)
16-Apr-2014 17:16:45 [Milkyway@Home] Scheduler request completed
16-Apr-2014 17:16:47 [malariacontrol.net] [task] result wu_4271_415_37036_0_1397679909_0 checkpointed
16-Apr-2014 17:16:56 [malariacontrol.net] [task] result wu_4271_414_37036_0_1397679909_0 checkpointed
16-Apr-2014 17:17:41 [Einstein@Home] [task] result p2030.20131122.G177.14+00.50.S.b3s0g0.00000_3792_0 checkpointed
16-Apr-2014 17:17:41 [Milkyway@Home] [coproc] ATI instance 0; 0.500000 pending for de_separation_86_DR8_Rev_7_4_002_1396965303_3647857_0
16-Apr-2014 17:17:41 [SETI@home] [coproc] ATI instance 0; 0.500000 pending for ap_09dc08ad_B0_P1_00257_20140416_15148.wu_1
16-Apr-2014 17:17:41 [Einstein@Home] [coproc] ATI instance 0; 0.500000 pending for p2030.20131122.G177.14+00.50.S.b3s0g0.00000_3792_0
16-Apr-2014 17:17:41 [Milkyway@Home] [coproc] ATI instance 0: confirming 0.500000 instance for de_separation_86_DR8_Rev_7_4_002_1396965303_3647857_0
16-Apr-2014 17:17:41 [SETI@home] [coproc] ATI instance 0: confirming 0.500000 instance for ap_09dc08ad_B0_P1_00257_20140416_15148.wu_1
16-Apr-2014 17:17:41 [Einstein@Home] [coproc] ATI instance 1: confirming 0.500000 instance for p2030.20131122.G177.14+00.50.S.b3s0g0.00000_3792_0
As you can see, the task was running on the HD3000, even though I now added an exclusion for it, I still would like to know how.
I'm amazed that anything was running on the GPUs because at that time a BURP task was hogging both cores, ended up having to go thru process explorer and change the cpu affinity on that task to get one freed up.
OK, I
)
OK, I see
There's some useful stuff in SETI task stderr:
etc. Milkyway says
Einstein doesn't give as much OpenCL diagnostic information, but in result 432253007 I see:
- I presume the second opencl_id will be the CPU driver.
All in all, I think it demonstrates that BOINC's management of the OpenCL platform still has some deficiencies - especially that jumping around between device numbers in the Assigning/pending/confirming stages.
My suspicion is that the project science applications are all doing their own OpenCL detection, and all are ending up on the same 'Cypress' card despite what BOINC may or may not be telling them to do.
You are running the latest alpha test version of BOINC (v7.3.15), and BOINC's OpenCL detection for CPUs was only added in this test cycle - I don't think any project has actually deployed an OpenCL CPU application we could use to test it yet. In the meantime, I think it would be helpful to report at least the most obvious bugs (BOINC over-committing the Cypress card with 1.5 instances) to the boinc_alpha bug reporting mailing list.
I could be wrong, but the
)
I could be wrong, but the instance number (0 or 1) is IMHO just an index of the GPU tasks in execution state, not related to any device id.
The way I read the log output, there is no evedence that anything is running on the HD 3000, instead two tasks, one from milkyway and one from E@H, both claim and get half an AMD OpenCL card, here a Cypress, so the dedictaed GPU.
The added support for CPU OpenCL tasks deserves some attention on the project side, tho, so this raises an interesting point indeed: Is the way that new BOINC clients enumerate OpenCL devices (now including CPUs if capable) still compatible with the way science apps enumerate OpenCL devices if they use older BOINC API versions (excluding CPUs), or do all projects now have to recompile their OpenCL apps with new BOINC APIs (when the current beta version is production ready)? And would a new BOINC API version then be compatible with old clients, because not all users will change to the new client instantaneously??? (if not we have a mess).
HB
RE: I could be wrong, but
)
I think we have a mess.....
I run a host with multiple GPUs (and with the current recommended v7.2.42 client):
I'm quite certain that:
NVIDIA instance 0 (running two SETI tasks) equates to NV Device 0
NVIDIA instance 1 (running one GPUGrid task) equates to NV Device 1
and the intel_gpu has its own numbering sequence, with another instance/device 0...
In general, it's working OK at the moment, and I don't think there's an immediate need to re-compile against a new API. But Darell's case, with three tasks running on the Cypress card, shows something's not quite right. I don't know if the problem is the non-OpenCL capable HD 3000, or the additional OpenCL CPU capability - Oliver might be interested in having a look.
RE: I'm quite certain
)
Yup.
I think it's partially a problem with the back-end here. The server should know that a 256MB non-OpenCL card should not be sent any work, no matter what, since the GPU/memory/drivers/capabilities don't meet the minimum requirements.
The BRP4G application should be intelligent enough to figure what it is running on, and if that piece of hardware is even capable of doing what it wants to do.
On the other hand, the BOINC client shouldn't show work being done on ATI instance 1 here either, since it's not OpenCL capable at all.
I'm sure we should flag this for all the developers involved.
But also, if Darrell could de-exclude the GPU for this project, then wait until he sees something similar happen again, stop BOINC and run everything through the client simulator (http://boinc.berkeley.edu/trac/wiki/ClientSim) and point out the run he had, one of us could point that to the developers as well.
Apropos, BOINC 7.2.28 had OpenCL for CPUs already. So the present recommended 7.2.42 does so as well.
RE: I think it's partially
)
The trouble is that the server doesn't allocate work at the device level, only at the host level. The host has a perfectly good GPU/memory/driver/capability resource available for use, so it's OK for the client to request - and for the server to allocate - work for the host in general.
It's only once the task is present on the host that the client has the subsidiary responsibility for allocating it to a particular device - and that's where it seems to be tripping over itself.
Yes, that seems to be exactly what it's doing.
GPU1 ________________________
)
GPU1
________________________________________________________________________________
Display device : ATI Radeon 3000 Graphics
Display driver : Catalyst 13.1
BIOS : 010.094.001.038
GUID : VEN_1002&DEV_9616&SUBSYS_00171565&REV_00&BUS_1&DEV_5&FN_0
GPU2
________________________________________________________________________________
Display device : ATI Radeon 3000 Graphics
Display driver : Catalyst 13.1
BIOS : 010.094.001.038
GUID : VEN_1002&DEV_9616&SUBSYS_00171565&REV_00&BUS_1&DEV_5&FN_0
GPU3
________________________________________________________________________________
Display device : ATI Radeon HD 5800 Series
Display driver : Catalyst 13.1
BIOS : 012.020.000.055
GUID : VEN_1002&DEV_6899&SUBSYS_00001002&REV_00&BUS_2&DEV_0&FN_0
Multi-GPU role : master
The above info is from MSI's afterburnner which appears to show that the integrated HD3000 is actually the Athlon cpu. Two cores, two gpus.