1024Mb of GPU memory is not enough for 3 BRP WU

astrocrab
astrocrab
Joined: 28 Jan 08
Posts: 208
Credit: 429202534
RAC: 0
Topic 195628

Hello,
i'm trying to use a new GTX 560 ti GPU with 3 simultaneous WUs, but often WU ends up with "Compute error" status and when i try to understand why i see thing like this "... Used by this application: 430 MB
[7099][ERROR] Couldn't allocate 3271556 bytes of CUDA HS summing memory (error: 2)!
"

so, 430 mb is much more when claimed 230 mb per WU.
what have i to do with it? reduce parallel number of WUs to 2, not 3?
thank you.

Jeroen
Jeroen
Joined: 25 Nov 05
Posts: 379
Credit: 740030628
RAC: 556

1024Mb of GPU memory is not enough for 3 BRP WU

Quote:

Hello,
i'm trying to use a new GTX 560 ti GPU with 3 simultaneous WUs, but often WU ends up with "Compute error" status and when i try to understand why i see thing like this "... Used by this application: 430 MB
[7099][ERROR] Couldn't allocate 3271556 bytes of CUDA HS summing memory (error: 2)!
"

so, 430 mb is much more when claimed 230 mb per WU.
what have i to do with it? reduce parallel number of WUs to 2, not 3?
thank you.

My GPU memory usage fluctuates a bit but is currently sitting at ~320MB per WU or ~960MB for three WUs. This would be cutting it close where 1GB of memory is available. With the 560, 2 WUs per GPU is probably what you will want to set to make sure there is enough memory available.

astrocrab
astrocrab
Joined: 28 Jan 08
Posts: 208
Credit: 429202534
RAC: 0

RE: With the 560, 2 WUs

Quote:
With the 560, 2 WUs per GPU is probably what you will want to set to make sure there is enough memory available.


and what will happen with overall WUs output of GPU?

Richard Haselgrove
Richard Haselgrove
Joined: 10 Dec 05
Posts: 2139
Credit: 2752669530
RAC: 1467484

RE: RE: With the 560, 2

Quote:
Quote:
With the 560, 2 WUs per GPU is probably what you will want to set to make sure there is enough memory available.

and what will happen with overall WUs output of GPU?


Try it and see. Anything other than the single WU per card designed by BOINC and the Einstein project is your responsibility, and at your own risk.

astrocrab
astrocrab
Joined: 28 Jan 08
Posts: 208
Credit: 429202534
RAC: 0

ok. what i see now: 1 WU per

ok. what i see now:
1 WU per GPU crunched for ~3300 seconds.
2 WU per GPU crunched for ~3300 seconds each i.e. 100% performance gain.
3 WU per GPU crunched for ~5000 seconds each so there is no performance gain compared to 2WU per GPU and there is often not enough memory for all 3 units.

ok, i will stick to 2 WU!

Gary Roberts
Gary Roberts
Moderator
Joined: 9 Feb 05
Posts: 5842
Credit: 109386416808
RAC: 35922810

RE: .... 2 WU per GPU

Quote:
....
2 WU per GPU crunched for ~3300 seconds each i.e. 100% performance gain.


Are you reporting CPU time or Run time? Are you running Windows or Linux? Your computers are hidden so I can't check for myself, sorry.

EDIT: I've found the answers - both times are listed as being pretty much the same, and, you're running Linux, so a full cpu core per GPU (unless you've modified that through the parameters in your app_info.xml). I used the TaskID link you posted in another thread to find your hostID :-).

It would seem to be a little weird that both CPU time and Run time are being reported as virtually identical numbers. I guess it's a byproduct of the Linux driver problem where a full CPU core has to be allocated, even though it isn't really needed.

Maybe you might be prepared to experiment and see what happens if you tried using a setting of say 0.5 cpus in app_info.xml? The worst you could do would be to trash a few tasks, I guess :-). You should be able to restrict the potential for damage by suspending the bulk of your cache so that a rapid failure can't cascade from task to task. Of course, don't do any of this if you don't want the risk of trashed tasks :-).

Cheers,
Gary.

astrocrab
astrocrab
Joined: 28 Jan 08
Posts: 208
Credit: 429202534
RAC: 0

RE: Are you reporting CPU

Quote:
Are you reporting CPU time or Run time? Are you running Windows or Linux? Your computers are hidden so I can't check for myself, sorry.


run time, linux 64, i opend it, so, please, check now. but there is no 100% performance gain, i have to check it more thoroughly and find only 50% gain (

Gary Roberts
Gary Roberts
Moderator
Joined: 9 Feb 05
Posts: 5842
Credit: 109386416808
RAC: 35922810

Your reply arrived whilst I

Your reply arrived whilst I was editing (with some distractions) my previous post.

When you run two GPU tasks simultaneously, do you see two CPU cores being tied up, or can both GPU tasks be fed using a single CPU core?

I wasn't really clear in my previous message. I was assumimg that you would have both and set to 1 and that perhaps two CPU cores would be tied up when you reduced to 0.5. Perhaps that's an incorrect assumption and you still only use 1 CPU core even when running 2 GPU tasks simultaneously. I don't know - I don't (yet) have any GPUs to observe :-). So I was just speculating on what would happen if both and were set to 0.5. I was just trying to get a better picture in case I decide to lash out and buy some GPUs :-). Even though I prefer to use Linux for crunching, I might use Windows if necessary to see a quad core still using 4 cores for CPU crunching HF tasks in addition to whatever was running on the GPU.

Cheers,
Gary.

astrocrab
astrocrab
Joined: 28 Jan 08
Posts: 208
Credit: 429202534
RAC: 0

editing and doesn't play

editing and doesn't play matter (
i run 2 WU per 1 GPU and it consumes 2 CPU cores by 100%

Oliver Behnke
Oliver Behnke
Moderator
Administrator
Joined: 4 Sep 07
Posts: 942
Credit: 25166626
RAC: 0

RE: Hello, i'm trying to

Quote:

Hello,
i'm trying to use a new GTX 560 ti GPU with 3 simultaneous WUs, but often WU ends up with "Compute error" status and when i try to understand why i see thing like this "... Used by this application: 430 MB
[7099][ERROR] Couldn't allocate 3271556 bytes of CUDA HS summing memory (error: 2)!
"

so, 430 mb is much more when claimed 230 mb per WU.

I should have written the info message somewhat more precisely I think (will do so!). The numbers given in the log represent the overall GPU memory status - regardless of what's running on it. Thus the log messages assume that you're only running that single GPU task that reports the numbers.

FYI, there's no way to determine actual GPU memory usage per process. The value of "Used by this application" is the difference of free global memory at startup and after our memory allocations. If some other task is launched simultaneously the numbers will be off...

Cheers,
Oliver

 

Einstein@Home Project

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.