All things Nvidia GPU

The Titan V on the list is still "stabilizing". It has been up and running for maybe a week or so with the new system with current settings that we think will be best. Its RAC is still going up, but slowly. We are interested to see where it ends up. 

KLiK- The GPU will only tell part of the story with some of the work right now. The O3 work right now is also very dependent on the CPU even though it also relies on the GPU. Not all projects are so dependent, but this one definitely is. 

Correct!

We retired an older system and got the 14900KS up and running (with KLEVV CRAS V RAM running stable at 8400MHz out of the box). Although the CPU has a bad reputation, it is incredible when undervolted and NOT running all-core (almost impossible to cool since the temp spikes in ~1 second). And, when not overclocked (and because it is undervolted), it is actually a relatively efficient system. Running MPS at 40%, but playing with that number a little bit but not noticing a difference. The final recalc step takes roughly 1 minute on the system across all 5 cores running tasks. Students put it all together and now fine-tuning. 

Right now, yes it is. It has not been fully optimized but here was our line of reasoning- anyone is welcome to chime in if there are flaws in it. That system runs 100% across all cores with cpu research (usually WCG when they are able to keep the system full). That is slowing the clock speed of the cpu, and we accept that. From our observations of other systems running this work, there are always periods of time where the gpu cores and memory bus are not fully utilized due to recalc steps. This is understandable and accepted on basically all of our systems even if the vram is ~80% full. We don't have that issue with the A6000 gpus because the vram is huge. If we run 12 tasks concurrent, then the gpu stays fully saturated (cores and memory bus), with only momentary dips. We understand that we CAN overload the gpu and tasks would slow down. We still have to refine the amout but we saw idle time at 10x so we set to 12x if I remember correctly. This still needs to be refined, along with MPS. It's running MPS ar 40% but I know that no single task will use more than 40% at any given time. We still have to refine the percent. 

If I am way off target with reasoning, let me know.  

Edit 11/21- We have already bumped it down to 10x. Might still go lower. I believe that when we originally tried 10x and saw idle time, all of the tasks started at roughly the same time and could not "naturally" separate from each other. Now, we are adjusting the concurrent task count on the fly while BOINC is open (read config file) and the tasks are already spaced out from each other. I think we still might go down from 10x but it is hard to know where it will end up. We will be almost completely offline next week so something we will eventually figure out. Not sure how to approach MPS percentage except trial and error.