Conversations about your/my setup

Anonymous

Water and electricity. 

Water and electricity.  Bold!!!

HAL
HAL
Joined: 9 Mar 20
Posts: 2080
Credit: 42265907
RAC: 38930

robl wrote: Water and

robl wrote:

Water and electricity.  Bold!!!

Yes I'm very bold. I live life on the edge all the time - anyone that knows me realizes that. Like this morning, I tried a brand of K-cup coffee I'd never even tasted before. So water cooling a computer isn't all that scary for me. :-)

I told all my friends I could teach them how to be funny, but they all just started laughing at me.

Tom M
Tom M
Joined: 2 Feb 06
Posts: 6555
Credit: 9643681408
RAC: 2841087

Pcie extender box

Pcie extender box

A Proud member of the O.F.A.  (Old Farts Association).  Be well, do good work, and keep in touch.® (Garrison Keillor)  I want some more patience. RIGHT NOW!

Boca Raton Community HS
Boca Raton Comm...
Joined: 4 Nov 15
Posts: 263
Credit: 10861074921
RAC: 11707276

We have some really

We have some really interesting setups here at our school for student usage that might be of interest to some. When they are not being used by students, they switch to work units (usually WCG for CPU- although not right now... and Einstein/GPUGrid for GPUs). Letting students freely use these systems for whatever (educational) task they want to do has been awesome to see. We are in the process of raising funds for a Threadripper system but we are still a ways off from completion. I want students to understand both Intel and AMD platforms. 

We have three main systems: 

1. Main system (pictures included)

  • Chassis/Containment: Dell Precision Tower 7920 inside Tripp Lite tower with active cooling (because of the noise of the system fans, the tower is insulated with foam board and we feed cold air into the front, and pull it out the back with our super high-tech Lasco fan)
  • Power: 208v outlet for increased efficiency and expandability
  • Display: 8K, 32 inches
  • CPU: Dual Intel Xeon Gold 6258R 2.7GHz,(4.0GHz Turbo, 28 core) = 56 cores, 112 threads 
  • RAM: 384G DDR4 2933MHz ECC 
  • GPU: Dual Nvidia RTX A6000, 48GB each
  • Storage: Four (4) M.2 1TB PCIe NVMe Class 50 Solid State Drive in RAID 10 along with a storage array that features TEN (10) 1.92TB SATA AG Enterprise Solid State Drives in RAID 10

(shroud removed- I know the memory was unbalanced in this picture, it was balanced after I took the image)

Overall, the temp of the system is really good! The air temp coming in is a consistent 17C (64F). System is really stable (except with Win 11). Right now, we are running GPU tasks in Windows and then CPU tasks in  virtual machines (Ubuntu). 

 

Setups 2 and 3. We have two of the following systems, but they are fairly standard in setup

  • Chassis: Dell Precision Tower 7810
  • CPU: Dual Intel Xeon Processor E5-2650 v3 (10 Core, 20 Threads, 25MB Cache, 2.3GHz Turbo) = 20 cores, 40 threads
  • GPU: Dual NVIDIA Quatro RTX 6000 in one workstation, dual NVIDIA RTX A4500 in the other
  • RAM: 64GB (8x8GB) 2133MHz DDR4 RDIMM ECC 

These has been rock-solid systems that we have pushed to 100% for years. Older CPUs but great. Not the fastest memory, PCIe, etc, but still run flawlessly. 

 

Let me know if you have any questions or suggested improvements!

 

mikey
mikey
Joined: 22 Jan 05
Posts: 12770
Credit: 1852383042
RAC: 884769

Boca Raton Community HS

Boca Raton Community HS wrote:

We have some really interesting setups here at our school for student usage that might be of interest to some. When they are not being used by students, they switch to work units (usually WCG for CPU- although not right now... and Einstein/GPUGrid for GPUs). Letting students freely use these systems for whatever (educational) task they want to do has been awesome to see. We are in the process of raising funds for a Threadripper system but we are still a ways off from completion. I want students to understand both Intel and AMD platforms. 

We have three main systems: 

  • CPU: Dual Intel Xeon Gold 6258R 2.7GHz,(4.0GHz Turbo, 28 core) = 56 cores, 112 threads 
  • RAM: 384G DDR4 2933MHz ECC 
  • GPU: Dual Nvidia RTX A6000, 48GB each
  • Storage: Four (4) M.2 1TB PCIe NVMe Class 50 Solid State Drive in RAID 10 along with a storage array that features TEN (10) 1.92TB SATA AG Enterprise Solid State Drives in RAID 10
  • (shroud removed- I know the memory was unbalanced in this picture, it was balanced after I took the image)

 

Overall, the temp of the system is really good! The air temp coming in is a consistent 17C (64F). System is really stable (except with Win 11). Right now, we are running GPU tasks in Windows and then CPU tasks in  virtual machines (Ubuntu). 

 

Setups 2 and 3. We have two of the following systems, but they are fairly standard in setup

  • Chassis: Dell Precision Tower 7810
  • CPU: Dual Intel Xeon Processor E5-2650 v3 (10 Core, 20 Threads, 25MB Cache, 2.3GHz Turbo) = 20 cores, 40 threads
  • GPU: Dual NVIDIA Quatro RTX 6000 in one workstation, dual NVIDIA RTX A4500 in the other
  • RAM: 64GB (8x8GB) 2133MHz DDR4 RDIMM ECC 

These has been rock-solid systems that we have pushed to 100% for years. Older CPUs but great. Not the fastest memory, PCIe, etc, but still run flawlessly. 

 

Let me know if you have any questions or suggested improvements!

Your students can really put out alot of workunits with those machines, they can ALSO do as you expect them to learn an whole lot about computers and being able to make them do what they want them to do as well. If I were a student I would be super impressed to come to a place that has those for me to use.

Boca Raton Community HS
Boca Raton Comm...
Joined: 4 Nov 15
Posts: 263
Credit: 10861074921
RAC: 11707276

For sure- that has been a lot

For sure- that has been a lot of fun to work with students on. We just installed the second CPUs on the older machines (figured "why not", they were super-cheap and doubled our cores in those machines) and students helped with the install and then the problem solving of getting it to boot back up (weird memory issue). Solving actual problems is so much better than artificially manufactured problems when it comes to learning. 

These students are also getting more and more involved and interested in distributed computing projects, which has been cool to see- they are even starting a club that focuses on this topic. 

I still feel its important to introduce them to the AMD platform but it will be a little of a struggle to get that off the ground. We have two Threadrippers (2970WX) that were donated and are working to figure out where to get the motherboards from. If anyone has ideas, let me know! They are out there- just really expensive. 

 

Tom M
Tom M
Joined: 2 Feb 06
Posts: 6555
Credit: 9643681408
RAC: 2841087

Boca Raton Community HS

Boca Raton Community HS wrote:

 

I still feel its important to introduce them to the AMD platform but it will be a little of a struggle to get that off the ground. We have two Threadrippers (2970WX) that were donated and are working to figure out where to get the motherboards from. If anyone has ideas, let me know! They are out there- just really expensive. 

 

There is a reason those were donated. Has a serious design flaw in memory access for higher than 50 percent of threads. I think.

Tom M

A Proud member of the O.F.A.  (Old Farts Association).  Be well, do good work, and keep in touch.® (Garrison Keillor)  I want some more patience. RIGHT NOW!

Keith Myers
Keith Myers
Joined: 11 Feb 11
Posts: 5013
Credit: 18905386205
RAC: 6681880

Tom M wrote: Boca Raton

Tom M wrote:

Boca Raton Community HS wrote:

 

I still feel its important to introduce them to the AMD platform but it will be a little of a struggle to get that off the ground. We have two Threadrippers (2970WX) that were donated and are working to figure out where to get the motherboards from. If anyone has ideas, let me know! They are out there- just really expensive. 

 

There is a reason those were donated. Has a serious design flaw in memory access for higher than 50 percent of threads. I think.

Tom M

You are wrong in your assertion.

 

Mr P Hucker
Mr P Hucker
Joined: 12 Aug 06
Posts: 838
Credit: 519661337
RAC: 15947

Ian&Steve C. wrote: if you

Ian&Steve C. wrote:

if you want to run dual PSUs, you should get an adapter like this to power up the non-MB PSU when the main system is powered on.

 

it will send the signal to power on the secondary PSU, so you can use the PCIe and other power connections from it. it will also shut down with the prime PSU when the system is powered up.

I just buy 12V only LED lighting PSUs - I get the 1kW variety.  Cheaper, very stable, even tweakable to get precise voltage, I shove all the GPUs on them.  I turn them on manually.

If this page takes an hour to load, reduce posts per page to 20 in your settings, then the tinpot 486 Einstein uses can handle it.

Mr P Hucker
Mr P Hucker
Joined: 12 Aug 06
Posts: 838
Credit: 519661337
RAC: 15947

Tom M wrote: Ian&Steve C.

Tom M wrote:

Ian&Steve C. wrote:

what made you come to that conclusion? All of my systems are still on ribbon risers. As they always have been, I don’t run GPUGRID often due to task availability and my focus on Einstein right now, but I do still run GPUGRID occasionally.

I assumed both the Pcie slot and the top power plugs had to be on the same PSU.

You disabused me of that in another thread.

I have moved 3 of the gpus top power plugs onto the secondary 1600 Psu and moved the modular ribbon cable onto the primary 1600 watt psu (so Pcie is not getting it from two PSU's).

I am not sure I have ever split a GPU between two PSU's on the top power plugs..

Tom M

I connect anything to anything, it's all 12V, they all share nicely.

If this page takes an hour to load, reduce posts per page to 20 in your settings, then the tinpot 486 Einstein uses can handle it.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.