I think the gamma ray CPU tasks use less ram per task. I believe that 64 tasks will fit into 64 gigs for the gravity wave CPU task. I ran into cache limits above 6-8 tasks.
The gravity wave CPU tasks slowed way down if I ran more than 6-8 on a 3950x CPU. I bought a 64 gig memory kit to discover that slow down.
Apparently due to a CPU cache limit.
Universe at home runs a lot gravity wave CPU tasks in a smaller amount of memory.
World community grid provides a lot of opportunities for socially beneficial research.
Tom M
A Proud member of the O.F.A. (Old Farts Association). Be well, do good work, and keep in touch.® (Garrison Keillor)
He mentioned possibly upgrading his CPU to a faster one. I previously bout a used 3700x and have been pleased with it's crunching performance. I wondered if I could run some tasks so we could compare.
Got my server running (VMs and ZFS pool on TrueNAS),has a 5900x (12 cores 24 threads). I was running BOINC via FreeBSD virtualization until yesterday (left disappointed) and am now back to Proxmox and running BOINC in a LXC container right now.
I want to run the machine on ECO mode most of the time, but I disabled it for some BOINC test runs.
cat /proc/cpuinfo | grep "cpu MHz" says I'm running 4.4Ghz +/- 50Mhz on all 24 cores while having 22 threads running. Thermals are okish at high 70s. I'm spoiled by ECO mode where I get low 50s on full 24 core load. Cooled by 240mm AIO.
Memory is currently running at 2667MT/s to rule out wierd board/cpu behavior because official support for my memory config is 2667MT/S although sticks are rated for 3200.
Only FGRP WUs right now because the server seems like he's out of GW tasks. Got memory for days (128GB Kingston 3200MT/S UDIMMs ECC).
Batch isn't done yet, but should be approx. 6 hours. Will update once I get some ECO vs. non-ECO and different memory bandwidth setups finished. I'm not sure if FGRP WUs are sensitive to memory bandwidth, we'll see.
The other VMs don't really like the load of 22x E@H. But this is for testing only. I will probably only run half the tasks for production use.
Got my server running (VMs and ZFS pool on TrueNAS),has a 5900x (12 cores 24 threads). I was running BOINC via FreeBSD virtualization until yesterday (left disappointed) and am now back to Proxmox and running BOINC in a LXC container right now.
Under TrueNAS, how many cores did you have allocated to the VM? last time I tried, bhyve could only give a maximum of 16 threads to a single VM. or did you use multiple VMs? or has bhyve increased the limit under TrueNAS?
Got my server running (VMs and ZFS pool on TrueNAS),has a 5900x (12 cores 24 threads). I was running BOINC via FreeBSD virtualization until yesterday (left disappointed) and am now back to Proxmox and running BOINC in a LXC container right now.
Under TrueNAS, how many cores did you have allocated to the VM? last time I tried, bhyve could only give a maximum of 16 threads to a single VM. or did you use multiple VMs? or has bhyve increased the limit under TrueNAS?
No that didn't change and with everything running now under Proxmox (Debian-based with KVM/QEMU), I get better performance in a 16 core container than I did with 15 threads via bhyve. Problem with TrueNAS is that it has a nasty habit of reserving memory to jails and VMs and not giving it back to system when the jail/VM shutdown/reboot. I always ended up with 10GB of ZFS ARC max and 90G of free memory after rebooting Jails/VMs several times. This appears to be a known issue.
Now TrueNAS is running as a VM with passthrough on NICs and drives without any virtualization. Runs fine. A bit of CPU overhead is noticeable but network throughput is higher than bare metal for some reason.
I did run several BOINC configs last few days and I settled on ECO mode with 12 threads running. ECO mode sets TDP to 65W down from 105W with a drawback of around 10-15% performance for FGRP tasks. Going for more than 12 tasks was around the advertised 25% boost from SMT cores, but with all the other VMs in need for CPU, it started slowing everything down either because of memory bandwidth,proxmox scheduling or plain CPU time, can't tell.
12 concurrent tasks is the sweet spot to run everything on the server without any any trouble. They finish after around 3.5hours when everything else is quiet on the server . Extensive CPU load from e.g. TrueNAS with compressing/decompressing data increases run times as we get really deep into SMT core territory. And my Proxmox has E@H at low priority, which wasn't possible via TrueNAS.
Looks like I'm running this config for some time now as everything is really smooth.CPU sits at 55°C and 120mm fans run at 800rpm. Only thing I would call noise are the writes from the Toshiba HDDs. Main goal is a low-noise and low-power homeserver that runs E@H with spare CPU time. Mission accomplished. Not millions of RAC, but should settle at around 70k with pure CPU as I only have on-board 2D server graphics .
edit: You can tweak bhyve to accept more cores, but apparently bhyve scales really badly with more than 12-16 cores that's why they put that limit there. FreeBSD has some awesome tech behind it, but bhyve really can't compete (yet)
I'm waiting for TrueNAS Scale. Linux based jails/containers should make many things a lot easier. and the ability to use KVM for virtualization rather than bhyve. I have a TrueNAS server that I use for non-BOINC purposes.
I have a TrueNAS server that I use for non-BOINC purposes.
Well BOINC doesn't really need a storage server unless you run a project yourself :-)
Quote:
I'm waiting for TrueNAS Scale. Linux based jails/containers should make many things a lot easier. and the ability to use KVM for virtualization rather than bhyve.
That's why I switched to Proxmox after 2 weeks. And I'm happy I did. Fully-fledged hypervisor with a clean and slick GUI. And the ability to passthrough anything and many features to manage a virtualized datacenter rather than your storage make this very professional. Live migration of VM disks on running VMs to other storage in your cluster is basically witchcraft to me. But for managing pure storage in a network, TrueNAS is just unbeatable. Proxmox has ZFS-on-root, but lacks many things for proper ZFS administration.
ESXi gets you to buy VMWare products at some point and TrueNAS scale will take some time to position itself among the competition.
I think the gamma ray CPU
)
I think the gamma ray CPU tasks use less ram per task. I believe that 64 tasks will fit into 64 gigs for the gravity wave CPU task. I ran into cache limits above 6-8 tasks.
The gravity wave CPU tasks slowed way down if I ran more than 6-8 on a 3950x CPU. I bought a 64 gig memory kit to discover that slow down.
Apparently due to a CPU cache limit.
Universe at home runs a lot gravity wave CPU tasks in a smaller amount of memory.
World community grid provides a lot of opportunities for socially beneficial research.
Tom M
A Proud member of the O.F.A. (Old Farts Association). Be well, do good work, and keep in touch.® (Garrison Keillor)
Just started up a used Ryzen
)
Just started up a used Ryzen 3700x compared to a 2700x.
Even with PBO disabled but CPU boost still on it is running much better boost speeds than the 2700x was.
About 4 GHz vs. 3.65 GHz.
Tom M
A Proud member of the O.F.A. (Old Farts Association). Be well, do good work, and keep in touch.® (Garrison Keillor)
Over in Milestones a
)
Over in Milestones a participant got really far using nothing but an Amd Ryzen 1700.
https://einsteinathome.org/content/milestones-vi?page=61#comment-190008
He mentioned possibly upgrading his CPU to a faster one. I previously bout a used 3700x and have been pleased with it's crunching performance. I wondered if I could run some tasks so we could compare.
His: https://einsteinathome.org/account/287639/computers 4.44 hours to 5.8 hours or so.
Mine: Computer 12847886 | Einstein@Home (einsteinathome.org) 3.8 to 4.44 hours or so.
So as predicted a 3700x crunches faster than a 1700 :)
Tom M
A Proud member of the O.F.A. (Old Farts Association). Be well, do good work, and keep in touch.® (Garrison Keillor)
Got my server running (VMs
)
Got my server running (VMs and ZFS pool on TrueNAS),has a 5900x (12 cores 24 threads). I was running BOINC via FreeBSD virtualization until yesterday (left disappointed) and am now back to Proxmox and running BOINC in a LXC container right now.
I want to run the machine on ECO mode most of the time, but I disabled it for some BOINC test runs.
cat /proc/cpuinfo | grep "cpu MHz" says I'm running 4.4Ghz +/- 50Mhz on all 24 cores while having 22 threads running. Thermals are okish at high 70s. I'm spoiled by ECO mode where I get low 50s on full 24 core load. Cooled by 240mm AIO.
Memory is currently running at 2667MT/s to rule out wierd board/cpu behavior because official support for my memory config is 2667MT/S although sticks are rated for 3200.
Only FGRP WUs right now because the server seems like he's out of GW tasks. Got memory for days (128GB Kingston 3200MT/S UDIMMs ECC).
Batch isn't done yet, but should be approx. 6 hours. Will update once I get some ECO vs. non-ECO and different memory bandwidth setups finished. I'm not sure if FGRP WUs are sensitive to memory bandwidth, we'll see.
The other VMs don't really like the load of 22x E@H. But this is for testing only. I will probably only run half the tasks for production use.
Thank you for the
)
Thank you for the post!
It's always good to see someone pushing the envelope.
Tom M
A Proud member of the O.F.A. (Old Farts Association). Be well, do good work, and keep in touch.® (Garrison Keillor)
Exard3k wrote: Got my server
)
Under TrueNAS, how many cores did you have allocated to the VM? last time I tried, bhyve could only give a maximum of 16 threads to a single VM. or did you use multiple VMs? or has bhyve increased the limit under TrueNAS?
_________________________________________________________________________
Ian&Steve C. wrote:Exard3k
)
No that didn't change and with everything running now under Proxmox (Debian-based with KVM/QEMU), I get better performance in a 16 core container than I did with 15 threads via bhyve. Problem with TrueNAS is that it has a nasty habit of reserving memory to jails and VMs and not giving it back to system when the jail/VM shutdown/reboot. I always ended up with 10GB of ZFS ARC max and 90G of free memory after rebooting Jails/VMs several times. This appears to be a known issue.
Now TrueNAS is running as a VM with passthrough on NICs and drives without any virtualization. Runs fine. A bit of CPU overhead is noticeable but network throughput is higher than bare metal for some reason.
I did run several BOINC configs last few days and I settled on ECO mode with 12 threads running. ECO mode sets TDP to 65W down from 105W with a drawback of around 10-15% performance for FGRP tasks. Going for more than 12 tasks was around the advertised 25% boost from SMT cores, but with all the other VMs in need for CPU, it started slowing everything down either because of memory bandwidth,proxmox scheduling or plain CPU time, can't tell.
12 concurrent tasks is the sweet spot to run everything on the server without any any trouble. They finish after around 3.5hours when everything else is quiet on the server . Extensive CPU load from e.g. TrueNAS with compressing/decompressing data increases run times as we get really deep into SMT core territory. And my Proxmox has E@H at low priority, which wasn't possible via TrueNAS.
Looks like I'm running this config for some time now as everything is really smooth.CPU sits at 55°C and 120mm fans run at 800rpm. Only thing I would call noise are the writes from the Toshiba HDDs. Main goal is a low-noise and low-power homeserver that runs E@H with spare CPU time. Mission accomplished. Not millions of RAC, but should settle at around 70k with pure CPU as I only have on-board 2D server graphics .
edit: You can tweak bhyve to accept more cores, but apparently bhyve scales really badly with more than 12-16 cores that's why they put that limit there. FreeBSD has some awesome tech behind it, but bhyve really can't compete (yet)
I'm waiting for TrueNAS
)
I'm waiting for TrueNAS Scale. Linux based jails/containers should make many things a lot easier. and the ability to use KVM for virtualization rather than bhyve. I have a TrueNAS server that I use for non-BOINC purposes.
_________________________________________________________________________
Ian&Steve C. wrote:I have a
)
Well BOINC doesn't really need a storage server unless you run a project yourself :-)
That's why I switched to Proxmox after 2 weeks. And I'm happy I did. Fully-fledged hypervisor with a clean and slick GUI. And the ability to passthrough anything and many features to manage a virtualized datacenter rather than your storage make this very professional. Live migration of VM disks on running VMs to other storage in your cluster is basically witchcraft to me. But for managing pure storage in a network, TrueNAS is just unbeatable. Proxmox has ZFS-on-root, but lacks many things for proper ZFS administration.
ESXi gets you to buy VMWare products at some point and TrueNAS scale will take some time to position itself among the competition.
In theory with an 8c/16th
)
In theory with an 8c/16th Ryzen 3700x you "should" be able to run 14 threads at full speed.
However, if I run more than 8 threads of the WCG Africa Rain Fall project on the CPU, the computation time jumps from about 12 hours to 14+ hours.
So I have 8 WCG ARF threads, 1 each of PrimeGrid and Machine Learning at home, and 4 "half" threads pushing my two Rx5700's (2X per GPU).
Tom M
A Proud member of the O.F.A. (Old Farts Association). Be well, do good work, and keep in touch.® (Garrison Keillor)