In the 4+ years I've been crunching here this is the first major (unplanned) problem/outage, you'll be hard pressed to find another project (and its team of programmers/scientists) as reliable and informative as Einstein@home.
Absolutely!
I have been amazed and one hardware glitch is not going to change my amazement.
All Uploads completed and reported. Back to normal, for me. :-)
According to the event log, I appear to've gotten some uploads done; but the server's apparently being hammered hard enough that after a few uploads it requests my client backoff for a few minutes. My remaining upload backlog in turn is preventing me from doing any fresh downloads.
Performance of the upload server (actually the filesystem) is still less than half than what it used to be. Fragmentation is <2%, defragmenting (the data space) wouldn't help. Apparently XFS has no way to fix/defragment/rebuild the "free inodes btree" of an existing filesystem, so it will stay that way until we completely clean the server and rebuild the whole filesystem.
FWIW ext4 performance is much worse than XFS when you have a large number of files in a directory, which occasionally happens in particular on that server.
Little bit more compact form of the XFS FAQ (at end of page):
Free inode btree
Starting Linux 3.16, XFS has added a btree that tracks free inodes. It is equivalent to the existing inode allocation btree
with the exception that the free inode btree tracks inode chunks with at least one free inode. The purpose is to improve lookups for
free inode clusters for inode allocation. It improves performance on aged filesystems i.e. months or years down the track when
you have added and removed millions of files to/from the filesystem. Using this feature doesn't impact overall filesystem
reliability level or recovery capabilities.
This feature relies on the new v5 on-disk format that has been considered stable for production workloads starting Linux Kernel 3.15.
It does not change existing on-disk structures, but adds a new one that must remain consistent with the inode allocation btree; for this
reason older kernels will only be able to mount read-only filesystems with the free inode btree feature.
The feature can be enabled with finobt=1 switch when formatting a XFS partition but requires the metadata checksum to be enabled as well
via the crc=1 switch. Therefore the following options are required to take advantage of both the free inode btree and metadata checksum
RE: In the 4+ years I've
)
Absolutely!
I have been amazed and one hardware glitch is not going to change my amazement.
All Uploads completed and
)
All Uploads completed and reported. Back to normal, for me. :-)
TimeLord04
Have TARDIS, will travel...
Come along K-9!
Join SETI Refugees
RE: RE: First time I ever
)
It looks like you only have a few tasks left so you can let your GPU crunchers take a rare nap until we get more work
I still have a couple hundred left
RE: All Uploads completed
)
According to the event log, I appear to've gotten some uploads done; but the server's apparently being hammered hard enough that after a few uploads it requests my client backoff for a few minutes. My remaining upload backlog in turn is preventing me from doing any fresh downloads.
Performance of the upload
)
Performance of the upload server (actually the filesystem) is still less than half than what it used to be. Fragmentation is <2%, defragmenting (the data space) wouldn't help. Apparently XFS has no way to fix/defragment/rebuild the "free inodes btree" of an existing filesystem, so it will stay that way until we completely clean the server and rebuild the whole filesystem.
BM
BM
FWIW ext4 performance is much
)
FWIW ext4 performance is much worse than XFS when you have a large number of files in a directory, which occasionally happens in particular on that server.
BM
BM
All applications and results
)
All applications and results should be visible again. Still not sending work, though.
BM
BM
Thanks for all of the work!
)
Thanks for all of the work!
And the patience to think through the scenarios.
.. reminds me of the time when all you had to do was to put in an update to the database.. and then.... :)
Thanks again,
Jay
@Magic : now there's an
)
@Magic : now there's an animal that knows it project preferences ! :-)
Cheers, Mike.
I have made this letter longer than usual because I lack the time to make it shorter ...
... and my other CPU is a Ryzen 5950X :-) Blaise Pascal
https://wiki.archlinux.org/in
)
https://wiki.archlinux.org/index.php/XFS
Little bit more compact form of the XFS FAQ (at end of page):
Starting Linux 3.16, XFS has added a btree that tracks free inodes. It is equivalent to the existing inode allocation btree
with the exception that the free inode btree tracks inode chunks with at least one free inode. The purpose is to improve lookups for
free inode clusters for inode allocation. It improves performance on aged filesystems i.e. months or years down the track when
you have added and removed millions of files to/from the filesystem. Using this feature doesn't impact overall filesystem
reliability level or recovery capabilities.
This feature relies on the new v5 on-disk format that has been considered stable for production workloads starting Linux Kernel 3.15.
It does not change existing on-disk structures, but adds a new one that must remain consistent with the inode allocation btree; for this
reason older kernels will only be able to mount read-only filesystems with the free inode btree feature.
The feature can be enabled with finobt=1 switch when formatting a XFS partition but requires the metadata checksum to be enabled as well
via the crc=1 switch. Therefore the following options are required to take advantage of both the free inode btree and metadata checksum
# mkfs.xfs -m crc=1,finobt=1 /dev/target_partition
According to the developers, the above -m crc=1,finobt=1 option will be the default mkfs option with the upcoming xfsprogs 3.3 release.