I constantly get "HTTP 504: Gateway Timeout" messages for the upload tries.
Maybe this message helps? I have a WU to upload which is scheduled for January 11. :/
For the techs: it looks like the root cause for our current performance problem is described here:
"After a large delete on a reasonably full filesystem the filesystem will have free inodes sparsely distributed through the inode btree. When a new file is created under these circumstances the inode btree will be walked (i.e.; searched) to find free inodes.
This is in contrast to a filesystem with a normal usage pattern, where an inode is allocated directly from the “free inode cursorâ€, and so searching to find a free inode to allocate is not necessary."
(Re)building the filesystem, as suggested in the article, of cause is not an option in our case.
For the techs: it looks like the root cause for our current performance problem is described here:
"After a large delete on a reasonably full filesystem the filesystem will have free inodes sparsely distributed through the inode btree. When a new file is created under these circumstances the inode btree will be walked (i.e.; searched) to find free inodes.
This is in contrast to a filesystem with a normal usage pattern, where an inode is allocated directly from the “free inode cursorâ€, and so searching to find a free inode to allocate is not necessary."
(Re)building the filesystem, as suggested in the article, of cause is not an option in our case.
BM
XFS is one of the few *nix filesystems that is subject to fragmentation. So, I have to wonder if running a defrag might help.
- Defragmentation might not help in this case as we suspect the inode structure itself, not the data. In such a case xfs_fsr won’t help. We started the fragmentation check but had to stop it in favour of a repair we’re currently running (we saw an FS error on Wednesday as well). We’ll check the fragmentation level as soon as the repair is done.
- We’re currently talking about 12 TB. We could in principle delete significant part of that but we’re not able to do that with the current FS performance…
I constantly get "HTTP 504:
)
I constantly get "HTTP 504: Gateway Timeout" messages for the upload tries.
Maybe this message helps? I have a WU to upload which is scheduled for January 11. :/
[edit]
Message log from Proxomitron:
Aloha, Uli
I was getting a lot of
)
I was getting a lot of HTTP/1.1 502 Bad Gateway earlier this morning, but it changed to HTTP/1.1 504 Gateway Time-out around 11:30 UTC.
I still have no problem
)
I still have no problem d/ling but rarely can get a finished task to u/l once again.
So that means I have plenty to do and it will just mean there will be thousands to send in very soon here.
For the techs: it looks like
)
For the techs: it looks like the root cause for our current performance problem is described here:
"After a large delete on a reasonably full filesystem the filesystem will have free inodes sparsely distributed through the inode btree. When a new file is created under these circumstances the inode btree will be walked (i.e.; searched) to find free inodes.
This is in contrast to a filesystem with a normal usage pattern, where an inode is allocated directly from the “free inode cursorâ€, and so searching to find a free inode to allocate is not necessary."
(Re)building the filesystem, as suggested in the article, of cause is not an option in our case.
BM
BM
I assume that the XFS
)
I assume that the XFS filesystem must be something new to E&H since this issue would have surfaced before now.
RE: I assume that the XFS
)
XFS has been in use @AEI before I started working there in 2002, long before Einstein@Home.
I can't remember this particular issue came up. I doubt that we ever had such a large single FS that required such performance ran full.
BM
BM
We'll shut down einstein4 for
)
We'll shut down einstein4 for a filsystem check. This may take a while.
BM
BM
RE: For the techs: it looks
)
XFS is one of the few *nix filesystems that is subject to fragmentation. So, I have to wonder if running a defrag might help.
XFS Defrag Article
I'm afraid the long-term
)
I'm afraid the long-term solution may be to backup, reformat as ext4, and restore the data. Any idea how many terabytes we're talking about here?
Just two quick replies: -
)
Just two quick replies:
- Defragmentation might not help in this case as we suspect the inode structure itself, not the data. In such a case xfs_fsr won’t help. We started the fragmentation check but had to stop it in favour of a repair we’re currently running (we saw an FS error on Wednesday as well). We’ll check the fragmentation level as soon as the repair is done.
- We’re currently talking about 12 TB. We could in principle delete significant part of that but we’re not able to do that with the current FS performance…
Stay tuned,
Oliver
Einstein@Home Project