- Today we paired a second server with the existing one. The setup is convoluted and hackish, and it still isn't clear yet how much it will help. We'll see on Sunday.
Looks like that solution is having some benefit.
For the last few Sundays, by this time (nearly 2.00am UTC) my logs have shown many of my hosts having upload difficulties. Today (fingers crossed :-) ), there hasn't been a single warning so far. Of course, having been stupid enough to make a comment, Murphy's Law will undoubtedly kick in and rectify that, just as soon as I post this :-).
Thank you for this interim solution. It's surely appreciated by many.
Are there any updates here? I still have over 200 files that need to be uploaded with no success (transient HTTP errors). They are going to start timing out soon (if they haven't already).
Just FYI, haven't seen any issues on either cruncher here in almost a week. Everything goes up nicely. Unlike the previous issue, which I think was pretty universal.
Are there any updates here? I still have over 200 files that need to be uploaded with no success (transient HTTP errors). They are going to start timing out soon (if they haven't already).
Please realise that this thread was started last year on Dec 20 by one of the staff to alert people to a problem happening on a Sunday only and with the Gamma-ray pulsar (GRP) search that uploads to a server called einstein4.
BTW: Uploads of GW GPU ("O2MDF") tasks go to a separate server and should not be affected.
The problem with GRP uploads seems to have been solved (read this whole thread if you want the details) so I presume you aren't talking about GRP uploads. I didn't see a single problem last Sunday and haven't seen one so far today. It could still happen though.
On the assumption you are actually referring to uploads for the GW GPU search, please realise that there was a recent change from the former version of this search (S2 in the task names and frequencies less than 500Hz) to a new search (S3, with frequencies > 500Hz) and this is now not showing validations and not allowing further uploads.
There were uploads for a while after the changeover - I know this for sure because every task I had for the new search was completed and returned. I had a 2 day cache so the upload problem took a while to show up.
When a new search starts, the Devs will usually allow uploads to accumulate without validation so that they will have time to test the whole validation process manually in case of any errors in the new search. My guess is that the volume of uploads has now exceeded the disk space available to store the files and this may not be resolved until someone physically does something about it.
So, no, there is no "update" about this that I've seen. You'll know about it before you read about it because all of a sudden uploads will probably start working when some new disk space is made available. This is just conjecture on my part but we'll all find out the real situation at some point.
You should stop stressing about deadlines. You haven't had tasks long enough for that and even if you had, you can be assured that the devs would handle it correctly.
*sigh* does ANY of you read further than one message back?
Problem started on Saturday evening.
Personnel of E@H will not be back before Monday (08:00 UTC+1 most likely)
E@H is a university based project, they let the servers run and if a problem like a not responding UL server pops up... you just have to WAIT until monday.
1: *sigh* does ANY of you read further than one message back?
2: Problem started on Saturday evening.
to 1: ... they don't know what that funny looking "wheel" on the mouse is for ...
to 2: ... I believe it started Saturday morning shortly before 01:00 AM ... if I can read the properties correctly ... not that it matters ... weekend is weekend.
AND they just read ONE post and don't read others, because they (the posts) might be boring (well, sometimes they are) ...
You all have a fine time waiting, which is supposed to be a sublime virtue!
Bernd Machenschalk wrote: -
)
Looks like that solution is having some benefit.
For the last few Sundays, by this time (nearly 2.00am UTC) my logs have shown many of my hosts having upload difficulties. Today (fingers crossed :-) ), there hasn't been a single warning so far. Of course, having been stupid enough to make a comment, Murphy's Law will undoubtedly kick in and rectify that, just as soon as I post this :-).
Thank you for this interim solution. It's surely appreciated by many.
Cheers,
Gary.
Are there any updates here? I
)
Are there any updates here? I still have over 200 files that need to be uploaded with no success (transient HTTP errors). They are going to start timing out soon (if they haven't already).
Thanks!
I'm unable to upload my
)
I'm unable to upload my finished 'GW-opencl-nvidia'-files for hours now.
31.01.2021 03:33:35 | Einstein@Home | Temporarily failed upload of h1_0539.90_O2C02Cl4In0__O2MDFS3_Spotlight_540.50Hz_468_1_4: transient HTTP error
.:morpheus:.
Just FYI, haven't seen any
)
Just FYI, haven't seen any issues on either cruncher here in almost a week. Everything goes up nicely. Unlike the previous issue, which I think was pretty universal.
ashealey wrote: Are there
)
Please realise that this thread was started last year on Dec 20 by one of the staff to alert people to a problem happening on a Sunday only and with the Gamma-ray pulsar (GRP) search that uploads to a server called einstein4.
On the same day in a followup message, it was pointed out that:-
The problem with GRP uploads seems to have been solved (read this whole thread if you want the details) so I presume you aren't talking about GRP uploads. I didn't see a single problem last Sunday and haven't seen one so far today. It could still happen though.
On the assumption you are actually referring to uploads for the GW GPU search, please realise that there was a recent change from the former version of this search (S2 in the task names and frequencies less than 500Hz) to a new search (S3, with frequencies > 500Hz) and this is now not showing validations and not allowing further uploads.
There were uploads for a while after the changeover - I know this for sure because every task I had for the new search was completed and returned. I had a 2 day cache so the upload problem took a while to show up.
When a new search starts, the Devs will usually allow uploads to accumulate without validation so that they will have time to test the whole validation process manually in case of any errors in the new search. My guess is that the volume of uploads has now exceeded the disk space available to store the files and this may not be resolved until someone physically does something about it.
So, no, there is no "update" about this that I've seen. You'll know about it before you read about it because all of a sudden uploads will probably start working when some new disk space is made available. This is just conjecture on my part but we'll all find out the real situation at some point.
You should stop stressing about deadlines. You haven't had tasks long enough for that and even if you had, you can be assured that the devs would handle it correctly.
Cheers,
Gary.
Jimbocous wrote:Unlike the
)
The previous upload issue was GRP, which you run. The current one is GW, which you don't.
GRP tasks uploaded correctly.
)
GRP tasks uploaded correctly. GW GPU tasks are still in the waiting list.
Tullio
tullio wrote: GRP tasks
)
Confirm
*sigh* does ANY of you read
)
*sigh* does ANY of you read further than one message back?
Problem started on Saturday evening.
Personnel of E@H will not be back before Monday (08:00 UTC+1 most likely)
E@H is a university based project, they let the servers run and if a problem like a not responding UL server pops up... you just have to WAIT until monday.
Because nobody is there to fix it on a sunday.
=Lupus= wrote: 1: *sigh*
)
to 1: ... they don't know what that funny looking "wheel" on the mouse is for ...
to 2: ... I believe it started Saturday morning shortly before 01:00 AM ... if I can read the properties correctly ... not that it matters ... weekend is weekend.
AND they just read ONE post and don't read others, because they (the posts) might be boring (well, sometimes they are) ...
You all have a fine time waiting, which is supposed to be a sublime virtue!