Losing progress on shutdown

I'm experiencing the same 'not checkpointing' issues with FGRPSSE #51.08 app running LATeah0052F_... and ...51F... tasks which run on a Windows 10 Pro, Xeon powered machine.

Ahead of machine shutdown, I checked the progress of the tasks that were running; one was at 89.989% with eight hours elapsed and 53 minutes to completion, another at 4.6%, 1:20hrs elapsed and 8:20hrs to go and six others that were 82%, 6:45hrs elapsed and 1:10 to completion (within a minute or two of each other). On starting the machine this morning, the latter six tasks have reverted to 0% complete - they did not checkpoint during execution nor as part of their 'app exit' routine.

I suffered similar issues, documented in another thread, for the O1OD1 (Gravity Wave Search v0.03) so deselected these from download until that issue got resolved.

From my 'contributor' perspective, I truly enjoy supporting this and other BOINC-based projects which run on my own kit at home, consuming electricity at a cost to me that is part of that contribution. However, the cost of processing a work unit for Einstein is rising rapidly, compared to other projects, due to the fact that non-checkpointing work units have to be reprocessed. Because I run multiple BOINC projects, this has a knock-on consequence for those other projects, too.

With that in mind, can someone undertake a code review to establish why some Einstein workunits checkpoint but others don't? Is(are) there some common, cross-application factor(s) e.g. has the checkpointing subroutine changed, is that routine common to many Einstein apps, where in an app's execution cycle is the 'checkpointing need' being assessed, is complier code optimisation causing the checkpoint to be bypassed... It should be noted that in this thread and the other O1OD1 threads, problems were observed and reported by people with Linux, Mac and Windows platforms.

Finally, and because in the end, it's about doing reliable processing to support the science, another great concern in all this is that if there are 'code problems' with checkpointing that seems to affect some workunit instances, but not others of the same generation, how reliable are the results?

Rgds,

Ged

@REDVIBE

I think the "Wed 20 Mar 09:09:01 2019 | Einstein@Home | Host location: none" just means that you haven't specified, in your Einstein Account->Preference if your computer (the Host) is at Home, Work or School.

Rgds,

Ged

This is correct but it needs to be mentioned that, for most people, there is no need to define a location.  The use of the word "none" is perhaps unfortunate.  There is a 'default' location used if one of the other locations of home, school, work have not been set.  The message would probably be less concerning if it read, "Host location:  The default location is in use."

The idea for having the 4 separate 'locations' (aka 'venues') is to allow people with more than a single computer to use different preference sets for different machines - if they so desire.  If you don't need that functionality, just ignore it.

As Mikey suggests, checkpointing (or the lack of it) is a function of the application and not BOINC.  Each project (Einstein, Milkway, etc) decides (for each individual search being conducted) if checkpointing is to be used and how it is to be implemented.

I'm not running CPU applications at the moment but I've previously done so for many years at Einstein.  In the past, each production run has always used checkpointing for the very reason of saving state regularly for long running tasks so I'd be extremely surprised if it wasn't being used for current production searches.

The two other comments I suggested you look at were simply to show that there may have been a temporary glitch with the O1OD1 search.  The followup there suggested that the issue had been resolved.  The absence of ongoing reports in those threads seems to confirm that.

The BOINC client is correctly running on your host - the event log shows that - and it is communicating with the Einstein servers.  The "scheduler request completed" message shows there was a successful two-way transaction.  Apparently, your host was requesting work for the old BRP (Arecibo) search and I think these days, that sort of work is designed for mobile devices like phones and tablets, etc.  Are you normally able to get those sorts of tasks?

In the last line there is a link to the scheduler logs that you can use to see a lot more details about the decision making process the scheduler goes through in deciding what work to send.  Did you have a look there to see if there was additional information?  There is this pinned thread in the 'Getting Started' forum which gives some information about interpreting the scheduler logs.

With reference to Ged's post, the 'no checkpointing' report was about the FGRP5 search.  That's quite different to your report about the O1OD1 search.  As far as I'm aware, there are no other reports about the FGRP5 search and there have been no changes to the app or the nature of the data there.  I would expect to see lots of reports if in general, checkpointing wasn't working correctly for that search.

I think the "generic" term might be specific to Einstein.  It's quite a while since I've run other projects so I'm not familiar with the naming used elsewhere.  In the past I've seen the default location expressed as "--" or maybe it was three dashes :-).

Whilst there have been cosmetic changes to long standing terminology (eg venues -> locations) that are directly attributable to BOINC devs, I think there are also server-side changes that individual projects make that further add to this potential for confusion.  I imagine that there are probably several nomenclature differences that now exist due to the fact that Einstein uses older (and heavily customised) server code versions.

I didn't actually say (or imply) that the name of the default location was "default" :-).  I said there was a 'default' location, but I steered clear of the thorny issue of what its proper name was :-).

For the benefit of people with a single computer, or for those with a small number where the same settings for each is appropriate, I would suggest that users carefully consider if additional locations are really needed.  They could tend to create the type of problems that archae86 mentions.  A user could check this out by going to the account page and clicking the right hand menu item of "Preferences".  On the next page click the 2nd sub-menu item "Project".

This will show the set of project preferences in play for the location shown in the "Preference set:" drop down box near the top left of the page.  For me, that shows "Generic".  Right down in the bottom right hand corner of the full page, there is a "(show comparison view)" link.  By clicking that you will be able to see if you are using more locations than just the generic one as there will perhaps be entries for any or all of the 4 different locations.  There are controls there for clearing any locations that aren't needed.

From time to time - particularly when applications change or new versions come along - it's useful to carefully review the project preferences to make sure they meet your expectations.  There is a lot of complexity there and also in other preference sets, like computing preferences, as well.