Fermi LAT Gamma-ray pulsar search #4 "FGRP4"

Stef
Stef
Joined: 8 Mar 05
Posts: 206
Credit: 110,568,193
RAC: 0

RE: The memory requirement

Quote:
The memory requirement is basically due to the FFT size. We could reduce this, but then would either loose resolution (sensitivity) or largely extend the computing time by ^2 (half size = quad time).

Any chance to make this user-configurable?
For me a thread could take 2-4GB, so it could run a lot faster at the same resolution, if I understood you right.

Jim1348
Jim1348
Joined: 19 Jan 06
Posts: 463
Credit: 257,957,147
RAC: 0

RE: For me a thread could

Quote:
For me a thread could take 2-4GB, so it could run a lot faster at the same resolution, if I understood you right.


I am all in favor of using more memory too when possible. It is a lot cheaper than a faster CPU when it works.

Stef
Stef
Joined: 8 Mar 05
Posts: 206
Credit: 110,568,193
RAC: 0

On a second thought I guess

On a second thought I guess this won't go the other way round, once the fft size is defined.

Bernd Machenschalk
Bernd Machenschalk
Moderator
Administrator
Joined: 15 Oct 04
Posts: 4,330
Credit: 251,181,133
RAC: 41,846

The point is that you can't

The point is that you can't compare (validation) or combine (post-process) tasks with different FFT baseline.

We once thought about offering two different "sizes" of WUs and found it to be too much of a hazzle on our side that wouldn't be worth the possible benefit. We might do this in the future with a different setup, but probably not in FGRP4.

BM

BM

Stranger7777
Stranger7777
Joined: 17 Mar 05
Posts: 436
Credit: 430,881,538
RAC: 67,977

I think you can divide the

I think you can divide the whole run on parts with different sensitivity levels, thus WUs will be smaller but will consume almost the same time on the same hardware as big ones but with lower level of sensitivity. This way you can resolve the validation problem (no need to validate different parts of a run with each other) and feed those of over machines that are hungry for big units.

But... on the other hand you can not predict which part needs to look at more closely therefore it is necessary to setup next run separately from the previous one, as we have exactly in S6.

So Bernd I suspect that one day your ingenious head will bring to life the idea of making WUs with different memory requirements, but this is so difficult task that it will be surely not in the nearest future...

And... I think this is not necessary at all. It is easier to setup a different run in this case.

Bernd Machenschalk
Bernd Machenschalk
Moderator
Administrator
Joined: 15 Oct 04
Posts: 4,330
Credit: 251,181,133
RAC: 41,846

We will update the FGRP4

We will update the FGRP4 application soon. The new application version will feature a minor fix to the computation (there was a typo in a constant) and checkpointing in the last ("follow-up") stage of the computation.

The modification is such that tasks ran with the current application version won't validate compared with that of the new version. So we'll finish the current
data set ("LATeah1052E") with the old version, then issue the new application version and only then issue the first WUs of the next data set ("LATeah1053E").

In preparation for the transition we lowered the deadline for the remaining WUs of LATeah1052E.

BM

BM

Gary Roberts
Gary Roberts
Moderator
Joined: 9 Feb 05
Posts: 5,874
Credit: 117,990,464,803
RAC: 21,145,515

RE: ... and only then issue

Quote:
... and only then issue the first WUs of the next data set ("LATeah1053E").


If I understand this correctly, this means there might have to be a gap, possibly a week or three, after the old tasks run out before new tasks can be issued. I imagine you will need every last resend task relying on the old app to be back before you can issue the new app and the next series of tasks. Even with a shortened deadline, a couple of rounds of resends could cause quite a delay. There will also be the resends from data prior to 1052E that will be trickling in for some time to come.

I had to wait a bit but I've now seen from a task received that the new deadline is 5 days. Participants doing FGRP4 who have set a 'too large' cache size might create problems for themselves with BOINC going into high priority and (temporarily) locking out other projects while the short deadline tasks get done. I imagine you will also see older 14 day E@H CPU tasks being held up whilst the newer short deadline tasks get done in preference. Perhaps some advice, either to reduce cache size, or opt out of FGRP4 temporarily until the changeover is complete, might be justified. Something that gets to people without relying on them reading Technical News (eg the Notices system).

I guess the biggest potential issue is what will happen with resends. Are you planning to process them 'in house' at some point?

Cheers,
Gary.

Bernd Machenschalk
Bernd Machenschalk
Moderator
Administrator
Joined: 15 Oct 04
Posts: 4,330
Credit: 251,181,133
RAC: 41,846

I wouldn't expect a "gap" of

I wouldn't expect a "gap" of more than a few days. When there are no more new WUs to be generated from "LATeah1052E" we will issue new tasks (increase "replication") for all the FGRP4 WUs that aren't finished by then, with even shorter deadlines (2-3d). People running tasks from these WUs with longer deadlines will still get credit. This will "waste" a bit of computing power because of redundancy, but should finish all remaining "old" WUs within a few days.

BM

BM

Filipe
Filipe
Joined: 10 Mar 05
Posts: 186
Credit: 409,234,230
RAC: 182,711

Are we still in the LAT1052

Are we still in the LAT1052 wu? Haven't seen any gap..

Bernd Machenschalk
Bernd Machenschalk
Moderator
Administrator
Joined: 15 Oct 04
Posts: 4,330
Credit: 251,181,133
RAC: 41,846

I'm not sure that you will

I'm not sure that you will actually notice a gap. One part of it (no "tasks to send" for several hours) was between 4PM yesterday and 7AM this morning (UTC).

The last workunits of LATeah1052 were generated yesterday. This morning we generated another two tasks with short deadline (2d) for each of the remaining "Workunits without canonical result" (13k at that time) to quickly finish these.

BM

BM

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.