Preparing S5R4 - No new Apps for S5R3

Bikeman (Heinz-Bernd Eggenstein)
Bikeman (Heinz-...
Moderator
Joined: 28 Aug 06
Posts: 3,522
Credit: 692,139,435
RAC: 20,855

RE: RE: RE: ... Is it

Message 82099 in response to message 82098

Quote:
Quote:
Quote:

... Is it possible to use grid files our clients already have ...

Well, the skygrid files make up only (roughly) 10% ....

Yes, the savings are small but I feel compelled to cache the skygrids and then roll them out to all hosts so that a new host (and also an existing one when changing frequency significantly) can at least save something.

Also, when you have as many hosts as I have, there is waste when several hosts in my farm are working on similar frequencies and probably have independently downloaded some of the same large data files. Not to mention the waste when the scheduler seems hell bent on shifting incessantly to adjacent data bands and downloading further large files at the same time - and marking files for deletion at some high sequence number instead of allowing the host to keep getting tasks for the same data at lower sequence numbers.

It would be very nice to have some sort of BOINC preference that allowed any request for data to check a local cache first just in case the data was locally available. Of course there would need to be a companion setting to tell the client to also cache what it was downloading so that the local repository could be filled progressively and automatically. This may need to be done by someone in the E@H camp since I'm not sure other projects package their tasks the same way E@H does and so it might not be appropriate for BOINC as a whole.

It would also be nice to have the whole data set available on DVD so that people like me could avoid monthly excess data charges that my ISP is now hitting me with. Probably time for me to change ISP to someone who has more sensible data limits. My ISP must be relying on inertia to keep its customers as it's really not competitive anymore with its plans and charges.

Maybe that would go nicely together with the idea of a "BOINC super host". In the meantime, those having large "farms" might consider using a caching web-proxy like SQUID as a central cache on a dedicated host with large diskspace.

As to the question of the datasets on local storage, I guess we are talking Blue-Ray instead of DVD here.

All the datafiles for S5R3 must have a size of approximately
3.2 MB * 2 * 1/0.05Hz * (1200Hz -50Hz) =~ 145 GB

( avg SFT file size x number of observatories x 1/frequency resolution x Bandwidth of search).

So the whole data would fit on approx 32 DVDs. While this is still manageable (hey, my collection of all M*A*S*H episodes came on a total of 30 DVDs), it would still be difficult to distribute.

Bernd Machenschalk
Bernd Machenschalk
Moderator
Administrator
Joined: 15 Oct 04
Posts: 4,276
Credit: 245,618,225
RAC: 10,987

RE: RE: And one more

Message 82101 in response to message 82097

Quote:
Quote:

And one more thing I have to ask here. Is it possible to use grid files our clients already have in the new run. This will significantly decrease the traffic and my money as well (in several places I have a satellite provider that is not the unlimited yet).

Well, the skygrid files make up only (roughly) 10% of the diskspace necessary to process a single Workunit, and they have a longer "lifetime" (are reused more frequently) than the other big files that are transferred. So the overall share of the skygrid files on the data traffic must be even less than 10%.


This will even get kinda worse in S5R4, as the size of the skygrid files will stay roughly the same while the instrument data volume will definitely increase - not by much I hope, but it will.

BM

BM

Ed1934158
Ed1934158
Joined: 10 Nov 04
Posts: 62
Credit: 14,481,483
RAC: 0

There is one thing that I

There is one thing that I don't understand. On S5R3 units I used optimized client that uses SSE and SSE2 instructions and during that time I crunched quite a lot of units, I haven't seen any errors, the same goes for most of the users (I haven't seen lot of topics discussing errors).
For new units there is no optimized client that uses all extra functions.
Now the question is, how long do you need to test these clients before all the users can get clients that use all the processor potential? In my case the difference is almost 100%, 16000-20000s per unit, while with ordinary client it takes almost 2 times more. I mean that is lot of power wasted, not using a client that works great.

Bikeman (Heinz-Bernd Eggenstein)
Bikeman (Heinz-...
Moderator
Joined: 28 Aug 06
Posts: 3,522
Credit: 692,139,435
RAC: 20,855

RE: There is one thing that

Message 82103 in response to message 82102

Quote:
There is one thing that I don't understand. On S5R3 units I used optimized client that uses SSE and SSE2 instructions and during that time I crunched quite a lot of units, I haven't seen any errors, the same goes for most of the users (I haven't seen lot of topics discussing errors).
For new units there is no optimized client that uses all extra functions.
Now the question is, how long do you need to test these clients before all the users can get clients that use all the processor potential? In my case the difference is almost 100%, 16000-20000s per unit, while with ordinary client it takes almost 2 times more. I mean that is lot of power wasted, not using a client that works great.

There's a misunderstanding. The new stock apps are actually almost identical to the power apps of S5R3. Note that the new stock apps come in bundles of two or three executables:

einstein*_0 is for legacy PCs (no SSE support)
einstein*_1 is used for SSE enabled PCs
einstein*_2 (Linux only at the moment) is for SSE2 enabled PCs

Intel-Macs are all SSE2 enabled.

The fact that workunits take longer now in S5R4 is due to the fact that they are designed differently than S5R3 workunits ("more science per workunit", so to speak).

CU
Bikeman

Ed1934158
Ed1934158
Joined: 10 Nov 04
Posts: 62
Credit: 14,481,483
RAC: 0

Oh, thank you Bikeman, that

Oh, thank you Bikeman, that is nice to know.

Byron S Goodgame
Byron S Goodgame
Joined: 16 Jan 06
Posts: 187
Credit: 56,581
RAC: 0

Would be nice if everyone in

Would be nice if everyone in Windows was able to run in SSE2 or beyond if they're able. Either thru Einstein or a 3rd party app, but I assuming that's just a matter of time. Is there a thread where this might be available or where they are talking about the production of one now?

Bikeman (Heinz-Bernd Eggenstein)
Bikeman (Heinz-...
Moderator
Joined: 28 Aug 06
Posts: 3,522
Credit: 692,139,435
RAC: 20,855

I'm not aware of any effort

I'm not aware of any effort outside the official developer team to work on a Windows app version. SSE2 support is only a facet of this, the Windows app is based on a slightly older version of the optimization code than the Linux and Intel OSX app, because the newest versions was not ported to the assembler syntax used by MS Visual C so far. It's planned to use a gcc variant for Windows apps as well in the future and abandon MS Visual Studio as a development platform.

CU
Bikeman

Mike Hewson
Mike Hewson
Moderator
Joined: 1 Dec 05
Posts: 6,552
Credit: 289,517,086
RAC: 102,542

RE: ... and abandon MS

Message 82108 in response to (parent removed)

Quote:
... and abandon MS Visual Studio as a development platform.


Presumably that would obviate the need to hand twiddle in assembler to avoid that 'store forward stall' delay issue w.r.t. instruction pipeline flushing? That is : gcc is a 'better' compiler for our purposes?

Cheers, Mike.

I have made this letter longer than usual because I lack the time to make it shorter ...

... and my other CPU is a Ryzen 5950X :-) Blaise Pascal

Byron S Goodgame
Byron S Goodgame
Joined: 16 Jan 06
Posts: 187
Credit: 56,581
RAC: 0

Thx for the info Bikeman and

Thx for the info Bikeman and Mike Hewson. What would be your best guess for Windows users to be able to use a more optimized version for S5R4 from past experience?

Bikeman (Heinz-Bernd Eggenstein)
Bikeman (Heinz-...
Moderator
Joined: 28 Aug 06
Posts: 3,522
Credit: 692,139,435
RAC: 20,855

@Mike: The "store forward

@Mike: The "store forward stall" issue was resolved, if I remember correctly, not so much by hand coding parts in assembler but by using a rather old version of VS.

I don't want to even start a discussion about relative performance of gcc, intel compilers, Microsoft compilers... The hotloos are all hand coded by now anyway for major platforms. Using gcc for *all* major platforms would be nice (in my opinion) mainly because it simplifies the development environment.

@Byron: I really can't make a prediction, as I cannot speak for the official dev team. I personally would *not* expect this to happen in the next 3 or 4 weeks or so.

CU
Bikeman

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.