Amazing Numbers of New Hosts

Gary Roberts
Gary Roberts
Moderator
Joined: 9 Feb 05
Posts: 4,993
Credit: 33,295,231,309
RAC: 31,360,285
Topic 193788

The project is now past the 99% complete mark with little more than 1 days primary work left.

With that in mind why would there be over 4000 new hosts added in the last 24 hours? According to the current server status page, there were 179 new users in the last 24 hours, most of which probably had 1 or 2 hosts. Even if you were generous and said that every new user had 5 hosts, that still leaves well over 3000 unaccounted for.

To my way of thinking, this probably serves to add weight to my earlier thoughts that there has been a very big conversion of Windows hosts to Linux and that this is still continuing at a fairly frenetic pace. The conversion of hosts in bulk like this automatically generates new hostIDs and so would seem to account for the statistic we are seeing. Of course, it could have just been several clusters of new hosts being added for S5R4 but this trend of large numbers of new hostIDs has been going on for a while now.

The question is why is it being done now rather than three months ago? The obvious conclusion is that the new apps for S5R4 will continue to perform significantly better under Linux. If that's the case, I'm glad I've converted the bulk of my hosts already.

I can take a host crunching under Windows and in around 15 minutes (sometimes less) it is firing up BOINC and downloading tasks and data under Linux. I have a single remastered CD which is loaded with a complete BOINC template so that, irrespective of what the hardware actually is, the only configuration I actually have to do during the install is to type in the hostname I want to use for this particular new host. Everything else is automatic or simply OK'ing a default (eg US keyboard, or use the available spare space on the HDD, etc) that is presented.

I prepared the CD on a desktop machine with a Gigabyte mobo with integrated graphics and IDE drives and recently I've used it to convert servers with serverworks dual processor mobos, SCSI HDDs and smart array controllers, without any issues. The different hardware all gets correctly detected during installation and since all my particular customisations are built into the remastered CD, there is no after installation configuration to be done other than setting the hostname. So as the installation CD is ejected and the new system boots from the HDD, all I have to do is set the hostname and click the "StartBoinc" icon that is sitting there on the desktop. The only annoying bit is that after the new host downloads a bunch of large data files and its very first task, I have to come back at a later stage and click the "StopBoinc" icon so I can edit the DCF from 1.000000 to 0.250000 so that the time completion estimates for tasks are at least in the ballpark :-).

Whilst I do hope the Windows app will be soon on a par with the Linux one, I also hope it never becomes significantly better. I don't want to be even slightly tempted into considering going back to crunching under Windows :-).

Cheers,
Gary.

Bernd Machenschalk
Bernd Machenschalk
Moderator
Administrator
Joined: 15 Oct 04
Posts: 3,904
Credit: 189,085,952
RAC: 21,672

Amazing Numbers of New Hosts

I haven't looked into the DB yet, but my guess is that a large part of these 'new hosts' comes from the 'e-science GRID', where every node of every cluster they submit an Einstein@home job on is potentially recorded as a new host at the server side (currently this 'participant' has ~250.000 hosts registered).

BM

BM

Gary Roberts
Gary Roberts
Moderator
Joined: 9 Feb 05
Posts: 4,993
Credit: 33,295,231,309
RAC: 31,360,285

RE: ... every node of every

Message 83035 in response to message 83034

Quote:
... every node of every cluster they submit an Einstein@home job on is potentially recorded as a new host at the server side (currently this 'participant' has ~250.000 hosts registered).

OK, thanks for the explanation. It seems "inefficient" (maybe it's not) to have what must be only a relatively small number of thousands of hosts turn into 250,000 registrations. You would have thought there would be some way to "reuse" the previous registration and keep that seemingly unwieldy number down to more manageable proportions.

Cheers,
Gary.

Bernd Machenschalk
Bernd Machenschalk
Moderator
Administrator
Joined: 15 Oct 04
Posts: 3,904
Credit: 189,085,952
RAC: 21,672

RE: RE: ... every node of

Message 83036 in response to message 83035

Quote:
Quote:
... every node of every cluster they submit an Einstein@home job on is potentially recorded as a new host at the server side (currently this 'participant' has ~250.000 hosts registered).

OK, thanks for the explanation. It seems "inefficient" (maybe it's not) to have what must be only a relatively small number of thousands of hosts turn into 250,000 registrations. You would have thought there would be some way to "reuse" the previous registration and keep that seemingly unwieldy number down to more manageable proportions.


I know that they now have their own management of hostids, but apparently this didn't work always as it should (remember the monster-host?).

BM

BM

Bikeman (Heinz-Bernd Eggenstein)
Bikeman (Heinz-...
Moderator
Joined: 28 Aug 06
Posts: 3,515
Credit: 444,027,504
RAC: 69,206

So one alternative would be

So one alternative would be to "somehow" mark those cluster computers so that they can be treated differently, e.g. periodically removed from the database (forced detaching) and therefore the statistics after a relatively short period of inactivity (sort of garbage collection).

Gary already mentioned somewhere else that even the scheduler might want to treat those short lived cluster nodes differently, as they are unlikely to compute long sequences of WUs from the same series if they are re-incarnated all over again many times, leading probably to excessive "fragmentation" of the WU pool.

CU
Bikeman

Gary Roberts
Gary Roberts
Moderator
Joined: 9 Feb 05
Posts: 4,993
Credit: 33,295,231,309
RAC: 31,360,285

RE: ... apparently this

Message 83038 in response to message 83036

Quote:
... apparently this didn't work always as it should (remember the monster-host?).

Yes, I certainly do :-).

I didn't really understand it at the time and I wasn't inclined to delve any further when it seemed to be some sort of "own goal" :-).

Makes a bit more sense now ...

Cheers,
Gary.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.