Hungry hosts could be a problem to Einstein?

Bernd Machenschalk
Bernd Machenschalk
Moderator
Administrator
Joined: 15 Oct 04
Posts: 4,034
Credit: 220,191,565
RAC: 44,403

RE: It would seem to me a

Quote:
It would seem to me a line count would be less taxing on resources than listing them out. But.. I am not a programmer.

The current "pendings" page is a nightmare on its own, both for the user and the system. I would rather spend time on fixing / improving this than extending the "tasks" pages. Don't know when I find the time for either, though.

BM

BM

juan BFP
juan BFP
Joined: 18 Nov 11
Posts: 839
Credit: 421,443,712
RAC: 0

Must of us a are returning to

Must of us a are returning to our homeland soon, but before we do that, i wish to thanks all for the good time we pass in this 3 weeks. Was a pleasure to meet so many new friends.

I´m realy hope you liked our contribution to the project in this time (about 350PFlops) and maybe in the future we could do more. If the door´s remains opens, why not? We know we could do better, but take some time to learn who to tunned our hungry hosts to crunch E@H a little better, and still a hard and long task to realy learn who to do that with eficience, to much to learn in so little time. We left a large number of pendigs, so that will be add a litlle more Tflops to the pot. Hope at least 1 discovery could be made with all that data.

Keep focus on the project, you all have a hard and long task to do, but i´m sure there is a lot of new discoveries waiting.

To the project admins, an all who give us a suport, thanks for all and hope we could see you all again.

We don´t close our participation in the project, but for now our main SETI team need us, but we will return. The last weeks was to good to forget.

lHj2ixL.jpg

 

mikey
mikey
Joined: 22 Jan 05
Posts: 7,908
Credit: 631,613,882
RAC: 226,455

RE: RE: It would seem to

Quote:
Quote:
It would seem to me a line count would be less taxing on resources than listing them out. But.. I am not a programmer.

The current "pendings" page is a nightmare on its own, both for the user and the system. I would rather spend time on fixing / improving this than extending the "tasks" pages. Don't know when I find the time for either, though.

BM

HOW does the current page work? Does it generate the page every time a user asks for it? If so that WOULD be a nightmare for an already busy server! ASSUMING that IS how it works maybe a second computer that you can periodically send that info to say once an hour, and then have that pc generate the report from there which would lessen the main Servers workload a bit.

soft spirit
soft spirit
Joined: 27 Oct 10
Posts: 113
Credit: 5,880,079
RAC: 0

I just thought a line count

I just thought a line count (which with some exception IS the information most of us look for on those pages.. MOST of the time..) would be much less taxing to generate than the lists themselves. And I would concur an occasional output of that information would in most cases be adequate. I also am sympathetic to programmers nightmares of code. Once upon a time I found myself thinking in Korn Shell and knew at that point I needed to change career directions.

Richard Haselgrove
Richard Haselgrove
Joined: 10 Dec 05
Posts: 2,042
Credit: 806,870,587
RAC: 1,310,340

RE: I just thought a line

Quote:
I just thought a line count (which with some exception IS the information most of us look for on those pages.. MOST of the time..) would be much less taxing to generate than the lists themselves. And I would concur an occasional output of that information would in most cases be adequate. I also am sympathetic to programmers nightmares of code. Once upon a time I found myself thinking in Korn Shell and knew at that point I needed to change career directions.


The beauty of databases is that the code is very easy - in this case, just one line in pending.php:

$results = BoincResult::enum("userid=$user->id AND (validate_state=0 OR validate_state=4) AND claimed_credit > 0"); But that one line can tie up a server for several seconds while it searches through three million tasks in the database and works out which belong to you (and are pending).

archae86
archae86
Joined: 6 Dec 05
Posts: 3,033
Credit: 5,205,836,636
RAC: 5,053,611

RE: But that one line can

Quote:
But that one line can tie up a server for several seconds while it searches through three million tasks in the database and works out which belong to you (and are pending).

Often database resource consumption gets down both to what keys are available in the chosen database structure, and just how the query is written.

If using a key selects only a small number of lines, even slightly wasteful subsequent processing may not be very costly. But anything which really requires a search on every line is anathema. I'm no database expert, but was doing "big data" on multi-gigabyte manufacturing databases before that term came into vogue, and even taught a couple of sessions of SQL classes. Aside from technical detail, the two major points I tried to drum into students' heads were the adroit use of keys, and the huge benefit of server-side summarization (as in our case the pipe from the database server to our desktops had astonishingly low capacity).

We, of course, were stuck with the key provision of the database implementation we were using. I'll hazard a guess that making pending counts or queries work well at Einstein might require revision either in key provision or in the formulation of the queries.

None of this commentary is aimed at either project staff or Richard, all of whom I assume to know much more than I on these matters--but rather as an aside to my fellow users.

Bernd Machenschalk
Bernd Machenschalk
Moderator
Administrator
Joined: 15 Oct 04
Posts: 4,034
Credit: 220,191,565
RAC: 44,403

RE: The beauty of databases

Quote:

The beauty of databases is that the code is very easy - in this case, just one line in pending.php:

$results = BoincResult::enum("userid=$user->id AND (validate_state=0 OR validate_state=4) AND claimed_credit > 0"); But that one line can tie up a server for several seconds while it searches through three million tasks in the database and works out which belong to you (and are pending).

One problem is that BOINC's DB interface (here BoincResult) partially ruins one possible benefit of having a DB by transferring every field of the result record (think of blobs like xml_doc, stderr_out etc.) to PHP memory even if only a few simple fields are ever shown on the web page.

Doing this without any limitation to the number of records to fetch kills the DB connection just by I/O from the DB if the number of records is large enough. But fixing the root cause of that would require to change the whole BOINC PHP DB interface, not just this page.

BM

BM

Bikeman (Heinz-Bernd Eggenstein)
Bikeman (Heinz-...
Moderator
Joined: 28 Aug 06
Posts: 3,516
Credit: 473,895,379
RAC: 89,610

RE: Must of us a are

Quote:

Must of us a are returning to our homeland soon, but before we do that, i wish to thanks all for the good time we pass in this 3 weeks. Was a pleasure to meet so many new friends.

I´m realy hope you liked our contribution to the project in this time (about 350PFlops) and maybe in the future we could do more. If the door´s remains opens, why not? We know we could do better, but take some time to learn who to tunned our hungry hosts to crunch E@H a little better, and still a hard and long task to realy learn who to do that with eficience, to much to learn in so little time. We left a large number of pendigs, so that will be add a litlle more Tflops to the pot. Hope at least 1 discovery could be made with all that data.

Keep focus on the project, you all have a hard and long task to do, but i´m sure there is a lot of new discoveries waiting.

To the project admins, an all who give us a suport, thanks for all and hope we could see you all again.

We don´t close our participation in the project, but for now our main SETI team need us, but we will return. The last weeks was to good to forget.

Thanks very much for your substantial contribution, and of course you are welcome to return any time. Needless to say, participating in the forum here is independent from contributing to the project (unless your RAC decays to 0 , for spam protection), so feel free to check out the forum and stay connected.

Cheers and feline greetings.
HB

Richard Haselgrove
Richard Haselgrove
Joined: 10 Dec 05
Posts: 2,042
Credit: 806,870,587
RAC: 1,310,340

RE: RE: The beauty of

Quote:
Quote:

The beauty of databases is that the code is very easy - in this case, just one line in pending.php:

$results = BoincResult::enum("userid=$user->id AND (validate_state=0 OR validate_state=4) AND claimed_credit > 0"); But that one line can tie up a server for several seconds while it searches through three million tasks in the database and works out which belong to you (and are pending).

One problem is that BOINC's DB interface (here BoincResult) partially ruins one possible benefit of having a DB by transferring every field of the result record (think of blobs like xml_doc, stderr_out etc.) to PHP memory even if only a few simple fields are ever shown on the web page.

Doing this without any limitation to the number of records to fetch kills the DB connection just by I/O from the DB if the number of records is large enough. But fixing the root cause of that would require to change the whole BOINC PHP DB interface, not just this page.

BM


Yes, I was surprised that the line I was reading didn't look like

SELECT result.id, result.workunitid, result.hostid, result.claimed_credit FROM result WHERE ....

juan BFP
juan BFP
Joined: 18 Nov 11
Posts: 839
Credit: 421,443,712
RAC: 0

RE: Thanks very much for

Quote:

Thanks very much for your substantial contribution, and of course you are welcome to return any time...

Cheers and feline greetings.


Could speak only for me, "i will back". I realy like to participate in your project. All reamining E@H in my host will be crunched in time.

My best wishes for your kitties, leave you with this song spacialy for them.

http://www.youtube.com/watch?v=_9uCP7jIa1I

lHj2ixL.jpg

 

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.