Write your own Einstein@home screensaver

Mike Hewson
Mike Hewson
Moderator
Joined: 1 Dec 05
Posts: 6537
Credit: 286446952
RAC: 96712

Mipmaps mip = multum in

Mipmaps

mip = multum in parvo or 'many in the small'. OpenGL at each frame rendering is going to have to deduce what texture picture elements - texels - are going to be used for rendering onto the screenbuffer as pixels ( or prior to rescaling for window/viewport placement what are called 'fragments' ). For magnification ( seeing close up ) fewer texels are going to appear on more pixels, for minification ( seeing from afar ) more texels are going to appear on fewer pixels. Rarely will texels-to-pixels be 1:1. Hence OpenGL has to sample or filter the texture : several options as regards averaging across texels, near enough is good enough etc. However all that is done on the fly as a frame by frame process. Trouble is for a large texture - 2048 by 2048 in my Earth case - when seen from afar then there can be quite alot of rigamarole to select a mere few 'representative' texels.

A solution is to provide OpenGL with many versions ( ie. mipmaps ) of the same image, pre-filtered if you like, that represent the same texture but at different levels of resolution or granularity. This normally goes by powers of two ( though 'non-square' images are dealt with too ) so for my 2048 x 2048 primary image one gets : 1024 x 1024, 512 x 512, 256 x 256, 128 x 128, 64 x 64, 32 x 32, 16 x 16, 8 x 8, 4 x 4, 2 x 2 and finally 1 x 1 for a total of 12 levels numbered 0 thru 11. By an algorithm ( I do not know what ) OpenGL makes a basic decision, per frame, as to which level to use and then filters that. Thus the runtime for this aspect is kept down, but at the price of more memory use to store the original texture and it's mipmaps. I can't be bothered working out the precise limit of the above sequence, but the extra memory used with mipmaps is going to be less than half that of the original ( or level 0 ) image.

The standard way to do that is to provide all those levels and introduce them to OpenGL. That's actually a bit painful to do, however the Graphics Utility Library ( GLU ) has a function that does all the work for you !!! It's generically called gluBuild2DMipmaps() - several varieties - that takes the original image and creates all the level images down to 1 x 1 ( single texel ), while also doing all the other required 'introductory' calls like glTexImage2D() per level for you. That only needs to be done once.

Anyway I've done that for the Earth's texture and it works a treat, and after playing with it for quite a while - moving in close and out afar - I can't yet find any artifacts/jumps/flashes etc that can occur with level changes. That's most likely due to beginning with a fairly high resolution image at level 0, giving OpenGL a choice of 12 to play with for various viewing distances.

FWIW I think this is probably the hardest OpenGL aspect that I will deal with for my purposes. Having done this for the Earth I now get the Sun for free so to speak : make a base/super class and/or parameterise upon the variants etc. Moon anyone ? This is fun, and it seems to be getting easier! :-)

Cheers, Mike.

( edit ) Well I did bother finding the series limit : it's 4/3 times the level 0 size. So my ~6MB Earth texture is needing no more than ~8MB on the video card for mip-mapping. The extra memory requirements go down dramatically with each level ie.

1 + 1/4 + 1/16 + 1/64 .....

[ partial sums are -> 1, 5/4, 21/16, 85/64, 341/256, 1365/1024 ....

or 1.0, 1.25, 1.3125, 1.328125, 1.33203125, 1.3330078125 ..... ]

as memory goes like the square of the side length for two dimensions ( so for 3D textures, say, that would be inverse cubes ). This is worth noting as an extra third upon the base image size is not too bad.

( edit ) For 1D textures the limit is that of :

1 + 1/2 + 1/4 + 1/8 + 1/16 ....

which is 2

for 3D textures you've got :

1 + 1/8 + 1/64 + 1/512 ....

which is 8/7 ~ 1.14

[ the pattern is for N dimensions the limit is 2^N/(2^N - 1) ]

Of course one has to be happy with the base image memory size in the first place. :-)

I have made this letter longer than usual because I lack the time to make it shorter ...

... and my other CPU is a Ryzen 5950X :-) Blaise Pascal

Mike Hewson
Mike Hewson
Moderator
Joined: 1 Dec 05
Posts: 6537
Credit: 286446952
RAC: 96712

Well it seems that I have

Well it seems that I have succeeded in re-jigging the repo, all my changes are on the branch mikesbest .... :-)

Cheers, Mike.

I have made this letter longer than usual because I lack the time to make it shorter ...

... and my other CPU is a Ryzen 5950X :-) Blaise Pascal

Oliver Behnke
Oliver Behnke
Moderator
Administrator
Joined: 4 Sep 07
Posts: 944
Credit: 25167626
RAC: 23

Alright, just attached your

Alright, just attached your repo to mine, looks good! Both branches, master and mikesbest, could be fast-forwarded, as expected from a true fork. Two questions though:

1) Why did you remove the starsphere stuff completely? Wouldn't it be nicer to just add your solar system as another "engine" to the existing package? This way people would just have to use one source tree (repo) in order to build all three incarnations. More importantly: the idea of having "coupled" repos was to eventually merge your changes back upstream into the official release - that can't be done when the original "engines" got removed.

2) What OS/filesystem do you use? Your files (code/text and others) all have a privileges setting of 755 and your commits changed existing file form 644 also to 755.

I can mail you the output logs (stats) of my merges tomorrow if you want. Gives a nice overview of what happened...

Cheers,
Oliver

 

Einstein@Home Project

Mike Hewson
Mike Hewson
Moderator
Joined: 1 Dec 05
Posts: 6537
Credit: 286446952
RAC: 96712

RE: Alright, just attached

Message 78101 in response to message 78100

Quote:
Alright, just attached your repo to mine, looks good! Both branches, master and mikesbest, could be fast-forwarded, as expected from a true
fork.


Good ! It took me a little while to get the ssh worked out, as there's an extra thingy to do on Linux machines ... ( testing with 'ssh -T ' gives success, but one gets a public key error upon commits. Now sorted. )

Quote:

Two questions though:

1) Why did you remove the starsphere stuff completely? Wouldn't it be nicer to just add your solar system as another "engine" to the existing package? This way people would just have to use one source tree (repo) in order to build all three incarnations. More importantly: the idea of having "coupled" repos was to eventually merge your changes back upstream into the official release - that can't be done when the original "engines" got removed.


Put that down to my naivety in these things, not thinking at a high enough level, that sort of thing .... :-)

It's no drama to re-insert that. I'll put the files back in and codewise only minor changes to the GraphicsEngineFactory switch/case etc - leave it with me.

[ Aha! The original build script ( to produce starsphere ) is going to be on the original master. I'm getting the idea now ..... ]

Quote:
2) What OS/filesystem do you use? Your files (code/text and others) all have a privileges setting of 755 and your commits changed existing file form 644 also to 755.


Linux Ubuntu 10 + updates. I'm assuming you're referring to the read/write/execute fields. I didn't know they got transferred - long time Linux/Unix users reading this are probably ROFLing by now. But I learn .... :-)

What settings are we after here ?

Quote:
I can mail you the output logs (stats) of my merges tomorrow if you want. Gives a nice overview of what happened...


That would be terrific! Thank you for your insights, they are sorely needed. :-)

Cheers, Mike

( edit ) ..... and I don't know why I didn't get the ssh public key problem first up, but had trouble this time around. I probably stuffed up the copy & paste of the contents of 'id_rsa.pub' is my guess, as I went back and repeated all the steps again ( as illustrated by GitHub ) - though including the part where one uses ssh-add .....

( edit ) FWIW : now that I think about it, make sure one uses a 'simple' text editor, like gedit say, that won't stuff about with inserting line endings and the like. The copy and paste from 'id_rsa.pub' must be pure.

I have made this letter longer than usual because I lack the time to make it shorter ...

... and my other CPU is a Ryzen 5950X :-) Blaise Pascal

Mike Hewson
Mike Hewson
Moderator
Joined: 1 Dec 05
Posts: 6537
Credit: 286446952
RAC: 96712

Yeah. Epiphany or doh !

Yeah. Epiphany or doh ! :-)

I've been thinking of today's fork, but not also of the some-time-later merge .... RTFM Mike. :-)

Cheers, Mike.

I have made this letter longer than usual because I lack the time to make it shorter ...

... and my other CPU is a Ryzen 5950X :-) Blaise Pascal

Mike Hewson
Mike Hewson
Moderator
Joined: 1 Dec 05
Posts: 6537
Credit: 286446952
RAC: 96712

This git/SHA1 business is

This git/SHA1 business is tres cool !! :-)

One type of object git uses is a 'tree', this is usually produced with each commit and essentially represents one directory layer - complete with metadata like file and pathnames to correspond with the blobs - and can reference other sub-directory layers by pointing to their tree objects etc recursively to any level of exhaustion. One can then form a SHA1 hash from a tree's own contents and thus that becomes a unique identifier like it does for blobs. Hence ( by logic previously outlined ) two trees with the same SHA1 are identical, and different SHA1's means distinct trees ! Now because the contents of a tree imply a certain (sub)directory structure, OS file/pathname correspondence, and includes each individual blob's own SHA1 too ... then the tree object immediately associated with a given commit is enough to kick off the process of entirely re-constructing a project's file content and directory structure - a true 'instance snapshot' - but ever so efficiently done.

[ So you could move a file, otherwise unaltered, to a different subdirectory. The blob of that file is the same, as there is no content change, but a reference to that blob will appear in a newly created tree object. The older tree object that did reference that blob in it's original directory position will not be deleted, it simply is not referenced by the new commit. ]

To transfer content to/from the OS's file system and git's internals, git only needs to keep an index that associates the pathnames to the blobs. Otherwise it need not care of the meaning of a filename or of a directory name - it only needs to arrange trees and sub-trees reflecting the directory structure at the time of the commit. Git just passes pathname strings to/from the OS and gets an input/output data stream for it's use. I am so impressed .... :-)

It hinges beautifully on SHA1 being an effectively unique identifier of the content of whatever you are calculating it upon, be it blob or tree or whatever.

Each commit points to it's parent ( easy to remember : children are born after their parents and thus parents represent an earlier state of the project ) so the 'current state' of the project is the most recent commit or child. But by traversing the linkages one can stride backwards in time through the revisions to the first commit ( which is thus the only one without a parent ). And by interrogating git one can then form other layers of abstraction/viewpoint above what git primarily manages - I use GitCola currently as a gui interface.

I think of it as a meld of linked lists and directed acyclic graphs ( well almost ), but in fact there is an exceptional analogy here with DNA, biological inheritance etc .... :-)

There are optional 'tag' objects which simply attach a human readable phrase ( say, 'Version 3.4' ) to the SHA1 hash for some commit. This is distinct from the message that ought accompany each commit to describe it.

Branches - master, mikesbest or whatever - are labels for lineages. So when I cloned the E@H screensaver repo it comes with the complete pedigree - who did what and when right back for some years. When I fork/branch I continue the legacy ....

Cheers, Mike.

( edit ) In case I haven't made it obvious enough : a 160 bit ( 40 byte ) SHA1 hash value is way shorter than the average file, directory structure representation, index etc. So this avoids a huge slab of time that would otherwise be spent on byte-by-byte pairwise comparisons of blobs, trees or whatever. Just compare 40 byte hashes instead. You still need to keep the content in the repo, and a file's content change will trigger a new blob to be stored with a new hash calculated too. That also happens with directory structure changes - additions/deletions/movements of files and/or subdirectories - thus a new tree object encapsulates that change and with it's SHA1 hash calculated anew. The project state is embedded in the pattern of the links/references between the git object types. Short of a lightning strike ( or hard drive failure etc ... ) the git operations are designed as 'atomic' : they happen entirely or not at all, thus avoiding the dreaded state of being inconsistent or 'half way' - the equivalent of incomplete/lost/scrambled birth certificates.

( edit ) Tracking files or not, ignoring files or not, staging files or not - are all precursors to a commit decision. This is how you manage the content relationship between your working or visible copy and the repository held within the hidden .git directory at the root of your project eg. why would one save build temporary files ? You could, but for what reason? So decide for yourself what is of intellectual significance and thus worth saving as integral to the project state. Perhaps ignore a file ( type/name/pattern ) completely - effectively invisible to git - or allow git to know of it/them but not bothering to track if they've altered. Staging is a proposal for change. Once you've committed the change you can't undo it. You can restore the content to a prior time using further commits and like functions ( after all what's the point if you can't venture back when you need to, before some prior mistake say ) but I really like the 'audit trail' that says who did what & when. :-)

( edit ) You could have more than one commit pointing to the same tree object. This implies the project's content hasn't changed. Say the boss comes in to inspect the work, he changes nothing, but likes it at that stage and thus 'signs off' on that project state with a commit to formalise/acknowledge that.

I'm still working on the screensaver code, I'm just doing a git detour. :-0

I have made this letter longer than usual because I lack the time to make it shorter ...

... and my other CPU is a Ryzen 5950X :-) Blaise Pascal

Oliver Behnke
Oliver Behnke
Moderator
Administrator
Joined: 4 Sep 07
Posts: 944
Credit: 25167626
RAC: 23

Before we get back to topic,

Message 78104 in response to message 78103

Before we get back to topic, my recommended git resources:

http://blip.tv/scott-chacon/git-talk-4113729 (1 hr video/talk!)
http://progit.org (book)

http://www.git-scm.com (git homepage)
http://gitref.org (reference)
http://hoth.entp.com/output/git_for_designers.html (for SCM newbies)

Personal favorites, you guessed it, the video and the book :-)

Best,
Oliver

 

Einstein@Home Project

Mike Hewson
Mike Hewson
Moderator
Joined: 1 Dec 05
Posts: 6537
Credit: 286446952
RAC: 96712

Ok, finally got my git-ducks

Ok, finally got my git-ducks in a row. The github repo is now all tidied up - I deleted it and pushed a better forked version with original master and my branch is now called 'solarsystem'. Thanks for the references Oliver - they are great and I'll work through them. I've also worked out what 'brevilo' means ..... :-)

Thanks for the links from killet, Robert! It's probably way more than I needed, in any case I've already rolled my own 3D geometry/vector/transform classes. Thanks for the thought. :-)

As for the GPU screensaver version - my rough guess from your description is contention on the card for resources with both the science app and the screensaver running - it keeps losing it's OpenGL context ??

Cheers, Mike.

I have made this letter longer than usual because I lack the time to make it shorter ...

... and my other CPU is a Ryzen 5950X :-) Blaise Pascal

Mike Hewson
Mike Hewson
Moderator
Joined: 1 Dec 05
Posts: 6537
Credit: 286446952
RAC: 96712

Have done a full overhaul of

Have done a full overhaul of the craft physics code : multiple inheritance no less of classes defining position/velocity in one line of descent merging with orientation/rotation in another to yield an AcceleratedPlatform (grand)child. Gives a 'gadget' that one can set in all the above kinematic/dynamic parameters - no second derivative of angle though. :-)

This will easily lead to a separate ( virtual ) Controller class that will be the interface b/w the keyboard inputs and craft behaviors. Different children of that Controller will thus behave as a different functioning 'joysticks'. A Craft object will then be composed of a Controller object and an AcceleratedPlatform object.

A fairly simple bit of code can then add in an acceleration component not involving keyboard input : this will be gravity!! At each frame/time-step an acceleration-due-to gravity vector can be calculated from the craft's position with respect to the deemed gravitating objects in the scene : the Earth and the Sun. As the solar system is not being done to true scale then I'll have to fake a few factors for niceness of behaviour. It's only a screensaver after all.

In any case that will allow 'unattended' evolution of the viewpoint, just punt it off from given ( maybe random ) initial conditions and let it wander about! All being well it'll model as a truly conservative field. Toss in a bit of 'scripting' to get it too look at certain things along the way .... :-)

Cheers, Mike.

( edit ) Hmmm ... gotta get my editor to swap tabs for spaces, except for the makefiles which require a tab character on indent if a following make command is to be recognised as such. It's a seriously annoying bit of history that makes ( pun !! ) two whitespace modes act differently.

I have made this letter longer than usual because I lack the time to make it shorter ...

... and my other CPU is a Ryzen 5950X :-) Blaise Pascal

robertmiles
robertmiles
Joined: 8 Oct 09
Posts: 127
Credit: 21134738
RAC: 14497

RE: Thanks for the links

Message 78107 in response to message 78105

Quote:

Thanks for the links from killet, Robert! It's probably way more than I needed, in any case I've already rolled my own 3D geometry/vector/transform classes. Thanks for the thought. :-)

As for the GPU screensaver version - my rough guess from your description is contention on the card for resources with both the science app and the screensaver running - it keeps losing it's OpenGL context ??

Cheers, Mike.

Looks like you still haven't noticed that the screensaver problem only occurs when the science app cannot run because Activity > Suspend GPU is set.

I believe that a moderator moved my earlier comments on the screensaver problem to some other thread, but without mentioning which other thread; I still haven't found it.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.