Would it be possible to support file compression on Einstein?
Rosetta Gzips "GZ" their files to make downloads more efficient.
Also hard disk space occupied by E@H files could be reduced somewhat if the "raw data" were downloaded and kept in a compressed state, at the expense of some extra overhead when starting the apps.
Would it be possible to support file compression on Einstein?
Rosetta Gzips "GZ" their files to make downloads more efficient.
We already do this to the reasonable extent. The skygrid files are compressed, the result files that are uploaded are compressed, too. The raw detector data is almost as resistant to compression algorithms as white noise - if you're lucky with your segment, you may squeeze out 15%.
If the download mirrors were set up using e.g. Apache's mod_deflate, newer BOINC clients will automatically and transparently (without any need to change the science app) download the input files in compressed form and unpack it when storing on disk. The compression is done on-the-fly by the webservers of the download mirrors. Neither the raw input files nor the science app need modification, it's a pure Webserver-configuration matter.
This is somewhat wasteful in terms of CPU resources on the server side, but on the other hand would conserve 25% bandwidth for servers and clients alike. It depends on the traffic patterns on the servers whether this is advisable or not.
Ahh yes, I remember dealing with the horrors of trying to get various telco services in the past... It all started with sharing one 33.6kbps modem off the server with a thin-net ethernet to clients. Over the years... ...
Ahh yes, I remember dealing with the horrors of trying to get various telco services in the past... It all started with sharing one 33.6kbps modem off the server with a thin-net ethernet to clients. Over the years... ...
Would it be possible to
)
Would it be possible to support file compression on Einstein?
Rosetta Gzips "GZ" their files to make downloads more efficient.
RE: Would it be possible to
)
Also hard disk space occupied by E@H files could be reduced somewhat if the "raw data" were downloaded and kept in a compressed state, at the expense of some extra overhead when starting the apps.
CU
Bikeman
RE: Would it be possible to
)
We already do this to the reasonable extent. The skygrid files are compressed, the result files that are uploaded are compressed, too. The raw detector data is almost as resistant to compression algorithms as white noise - if you're lucky with your segment, you may squeeze out 15%.
BM
BM
Bring on the changes -- We'll
)
Bring on the changes -- We'll muddle through somehow. Remember, we (the crunchers) are here for the project ...
If I've lived this long - I gotta be that old!
I got 25% compression rate
)
I got 25% compression rate for the raw data (using gzip). Still...is it worth it?
Here's an interesting alternative that comes at virtually no extra overhead:
http://boinc.berkeley.edu/trac/wiki/FileCompression
If the download mirrors were set up using e.g. Apache's mod_deflate, newer BOINC clients will automatically and transparently (without any need to change the science app) download the input files in compressed form and unpack it when storing on disk. The compression is done on-the-fly by the webservers of the download mirrors. Neither the raw input files nor the science app need modification, it's a pure Webserver-configuration matter.
This is somewhat wasteful in terms of CPU resources on the server side, but on the other hand would conserve 25% bandwidth for servers and clients alike. It depends on the traffic patterns on the servers whether this is advisable or not.
CU
Bikeman
RE: Ahh yes, I remember
)
Sounds almost like the outcome of this YouTube video: Cingular is Now Turning into the new AT&T!
RE: RE: Ahh yes, I
)
Hehe, that was a funny one! I never saw that spot when it first aired, so that was definitely good for a laugh. :)