Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

comment on

( #3333=superdoc: print w/replies, xml ) Need Help??

There are delta transfer algorithms (see rsync and librsync) where the client updates a local file according to the difference sent by server. They use rolling checkums to only transfer the differing bits. There's BitTorrent where the client knows the checksum of each fragment of the file and only asks the server for the chunks that it doesn't have yet. Both of these methods require some kind of prior knowledge on the side of the client.

There are domain-specific data compression methods that may outperform generic lossless compression methods. For example, if you want to store an m by n matrix and know for sure that is has a low rank p, you can use truncated SVD and only store two much smaller m by p and p by n matrices, then compress the residuals (supposedly small normally distributed noise) using arithmetic coding. You can also train an autoencoder neural network and hope it's sufficiently smart to figure out whatever structure your data has. These methods require the data to follow some kind of model in order to compress the data well.

Either way, you cannot compress the data below its information content. If you're transmitting pictures of a street light and all you care about is whether it's on, you can transmit one bit instead of the whole picture. But if these are going to be pictures of different street lights and every aspect of them is going to be important, including some you can't predict in advance, this adds up to many bits which you can't avoid transferring. In this case, you probably won't outperform generic image compression methods.

Perhaps if you're more specific (what's your actual use case?), we may be able to offer better advice than that.


In reply to Re^14: Reverse download protocols by Anonymous Monk
in thread Reverse download protocols [solved] by tomasz

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Are you posting in the right place? Check out Where do I post X? to know for sure.
  • Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
    <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
  • Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
  • Want more info? How to link or How to display code and escape characters are good places to start.
Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (4)
As of 2022-12-04 11:32 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?