Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister

Re^2: Storable: Byte order is not compatible

by macnod (Initiate)
on Feb 14, 2012 at 22:41 UTC ( #953792=note: print w/replies, xml ) Need Help??

in reply to Re: Storable: Byte order is not compatible
in thread Storable: Byte order is not compatible

I ran into the following problem: in our database, we have some frozen Perl hashes that were created with an old 32-bit version of Storable. While moving our application to a newer version of Linux and Perl and Storable, we discovered that the newer version of Storable, 64-bit, couldn't thaw the old frozen hashes.

I developed the following Perl solution, which works for the data that we have. The code can take something that an old, 32-bit Storable freeze created and convert it into something that a new, 64-bit Storable can thaw.

sub fix_frozen { my $frozen= shift; my $new= join( '', (qw/04 08 08 31 32 33 34 35 36 37 38 04 08 08 08/, (map {sprintf("%02x", ord($_))} (split //, substr($frozen, 11))))); $fixed_frozen=~ s/0{7}b/0{15}b/sg; join('', map {chr hex $_} ($fixed_frozen =~ /[0-9a-z]{2}/gi)); }

Keep in mind that this code will not work for all cases and that I've only ever tested it with the data that we have.

Replies are listed 'Best First'.
Re^3: Storable: Byte order is not compatible
by Aristotle (Chancellor) on Apr 26, 2012 at 14:02 UTC

    in our database, we have some frozen Perl hashes that were created with [] Storable

    That right there is your problem. Storable makes no promises about binary compatibility across different releases or different perls. You do not want to use it as a serialisation mechanism. It is not a good idea for anything except intra- or interprocess communication between forks of the same process.

    Most likely you want JSON. That will also decouple your data from perl. JSON::XS even happens to be faster than Storable, and by no small margin (which is where the simplicity of the JSON data model pays off).

    Makeshifts last the longest.

Re^3: Storable: Byte order is not compatible
by Anonymous Monk on Jun 19, 2012 at 21:58 UTC

    The above function doesn't look correct or useful. I wrote one to convert Storable 2.30 32-bit x86 data to 64-bit architecture. It seemed to work correctly for my data but was not thoroughly tested. Hope someone finds it useful. You should be able to tweak it as needed for other systems.

    # Convert Linux (little-endian) 32-bit Storable format to 64-bit. # Doesn't yet handle a few Storable types. # Tweak xxx lines for your architectures. # # Greg Ubben, 18 June 2012 # sub fix_frozen { local $data = shift; local $pos = 15; # xxx length of new header below local $lev = 0; # version, byte order, sizes (int, long, ptr, double) $data =~ s{^\x04\x07\x041234\x04\x04\x04\x08} # xxx {\x04\x08\x0812345678\x04\x08\x08\x08} # xxx or die "not a 32-bit x86 Storable"; object(); die "length error" if $pos != length( $data ); return $data; } sub object { local $_ = chr( 64 + byte() ); local $lev = $lev + 1; #printf "%8d %*s\n", $pos, $lev * 2, $_; # DEBUG return if /E|N|O|P/; return $pos += 1 if /H/; return $pos += 4 if /@|I/; return $pos += byte() if /J|W/; return $pos += len() if /A|X/; return object() if /D|T|K|L|M/; return object( $pos += vnum()) if /Q/; return object( vnum() ) if /R/; return fix_integer() if /F/; return fix_double() if /G/; if (/B|C|Y/) { # array or hash my $n = len(); $pos++ if /Y/; while ($n--) { object(); $pos++ if /Y/; $pos += len() if not /B/; } return; } die sprintf "Type %d unknown at pos %d\n", ord()-64, $pos-1; } sub byte { return ord( substr( $data, $pos++, 1 )); } sub len { my $len = unpack 'L', substr( $data, $pos, 4 ); # xxx 'V' or 'N' + ? $pos += 4; return $len; } sub vnum { my $n = byte(); return ($n < 128 ? $n : len()); } sub fix_integer { # xxx my $n = unpack 'l', substr( $data, $pos, 4 ); # was 32-bit substr( $data, $pos, 4 ) = pack 'q', $n; # now 64-bit $pos += 8; } sub fix_double { # xxx $pos += 8; # assume same floating-point format }
      Nice code ... helped me ... tnx

      I forgot, if the byte order differs, len() also needs to update the length in the other order. E.g.:

      sub len { my $len = unpack 'V', substr( $data, $pos, 4 ); substr( $data, $pos, 4 ) = pack 'N', $len; $pos += 4; return $len; }

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://953792]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (2)
As of 2022-01-25 02:13 GMT
Find Nodes?
    Voting Booth?
    In 2022, my preferred method to securely store passwords is:

    Results (65 votes). Check out past polls.