http://qs321.pair.com?node_id=295844


in reply to Storable: Byte order is not compatible

You could try re-serializing your data using a different mechanism before you convert the database. FreezeThaw and Data::Dumper come to mind. Using the latter, you could even see if anything went awry during the conversion.

Makeshifts last the longest.

  • Comment on Re: Storable: Byte order is not compatible

Replies are listed 'Best First'.
Re^2: Storable: Byte order is not compatible
by macnod (Initiate) on Feb 14, 2012 at 22:41 UTC

    I ran into the following problem: in our database, we have some frozen Perl hashes that were created with an old 32-bit version of Storable. While moving our application to a newer version of Linux and Perl and Storable, we discovered that the newer version of Storable, 64-bit, couldn't thaw the old frozen hashes.

    I developed the following Perl solution, which works for the data that we have. The code can take something that an old, 32-bit Storable freeze created and convert it into something that a new, 64-bit Storable can thaw.

    sub fix_frozen { my $frozen= shift; my $new= join( '', (qw/04 08 08 31 32 33 34 35 36 37 38 04 08 08 08/, (map {sprintf("%02x", ord($_))} (split //, substr($frozen, 11))))); $fixed_frozen=~ s/0{7}b/0{15}b/sg; join('', map {chr hex $_} ($fixed_frozen =~ /[0-9a-z]{2}/gi)); }

    Keep in mind that this code will not work for all cases and that I've only ever tested it with the data that we have.

      in our database, we have some frozen Perl hashes that were created with [] Storable

      That right there is your problem. Storable makes no promises about binary compatibility across different releases or different perls. You do not want to use it as a serialisation mechanism. It is not a good idea for anything except intra- or interprocess communication between forks of the same process.

      Most likely you want JSON. That will also decouple your data from perl. JSON::XS even happens to be faster than Storable, and by no small margin (which is where the simplicity of the JSON data model pays off).

      Makeshifts last the longest.

      The above function doesn't look correct or useful. I wrote one to convert Storable 2.30 32-bit x86 data to 64-bit architecture. It seemed to work correctly for my data but was not thoroughly tested. Hope someone finds it useful. You should be able to tweak it as needed for other systems.

      # Convert Linux (little-endian) 32-bit Storable format to 64-bit. # Doesn't yet handle a few Storable types. # Tweak xxx lines for your architectures. # # Greg Ubben, 18 June 2012 # sub fix_frozen { local $data = shift; local $pos = 15; # xxx length of new header below local $lev = 0; # version, byte order, sizes (int, long, ptr, double) $data =~ s{^\x04\x07\x041234\x04\x04\x04\x08} # xxx {\x04\x08\x0812345678\x04\x08\x08\x08} # xxx or die "not a 32-bit x86 Storable"; object(); die "length error" if $pos != length( $data ); return $data; } sub object { local $_ = chr( 64 + byte() ); local $lev = $lev + 1; #printf "%8d %*s\n", $pos, $lev * 2, $_; # DEBUG return if /E|N|O|P/; return $pos += 1 if /H/; return $pos += 4 if /@|I/; return $pos += byte() if /J|W/; return $pos += len() if /A|X/; return object() if /D|T|K|L|M/; return object( $pos += vnum()) if /Q/; return object( vnum() ) if /R/; return fix_integer() if /F/; return fix_double() if /G/; if (/B|C|Y/) { # array or hash my $n = len(); $pos++ if /Y/; while ($n--) { object(); $pos++ if /Y/; $pos += len() if not /B/; } return; } die sprintf "Type %d unknown at pos %d\n", ord()-64, $pos-1; } sub byte { return ord( substr( $data, $pos++, 1 )); } sub len { my $len = unpack 'L', substr( $data, $pos, 4 ); # xxx 'V' or 'N' + ? $pos += 4; return $len; } sub vnum { my $n = byte(); return ($n < 128 ? $n : len()); } sub fix_integer { # xxx my $n = unpack 'l', substr( $data, $pos, 4 ); # was 32-bit substr( $data, $pos, 4 ) = pack 'q', $n; # now 64-bit $pos += 8; } sub fix_double { # xxx $pos += 8; # assume same floating-point format }
        Nice code ... helped me ... tnx

        I forgot, if the byte order differs, len() also needs to update the length in the other order. E.g.:

        sub len { my $len = unpack 'V', substr( $data, $pos, 4 ); substr( $data, $pos, 4 ) = pack 'N', $len; $pos += 4; return $len; }