Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

[SOLVED] JSON::XS parsing error when incrementally parsing JSON file - "panic: sv_chop"

by ateague (Monk)
on Nov 08, 2016 at 23:29 UTC ( [id://1175565]=perlquestion: print w/replies, xml ) Need Help??

ateague has asked for the wisdom of the Perl Monks concerning the following question:

UPDATE:
Just for completeness for any that may come after, this issue has been fixed in version 3.03:

- fix a bug introduced by a perl bug workaround that would cause incremental parsing to fail with a sv_chop panic.

UPDATE:
I tried replacing JSON::XS with JSON::PP and got the following error message instead:

unexpected end of string while parsing JSON string, at character offset 5 (before "(end of string)") at parse_json.pl line 40.

Good morning!

I am working through a little script that will incrementally parse a multi-gigabyte JSON file using the JSON::XS module. I am using the incremental parsing example provided on CPAN at the bottom of this example section here and I am running into errors when I try to parse the JSON data.

In the example test case below, Perl is aborting with a panic: sv_chop error after the first document. What is causing this particular error? Am I doing something wrong?

EDIT: add actual error message

End of document 1 panic: sv_chop ptr=4488b1, start=422038, end=4221f8 at parse_json.pl l +ine 40, <DATA> chunk 886.

I am using Strawberry Perl 5.024 x64 on Windows 8.1 with JSON::XS version 3.02

Example program:
#!/usr/bin/perl use 5.024; use JSON::XS; use strict; use warnings; ################# # CHUNK_LENGTH is intentionally tiny to highlight the error my $CHUNK_LENGTH = \1; ################# ################### ## MAIN { my $json = new JSON::XS; my $buffer; my $statement_count = 0; # Incrementally parse an array of JSON objects local $/ = $CHUNK_LENGTH; # first parse the initial "[" INITIAL_PARSE_LOOP: while ( 1 ) { $buffer = <DATA>; $json->incr_parse($buffer); # void context, so no parsing # Exit the loop once we found the initial "[". # In essence, we are (ab-)using the $json object as a simple scala +r # we append data to. last INITIAL_PARSE_LOOP if $json->incr_text =~ s/^.*?\[//msx; } # now we have the skipped the initial "[", so continue # parsing all the elements. PARSE_LOOP: while ( 1 ) { # clean up whitespace and padding $json->incr_text =~ s/^\s*(.+)\n/$1/gms; # in this loop we read data until we got a single JSON object STATEMENT_PARSE_LOOP: while ( 1 ) { if ( my $obj = $json->incr_parse ) { say "End of document ".++$statement_count; last STATEMENT_PARSE_LOOP; } # add more data $buffer = <DATA>; $json->incr_parse($buffer); # void context, so no parsing } # in this loop we read data until we either found and parsed the # separating "," between elements, or the final "]" CLEAN_UP_LOOP: while ( 1 ) { # first skip whitespace $json->incr_text =~ s/^\s*//; # if we find "]", we are done with the file last PARSE_LOOP if $json->incr_text =~ s/^\]//; # if we find ",", we can continue with the next element last CLEAN_UP_LOOP if $json->incr_text =~ s/^,//; # if we find anything else, we have a parse error! if ( length $json->incr_text ) { die "parse error near ".$json->incr_text; } # else add more data $buffer = <DATA>; $json->incr_parse($buffer); # void context, so no parsing } } } __DATA__ [ { "name": "xxxxxxxxxxxxxxx", "id": "xxxxxxxxxxxxxx", "previousBalance": "xxxxx", "currentMonth": "xxxxxxx", "total": "xxxxxxxx", "billingDate": "xxxxxxxxxxxxxx", "billingPeriod": "xxxxxxxxxxxxxxxxxxxxxxx", "invoiceDate": "xxxxxxxxxxx", "dueDate": "xxxxxxxxxxx", "autopay": false, "address": { "line1": "xxxxxxxxxxxxx", "line2": null, "city": "xxxxxxxxxxxx", "state": "xx", "zip": "xxxxxxxxxxx" } }, { "name": "xxxxxxxxxxxxxxx", "id": "xxxxxxxxxxxxxx", "previousBalance": "xxxxx", "currentMonth": "xxxxxxx", "total": "xxxxxxxx", "billingDate": "xxxxxxxxxxxxxx", "billingPeriod": "xxxxxxxxxxxxxxxxxxxxxxx", "invoiceDate": "xxxxxxxxxxx", "dueDate": "xxxxxxxxxxx", "autopay": false, "address": { "line1": "xxxxxxxxxxxxx", "line2": null, "city": "xxxxxxxxxxxx", "state": "xx", "zip": "xxxxxxxxxxx" } } ]
Thank you for your time.

Replies are listed 'Best First'.
Re: JSON::XS parsing error when incrementally parsing JSON file - "panic: sv_chop"
by kcott (Archbishop) on Nov 09, 2016 at 02:45 UTC

    G'day ateague,

    I tried running your code but found I didn't have JSON::XS installed under v5.24. On attempting to install using cpan, I got a message which in essence said:

    "... standard perl versions 5.022 and up are not supported by JSON::XS."

    The full message, in the spoiler below, was far more detailed.

    As the installation ran to completion, I tried your script and got:

    $ pm_1175565_json_xs_sv_chop_panic.pl End of document 1 panic: sv_chop ptr=7faa01441e79, start=7faa0140eb30, end=7faa0140ecf0 +at ./pm_1175565_json_xs_sv_chop_panic.pl line 41, <DATA> chunk 886.

    The "panic: sv_chop %s" diagnostic was introduced in v5.12 (see "perl5120delta: New Diagnostics"):

    • panic: sv_chop %s

      This new fatal error occurs when the C routine Perl_sv_chop() was passed a position that is not within the scalar's string buffer. This could be caused by buggy XS code, and at this point recovery is not possible.

    I don't have sufficient knowledge of Perl's 'C' internals to help further on that front.

    You could try installing JSON::XS under a version of Perl earlier than v5.22; however, given ++stevieb's results under v5.16.3, you should probably look more closely at your script for other issues.

    Also note that the current version of JSON (2.90) has (in its VERSION section):

    "This version is compatible with JSON::XS 2.34 and later. (Not yet compatble to JSON::XS 3.0x.)"

    So, perhaps you might have more success with earlier versions of both Perl and JSON::XS.

    — Ken

      I tried running your code but found I didn't have JSON::XS installed under v5.24. On attempting to install using cpan, I got a message which in essence said: "... standard perl versions 5.022 and up are not supported by JSON::XS."

      That is interesting.

      I was able to install it on Windows without any issues with Strawberry Perl's cpan and ActivePerl's ppm

      The "panic: sv_chop %s" diagnostic was introduced in v5.12

      I saw that too on Perldiag. Unfortunately, it only tells *what* is happening, not *why* it is happening...

Re: JSON::XS parsing error when incrementally parsing JSON file - "panic: sv_chop"
by stevieb (Canon) on Nov 09, 2016 at 00:49 UTC

    I have not had a chance to go through the code in detail yet, but thought I'd share my observations.

    On Linux Mint 18 with:

    This is perl 5, version 24, subversion 0 (v5.24.0) built for x86_64-li +nux

    ...under Perlbrew, and:

    perl -MJSON::XS -E 'say $JSON::XS::VERSION' 3.02

    I get the same result:

    perl jsonxs.pl End of document 1 panic: sv_chop ptr=16e6219, start=16ebf20, end=16ec0e0 at jsonxs.pl li +ne 41, <DATA> chunk 886.

    Removing your requisite use 5.24.0; and after installing 5.16.3, I get the same error (note I included use feature 'say';):

    End of document 1 panic: sv_chop ptr=1ed94c9, start=1d31480, end=1d31640 at jsonxs.pl li +ne 41, <DATA> chunk 886.

    So it doesn't seem related to the platform, nor the version of Perl. Only thing that's the same is the version of JSON::XS.

      Thanks

      I tried swapping out JSON::XS with JSON::PP without any success and got a different error:

      unexpected end of string while parsing JSON string, at character offset 5 (before "(end of string)") at parse_json.pl line 40.

      Not quite sure why the pure-Perl version is failing in this manner...

      I don't know if this is a red herring, but I notice the errors posted for the XS version all have the same length between start and end: 0x1C0

      That may (or may not) be significant...

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://1175565]
Approved by Paladin
Front-paged by stevieb
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others about the Monastery: (5)
As of 2024-03-28 22:54 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found