http://qs321.pair.com?node_id=11146255

nataraj has asked for the wisdom of the Perl Monks concerning the following question:

Hi monks!

I am looking for some framework that will allow to create Moose-compatible object out of perl-hash that came from json parser.

The idea is following: Some API-specific module receives a json via REST-API, make an object out of it including access methods, adds object specific methods, some internal logic, processing etc... Then pass it back to user code, for further usage

I found module that allow to do similar thing for XML: MooseX::DOM. I also found MooseX::Role::JSONObject that looks like something useful, but it is almost not used in other modules, and it adds some doubts, if I am on the right way

Have I missed something? May be there is good module for making Moose-compatible object out of json, and I simply did not look for it in a right way?

  • Comment on Framework for making Moose-compatible object from json

Replies are listed 'Best First'.
Re: Framework for making Moose-compatible object from json
by haj (Vicar) on Aug 20, 2022 at 19:30 UTC

    A difference between a JSON data structure and a Perl object is that Perl objects are blessed to a class. In your other comment you write that the Class hierarchy is predefined. So it appears the actual task is to find the correct class for each of the references you find in your JSON data structure. But then, there's the phrase adds object specific methods, some internal logic which creates some friction. If the class hierarchy is predefined, then every object should be happy with the methods this class provides, and the user code should already contain the internal logic.

    Maybe what you need is just a set of rules which map parts of the structure to their corresponding class. Users of MooseX::DOM create a DOM to define these rules for XML data. With MooseX::Role::JSONObject you can convert JSON data to objects if the JSON data itself contain the rules, which they do if the JSON data have been created by the very same module. That doesn't seem to be the case for your JSON data.

    There is a chance that Moose itself is all the "framework" you need: If your classes are written as Moose-classes matching the JSON hashes, then you can just feed the JSON hash to the class constructor. The attribute declaration in the class can then use coercion to turn JSON hash references to objects if an attribute is supposed to be an object. Or, slightly more messy, you can use the BUILDARGS methods.

    Here's some code, based on your JSON example:

    use 5.028; use warnings; package MyCoolRestObject; use Moose; use Moose::Util::TypeConstraints; has body_raw => ( is => 'ro' ); has url => ( is => 'ro' ); has datetime => ( is => 'ro' ); has entry_id => ( is => 'ro' ); has subject_html => ( is => 'ro' ); coerce 'MyCoolRestObject::Poster' => from 'HashRef' => via { MyCoolRestObject::Poster->new(%$_) }; has poster => ( is => 'ro', isa => 'MyCoolRestObject::Poster', coerce => 1, ); coerce 'MyCoolRestObject::Icon' => from 'HashRef' => via { MyCoolRestObject::Icon->new(%$_) }; has icon => ( is => 'ro', isa => 'MyCoolRestObject::Icon', coerce => 1, ); has subject_raw => ( is => 'ro' ); has body_html => ( is => 'ro' ); # -------------------------------------------------------------------- +-- package MyCoolRestObject::Icon; use Moose; has url => ( is => 'ro' ); has comment => ( is => 'ro' ); has picid => ( is => 'ro' ); has keywords => ( is => 'ro' ); has username => ( is => 'ro' ); # -------------------------------------------------------------------- +-- package MyCoolRestObject::Poster; use Moose; has display_name => ( is => 'ro' ); has user_name => ( is => 'ro' ); # -------------------------------------------------------------------- +-- package main; use JSON::PP; use Data::Dump qw/dump/; $/ = undef; my $json = decode_json <DATA>; my $obj = MyCoolRestObject->new(%$json); print $obj->icon->username; __DATA__ { "body_raw" : "This is <b>Entry 1</b>", "url" : "http://nataraj.nataraj.hack.dreamwidth.net/459.html", "datetime" : "2022-08-20 16:36:00", "entry_id" : 459, "subject_html" : "Entry 1", "poster" : { "display_name" : "nataraj", "username" : "nataraj" }, "icon" : { "url" : "http://www.nataraj.hack.dreamwidth.net/userpic/1/3", "comment" : "4", "picid" : 1, "keywords" : [ "3" ], "username" : "nataraj" }, "subject_raw" : "Entry 1", "body_html" : "This is <b>Entry 1</b>" }
      Maybe what you need is just a set of rules which map parts of the structure to their corresponding class.

      That's exactly what I need. I only did not managed to put it into words properly

      There is a chance that Moose itself is all the "framework" you need

      This looks like a good idea, but it would miss backward serialization (object->json). So you can do something like that:

      my $obj= MyCoolRestObject::UserInfo->new($api->get("/path/to/proper/ +call")); $obj->phone1("+722233344555"); $api->put("path/to/put/call",$obj->as_json);

      So as far as I can get, there is no such mapper for JSON, that would work out of box the same way MooseX::DOM works for XML. And may be I should write it first, and then use it

        And may be I should write it first, and then use it

        There's nothing wrong with that approach, of course. On the other hand, object->JSON conversion isn't all that difficult, so I added it to my example (I also fixed a typo where I had user_name instead of username).

        In my example, the Moose class definition plays the role of the DOM, so you don't need to store class metadata in JSON. JSON does not know about classes, so you need to tell the JSON encoder what to do if it hits an object. To do this, I use the convert_blessed property of the JSON::PP encoder and define a TO_JSON method which just reduces an object to its underlying hash reference.

        A final note before we get to the code: I've been using JSON::PP because it's in Perl core. The code works just as fine with JSON which will use one of the faster XS implementations if they're available.

        use 5.028; use warnings; package MyCoolRestObject; use Moose; use Moose::Util::TypeConstraints; with 'MyCoolRestObject::Serializer'; has body_raw => ( is => 'ro' ); has url => ( is => 'ro' ); has datetime => ( is => 'ro' ); has entry_id => ( is => 'ro' ); has subject_html => ( is => 'ro' ); coerce 'MyCoolRestObject::Poster' => from 'HashRef' => via { MyCoolRestObject::Poster->new(%$_) }; has poster => ( is => 'ro', isa => 'MyCoolRestObject::Poster', coerce => 1, ); coerce 'MyCoolRestObject::Icon' => from 'HashRef' => via { MyCoolRestObject::Icon->new(%$_) }; has icon => ( is => 'ro', isa => 'MyCoolRestObject::Icon', coerce => 1, ); has subject_raw => ( is => 'ro' ); has body_html => ( is => 'ro' ); # -------------------------------------------------------------------- package MyCoolRestObject::Icon; use Moose; with 'MyCoolRestObject::Serializer'; has url => ( is => 'ro' ); has comment => ( is => 'ro' ); has picid => ( is => 'ro' ); has keywords => ( is => 'ro' ); has username => ( is => 'ro' ); # -------------------------------------------------------------------- package MyCoolRestObject::Poster; use Moose; with 'MyCoolRestObject::Serializer'; has display_name => ( is => 'ro' ); has username => ( is => 'ro' ); # -------------------------------------------------------------------- package MyCoolRestObject::Serializer; use Moose::Role; sub TO_JSON { +{ shift->%* } } # -------------------------------------------------------------------- package main; use Cpanel::JSON::XS; $/ = undef; my $json = decode_json <DATA>; # Cpanel::JSON::XS -> Object: my $obj = MyCoolRestObject->new(%$json); say $obj->icon->username; # Object->Cpanel::JSON::XS: my $encoder = Cpanel::JSON::XS->new->utf8->pretty->convert_blessed; my $json_out = $encoder->encode($obj); say $json_out; __DATA__ { "body_raw" : "This is <b>Entry 1</b>", "url" : "http://nataraj.nataraj.hack.dreamwidth.net/459.html", "datetime" : "2022-08-20 16:36:00", "entry_id" : 459, "subject_html" : "Entry 1", "poster" : { "display_name" : "nataraj", "username" : "nataraj" }, "icon" : { "url" : "http://www.nataraj.hack.dreamwidth.net/userpic/1/3", "comment" : "4", "picid" : 1, "keywords" : [ "3" ], "username" : "nataraj" }, "subject_raw" : "Entry 1", "body_html" : "This is <b>Entry 1</b>" }
Re: Framework for making Moose-compatible object from json
by kcott (Archbishop) on Aug 20, 2022 at 22:11 UTC

    G'day nataraj,

    Would something like this suit your purposes:

    package Local::MooseJson; use strict; use warnings; use Moose; use JSON; use namespace::autoclean; around BUILDARGS => sub { my ($orig, $class, @args) = @_; # Validate @args: 1 element only; JSON text my $json_to_perl = decode_json $args[0]; my $new_hashref = {}; for my $key ($class->meta->get_attribute_list()) { $new_hashref->{$key} = delete $json_to_perl->{$key}; } for my $extra_key (keys %$json_to_perl) { warn "IGNORED! Unknown JSON property: '$extra_key'\n"; } return $class->$orig($new_hashref); }; has body_raw => (is => 'ro', isa => 'Str', required => 1); has entry_id => (is => 'ro', isa => 'Int', required => 1); has icon => (is => 'ro', isa => 'HashRef', required => 1); __PACKAGE__->meta->make_immutable; 1;

    Here's a test script:

    #!/usr/bin/env perl use strict; use warnings; use FindBin qw($RealBin); use lib "$RealBin/../lib"; use Local::MooseJson; chomp(my $rest_json = <<'EOJ'); { "body_raw" : "This is <b>Entry 1</b>", "url" : "http://nataraj.nataraj.hack.dreamwidth.net/459.html", "datetime" : "2022-08-20 16:36:00", "Surprise!" : "You didn't expect this.", "entry_id" : 459, "subject_html" : "Entry 1", "poster" : { "display_name" : "nataraj", "username" : "nataraj" }, "icon" : { "url" : "http://www.nataraj.hack.dreamwidth.net/userpic/1/3", "comment" : "4", "picid" : 1, "keywords" : [ "3" ], "username" : "nataraj" }, "subject_raw" : "Entry 1", "body_html" : "This is <b>Entry 1</b>" } EOJ my $mj = Local::MooseJson::->new($rest_json); print 'Username: ', $mj->icon()->{username}, "\n";

    Output:

    ken@titan ~/tmp/pm_11146255_moose_json/bin $ ./moose_json.pl IGNORED! Unknown JSON property: 'subject_html' IGNORED! Unknown JSON property: 'url' IGNORED! Unknown JSON property: 'poster' IGNORED! Unknown JSON property: 'subject_raw' IGNORED! Unknown JSON property: 'Surprise!' IGNORED! Unknown JSON property: 'body_html' IGNORED! Unknown JSON property: 'datetime' Username: nataraj

    When you add in the missing attributes, all but one of the "IGNORED" lines will disappear (i.e. you'll still get a Surprise!). You can then decide if you're getting bad data back from the API or if you need to update the module. This acts as a sanity check for both your module and the incoming data.

    In case you were wondering how all of that fits together, here's the directory structure.

    ken@titan ~/tmp $ ls -lR pm_11146255_moose_json pm_11146255_moose_json: total 0 drwxr-xr-x 1 ken None 0 Aug 21 07:43 bin drwxr-xr-x 1 ken None 0 Aug 21 07:04 lib pm_11146255_moose_json/bin: total 4 -rwxr-xr-x 1 ken None 846 Aug 21 07:43 moose_json.pl pm_11146255_moose_json/lib: total 0 drwxr-xr-x 1 ken None 0 Aug 21 07:40 Local pm_11146255_moose_json/lib/Local: total 4 -rw-r--r-- 1 ken None 788 Aug 21 07:40 MooseJson.pm

    — Ken

Re: Framework for making Moose-compatible object from json
by NERDVANA (Chaplain) on Aug 20, 2022 at 14:17 UTC

    If you mean that you have Moose classes/roles available and you want to detect incoming objects in your JSON and call  ->new on the appropriate class, maybe also calling apply_all_roles to customize the object a bit, then you can do that using the JSON module's filter_json_object feature.

    On the other hand, if you mean that you want to create a brand new Moose class with one attribute for each of the keys seen in the incoming hash, and a metaclass that enumerates these attributes, you have a harder problem. The hard part of the problem isn't creating the Moose class; the hard part is that once you create a Moose class it remains in memory for the duration of the program, because classes are global. You would want to at least recognize when the set of attributes was the same as a previous class and re-use that class. But, a malicious client could connect to your service requesting infinite different attributes, and run you out of memory. A possible solution here would be to create the class with a DESTROY method that deletes the entire class when the one-and-only instance of that class goes out of scope.

    An additional problem is that some names are reserved by Perl or Moose. So if an object came in looking like

    { new => 1, meta => 2, BUILDARGS => 3, DESTROY => 4 }
    blindly creating attribute accessors for it would be bad. I think you could still access the attributes using the MOP, as long as you didn't create accessors.

    So if you only access the attributes using the MOP, then you might actually have a path forward, because you could make a subclass of the MOP classes which return dynamic info about the instance instead of static info about the class. Then you only need one Moose class, and it generates dynamic MOP on demand when you call  ->meta.

    Then, the final thing you need to ask yourself is whether all of this is actually solving your problem, or just creating more work :-)

      for each of the keys seen in the incoming hash
      No, I would consider, that key list are known in advance, and Class hierarchy is predefined. But I guess I will need some framework to quickly map json keys to class attributes. See example I've posted in a comment above...
Re: Framework for making Moose-compatible object from json
by LanX (Saint) on Aug 20, 2022 at 13:31 UTC
    Best you provide an SSCCE of what you want to achieve.

    Moose is compatible to use plain Perl classes so I'm not sure what the requirements are.

    The devil is in the details of the actual JSON semantics and what

    > adds object specific methods, some internal logic, processing" etc

    is supposed to mean.

    edit

    normally a converted JSON structure consists of a tree of nested Array's and Hashes and probably blessed Booleans.

    • Do you want Perl wrapper objects for special serializations?
    • Or a walker to search that tree?
    Please be more specific.

    Cheers Rolf
    (addicted to the Perl Programming Language :)
    Wikisyntax for the Monastery

      Generally I wonder, I am not the first one who met json<->moose mapping problem. There should be some existing solutions, and I should look at them and choose one that fits me better.

      But if I imagine some ideal solution for me it would be something like that:

      { "body_raw" : "This is <b>Entry 1</b>", "url" : "http://nataraj.nataraj.hack.dreamwidth.net/459.html", "datetime" : "2022-08-20 16:36:00", "entry_id" : 459, "subject_html" : "Entry 1", "poster" : { "display_name" : "nataraj", "username" : "nataraj" }, "icon" : { "url" : "http://www.nataraj.hack.dreamwidth.net/userpic/1/3", "comment" : "4", "picid" : 1, "keywords" : [ "3" ], "username" : "nataraj" }, "subject_raw" : "Entry 1", "body_html" : "This is <b>Entry 1</b>" }
      package Moose::MyCoolRestObject; use MooseX::JSONBase; has_json_string url => (); has_json_datetime datetime => (); has_json_int id => (); has_json_object icon => ("Moose::MyCoolRestObject::Icon");
      And then use it like this:
      my $obj = Moose::MyCoolRestObject->new($json); print $obj->icon->username;
      I guess I can write something like this, if nothing similar does not exist...
        > who met json<->moose mapping problem

        The bidirectional mapping looks like you are interested in an "object serialization".

        NB: Then the JSON structure will originally derive from serializing your OOP system.

        HTH narrowing your search! :)

        EDIT

        e.g. MooseX::Storage came up after a quick search.

        Cheers Rolf
        (addicted to the Perl Programming Language :)
        Wikisyntax for the Monastery

        ) Your question was originally worded the other way round.

        > OP > to create Moose-compatible object out of perl-hash that came from json parser.

        has_json_string url => (); has_json_datetime datetime => ();
        What is the significance of "has_json_datetime"? How is that different from
        has datetime => ( is => 'rw', isa => 'DateTime', coerce => 1 );
Re: Framework for making Moose-compatible object from json
by awncorp (Acolyte) on Aug 23, 2022 at 16:44 UTC

    I can't go into a lot of detail at the moment, but if you're open to using something non-Moose based (at least for inflating the nested data structures), Venus can do all of this for you via the "Venus::Role::Coerciable" role. If you want to stick with Moose you could try reverse engineering "Venus::Role::Coerciable" and using it with your Moose classes. Here's how to do it in Venus:

    package JsonData; use Venus::Class; with 'Venus::Role::Coercible'; attr 'icon'; attr 'url'; attr 'datetime'; attr 'subject_html'; sub coerce { { ..., icon => 'JsonData/Icon', url => 'Mojo/URL', datetime => 'Mojo/Date', subject_html => 'Mojo/DOM', ..., } } package JsonData::Icon; use Venus::Class; with 'Venus::Role::Coercible'; attr 'url'; attr 'comment'; attr 'picid'; attr 'keywords'; sub coerce { { ..., url => 'Mojo/URL', ..., } } 1;

    Venus::Class

    Venus::Role::Coercible

    "I am inevitable." - Thanos
Re: Framework for making Moose-compatible object from json
by nataraj (Sexton) on Sep 04, 2022 at 11:28 UTC

    Actually I started wiriting one MooseX::Embody::JSON and MooseX::Embody::JSON::Role

    This allows me to do things exactly I would expect it:

    package WebService::DreamWidth::Entry; use Moose; use MooseX::Embody::JSON; use WebService::DreamWidth::Icon; use WebService::DreamWidth::Poster; # this one is not part of Embody::Json yet has "tags" => (is=>'ro', isa => 'ArrayRef[Str]'); j_obj "icon" =>(class => "WebService::DreamWidth::Icon", is =>'ro'); j_obj "poster" =>(class => "WebService::DreamWidth::Poster", is =>'ro' +); j_attr "security" => (is => 'ro'); j_attr "body_html"=> (is => 'ro'); j_attr "subject_raw"=> (is => 'ro'); j_attr "subject_html"=> (is => 'ro'); j_attr "allowmask"=> (is => 'ro'); j_attr "body_raw"=> (is => 'ro'); j_attr "icon_keyword"=> (is => 'ro'); j_attr "entry_id"=> (is => 'ro'); j_attr "datetime"=> (is => 'ro'); j_attr "url" => (is => 'ro');

    This is early prototype, I need to play with it a lot, before releasing in even as beta, and some internals may be needs changing a log, but is is exactly the thing I think is needed for creating JSON-based REST clients.

    And yes, comments and suggestion are appreciated.

      I'm curious about why MooseX::Storage (mentioned previously in this thread) wasn't up to the task?
        I'm curious about why MooseX::Storage (mentioned previously in this thread) wasn't up to the task?

        Because I consider that case "object inside object inside object" is common case, and should be solved without any Moose code, just by mentioning "This is a subobject of following type". Like this:

        j_obj "icon" =>(class => "WebService::DreamWidth::Icon", is =>'ro');

        I even thinking about removing "is =>'ro'", making it default. In simple case developer should just list attributes and child objects. This is not case for MooseX::Storage