I see many pitfalls for a General Conversion API, or rather only a single, as it will be either too generic, because it has to accomodate everything, or too special for its title.

The pipelines I know are:

• AxKit - XML::SAX::Machines - an XML transformation pipeline
• My homegrown data transformation and reconcilation engine (written in Python for work). It takes input files, and converts them into hash-lists, the single filters operate on these hashes or the whole list, and the output is a report in the end. It has evolved far enough so that a reconcilation report can now be written in a mostly declarative style.
• A realtime movie/animation transformation engine, much like Adobe AfterEffects, but realtime. This is a toy idea I tinker with. Here, the problem would be that the consumer/producer relation is not entirely clear and must be arbitrated somehow, as you have a target framerate and a (different) framerate at which every producer works, and the arbitrator must make good decisions about where to spend the CPU resources, and every filter involved has to at least export some metadata on how it intends to spend resources allocated to it.

As long as you reduce your problem domain close enough, you can keep most of the meta-data problem out of your house, as the meta-data is common to all filters involved.

As soon as you go "general", your filter API can be reduced to the following interface:

  sub filter::can_connect_to {
    my ($self,$input) = @_;
    return 1;
  };

  sub filter::connect_output {
    my ($self,$next_filter) = @_;
    # ... do magic
  };

  sub filter::on_input_connect {
    my ($self,$input) = @_;
  };

  sub filter::process {
    my ($self, $data) = @_;
    # ... do magic
  };

  sub filter::describe {
    return "Magic filter";  
  };
[download]

This hides all the "arbitration magic" in the connect and on_connect methods, where both filters have to decide on something they can do to each other... I think that without a specific goal, you won't get far with a general concept.

perl -MHTTP::Daemon -MHTTP::Response -MLWP::Simple -e ' ;    # The  
$d = new HTTP::Daemon and fork and getprint $d->url and exit;#spider
($c = $d->accept())->get_request(); $c->send_response( new   #in the
HTTP::Response(200,$_,$_,qq(Just another Perl hacker\n))); ' #  web
[download]

I see many pitfalls for a General Conversion API, or rather only a single, as it will be either too generic, because it has to accomodate everything, or too special for its title.

I recall similar arguments being used when DBI's development got started. Sure, there will be big bumps to take, but I think this is worthwhile pursuit.

I would definitely investigate first which similar approaches have been done in the past: there are several packages capable of conversion between different image formats already. This API could serve as a glue for such conversions.

I think it would also be important to focus on capabilities rather than efficiency. There will always be more efficient ways to do specific conversions. The generic API is for ease of use, efficiency can be built in later if people have an itch for it.

Liz

It is an interesting idea, and probably worth a try.

I envision it as a set of XML tag describes things like:

• Steps
• The original format for each step
• The destination format for each step
• The tool used for each step

And a set of programs interpret those tags and execute them.

I suggest you spend time to make the tags right, and their meanings of really clear. Do it right from the beginning.

At the very least, for my own needs, I need a tool that can auto-detect namespaces and the stylesheets that can get them from a source format to a sink. The other transformation ideas may be desireable, but I haven't even sold myself on those.

I may end up just doing the XML namespace bit and leave it at that and perhaps continue to hypothesize about the other until I get tired of the idea or come up with a clever solution.

So, does such an API sound interesting? Why? Why not? What caveats would you anticipate? I've thought of several, but I want to make sure I know the risks. Anyone have a suggestion for an API or utility that is similar in Perl or another language? (That is, metaconversion tools, not actual converters like XSLT or ImageMagick.)

Just a quick thought, but you could build something on top of the Pipeline project. At least the infrastructure for asynchronous transformation declarations is there.

Cool. This is actually pretty similar to what I was thinking. Perhaps, this would work as a foundation. I don't know. I'm starting to backpedal out from under this idea as I've found another way around the problem using a faster/easier, but less idealistic solution.

I'll keep my brain on this one since it is still an interesting idea to me.

Thanks to everyone for your replies. Your suggestions have been instructive and constructive.