http://qs321.pair.com?node_id=73030

vroom has asked for the wisdom of the Perl Monks concerning the following question:

I was trying to come up with a Perl project to work on and I thought of developing a system for building agents which would be able to be run by the cron and then gather and process information. I'd like to keep the framework as generic as possible so it could be used for a variety of things.

An agent will get a listing of data, check which items are already stored, and add new entries to the database as necessary. A user could then mark each item as being of interest, or delete it. Some types of agents might require a user to answer a series of questions about each item the first time they view an item. This information could be used to query and filter the data later. The agent could do some of this automatically or use a heuristic to rank each item as they come down the pipe.

Different driver programs could be used to provide different interfaces for the information stored within the database. A command line interface, and web based interface could then manipulate the same set of data.

I'm interested in hearing ideas about how to best set up the object framework for this system. I'm also interested in hearing ideas for additional features.

vroom | Tim Vroom | vroom@cs.hope.edu

Replies are listed 'Best First'.
Re: RFC on a system for automated agents
by AgentM (Curate) on Apr 17, 2001 at 08:04 UTC

    Being an automated agent, I feel qualified to answer your question.

    This project seems to have alot of similarity to CGI::Application in which "states" are assigned to various user-interactions which may be intertwined. But it seems that you want your "agentm" to also be able to have drivers for other display methods. Correct me if I'm wrong, but perhaps you're looking for the database entity "view" in which an SQL query is stored for later use and data sets are constructed- perhaps for different users or for different requests. Perhaps you're looking for some answers where to make divisions in your implementation so that your model is extensible later? I see three basic steps:

    1. Info retrieval (db query)
    2. data processing (agentm at work)
    3. user-level representation

    This may seem a little vague or cliche/useless, so each section deserves some discussion. 1. This info retrieval, depending on the complexity of the app, may involved stored queries- in fact, I recommend it. The SQL queries may even be built from the user input as you request on the fly. This is simple to implement and the best way to lead into 2. which may include templates, sorting, and your heuristics. Such an agent would, of course, be a module and be "hot-swappable", i.e. this running instance needs Agent #X while the next one requires #Y. Essentially, you would need an Über-Agent watching the under-agents perform their regular duties. I would make this Über-Agent an IPC-reachable daemon, so that it has the potential to take have a command-line client and other programs may check on various stati, or request permission for a "session". Obviously, signals won't cut it. You say that processing may take place later- that's fine. I probably wouldn'y push it as a cron job since that incurs an amount of overhead and is generally unnecessary for one time tasks. Instead, allow your Über-Agent to handle it. 3. Obviously, the agent-filtered data will need to be fed through various lines, whether it be CGI or Curses. For this task, I might recommend XML, so that your end graphical clients might easily pick up on the what's to be displayed. In fact, this would allow for later innovation in the area of remote GUI clients.

    You'll probably come to the conclusion that the agent's design will eventually be built around the incoming data instead of much else- at least that's how it sounds to me. I see incoming data being individually processed by a human- correct me if I'm wrong. Along those lines, it might be worth reconsidering whether you actually want a web interface since it is grossly limited in its ability to interact with the user. Eventually, it would become the "dumbed-down" interface. I urge you not to limit your GUI drivers to this limitation since you might eventually find that the command-line interface or even the X/Gtk/Tk/whatever interface may be easier to use (heck, I'll say it, how 'bout a Java-applet interface? Still a "web page", right?)

    Now that I think of it, your SQL-level queries could actually be a subset of the agent, but this MAY limit you later when it comes to more robust agents that do something other than crunch numbers. Either way, DBI shortcuts are the way to go. But like I said, keep an extra layer of abstraction between the GUI and agent.

    Additional features? Well, I already have a genie agent crunching my data before I view it, could there be actual user-level controls for this agent? Perhaps this is unrealistic, but if the agent is designed as a base module built under an easy to understand Perl interface, certainly it's not unimaginable that the user could build his own agents including heuristics or whatever else is used to build the end-user-result. And if you get the GUI-independent display off the ground, that could be a hit module in itself. This actually sounds alot like what I'm working on now- only more impressive. Good luck! Update: Perhaps DBIx::Renderer could find a use in your program?

    AgentM Systems nor Nasca Enterprises nor Bone::Easy nor Macperl is responsible for the comments made by AgentM. Remember, you can build any logical system with NOR.
Re: RFC on a system for automated agents
by idnopheq (Chaplain) on Apr 17, 2001 at 07:09 UTC
    Well, portability will be an issue. Need to find tools to run for *NIX and Win32 and other platforms. Re: Scheduling at NT has a link to a perl implementation for cron, workable on a Win32 platform. The PPT will give alot of the command line tools. MySQL can be the database backend.

    Something to consider on the heuristic/ranking end, so far as implementing such a system, is the GNUS mail/news package for EMACS. Granted, it is LISP, but the logic could easily be ported over (or so I hope after such a grandiouse statement). Very similar to what you describe from a messaging background.

    A Tk interface is worthwhile as well as the command and web interfaces.

    The proposed back-end querying could save us corporate slaves hours in reports and time tracking, not to mention better inventories, etc.

    Please summarise the comments when you get them collected, in your copious free time.

    HTH
    --
    idnopheq
    Apply yourself to new problems without preparation, develop confidence in your ability to to meet situations as they arrise.

Re: RFC on a system for automated agents
by arhuman (Vicar) on Apr 17, 2001 at 11:25 UTC
    To my mind agentM already summarized pretty well :

    • Data gathering
    • Data processing
    • Result feeding

    I would just add some features which could make this agent really versatile :
    • A kind of scheduler to manage execution of the agents (make hem sleep, wait for another info...)
    • Add a 'state' to the agent (beccause your log reader agent shouldn't read logs the same way when everything is normal as when there's an ongoing intrusion...) for managing behaviour AND/OR priority.
    • The code of the Agent should be easily AND securely upgradable (to dynamically/remotely add new features)


        "Only Bad Coders Badly Code In Perl" (OBC2IP)
Re: RFC on a system for automated agents
by Anonymous Monk on Apr 18, 2001 at 01:29 UTC
    The AAFID2 system is an IDS based on agents written in Perl. Check it out for their framework.

    I'm just getting to know more about agents now, and coming back to Perl scripting after a long time (I did one VERY simple script a few years ago). Right now I'm writting a Perl version of an old kids entertainment program I saw years ago :)

    Oh: Links to AAFID:
    http://www.cerias.purdue.edu/homes/aafid/

    Come to think of it... its creator might actually end up reading this.