Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

Changes to the User Nodes ticker and introducing the NodeRep XML ticker

by demerphq (Chancellor)
on Nov 10, 2005 at 08:16 UTC ( #507312=monkdiscuss: print w/replies, xml ) Need Help??

Hi folks.

Performance reasons mandated a change to the user nodes xml ticker. Users with many posts were causing undue load on the DB server so we had to change it. The changes are as follows:

  • The ticker returns no more than 100 records at a time. You can use the limit parameter to request less than this number per fetch if you like.
  • New arguments for paging through the result are available. You may use the offset parameter to controll how far into the data set you want.
  • Additionally you can control the result set ordering by using the order parameter which accepts the arguments 'asc' or 'desc'.
  • Lastly you can request all new nodes since a particular node id by using the fromid parameter. This is the recommended mode of use as it enables you to write an client that querys in an incremental fashion, without requerying all the stuff that it has already seen.

Its well known to the gods that the primary existing use of the user nodes ticker is to monitor reputation changes in the users posted nodes. We would prefer that people stop doing this, as it requires fetching an awful lot of nodes. In order to get people to stop doing this we have created the noderep xml ticker which is designed to provide data on only those nodes that have changed since the last fetch. Please read the following list to see how it works:

  • On the first fetch the ticker will return the results on the last 24 hours voting.
  • On subsequent fetches the ticker returns any changes since the last fetch or the last 24 hours, whichever is more recent. The ticker stores the time since the last fetch by a given user internally. This means you cannot have two clients consuming data from it simultaneously. They wont get the same results. This is a deliberate design decision, so don't expect it to change.
  • You may not fetch from the ticker more frequently than every 600 seconds. Attempts to fetch more frequently will result in an xml error message being returned. This restriction is partly to keep he DB load down, but mostly to discourage people from writing "who downvoted me" tools. If we feel such tools are being produced the fetch period may be extended. Client writers should use the data in the 'info' tag of the result set to automatically control the interval between fetches.
  • In order to make designing new clients for this ticker easier the two restrictions above are relaxed somewhat for the first 50 fetches. In this time period you may request the data set whenever you like, and use the clear option to reset the internal state of the ticker to "never fetched before". This should allow client writers sufficient opportunity to get their clients right before the polling frequency restriction kicks in. (At request the gods may reset the counter if you can justify why.)
  • Important: as this is a new release we reserve the right to make any changes necessary to the feed as the circumstances arise. IOW, if it turns out that this ticker is as much of DB pig as the user nodes ticker used to be then the ticker will be modified or even disabled. Hopefully nothing like this will be necessary.

So now all you XP whores can use the ticker to watch your nodereps in something close to real time. Enjoy. ;-)

---
$world=~s/war/peace/g

  • Comment on Changes to the User Nodes ticker and introducing the NodeRep XML ticker

Replies are listed 'Best First'.
Re: Changes to the User Nodes ticker and introducing the NodeRep XML ticker
by thor (Priest) on Nov 10, 2005 at 12:16 UTC
    I don't personally use such things, but with the restriction on the interval between fetches, would it be possible to have some sort of "development" version that doesn't hit the db, but still spits out XML in the form that this ticker does? I know that if I were to create a client for this, I'd probably need more than 50 tries before the time restriction kicked in. Moreover, it'd be nice to continue development on such a thing and not have to deal with waiting 10 minutes in between authorized refreshes only to find that you forgot something.

    thor

    Feel the white light, the light within
    Be your own disciple, fan the sparks of will
    For all of us waiting, your kingdom will come

      When developing against web services, I capture the server responses to disk and replay them from disk. That way, I get exactly reproducible results and errors, and it also helps me to separate the gathering of input from the program logic, as I have to be able to easily replace a socket connection with a filehandle.

      If you're using any network based transfer and can install libpcap or Winpcap, using Net::Pcap to capture and feed your input makes replaying sessions very convenient, as long as they don't need any interactivity.

Re: Changes to the User Nodes ticker and introducing the NodeRep XML ticker (simple client)
by demerphq (Chancellor) on Nov 11, 2005 at 13:20 UTC

    Here is a simple client that maintains a Node DB from the user nodes ticker and updates it from the Noderep ticker. It populates the node db _first_ so if you have made a lot of posts, it could take a while before it picks them all up and starts polling the noderep ticker.

    Note the client is compliant in that it determines its polling period automatically based on the information in the 'info' tag from the previous fetch.

    Apologies to davido and bobf to not using their code for this, I wanted to see how long it would take to write from scratch (not too long :-)

    Update: I modified the code slightly as it blew up when only one node was returned by either ticker.

    ---
    $world=~s/war/peace/g

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: monkdiscuss [id://507312]
Approved by Corion
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others rifling through the Monastery: (3)
As of 2023-03-21 21:08 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Which type of climate do you prefer to live in?






    Results (60 votes). Check out past polls.

    Notices?