My usual plug here for using NetAddr::IP for parsing IP addresses and generating Cisco's wildcard notation rules is in order :)

Additionally, I've been doing this kind of analysis recently. Beware how you create and apply those rules, as the number of them could be overwhelming. I would suggest using some kind of automatic expiration time on the rules, so that they clean themselves automatically.

Best regards

-lem, but some call me fokat

So my first question is, what kind of data structure should I use to store the data collected

Sound like a job for a hash-of-arrays to me. Key is the IP address, and the elements are the number of errors and some way of tracking the time intervals between each error (perhaps the average interval is good enough?)

----
I wanted to explore how Perl's closures can be manipulated, and ended up creating an object system by accident.
-- Schemer

Note: All code is untested, unless otherwise stated

I tend to agree with hardburn, actively tracking the IPs in a hash-of-arrays is probably much more efficient than storing every single error in a DB and then calculating error-rates after the fact.

Perhaps a sampling mechanism could be employed. Pick a period (5 minutes for instance). When you see an error on a given IP, you store the IP, the time you saw it, and increment the error counter. Now as long as that IP's error counter is less than 5 minutes old, you continue adding subsequent errors to that counter. When the counter is 5 minutes old, you check the error count, decide if the error-rate for 5 minutes is exceeded and decide to allow or ban that IP. Then you remove that hash entry and start again with that IP.

When you decide to ban IPs, you can create another hash with IPs and the time they were banned. This hash is used to generate the list of IPs to ban. Once their ban is done (30 minutes) you can drop them from this hash and the program can remove the iptables rules.

You can dump these hashes to files at a given interval (or via a signal handler so your iptables generating script can log on, issue a kill with a given signal and expect your script to spit out the list of banned IPs).

i have half a dozen or so daemons that watch various streams of info and then block the evildoer through various means.

some general advice:

• if you go for a Database, use PostgreSQL instead of MySQL. Postgres has builtin datatypes for MAC addresses and CIDR notation so you don't have to craft fancy SQL to determine if an IP addresss is within a CIDR block (or worse yet, fetch all the ips and find the containing block in your code).
• look into Cache::Cache, Net::Patricia, Net::IP, and Net::Netmask. update: forgot about File::Tail

when you see a possible evil IP, check the Cache for your statistics on the IP, if there are none then create a Cache entry for your IP and start collecting the numbers. i keep a $ip_short, $ip_medium, and $ip_long counts with seperate Cache entries of varying lifetimes (short = 1min, med = 10min, long = 30min). then while processing each IP you fetch the short entry, add your numbers, and if they pass the short threshold you request a block, otherwise put the entry back in the Cache. do this for each of the short, med, and long counts. if you do make a block request, add a Cache entry for $ip_blocked so you don't keep requesting the same block over and over.

a block request simply puts the offending IP in a database with some info as to why it's being blocked. a blocker process wakes up every 5 minutes and checks the block requests and performs any that it finds. (for me that means external hosts get blocked in a filter on the border router, internal hosts on the Switch Fabric get their port disabled and their MAC address disabled (so they can't just move to another port), internal hosts not on the Switch Fabric get blocked at the router-port closest to them (this is harder and takes much more work).

using the Cache objects keeps you from having to do lot's of time calculations. if the short entry is there when you check, it's been less than a minute since their last failure... so you can easily catch the fast evil in the short entry, and the long slow evil in the long entry.