This is what perl is great for.

Probably the most direct approach would be to parse each line with a regex to get the parts you care about (IE not the timestamp). Create a hash where the key is the relevant part of the line and the value is a number that you increment every time you find the same message. Notice you're not keeping the whole file in memory, just one instance of each line and a number.

--Pileofrogs