Let's not quibble too much about the details.

The main point is that using SQL especially in conjunction with a real DB is definitely "on the table" as a reasonable solution for the OP's problem. So I don't think that is OT at all. This approach will scale well into much, much larger files. Functionality like "build histogram" and "print only the first time" are heavily optimized and a real DB is pretty smart about how it uses the memory available to it.

I don't think we have heard back from the OP about requirement questions, so this is really just an exercise for our own edification and enjoyment.

so this is really just an exercise for our own edification and enjoyment.

So let me just enjoy some more light quibbling:

Hipp, SQLite-man extraordinaire:

"SQLite was a spin-off of postgres, a conceptual fork"

"We continue to look up at postgres as our mentor"

(which makes sense, why invent the wheel? I saw (just a few days ago) that the SQLIte devteam/Hipp are implementing some JSON operators, taking the example from postgres. Good!)

More fun: a youtube talk by Hipp (admittedly talking to a postgres-audience, at PGcon 2014):

PGcon 2014 talk by Hipp

Well....I don't know about the postgres foundation for SQLite...check this interesting story out from Richard Hipp: Origin of SQLite

My SQL-Foo is weak. OMG this can get complicated!
As an example, I present jaro-winkler.sql - this is so far above my SQL pay grade, I get a nose bleed just thinking about it!:

Read more... (4 kB)

I have a tutorial on postgres that I am slowly working my way through. SQL is a complex subject. Admin of a DB is a complex subject. I have certain tasks that I want to learn how to do in SQL. But this is very much a "work in progress". I am now able to do simple SQL tasks and I have found that the combo of Perl + SQL is super powerful. SQLite is my "go to" DB because there is no admin or installation hassle.

I like this Hipp quote: "Fortunately, I didn’t know any experts — and so I did it anyway.”