Actually the algorithm presented is the real one I am using. I don't have access to the data on the database level. I am profiling the script right now and checking the bottlnecks. Also I am trying to use an intermediate tmp database as Jenda said, and trying to use Redis as well.