I suppose you are not interested in some technical tips to speed up remove_unwanted because most time is consumed within the ->$filter()s?

Like avoiding array-copies and faster method-calls?

Well hard to tell with the infos given, but the filters could dynamically record their performance in previous runs and you sort them accordingly.