Most of the time, I'm testing things that haven't changed.

Most of the time you are testing things which you think haven't changed. There's a difference.

If you really only want to test the bits which you think have changed, then just run those test files. How many files are there in your test suite? Just run the one which tests what you think has changed and only run the full test suite when the individual script passes. Or have the full test suite run on-demand but in the background so you can get on with other things. There are so many ways to approach this without going to the complexity of dependency-driven tests.