The oldest trick in the book is probably using a flag file in the filesystem with something like while(1){sleep 1; -f $FLAGFILE and do{ # open file if flag file exists, etc }}. The flag is created by the writer when the data files have been fully created. Have the reader process (your background program) check the flag and remove it when done.

Other ways to communicate, queue or synchronize asynchronous processes are signals or ipc (mmap to shared memory; perlipc). Check e.g. the recent mmap thread.

At a in-process level, you can use the same concepts based on threads instead of processes: threads. Be a bit careful in mixing multiprocessing and threads (-> signals...), and consider dedicating some thread(s?) for specific IO-tasks.

The bold words above should cover the more interesting keywords to use in searching for further ideas, modules, or frameworks.