You expect open to identify and reject all invalid names. It is not surprising that it is not always able to do this. A far better approach is to write a function that tests the name against the specification https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file#naming-conventions, for windows file names.