This explains something I ran into recently. I stick with older versions of macOS for a variety of reasons, and El Capitan cannot rename or delete a file that has a NUL character in its name. Earlier versions of macOS could do so. This implies they rewrote the unlink function and made a rookie C programmer mistake in the process. Why rewrite unlink? The mind boggles.
What do you mean 'rewrite unlink'? As far as i know, none of the standard POSIX/C file-manipulation functions, including unlink(2), have ever supported NUL bytes embedded in file names. Path arguments to those functions are all NUL-terminated strings, as are exec*() process arguments, so it would be impossible to manipulate or reference those files using any of the standard APIs or command-line tools.
I can't think of how macOS could have previously supported NUL bytes in file names without those files existing in a separate universe that can only be interacted with by software that uses some proprietary Apple API. And if that's really how it worked, i feel like that was the real mistake all along...?
I do not know details here. But there is, in fact, a separate “proprietary” API for interacting with files on macOS, provided for backwards compatibility. This API exists because the POSIX file functions do not provide similar enough functionality to the earlier API. This API is called the File Manager. Documentation is hard to find on the Apple website, but you can find some fairly extensive documentation for it in the old Inside Macintosh series (and sometimes you can dig HTML or PDF versions of this on the Apple website, but whether this is possible, and the exact location, seems to change every couple years).
The way you would specify a file on old versions of macOS used a length-delimited string for the filename, and referred to directories by 32-bit IDs. This was packed into an FSSpec structure which was passed to most functions which operated on files. Being length-delimited, it is possible to pass a NUL byte.
From what I understand, the macOS kernel exposed some additional entry points for these functions to operate correctly.
Pre-posix macOS (previously known as ‘system N’ where N was the version number) did indeed use pascal calling conventions, so the posters assertions aren’t unreasonable.
