Pip and pyproject.toml have no way of helping you to get scripts to your system.
Pip doesn't really know how to install programs. Pyproject.toml is completely irrelevant to the problem. What pip can do is install (generated) files from the scripts section of the Wheel it's installing into the directory for executables known to your Python environment. In most cases this directory will not be on system path, and even if it is, you are better of not using this functionality, instead you'd need to rely on tools from your system packaging to install files there, so that the system packaging tools can track them, deal with conflicts caused by upgrades / downgrades, remove them, audit them etc.
> virtualenv
Whoa, this fossil is still alive somewhere? I think, you probably meant venv. virtualenv is a throwback to the Python 2 era. Not that its bad because of that, but you should probably warn your readers about this detail.
> pip vs Bazel or Buck
Are you sure you understand what these tools are supposed to do? pip installs Python packages. Bazel and Buck build (mostly Java) packages. The analogue in Python world to Bazel and Buck would be SCons, maybe setuptools.
In other words, pip doesn't know how to build Python packages. Sometimes it wants to build them (which is bad, and you should never do that), but it never does it on its own -- it uses other tools to do that, and the tools could be anything, setuptools, CMake, MSVC, rustc... whatever the authors of that particular library chose to use to build it. In particular, pip could, in principle, call Bazel to build a package (would be a weird twist, but not impossible).
On the other hand, tools like Bazel or Buck would usually use something else to install packages, if those are needed during build, eg. Maven.
Must be doing it much more and much better than you do.
I've started my familiarity with Python after Peter Norvig promised that Python can be an OK substitute to Common Lisp. That promise turned out to be a bold-faced lie, but learning some Python made me more employable, so, I'm not complaining. I've made my first steps using Python when Twisted was popular, there were "old-style" and "new-style" classes and you could raise whatever you wanted (not necessary an exception), setup.py files were written in such a way as to use distutil if setuptools wasn't installed.
I've also contributed to CPython (reported bugs mostly). Wrote a bunch of C, some C++, Rust and Go code that produces Python modules as well as contributed to pip, setuptools, conda... Again, mostly bug reports or small patches for specific bugs, but still.
At day job, my role is in infrastructure, which is mostly written in Python, so, I deal with stuff like Linux kernel to userspace interface, various system utilities, or cloud-related stuff, mostly OpenStack. Another aspect I'm involved with at day job is CI and packaging. Perhaps the utility I wrote that's seen the most use is one that deals with combining multiple wheels into a single wheel to speed up deployment. It's not sophisticated, but turned out to be very useful. Another popular utility is used to dismantle Linux storage so that it can be re-defined and re-assembled. What it does is it traverses /sys/block looking for various devices and connections between them, finds the right order in which these devices need to be stopped / removed / disassembled and does that. Again, this isn't very exciting, but turned out to be useful.
I find it solves the installation problem really well: each installed program gets its own virtual environment, but the single binary is still added to your path.
I would never want that. This is the exact opposite of what an installer should do. The whole point of using Python is to rely on the previously installed stuff: both the runtime and other libraries. If I'm making a Python program, I package it as a DEB or RPM (for work, this is what we support). I'm not saying these are great tools or are pleasant to work with, but I find the end result to be acceptable.
Similarly, I guess, I'd make an MSI or w/e is the modern way to install on MS Windows, if I ever have to. I don't know what's the equivalent is on other systems.
The separate environment thing ensures you get the exact versions of the libraries that are guaranteed to work with the tool, without risk of upgrading a library in a way that breaks something else.
I wonder if pipx and venv could grow functionality based around symlinks that allowed installations using the exact same package version to avoid having two copies of the files?
A compression based file system hack might provide a better result though, since it could optimize differences between two dependency versions with only minor changes.
> The separate environment thing ensures you get the exact versions of the libraries that are guaranteed to work with the tool,
1. I think it's the wrong goal.
2. It's easier to achieve that in ways that don't require bloat on my system.
It's much nicer for users if the library interfaces with its dependencies respecting the versioning rules: doesn't use undocumented, unreleased stuff, doesn't rely on undocumented side-effects etc. Only relies on what's tested and released. In this way, there's no need to be very selective about versions you have.
Unfortunately, this is not how the world is. The reality is that especially in popular environments like Python, you get a crapload of very low-quality libraries, with poorly defined dependencies, with people who don't understand the infra side of things and have convoluted dependency requirements. But, I usually try to fight back. If I absolutely have to have a library with convoluted requirements, I fork it and fix the nonsense. Or vendor it.
Another unfortunate quite ironic side-effect of this situation is that people are so dedicated to specifying nonsense requirements (eg. including patch in requirement specification even though Python doesn't even have a patching mechanism) is that popular programs used to install libraries are optimized for these absurdly specific requirements. I.e. it's faster to install requirements with pip or conda if you give them the exact list, preventing the solver from putting it much work. This puts people who want to make good libraries at a disadvantage because their libraries will take forever to install.
Which leads me to the following conclusion: if I want super-precise requirements, I don't need pip or conda. I can simply curl -o the packages I need, and it will be much faster and a lot more reliable.
> without risk of upgrading a library in a way that breaks something else.
I'm not afraid. I usually know what I'm installing. If it breaks, I'll fix it. I actually want to know when and why it breaks, so this is also an anti-feature for me.
> A compression based file system
Do you mean deduplication / CoW filesystem? Not sure why is this a hack. Compression in filesystems typically compresses individual blocks. It won't help you if different files have the same contents, the theoretical benefit comes from entropy within a file, not because some files share contents.
Using Python with Bazel is fairly common at big SV companies -- they use rules_python with it (https://github.com/bazelbuild/rules_python). It does rely on pip for grabbing dependencies but handles building modules and can integrates well with rules_docker/rules_oci for building container images from your code.
I honestly don't know why anyone would use that... as in what does Bazel do better than virtually anything else that can provide this functionality. But, I used to be an ops engineer in a big company which wanted everything to be Maven, regardless of whether it does it well or not. So we built and deployed with Maven a lot of weird and unrelated stuff.
Not impossible, but not anything I'd advise anyone to do on their free time.
Specifically wrt' the link you posted, if you look here: https://github.com/bazelbuild/rules_python/blob/main/python/... it says that only pure Python wheels are supported, but that's also a lie, they don't support half of the functionality of pure Python wheels.
So, definitely not worth using, since lots of functionality is simply not there.
Pip doesn't really know how to install programs. Pyproject.toml is completely irrelevant to the problem. What pip can do is install (generated) files from the scripts section of the Wheel it's installing into the directory for executables known to your Python environment. In most cases this directory will not be on system path, and even if it is, you are better of not using this functionality, instead you'd need to rely on tools from your system packaging to install files there, so that the system packaging tools can track them, deal with conflicts caused by upgrades / downgrades, remove them, audit them etc.
> virtualenv
Whoa, this fossil is still alive somewhere? I think, you probably meant venv. virtualenv is a throwback to the Python 2 era. Not that its bad because of that, but you should probably warn your readers about this detail.
> pip vs Bazel or Buck
Are you sure you understand what these tools are supposed to do? pip installs Python packages. Bazel and Buck build (mostly Java) packages. The analogue in Python world to Bazel and Buck would be SCons, maybe setuptools.
In other words, pip doesn't know how to build Python packages. Sometimes it wants to build them (which is bad, and you should never do that), but it never does it on its own -- it uses other tools to do that, and the tools could be anything, setuptools, CMake, MSVC, rustc... whatever the authors of that particular library chose to use to build it. In particular, pip could, in principle, call Bazel to build a package (would be a weird twist, but not impossible).
On the other hand, tools like Bazel or Buck would usually use something else to install packages, if those are needed during build, eg. Maven.