I've been using go modules in my company for several months now. Everything has "just worked." It's at least 1 order of magnitude faster than dep.
It's worth noting that go modules use a novel dependency resolution algorithm which is extremely simple to reason about/implement, fast, and produces more reliable builds than npm/bundler/cargo. That's why I was excited about it, anyway. It removes the ever-present NP-complete assumptions in this space, so from a computer science perspective it's extremely interesting.
I've heard/read this, but I can't tell what is necessarily novel about it...to me, it reads like old-school/boring Maven transitive dependency resolution.
(Not holding out maven as best practice, it's just what I know best in terms of pre-version ranges, pre-lock file dependency management, once those features become state of the art in ~2010.)
...that said, Maven does actually support version ranges; ~10 years ago when I last used it, either it didn't support them then, or we didn't use it, so perhaps that is why vgo seems so familiar. Or I just have a terrible memory.
Anyway, if anyone can correct me on my fuzzy assertion that "vgo is like maven w/fixed versions", I'd appreciate it!
Can you elaborate on why it’s easier to reason about than Cargo et al? I’ve heard a lot of theoretical criticism of Go modules for not taking Cargo’s approach so I’m surprised to hear an experience report to the contrary.
I can't do much justice to the topic in a HN post, but this post specifically describes Go module's dependency resolution algorithm and compares it against SAT solvers like Cargo's: https://research.swtch.com/vgo-mvs. Click the "Go & Versioning" link in the header, and you'll find a whole series explaining the design decisions.
From cargo's source: "Actually solving a constraint graph is an NP-hard problem. This algorithm is basically a nice heuristic to make sure we get roughly the best answer most of the time." (https://github.com/rust-lang/cargo/blob/master/src/cargo/cor...)
vgo's more-constrained specification for dependencies means there is exactly one right answer, and it can be easily and quickly calculated by both computer and human.
Whether or not this will turn out to matter in practice is still an open question.
Not very the parts that make it NP hard are allowing libraries to specify maximum versions (and other more complex version ranges). Most of the time libraries use minimum constraints (~) or (^) which allows the heuristic to work like go's algorithm. For rust, node, and other languages libraries can be imported twice as different versions (without requiring a major version renaming like go) this also allows the heuristic to have an out: if it reaches a really complex case it can just give you both versions. Beyond that package management is a barely disguised 3-SAT solver which we have good, fast solvers. There are definitely some edge cases, but when's the last time you ran any of the following package managers and worried about dependency solve speed? cargo, apt-get, npm (and yarn), dnf, zypper. IO far and away dominates the profiles of these programs, solver speed is basically a non-issue in practice.
It does. There are ways to mark a package as "only once" in the dep graph. For instance, C libraries are required to be marked in this way.
The only once constraint also has a nice out for the SAT solver, if you reach a conflict or something that can't be solved cheaply you just make the user select a version that may not be compatible with the constraints. Bower, dep, and maven work that way.
You anticipated where I was going, which is mutable state + multiple copies of packages seems like a recipe for trouble.
So, I'm not sure how happy I would be as a user if my package installer bailed out and asked me to choose!
Out of curiosity, how do you mark your package as "only once" in cargo? I tried googling, and didn't find an answer, but did find a bug where people couldn't build because they ended up depending on two different versions of C libraries!
It does make wonder if MVS will solve real pain in practice. :-)
> So, I'm not sure how happy I would be as a user if my package installer bailed out and asked me to choose!
Its definitely not a great UX, but at the end of the day the problem can only be solved at the language level or by package authors choosing new names. For instance in java you can't import 2 major versions of a package. Solving for minor versions having to bail out has been incredibly rare in my experience. I only see it when there's "true" incompatibilities, e.g.
foo: ^1.5.0
bar: foo (<= 1.5)
> Out of curiosity, how do you mark your package as "only once" in cargo? I tried googling, and didn't find an answer, but did find a bug where people couldn't build because they ended up depending on two different versions of C libraries!
I think its the `links = ""` flag. It may only work for linking against C libraries at the moment, but cargo understands it!
> It does make wonder if MVS will solve real pain in practice. :-)
Not by itself, the semantic import versioning is the solution to the major version problem, by giving major versions of package different names. Go packages aren't allowed to blacklist versions, though your top level module is. This just means that package authors are going to have to communicate incompatible versions out of band, and that the go tool may pick logically incompatible versions with no signal to the user beyond (hopefully) broken tests!
> Go packages aren't allowed to blacklist versions, though your top level module is. This just means that package authors are going to have to communicate incompatible versions out of band, and that the go tool may pick logically incompatible versions with no signal to the user beyond (hopefully) broken tests!
Yeah, it seems if the Go system ends up not working out in practice, this will be why.
But because of the minimal nature of MVS, you won't run into this problem unless something else is compelling you to go to the broken version. And by the time that's happening, you'd hope that the bug would've been reported and fixed in the original library.
It'll be interesting to see how it plays out in practice.
(Also, if I have a library A that depends on B, which is being reluctant or slow about fixing some bug, I can always take the nuclear option and just fork B and then depend on that. Basically the explicit version of what Cargo would do through deciding it couldn't resolve w/o creating two versions. But I think the incentives might be set up right that the easy/happy path will end up getting taken in practice.)
Packages are made up of modules, and modules can have global state. But doing so directly is unsafe, specifically because it can introduce a data race. Rust also does not have “life before main”, so it doesn’t get used in the same way as languages that do. I’m not sure if Go does?
(I replied but the reply vanished. If it reappears, apologies for the dup.)
Yeah, go has a magic function `func init()` which gets called before main. (You can actually have as many init's as you want, and they all get called.)
Probably evil, though so far it hasn't hurt me in the same way as, e.g., c++ constructors have. Maybe because it's more explicit and thus you're less likely to use it in practice.
1. vgo focuses on the wrong issue (if you're spending a ton of time resolving and re-resolving your dependency graph, maybe the issue is your build process).
2. vgo will get the wrong answers and/or make development much harder
I'd say there's about a 3% chance vgo ends up being a smashing success that revolutionises package management and gets copied by everyone else, a 30% chance that vgo works well for golang due to their unique requirements but has nothing to offer anyone else, and about a 67% chance it ends up being a failure and being scrapped or heavily revised to scrap the novel, controversial and (arguably) fundamentally broken ideas that set it apart from every other package manager.
But fundamentally, the reason the cargo people aren't copying it right now is that it doesn't even really claim to have advantages over cargo for rust. (There are some quirks in the golang ecosystem which mean you end up analyzing your dependency tree way, way, more than you do in basically any other common language. That makes speed important for golang, but for everyone else, it's almost meaningless.) "We make the unimportant stuff fast at the expense of getting the importing stuff wrong" isn't very compelling. :)
Of course, the vgo people would phrase it as "we make the important stuff fast and we get the important stuff right", so...time will tell. But don't expect anyone to copy this quickly; it remains to be seen if it'll even work for golang, and it'd need to be a huge step up from the current state of the art to make it worth switching for other languages and ecosystems.
> There are some quirks in the golang ecosystem which mean you end up analyzing your dependency tree way, way, more than you do in basically any other common language.
Basically, dep/glide do a bunch of stuff, including recursively parsing import statements because of How Go Works (tm). Other package managers don't, because they have lock files, and central repositories. Go expects you to just be able to throw a ton of raw code into your GOPATH and have it all magically fetched from github, which is super cool, but also very hard to do quickly, and not really something other languages are clamouring to support.
(A lot of attention has been focused on vgo's solver, and it is much faster, but the solver isn't what takes up all the time; the speedup from dep/glide to vgo seems to be almost entirely related to the changes in how dependencies are declared. Saving 10ms on a more efficient solver algorithm means nothing if the overall process is spending 12s grinding through slow disc and network access.)
And when you survey the language ecosystem, you see a lot of languages very enthusiastically committed to traditional package managers (with lock files) and centralised repositories. Cargo, composer, npm/yarn, bundler/ruby gems - recent history is full of languages happily moving in that direction. Go is an exception, and I don't see anyone actively copying that quirk any time soon.
The catch to the vgo approach required that no package in the ecosystem ever have even unintentional backwards incompatibilities, because you can't do anything other than specify minimum versions. Or, rather, it makes the resulting problems something that need to be addressed outside the scope of dependency specification and resolution.
When you just decide not to address a significant part of the problem, the solution becomes simpler.
You mean a bug? Because that's what that is and it is no different from any other bug, and like any other bug they are outside the scope of dependency specifications as they are unintended.
Maybe they should be. If you can make it work, fixing the bug seems like the obviously superior solution compared to letting it fester and working around it locally with incompatibility declarations, slowly degrading the ecosystem up to the point where you have lots of little islands that can't be used together anymore in a sane manner.
> If you can make it work, fixing the bug seems like the obviously superior solution compared to letting it fester and working around it locally with incompatibility declarations
Fixing the bug creates a new version. Unless you are going to create the mess of unpublishing packages or replacing packages with new different ones with the same identified version (both of which are problematic in a public package ecosystem), the fact that maintainers should fix bugs that occur in published versions doesn't , at all, address the issue for downstream projects that is addressed by incompatibility declarations in a dependency management system, even before considering that downstream maintainers can't force upstream maintainers to fix bugs in the first place.
I’m no expert, and I might even be very wrong, but I read the post about it and it seems to hinge on only resolving a minimum version and assuming all future packages with the same import path are backwards compatible. If I’m reading right it basically treats path/to/package and path/to/package/v2 as entiresly different packages.
It's worth noting that go modules use a novel dependency resolution algorithm which is extremely simple to reason about/implement, fast, and produces more reliable builds than npm/bundler/cargo. That's why I was excited about it, anyway. It removes the ever-present NP-complete assumptions in this space, so from a computer science perspective it's extremely interesting.