Why Language Specific Package Managers Exist

This post has moved! Click here to view the new home of this post.

If you are a developer for any language but C, you have probably seen language specific package managers; pip, luarocks, gem, easy_install, the list goes on. But while they may be convenient in the short term I will explain why they suck, and why they are necessary.

This will be a very long and drawn out post, but please stick with me. I have thought about focusing on one area of the topic but that doesn't do justice and doesn't allow me to complete my argument. So get ready for the whole story.

Why they are needed

Language specific package managers let you get the freshest dependencies for your project. The core problem here is that the library authors don't have a easy way to push to all of the distributions repos. So they use a language-specific repo that everyone can access.

Why they suck

Sounds good so far, right? Well I have covered the one advantage, everything from here on in is why they suck.

• They don't update with your system.
• They don't have many features.
• They don't integrate with your system.

They don't update

Let's kill the easy issues first. Language specific package managers don't update with your system. When I run pacman -Syu I expect everything to be up to date. With all of these packages I need pacman -Syu; gem update; npm update; ... I think you get the idea. This is the thing I miss the least about windows, but with these package managers I am back to updating 50 programs every time I log in.

They don't have many features

These programs are usually garbage when compared to the real packages managers that drive unix systems. It sounds weird but using a package manager to check what program "owns" a file is fantastic, I haven't seen one language-specific package manger that can do this.

They don't integrate with your system.

Okay, you have an awesome app and want to distribute it. Great! put it in package managers, but now you have a problem, your dependencies are in a language specific package manager, and you can't access it from your distribution's one. So you are going to have to repackage your dependencies. However, when you try to install your dependencies they conflict with the files that were installed by your language-specific package manager. This has to be one of the most annoying things to happen.

The Solution

There are two main problems that have to be solved. Unfortunately, they will require huge changes to how most package managers work. But, if it happens we can finally get rid of these language specific package managers for once and for all. If we can write a new package manager, and get all distros to use it the world would be a better place.

Easy Publishing

We need to allow the developers of software to be able to publish easily. This comes down to automated, no review publishing. This has its own problems though. We can expect the repo to become large very quickly. So the current format of downloading an entire list of packages has to go. This will mean that searching is no longer a local operation, but honestly searching is pretty useless if you can't install anything so you are probably going to be online anyway.

Versioning

This is why a global package archive won't currently work. People suck at versioning packages. This hasn't changed in forever, if you break compatibility, bump the major version number, if you add a feature, bump the minor number, if you change anything, bump the patch. Simple. If you need a more in depth explanation see semantic versioning. Basically, if your program works with library foo version a.b.c, it should work with any other version of foo a.x.y, provided that x >= b. In other words, if the major is the same, and the minor is the same or greater it should work, the patch number doesn't matter for compatibility just for identifying releases.

The other problem with versioning is installing different versions side by side. If you design your program well it should work side-by-side with any future and past versions. The easiest way to do this is to treat the major version number as part of the applications name. Don't use the directory /usr/lib/foo/, use /usr/lib/fooa/ where a is your version number. That way foo 1 and foo 2 can live side by side, and applications can use the one they need.

If you follow these simple rules versioning is no longer an issue, and if versioning is not an issue we don't need all of our programs to use the same version of libraries and we can install the same programs on different distros without issues.

Putting it all together

So what does our mythical package manager look like? There are only two differences between traditional package managers, the package list is not stored locally and the major version number is part of the package name. I will write a post on the ideal package manager in a bit, but at least we have covered the issues that language-specific package mangers "solved".