Modular malleability
This is a contribution to the Challenge problem: Fearless extensibility by the Malleable Systems Collective.
As superheroes know very well, with great power comes great responsibility. Malleable systems offer a lot of power to their users. In exchange, users have to take responsibility for the system they have tailored to their needs, since there is nobody else to blame. If you are the only user of your malleable system, the main cause of worry is that your adaptations might have undesirable side effects that you didn't foresee. You could have broken some functionality of the system that you don't understand very well. Or you might have increased the system's attack surface for malware. If you are the administrator of a system used by others, the changes that you see as improvements may also be a source of trouble for other users, for example by breaking compatibility for their extensions.
In the following, I will outline how a malleable system architecture could reduce the risk of such undesirable side effects. The risk will never be zero, or course. As long you have power over your system, you will also retain responsibility for it. A good system architecture can help you, but not make the system foolproof. I focus on malleable systems that make no fundamental distinction between developers and users, as oppopsed to two-level systems that offer "plugins" or "extensions" to users but reduce their power relative to the main code of the system. An example for the kind of system I have in mind is Emacs, whereas a Web browser is a good example for a plugin-based system.
They key concept in my design is modularity. To illustrate it, I will start from a top-notch malleable system that is not modular: Smalltalk. Its early implementations from Xerox PARC, up to Smalltalk-80 were small systems intended to be used by a single person and, more importantly, intended to be fully understandable by a single person (see Design Principles Behind Smalltalk by Smalltalk co-creator Dan Ingalls). Its later descendants, with the exception of Cuis Smalltalk, have abandoned this idea, aiming instead to become full-featured software development environments for use by software professionals. However, they retained the simple architecture of Smalltalk-80: one "sea of objects", one system dictionary for class names, a single namespace for methods, etc. You can change the behavior of fundamental values such as true just as easily as you can change your own code, and it's in fact quite easy to crash a Smalltalk system with such techniques (though it rarely happens by accident).
Let's look at how this lack of modularity can hurt in practice. All of the following mistakes and accidents have happened to me. Some out of ignorance of best practices, when I was a Smalltalk newbie. Others accidentally, when I wasn't careful enough when making changes to my code. Yet others because I wanted to change parts of the system that I didn't understand them well enough to foresee the consequences of my changes.
First example: you have some method in your code that you want to rename. In Smalltalk, you don't do that in a text editor. A Smalltalk system treats code more like a database, and offers specific tools for tasks such as renaming. So you tell such a tool to replace the old method name by the new one. This is a global operation on the system. If someone else has used the same method name in other parts of the system, it becomes part of the operation, with unforeseeable consequences. The renaming tool shows you all the affected locations, and lets you select which ones to change, but it's easy to make a mistake especially if the list is long. A good renaming tool will let you restrict the operation to selected packages (which are groups of classes that have been added to the system in a single operation), but that's not always the restriction you actually need.
Second example: you change your code and break some code elsewhere that depends on yours. This can happen in any programming system, of course, but in Smalltalk it happens more easily because there is no notion of an interface definition between different parts of the code. Everyone's code is just a bunch of classes added to the system. You cannot easily know what features of your code someone else relies on, nor signal clearly to others which features of your code you consider stable and safe for others to use.
Third example: you want to add two packages to your system that depend on the same dependency but requiring different versions. You cannot have multiple classes with the same name in your system, so this is impossible. What actually happens if you try is that the version loaded last replaces earlier versions, breaking the code that depends on that earlier version. This is a classic example of dependency hell, a problem shared by most programming systems, malleable or not.
My proposal for reducing the risk of such events is to introduce a hierarchical module structure for everything: functions, classes, method names, and maybe even objects. A module is roughly a subsystem, a well-defined part of the system that has a well-defined interface. "Hierarchical" means that modules can themselves contain submodules. The submodules of a module would be an implementation detail, i.e. not part of the module's interface.
One safety feature provided by such modules is that each module can be locked or unlocked. Locked modules can be used and inspected, but not changed. You have to unlock the module first, which should make you think if that's really what you want to do. In a multi-user system, unlocking a module would be subject to authorization. Modules could also be locked against read access, for example to be used as permission tokens in an object capability system.
Another safety feature is the isolation of submodules. If both module A and module B use the same submodule X, then the two copies of X are independent. A and B could use different versions of X (a big step out of dependency hell), and C could be locked in A but unlocked in B. If you want to hack on X, you wrap a it in a test module, with the rest of the system continuing to use a stable release.
As an example, suppose I am writing a Zotero client for my malleable system (which is something I have actually done in Smalltalk). My client lives in a new module, initially without any interface, meaning that no other code can interact with it. It's there only for the "end user", accessible via a REPL or via GUI elements. The Zotero Web API communicates via JSON data, so I need a JSON parser. That's my first submodule, taken from an existing module library. Normally I'd keep it locked, but I might temporarily unlock it if I suspect a bug in it, or if I want to tweak it for debugging my own code. The HTTP client for accessing the Zotero API is dealt with in the same way. I also want my client to have a GUI, so I add a suitable GUI toolkit as another submodule. My development universe is my code plus a handful of submodules. While I am working on my Zotero client, I don't touch code anywhere else, though I may well look at other parts of the system for inspiration, e.g. other modules that use Web APIs and JSON data. The system's development tools therefore grant read-only access to the whole system.
Some of these ideas have been implemented elsewhere. Common Lisp, for example, has a package system that permits creating namespaces for variables, functions, classes, etc. There is also a non-standard mechanism for package locks that has some of the features of my module locks. Moreover, today's implementations are file-based, with a source-code file acting as a unit of code for editor operations. That provides another level of protection against accidental modification, though not malicious attacks.
However, I am not aware of any malleable system that implements my idea fully, nor any that would allow me to retrofit hiearchical modularity easily enough that I'd be tempted to try. There is of course a good chance that some out-of-mainstream system does provide everything I am asking for. If you know one, please leave a comment!
One obstacle is that most systems have a global namespace. If you want hierarchical modularity, then every name must be resolved in the context of the containing module. Names that resolve identically everywhere are a problem. In Smalltalk, as in Emacs, all namespaces are global. In Common Lisp, it's only the namespaces for packages and systems that are global, but that's enough to make hierarchical modularity difficult to implement.
As an example of a system architecture without a global namespace, consider Nix or Guix, which are Linux systems that abandon the global namespaces of standard distributions, i.e. global directories such as /bin or /lib, for getting out of dependency hell. They let users define any number of software environments, which can be attached to a user account or run as containers. In the latter case, the environments are completely isolated from each other. Guix (and maybe Nix, which I know less well) allows the creation of containers from inside another container, making hierarchically structured containers possible. Guix containers thus come very close to my submodules in terms of modularity and safety features. But since they live in the realm of Linux processes, at the level of binary executables, they cannot be considered a programming system, let alone a malleable one. They are meant for deploying software, not for creating or modifying it.
A second obstacle is global data types. In Smalltalk, an object is accessible from everywhere in the system. It carries its class and thus all of its methods with it. In a modular ystem, if the interface of module A hands out objects defined in submodule X, then it exposes implementation details of its particular incarnation of X. This problem is not limited to object-oriented systems. Any system that attaches names to data types and interprets them at the system level, in whatever way, has to deal with similar issues.
One solution is to restrict data types used at interfaces to a fixed set of foundational data types that are common to all modules. That comes down to something like the JSON data model. That's what Web APIs are based on, so it's certainly doable. And the approach is well aligned with the idea of malleability, because glue code between modules can easily tweak such data items. Any type checking, static or dynamic, must be structural rather than nominal. The main disadvantage I see is restrictions on generic programming techniques.
Another solution could be to allow modules to adopt the data types of their submodules and label them as their own for the outside world. I haven't seen anything similar implemented anywhere, so it's safe to suppose that there are problems with this idea that I didn't spend enough time searching for.
More generally, the kind of modularization I propose here raises many questions of interface design. Some of today's popular techniques become impossible, and many others may need to be revised for pragmatic reasons, for example because they lose their power or convenience. In exchange, new options that weren't possible before will likely be discovered. It may turn out to be necessary, or at least desirable, to have explicit language elements for interfacing modules (see e.g. this paper). In other words, exploring this space looks more like research than like design or development.