Conviviality in computational science

Convivial technology was defined by Ivan Illich in his 1973 book "Tools for conviviality" as technology that supports a convivial society, which is a society that strives to grant each of its members as much agency as is possible without infringing on other members' agency. Conviviality is thus about equality, about the absence of dominance relations. Convivial technology is shaped by its users according to their needs, rather than being controlled by entities such as companies or governments, which then derive power over the user base by exercising control.

One of Illich's examples is transportation, with bicycles being convivial whereas railways and cars are not. Cars in particular have turned into what Illich calls a "radical monopoly": a technology that imposes itself on everyone. Once a society has adapted its landscape and infrastructure to cars, walking or cycling become insufficient as a means of locomotion for most people, if only because typical distances are now typical distances for driving, not walking. Moreover, the total societal cost for car-based mobility is enormous, if you count in the cost of road construction, traffic accidents, environmental pollution, and much more.

A recent paper entitled "Conviviality for Digital Degrowth", by Sophie Quinton and Jean-Bernard Stefani, discusses how today's digital technology is not convivial, and outlines how this could change as part of a transition to a degrowth society. It motivated me to finally write down my personal story, which is about something much more modest: the conviviality of digital technology in scientific research. It's something I have been thinking about for thirty years, even though I wasn't aware of Illich's work and terminology until recently.

Let me start with the observation that most pre-digital technology in scientific research is convivial. Theoretical tools (theories, models, etc.) are developed and evolved completely inside the scientific community and belong to no individual nor any institution. Scientific instruments and experimental setups are designed either by scientists, or explictly for scientists and in close collaboration with them. Neither kind of tool is controlled by outside entities, with the possible exception of very large instruments such as CERN. Nobody can decide that scientists may no longer use NMR spectrometers, nor that they have to replace all pre-2000 microscopes by new ones. This has changed with the adoption of digital tools and the integration of digital technology into scientific instruments. Theoretical tools are now often software, whose complexity makes its behavior inscrutable to its users and puts them at risk of losing their tools to software collapse. Scientific instruments increasingly rely on built-in computers that create exactly the same issues. Finally, digital technology has enabled industrial-scale production of data, e.g. in DNA sequencing, and that technology is itself not convivial either.

Conviviality matters for science for multiple reasons. One of them is epistemic: if you want to derive knowledge from your work, you need to know exactly what you are doing, and that includes a detailed understanding of your tools. Moreover, research is much facilitated if you also have the inverse: the ability to create a tool that does exactly what you want to do. And since science is a collective activity, in which participants critique and build on each other's work, the understanding of tools needs to be shared inside a discipline. There have always been limits to this shared understanding, in particular concerning specific physical devices or unique experimental setups, but building shared understanding on a best-effort basis has always been one of the tacit underpinnings of science. This best effort has been abandoned in the digital era, as I discuss in an analysis of trust issues with scientific software,in which conviviality plays an important role.

When I started doing computational studies of colloidal suspensions in the late 1980s for my master's degree and then my PhD, research software was still quite convivial. Like most PhD students, I wrote medium-size Fortran programs, which ran on any computer with a Fortran compiler, from the Atari ST I had at home to the Cray X-MP that I used for production runs. Other scientists could read and understand my code in a few days, given sufficient motivation, and I know that some actually did, because I received questions from them by e-mail. It was also quite common for PhD students to look at and comment each other's programs. Publishing software was still exceptional, but publication venues did exist, and I ended up publishing the main code library underlying my work in low-Reynolds-number hydrodynamics in 1993. Unfortunately I didn't publish, nor properly archive, the small bits of code that did the actual computations for concrete specific systems, and that is why the results of my papers aren't reproducible any more. But the library still works exactly as it did in 1993, and still finds new users.

When I moved on to a postdoc in another field, biomolecular simulation, I discovered a very different world. There were only three big simulation programs that everybody worked with: AMBER, CHARMM, and GROMOS. Only a very small number of researchers understood them in detail and could modify them. Everyone else computed whatever the software allowed them to compute, rather than what they actually wanted to compute. But even the correct use of the software was a challenge if you weren't in personal contact with the development teams, as documentation tended to be incomplete and outdated. Biomolecular simulation software was clearly not convivial, an observation that I attributed to the complexity of the underlying theoretical models. I was also finding out about the politics favoring the concentration of power over software, but I didn't make the connection at the time.

The objects of biomolecular simulations, proteins and nucleic acids, were much more complex than the hard-sphere colloids I had studied in my PhD. Managing protein structures and the force fields defined on them in Fortran 77 is difficult and laborious. Maybe we could make biomolecular software more convivial by using a high-level programming language? That idea lead me to discover the Python language, become a founding member of the Matrix-SIG that developed Numerical Python, the precursor to today's NumPy, and write one of the first scientifc libraries in Python, the Molecular Modelling Toolkit (MMTK), first published in 1997.

From a technical point of view, MMTK did enable convivial biomolecular simulation. I have been in contact with various researchers, mostly PhD students, who implemented new simulation methods on top of MMTK and shared their work as add-on Python modules. However, I also found out that the majority of researchers in my field didn't care about conviviality at all. The power gradient between the big groups that developed the main software packages and the smaller groups of users was part of the research system, interwoven with apprenticeship relations, grant reviews, etc. Most researchers didn't choose a software package on its scientific or technical merits, but on the political merits of joining its user community. Among Illich's five threats to conviviality, I observed polarization and radical monopoly. As an illustration of the latter, some PhD students who contacted me with questions about MMTK asked me not to talk to their supervisors about their use of MMTK, because "for political reasons, I am supposed to use software X".

An individual or a small group cannot hope to address the social issues that encourage dominance structures over conviviality. Conviviality can only happen if a majority of a community adopts it as a value. What small groups of people can do, however, is develop and use convivial tools at their modest scale, to demonstrate that it is possible, and to provide a model that others can learn from if they want to. I was fortunate enough to have a stable position in French public research that allowed me to maintain this attitude in spite of its risk of reduced productivity. What I hadn't expected, however, because I didn't know about Illich's work yet, is the destruction of conviviality from the outside that followed.

The history of the scientific Python ecosystem is an interesting case study for Illich's conviviality framework. He describes two watersheds that institutions and technologies pass through as they gain in importance:

Any industrialized institution will go through two watershed moments. At first, its progress provides clear and substantial benefits to society. But second, its overdevelopment begins to run counter to its original goal and in fact becomes destructive to society.

Python for science reached the first watershed around 2000, only five years after the first release of Numerical Python. There was a solid foundation consisting of Python, Numerical Python, a few general-purpose utilities (plotting etc.), and domain-specific libraries for a few disciplines. Researchers could convivially develop and share Python scripts and modules, including if necessary so-called "extension modules" written in C or Fortran for performance.

Five years later, development shifted to NumPy, a new project aiming at a unification of Numerical Python and its offshoot numarray, which catered for different application domains with different priorities. One of NumPy's explicit goals was to encourage further growth of the user community, by making it more easier to learn for users of its main commercial competitor, Matlab. That was the point at which I started to feel uncomfortable with the ecosystem's direction. Numerical Python had a small and consistent API, inspired by APL. NumPy added an alternative API inspired by Matlab, and made breaking changes to the API inherited from Numerical Python. This meant imposing adaptation work and a higher cognitive load on existing users for the sole benefit of attracting new ones. Growth took priority over the qualities that make software convivial.

The second watershed was reached between 2010 and 2015. Due to a combination of growing ecosystem complexity, growing corporate influence on development decisions (Google in particular became an important sponsor), and the rise of a breaking-change version of Python (Python 3), the scientific Python ecosystem flipped from a stable infrastructure for research projects to an unstable software layer whose frequent breaking changes required researchers to invest more and more time just to keep their code in a usable state. Conviviality was lost.

In the following years, the Python developer community first encouraged and then increasingly forced authors of Python software to migrate to Python 3. In the FOSS spirit, Python 3 should have been considered a fork of Python 2, and both versions should have been allowed to coexist for as long as there were people willing to maintain them. But many people rightly recognized that this would have split the Python community into two competing factions. The bolsheviks, supporting Python 3, decided to kill Python 2 by various means, including highly questionable methods such as the Python 3 Wall of Shame, an online pillory listing projects that had not yet made the migration. This was possibly the most destructive event in the history of FOSS, and in particular a lot of domain-specific research software, the kind that only a handful of people would ever have heard about, was made unusable. Today, scientific Python is a typical industrial software product that happens to be free (as in beer). It is a good support for large corporate libraries such as PyTorch, but no longer a good choice for typical research teams that don't have the resources for dealing with high rates of tech churn (Illich's obsolescence). It is now more difficult to run a five-year-old Python script than a 40-year-old Fortran program, and even if it runs, it may not produce the same results as it did in the past.

My own MMTK library became practically unusable with the demise of Python 2. Porting it to Python 3 would have been a major effort, and I wasn't motivated to do that work. It would have been difficult (e.g. check line by line for divisions whose semantics had changed) and laborious (the C extension modules, written for Python 1.4, would have to be rewritten from scratch). But most of all, it would have been only the first step into a treadmill of continuous software collapse and repair. Together with a handful of colleagues that depended on MMTK, I looked for funding to have someone else do a port to Python 3 and maintain it, without success. MMTK is now a museum piece. You can still run it via reproducibility infrastructure such as Guix, but it is no longer a reasonable basis for new research projects.

The scientific Python ecosystem is the example I know best for progressive loss of conviviality, but the phenomenon is much more widespread. For another illustration, see the historical account of developer-user relations in computational chemistry by Wieber and Hocquet (preprint) that outlines how the conviviality of computational chemistry in the 1960s was lost as licenses, limited access to the source code, and ultimately the transition to software as a service increasingly restrained the agency of researchers.

After the end of my long and ultimately failed Python-for-conviviality experiment, I have been playing with a few other ideas for convivial computational science. One of them is Digital Scientific Notations. This is mostly a new label for what computer scientists call formal specification languages. Mostly but not quite: no existing formal specification language I know of would qualify as a Digital Scientific Notation, simply because existing languages were made for different application scenarios. And that's why I designed my own Digital Scientific Notation, called Leibniz, for my experiments.

The basic idea is simple: the human-computer interface for many aspects of computational science should not be code, but specifications. The relation between specifications and the code that implements them is roughly the same as the relation between a set of mathematical equations and a function that solves them (see here for a longer explanation). Specifications are often simpler than their implementations, and in general more modular: you can just throw any two specifications together, assuming coherent notation, and you get a new specification (which may or may not be useful). Researchers discussing computational models would never have to leave the level of specifications, leaving the technicalities of implementation to specialists (software engineers) or to computers (if you think "AI" now, you are not wrong but there are also much older deterministic techniques to "solve" specifications, such as the Bird-Meertens formalism). Leibniz is designed to resemble mathematical notation more than programming languages, hoping that people will feel more familiar with it. But I am not yet at the point of having done real computational science in Leibniz. For now, all I have is implementations of toy problems.

My second conviviality-related project is HyperDoc. It addresses the problem of scientific publishing in the digital era. The idea that data and code should be published along with an article is almost mainstream by now, but most people imagine three different entities (paper, code, data), published in different places and at best linked to each other. But in a convivial setting, code is written primarily for humans, not machines. It should be part of the paper, or part of what replaces the paper, and reviewed exactly like a paper (see here for details). Data should be explorable as well right from the discussion of the science it supports. Moreover, these papers on steroids should be composable: you want to re-use data and code of papers you cite, and allow the reader to navigate freely across citations. Putting all these requirements together leads to a hypermedia structure, where code becomes a medium alongside text, graphics, videos, etc. For a more detailed discussion of the foundations, see my Substrates 2026 paper, and for a direct experience, play with the demo server.

What remains to be done before I can envisage using my new toys for a real research project is integrating Leibniz into HyperDoc. Not difficult, but a bit laborious. Maybe I will profit from the quiet summer period to get started.

Back to the paper by Quinton and Stefani. It takes a much broader view of digital technology at the societal level, and discusses the relation of conviviality to degrowth. At the smaller scale of computational science, there is a similar relation: conviviality requires a limit to the scale of computations. Much of high-performance computing, for example, looks difficult or even impossible to make convivial. It requires optimizations, sometimes specific to one machine, that severely increase the code's opacity, and that is an obstacle to conviviality. The computational resources themselves are another obstacle, making it difficult to impossible for researchers to repeat, possibly with variations, the work of their peers. As Illich points out concerning industrial processes in general, this doesn't mean that we have to stop doing HPC, but we have to take into account non-conviviality as a problem that needs to be compensated by conviviality-restoring measures such as democratic governance. That holds even more for the rapidly growing use of extreme-scale machine learning techniques, usually referred to as "AI", which push polarization and obsolescence to another level, and which, if widely adopted, will establish a radical monopoly impacting not only scientific research, but all of our societies' knowledge management.