What are we going to do about programming?
Seriously it’s very bad right now. Right?
1
I’m annoyed about the state of programming.
For instance, it takes an unreasonable amount of effort to do anything at all with software, and we still don’t do it particularly well. How many millions of person-hours are spent every year on fixing pointless bugs, and understanding confusing code? And why is it that every significantly-sized project has enough complexity to require a team of specialists to support it, and still there is no useful overlap with the complexity of any other project.
And why is every company and organization is duplicating each other’s work because almost none of the work is shared? And why is every advance in hardware efficiency is canceled out by inefficiencies in software, so that everything is barely performant enough, all the time, and gobbles up any and all energy we have available to give it?
What a waste of human labor. It is all working, kinda, in that humanity is churning out software and solving problems and making money. But it’s not working well. Humanity is spending too much of effort to make software that doesn’t work very well and doesn’t do very much slowly.
If you work in the industry this should all be obvious. If you write code and don’t feel like most of your time is wasted dealing with problems that shouldn’t exist in the first place, I don’t know what to tell you. It’s really bad.
2
Really it is that there is something medieval about how we write code. I imagine that we’re still in the dark ages, like mathematics before algebra and calculus were discovered. The way software is written in five hundred years—if we haven’t run out of breathable air or microchips or whatever—will, I expect, be mostly unrecognizable compared to how it’s done today. At best we are, as a species, 20% of the way along that path. (My guess is that we’re at like 15% in most places and then React.js pushes the number to 20%.)
Here’s a test for assessing how good humanity is at writing software:
Suppose that humanity builds out a spaceship of 1000 colonists is traveling to another star system, many light-years away.
And suppose this ship has to be totally self-sufficient, including having the ability to support all of its software, fixing bugs and improvising solutions to whatever comes up on the journey, and building out whatever is needed once they get there.
Would you get on that ship?
The answer is: hah! definitely ‘no’. They will all die. RIP. Definitely don’t sign up for that trip.
The answer needs to be ‘yes’, of course, if we are going to colonize other star systems. I don’t want to ship off to another star system only to die partway of, like, an unfixable integer overflow in the life-support system. There is no way you can fill out the roster of the ship with expert software engineers, but there is no way a roster of non-experts, even if they are geniuses in other fields, can understand even one component of the ship end-to-end. So we have work to do. It’s probably possible to fix this, but it will take some serious advances to get there.
3
An analogy can be made to mechanical engineering. I don’t really know how my (gas-powered) car works. But if I open the hood and look at the engine, I feel like I have at least a hope of figuring it out. Apart from, well, the electronics, I can clearly tell which parts interact with which other parts, and approximately what they do to each other, and I can learn the principles of chemistry and combustion to the point where I can maintain and rebuild the thing.
If I had a good manual and a science-fiction-grade fabrication machine to make the parts, I can probably fix it if it breaks. The fact I can make progress at all is valuable. If I took a long road trip away from civilization with just a box of tools and spare parts I have at least a hope of handling whatever comes up.
The difference, I think, is that physical machines are constrained by fundamental requirements of causality. For a widget to affect a gizmo, it has to, like, touch it, and there has to be some motive force between the two, which I can view and manipulate myself. And if you take the widget apart, its internal components have the same property. Of course this falls apart when chemistry gets involved; you actually do need some specialized knowledge to make sense of, say, the actual combustion process. But for mechanical systems, it’s a start.
Physical interactions have this fundamental property that I like to call “scrutability” which allows me to make progress on understanding them. Scrutability means: components interact through other components through well-defined connections, pushes and pulls and the like. And components are made of subcomponents which do the same thing, compositionally, all the way down.
Software today has no such property. It works exactly how it works, and good luck figuring it out from the outside.
The closest thing to “scrutability” I know of in software is understanding a website via Devtools, but it barely counts. Someday, figuring out how software works needs to be as natural as taking apart and tinkering with an old engine.
As should be clear from this comparison, it’s not enough that software is open-source (although that’s a start). It must also be conceptualized and built in a way that makes its functionality “scrutable”. It needs to be split into scrutable modules that ‘push’ and ‘pull’ on each other in a way that we can follow, and those modules should have submodules with the same property.
And most importantly, it needs to be constructed in such a way that allows for the digital equivalent of ‘opening up the hood and looking inside’, and we need to have the tools at hand to do so.
4
Here’s where I think progress is happening:
The most “scrutable” system I know of is Chrome Devtools, because it lets you see your website as it is, and modify it. Unfortunately it doesn’t let you write code (or really search for it properly, or really modify anything in a way that doesn’t get reversed the next time a callback is triggered… yeah, it’s not great). But it does something the rest of them don’t, which is let you record every piece of code that’s run on a page and inspect it to see what happened, which IMO is the bare minimum for scrutability.
(Nevermind that this process is janky and error-prone, and obfuscated, and the backend is invisible, and making sense of the execution after you’ve recorded it is a mindfuck– at least it exists. There is no future in having to add print / console.log statements to find out what your code did, or in reading imperative code to figure out what state a UI ended up in. You have to be able to look at everything, as it exists.)
The most “scrutable” way of writing code that I know of is React. The declarative model is the right way to reason about UI code. React Devtools are reasonably good at looking at something while it’s running, and, in some cases modifying it. Hooks are better than any other way I’ve ever seen to reason about side effects, although in every case the whole philosophy is hamstrung by being implemented in Javascript and having to transpile to the DOM. And the problem of data processing and externalities is, as far as I know, still an unsolved problem, despite the efforts of the Redux/Flux/state management ecosystem.
At least when it comes to UI development, I believe there is a future in which Chrome Devtools, React Devtools, a design tool like Figma, and your IDE are all the same piece of software. And I think that in this world, user-facing code no longer has anything like unit tests, because it’s a waste of time to test code when you can look at it and observe it’s correct: it’s correct by definition, because every building block is correct and they’re obviously laid out like you wanted them.
The best shell I know of is, I guess, Python. Bash and its descendants are a disaster and the world would be better off if they were entirely replaced. In the future there is no way that we’re going to be working in languages that use $PATH variables, that pipe unformatted string data through bizarrely-named commands inflected by obscure flags, or that require in-band signalling with strings like \u001b[31m to colorize text. I mean, my god. (Once upon a time I had high hopes for TermKit but it never really got off the ground.)
I think ‘time-traveling’ and ‘deterministic’ debuggers such as RR are due for another life, but really it all needs to be combined with the need for determinism to make an ecosystem where you’re just looking at your code instead of replaying it.
Finally, I know this: most of the code written today isn’t any good, compared to what will be possible in the future.
It’s simply not possible in today’s ecosystems to write something scalable, maintainable, and resilient to errors. It’s up to the frameworks and paradigms to develop the art of programming to the point where it’s actually an efficient and accessible craft instead of a massive timesink for the whole human race.