I was thinking a lot about software malleability - but from a technical perspective. I am on the verge of building something useful - only if I could find the time to do it.
Here's my premise - if you use something like a game engine, say Unity, and Unreal, you basically have the ability to modify everything in real time, and heve it reflected inside the editor immediately - you could change textures, models, audio, even shaders (which are a kind of code), and have the editor reload just that tiny resource instantaneously.
But not code code - for some reason computer code must go through a compilation, optimization and linking process, creating a monolithic executable piece of code, that cannot be directly modified. This is even true of dynamic languages like Js/Ts, which support modification on the fundamental level, yet somehow lose this ability when using advanced toolchains.
Which is weird since most compilers/OSes support this dynamism at a fundamental level - the machine interface unit of the C compiler is a function, the replacement unit in most OSes is a dynamic library, a collection of said functions - yet changing this at runtime is almost unheard of and most of the times suicidal.
This is because of a couple problems - memory allocation - replacing parts of a program at runtime can lead to leaks if we don't clean that up, resource allocation - this once again can be solved by tying resource lifetimes to either outside factors, or the lifetime of the function or its containing unit.
A demonstrated analog of this is OS processes, which can be terminated abruptly, their binaries replaced without fear of resource leakage.
The final problem of data corruption can be solved by making such program parts stateless, and making them use a store with atomic transactions.
I have a pretty good idea on how to build such an environment on the low level, whose core idea is having process-like isolation barriers isolating small pieces of programs, and an object database-like datastore that can never be corrupted due to transactional changes (which can be rolled back, enabling stuff like time-travel debugging). Said processes could communicate either via messages/events or sharing parts of their memory.
Such a system would allow you to fearlessly change any part of the source code of a running application at runtime - even if you mess up the code of a said component - say event to a point that it doesn't compile - all that would happen would that single component would cease to function without affecting the rest of the app.
But not code code
but that's not true: smalltalk, lisp, pike, erlang and some other languages allow you to change code at runtime, only requiring the recompilation of the changed unit of code (depending on the language. in pike it's at the class/object level)
process-like isolation barriers isolating small pieces of programs, and an object database-like datastore that can never be corrupted due to transactional changes (which can be rolled back, enabling stuff like time-travel debugging).
doesn't smalltalk do pretty much that? i'd be really interested in learning how your idea differs. you may also want to look at societyserver/open-Team: https://news.ycombinator.com/item?id=42159045
it's a platform written in pike that implements an object storage, and allows code objects in that to be modified at runtime. transactions are at the object/class level. (if the class fails to compile, the objects are not replaced). it stores versions of classes so a rollback is possible, although not implemented in the interface. (means right now, if i want an older version i have to rollback manually)
Such a system would allow you to fearlessly change any part of the source code of a running application at runtime - even if you mess up the code of a said component - say event to a point that it doesn't compile - all that would happen would that single component would cease to function without affecting the rest of the app.
smalltalk does that, as does societyserver/open-Team, or the roxen webapplication server (also written in pike) and i am pretty sure some lisp and erlang systems do as well.
Tbh, I have never heard of Pike or societyserver before, will check those out!
As for smalltalk, I am also not intimately familiar with the language, but what I have in mind is somewhat lower level, with emphasis on C-like struct layouts stored in a POD way (so raw structs inside arrays and the like).
I'd say a key difference is in my language (working name Dream (because I started the project as my 'dream' language, and picking names is hard)), is that these isolation contexts are explicit, and you pointers can't really cross them.
There are special 'far' pointers that do have the ability to reference external objects in different context, but there's an explicit unwrap operation that needs to happen that can fail, as that object is not guaranteed to be reachable for whatever reason. Processes can be explicitly deleted, meaning all reference operations to them will fail.
To be clear, when i say process, i mean my lightweight internal isolation thing.
So in summary, my langage is procedural inside processes, with in-process garbage-collection, C-like performance and explicit method calls. Between processes, you either have smalltalk-like signals, or you can do Rust-style borrows, where you can access objects inside the process for the duration of a method call.
It has erlang-like 'just let it crash' philosophy, but again is a C-like procedural language (or shall I say Go-like, since it has total memory safety and GC).
It also has familiar C-like syntax, and quite a small(ish) feature set outside of the core stuff.
I have a huge doc written up on it, no idea if it would work and if it did, it would be useful, but I do have some tentative confidence in it.
(Also no claims on being original or inventive.)
I have never heard of Pike or societyserver before
pike/roxen had a brief window of growth in the 90s but the leaders at the roxen company (not the devs) missed the opportunity to work with the FOSS community.
pike is fully C-syntax, and it is very performant, so that may be interesting for you.
societyserver is my fork/continuation of a university project called open-sTeam that stopped development more than a decade ago. i continue to use it and when i am not busy earning money try to work on it, but i haven't yet been able to build a community around it.
the process isolation you talk about sounds like something that erlang promises as well, but i don't know enough about erlang to tell. i'd be curious to learn more though.
open-sTeam/societyserver built an object-level access control system. method calls on others object are being intercepted and only allowed to pass if the caller has the necessary permission to access that object.
it's not process isolation, but also a concept i find interesting
Things like this exist. They're not that useful in practice, because it's like live editing PHP directly on the server, only more so. Most of the value of edits to a piece of code doesn't come from running it once, it comes from having that edit in a durable, managed place. And however much effort you put into your code management database, it's hard to beat the amount of tooling that exists for the worse-is-better "mostly-ascii files on a unix filesystem" model.
I mean, Tcl/Tk has had this since the 90s. Rewrite your procs (functions) on the fly, delete GUI items on the fly, generate new events, create listeners on the fly, etc, etc.
Quite easy to create a GUI that's interactive AND a console you can script on at the same time to inspect / edit / change code.
For example, don't like your window attributes? Write code to destroy it, and re-create it and keep your "live" data unchanged, and it will redisplay in the new style / layout.
And sure, you could code up atomic transactions quite easily.
Itcl even lets you create / add /remove classes or specific class instances on the fly, or redefine class methods.
I concur that TclTk is enormously versatile and productive. I've always regarded Tcl as a Lisp-like language. Tcl and Lisp share characteristics like homoiconicity that enable benefits you describe.
Tcl isn't as widely known and used as it deserves to be. I think that's in part due to its syntax being sufficiently different from "mainstream" languages. The learning curve isn't particularly steep, but enough so that developers question whether it's worth the effort to go there.
FWIW Tcl 9.0 has recently been released. The language has been enriched with sophisticated object-oriented capabilities, coroutines, full math tower, etc. It's also rather easy to write extensions in C.
Anyway, the GUI toolkit (Tk) has been "borrowed" by many other languages (e.g., Python's tkinter), so quite a few programmers use TclTk, know it not.
Never claimed to be innovative, but sadly all these cool features are nowhere to be found in modern languages. And for some reason, they never appeared in a fast(ish) language, even though I'm sure the JVM is very well equipped to handle this kind of dynamism.
Recompiling a method[1], popping the stack frame, and re-entering the new method is a very, very common debugging pattern on the JVM. I miss it every day that I'm on vastly dumber platforms
https://www.jetbrains.com/help/idea/altering-the-program-s-e... -> https://www.jetbrains.com/help/idea/pro-tips.html#drop-frame
1: pedantically, you're recompiling the whole class, but usually it's only one method changing at a time unless things are really going bananas
DCEVM (RIP) allowed swapping the method signature, too, but that is a lot more tricky to use effectively during debugging (e.g. popping the stack frame doesn't magically change the callsite so if you added extra params it's not going to end well) e.g. https://github.com/TravaOpenJDK/trava-jdk-11-dcevm#trava-jdk...
For anyone interested in this: Tsoding (Twitch and YouTube streamer of "recreational programming") demonstrates this in one of his projects, where he hot reloads a dynamic library without interrupting the main program, to test different functionality.
Here's an operating system kernel in Rust that can hot load/unload modules at ELF object boundaries, made safe by trusted compiler allowing only safe Rust:
Upgrading a module in a way that changes its data structures still requires writing a converter from the old format to new.
This is cool and all, but the problem with doing this in C, is if you accidentally do a memory corruption bug while you're just messing about with the code, now you're forced to restart.
Not a problem in a toy app, but in something like a huge program, it can be a PITA to reload everything and get back to where you were.
Yeah, C really does not support this as a language (or implementation) feature - it's something that can be hacked in, with a lot of difficulty, inconvenience, and loss of safety guarantees.