In Java, there are two kinds of Throwable instances[1]: Error and Exception. As the name suggests, OutOfMemoryError is a subclass of Error. In contrast to Exception, an Error "indicates serious problems that a reasonable application should not try to catch"[2]. For this reason, it's considered bad practice in Java to catch all Throwable instances (or catch Error instances explicitly).
> My instinct would be to catch OOM early
OutOfMemoryError subclasses VirtualMachineError, and when a VirtualMachineError is thrown, your program seems to be firmly in undefined behavior territory. Quoting the JVM specification [3]:
A Java Virtual Machine implementation throws an object that is an instance of a subclass of the class VirtualMachineError when an internal error or resource limitation prevents it from implementing the semantics described in this chapter. This specification cannot predict where internal errors or resource limitations may be encountered and does not mandate precisely when they can be reported.
[1] https://docs.oracle.com/en/java/javase/21/docs/api/java.base...
[2] https://docs.oracle.com/en/java/javase/21/docs/api/java.base...
[3] https://docs.oracle.com/javase/specs/jvms/se21/html/jvms-6.h...
Is this a hole in the safety guarantees?
Safety can mean many things. I'm fairly sure JVM will keep the C-level memory safety guarantee after an OOM, but you can still end up with Java objects in inconsistent states that will start throwing exceptions when you poke them.
The reason why these exceptions are so "unsafe" is that they can occur at any moment. Maybe you were in the middle of modifying some data structure which will enter an infinite loop the next time you touch it. It's a bit like signals in C where you have to be extremely careful about the APIs you use, except worse because exceptions unwind the stack. At least in C you can still safely call kernel syscalls regardless of userspace state.
Yep, it sounds ambiguous. So if there's potentially unsafety allowed in the semantics it would already be interesting. If it turns out that the specification permits "all bets are off" in OOM, eg if it can create conditions where an adversary can gain control of execution in some way, like can happen with synchronisation bugs on some (eg Go) runtimes.