Unless they add multi-processing to the mod system, they can't. You can't "sandbox" a Java class like that. You can't deny it access to JNI. You can't deny it access to java.io. You can have the host OS do this, of course, but now we're talking something far, far more complex than any mod system ever.
Which is, frankly, an absurd demand.
Mojang should have a mod API, but that won't do diddly squat to improve saftey. Don't execute code if you don't want that code to run.
Of course you can deny it access to JNI [1]. The Java platform has had this built in since forever. This is how applets work. Aside from sandbox exploits classes in Java can be locked down so much that they can pretty much only allocate memory and burn cycles.
Java Applets? That thing that had exploits like clockwork?
Yeah, that worked great
Regardless did you even bother looking at the documentation of SecurityManager? That would be a disaster to try and support. It can't clamp down per-library, it's all about current thread. Mods run in the same thread, they have to. So now you're suggesting that every. single. call into or out of a mod is guarded manually by updates to a permission manager that you are preventing the mod from modifying via reflection through... magic? Yeah, there's no way in hell that would ever work.
And JavaScript runs in a VM designed specifically to handle untrusted code.
Neither applies in the slightest to the Java VM or Minecraft mods which, by definition, occupy the same process space with the same permissions as Minecraft itself. It is not a web browser, it does not have process sandboxing, it does not have a seperate VM specifically for mods, etc...
Which is, frankly, an absurd demand.
Mojang should have a mod API, but that won't do diddly squat to improve saftey. Don't execute code if you don't want that code to run.