It is learning to express what it is you want, clearly and unambiguously enough so that there is as little room for error as possible; and even before then, learning to understand for yourself inside your own head what it is you want in sufficient detail that you can verbalise it, and maybe reason through how it might be achieved and what the possible outcomes, whether good or bad, might be.

Tooling can mainly help here by getting out of our way, so that we are free to focus on thinking about the actually important parts of what we want to happen without distracting details of how specifically it might be implemented such as the memory safety you mention.

However, with all those details gone, the actually hard problem of bringing your creation from your head into the outside world still remains. It predates computers, and is still present even if you remove computers from the equation entirely: it is why, e.g., lawyers as a profession exist. It arises not from the technology or area of activity, but from the nature of our own minds.

I don’t understand why AIs wouldn’t use the languages humans already use to program?

Or do you mean, if an AI designed a programming language, it would be better than anything humans have come up with? Why? I’d expect it would be more like some unholy mashup of C, Javascript and Python.

Can you elaborate on the difference?

Why not train the AI on machine language and have it generate binaries directly?

An AI that generates binaries directly is a compiler. Or a compiler/assembler/linker in one, whatever.

So it raises the question of what language humans should use to communicate with this compiler. What language does it compile?

If the language is some existing programming language, then how can the "AI" technology help much? Suppose it compiles C. The semantics of the existing language have to be maintained or it isn't C. So maybe AI can improve optimization by 0.0001%. Or do something really ridiculous based on the undefined behavior loophole. Who cares?

English (or Chinese or Norwegian) are not programming languages - they are too ambiguous.

Therefore it makes sense to me that a higher-level programming language than existing ones would be useful to talk to this hypothetical compiler - a language that gives the AI more freedom than the existing languages, but more precise than languages used by humans to talk to each other.

Maybe someone already has invented one, but probably the last word hasn't been written on the subject yet.

Maybe that's the answer I was looking for, AI can "interpret" sentences referring to the problems/needs to be solved and it may lack the way to solve for those with the existing programming languages and it could come up with it's own "abstraction" on top of processes - or maybe not, and will plainly speak low-level 0-1s directly.

"Let's see now. That human obviously wants me to produce a binary for a self-driving car... or perhaps it was a binary for a game of Chess ... or perhaps it was a binary for a new version of Windows .... or perhaps .... Well, I don't know!"

Don't ruin good Hype by bringing in real practicalities.