>"During the years there have been many attempts to improve Python speed; generally they fall into two categories:
Implement "full Python". To be able to support all dynamic features and be fast, they usually employ a Just In Time (JIT) compiler.
Examples are PyPy, GraalPy, Pyston, and CPython's own JIT.
Implement a "subset of Python" or "variant of Python", either as an Ahead of Time (AOT) or JIT compiler which is able to produce fast code. The usual approach here is to remove many (if not all) of the dynamic features which make Python hard to compile.
Examples are RPython, Mypyc, Cython and Numba.
The problem of "full Python" JIT compilers is that sometimes they work very well and produce huge speedups, other times they don't produce any speedup at all, or might even introduce slowdowns, or they might use too much memory, or they are slow to "warm up".
The problem of the subset/variant approach is that by removing the dynamic features of Python, you end up with something which does not feel pythonic, and in which many typical and idiomatic Python patterns just don't work. You often end up with "Java with Python syntax" (nothing in particular against Java, but I hope it gives an idea of what I mean)."
Isn't that interesting!
You know what I'd love to see?
A Python compiler that works on a function-by-function basis...
That is, for each function, make the determination if a straight "clean" compilation of the function is possible -- that is, if it doesn't use special Python features that make compilation difficult/challenging/complex/impossible -- or have dependencies on other functions that do...
So if you have a Python function that adds simple integer variables, let's say, then that can be converted to C or Assembler or VM or LLVM or QBE code in a straightforward way, but if a function uses objects and lambdas and higher-level/complex constructs in a way that makes compilation challenging, complex or impossible, then flag those functions (write them out in a log, so the programmer could optionally simplify them, etc.) -- those will continue to be interpreted like they always are...
Yes, some or all of that infrastructure might already exist... I'm just thinking aloud... :-)
>"The first problem is that Python is extremely dynamic. I'm not talking only about dynamic typing, but also about the fact that in a given Python process, the "world" is a moving target and "everything can change at any time" "
(If we had a future language that supported both interpreted and compiled functions, then the interpreter should have the concept of "locked" functions and objects -- that is, once the compiler has compiled something, it's now "locked", and changes (if we want to keep it dynamic) to it mean invoking the compiler again post-change, at runtime... it would be messy, but there could be a hybrid dynamic interpreted yet JIT compiled language... but it would require runtime interaction between the interpreter and compiler... which could be messy, to say the least... and yes, some of that infrastructure might already exist...)
Implement "full Python". To be able to support all dynamic features and be fast, they usually employ a Just In Time (JIT) compiler.
Examples are PyPy, GraalPy, Pyston, and CPython's own JIT.
Implement a "subset of Python" or "variant of Python", either as an Ahead of Time (AOT) or JIT compiler which is able to produce fast code. The usual approach here is to remove many (if not all) of the dynamic features which make Python hard to compile.
Examples are RPython, Mypyc, Cython and Numba.
The problem of "full Python" JIT compilers is that sometimes they work very well and produce huge speedups, other times they don't produce any speedup at all, or might even introduce slowdowns, or they might use too much memory, or they are slow to "warm up".
The problem of the subset/variant approach is that by removing the dynamic features of Python, you end up with something which does not feel pythonic, and in which many typical and idiomatic Python patterns just don't work. You often end up with "Java with Python syntax" (nothing in particular against Java, but I hope it gives an idea of what I mean)."
Isn't that interesting!
You know what I'd love to see?
A Python compiler that works on a function-by-function basis...
That is, for each function, make the determination if a straight "clean" compilation of the function is possible -- that is, if it doesn't use special Python features that make compilation difficult/challenging/complex/impossible -- or have dependencies on other functions that do...
So if you have a Python function that adds simple integer variables, let's say, then that can be converted to C or Assembler or VM or LLVM or QBE code in a straightforward way, but if a function uses objects and lambdas and higher-level/complex constructs in a way that makes compilation challenging, complex or impossible, then flag those functions (write them out in a log, so the programmer could optionally simplify them, etc.) -- those will continue to be interpreted like they always are...
Yes, some or all of that infrastructure might already exist... I'm just thinking aloud... :-)
>"The first problem is that Python is extremely dynamic. I'm not talking only about dynamic typing, but also about the fact that in a given Python process, the "world" is a moving target and "everything can change at any time" "
(If we had a future language that supported both interpreted and compiled functions, then the interpreter should have the concept of "locked" functions and objects -- that is, once the compiler has compiled something, it's now "locked", and changes (if we want to keep it dynamic) to it mean invoking the compiler again post-change, at runtime... it would be messy, but there could be a hybrid dynamic interpreted yet JIT compiled language... but it would require runtime interaction between the interpreter and compiler... which could be messy, to say the least... and yes, some of that infrastructure might already exist...)