Not to mention the most difficult issue that even mainstream libraries (like libgcrypt which is used by gpg) get wrong: implementing rsa in constant time as to avoid timing side channel attacks. I would argue that understanding and implementing elliptic curves (the modern ones especically) correctly is much easier.
Do you have a reference to any practical side channel attack on, say, 2048 bit RSA using the current version of libgcrypt? If so it should be filed as a bug.