^Someone recommended this article to me, and I thought I'd share, since you mentioned quaternions.
For example, most game studios would benefit from the ability to build large continuous 3D world meshes, with multiple artists working on the same mesh at once—or methods of editing triangular meshes to ensure that cracks and holes do not appear.
Mathematical knowledge. A programmer just isn’t going to be competent in a modern game without a decent grasp of basic linear algebra,4 as well as geometry in 2D and 3D. We often use 4D representations for basic operations (4D homogeneous coordinates for general linear transformations, and the quaternions to represent rotations5) so the ability to reason about higher dimensions is extremely useful. Basic calculus is necessary for all kinds of simulation and rendering tasks. For many rendering tasks, signal-processing mathematics is very important—both linear signal processing6 as well as the murkier study of spherical harmonics.7 For any kind of sophisticated simulation, you’ll want experience with numerical analysis and differential forms. For networking, information theory and the statistics behind compression and cryptography are necessary to build a robust system.
basic linear algebra
geometry in 2D
geometry in 3D
4D homogeneous coordinates for general linear transformations, and the quaternions to represent rotations
>linear signal processing
numerical analysis and differential forms
information theory and the statistics behind compression and cryptography
Algorithmic knowledge. A good engine programmer should have working familiarity with a great many algorithms—so many that attempting to list them here would be silly. The most necessary algorithms perform tasks like spatial partitioning, clustering, and intersection and clipping of geometric primitives. Most algorithms will be mainly focused on one task area, like rendering or physics, but these algorithms are often very deep and take a while to master. For years we have been mining academic research to find and modify appropriate algorithms. However, a game engine must meet soft realtime requirements, and most academic work in the relevant subject areas is geared toward batch computation. (Most of the past research in graphics has applied to offline cinematic rendering. Most physics algorithms are unstable and can fail outright, which is solved in a batch setting by tweaking the initial conditions and trying again. These algorithms do not adapt successfully to a soft realtime setting.) As games are now starting to be taken seriously by the academic community, this is beginning to change, but most academic research is still pointed in directions that don’t do us much good. So, creating a technically ambitious game engine will often require a substantial amount of original research.
>intersection and clipping of geometric primitives
Google, here I come.