I din't really use any book.. Only book I've read is about network programming for multiplayer games. There is lots of OpenGL/DX Resources flowing on internet.. Most of them are outdated tho. Many suggest Orange Book for learning modern OGL and it's shaders.
For math's you should learn matrixes and quaternions. I suggest starting with some matrix library tho so you get the hang of it.
