nonEuclidean geometry (especially Minkowskian), Riemann surfaces, complex and hypercomplex numbers, ndimensional manifolds, symmetry groups, number theory (for quantum aspects), and of course calculus and tensors.As is often the case, most of the math was developed by mathematicians before anyone figured out there was a need for it.String theory raised the bar for the degree of mathematical knowledge and skill needed to practice theoretical physics. See the reference to get an idea of how overwhelming it really is.
