Duality and mock modularity

We derive a holomorphic anomaly equation for the Vafa-Witten partition function for twisted four-dimensional \mathcal{N} =4 <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:mrow><mml:mstyle mathvariant="script"><mml:mi>𝒩</mml:mi></mml:mstyle><mml:mo>=</mml:mo><mml:mn>4</mml:mn></mml:mrow></mml:math> super Yang-Mills theory on \mathbb{CP}^{2} <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:msup><mml:mstyle mathvariant="double-struck"><mml:mi>ℂ</mml:mi><mml:mi>ℙ</mml:mi></mml:mstyle><mml:mn>2</mml:mn></mml:msup></mml:math> for the gauge group SO(3) <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:mrow><mml:mi>S</mml:mi><mml:mi>O</mml:mi><mml:mo stretchy="false" form="prefix">(</mml:mo><mml:mn>3</mml:mn><mml:mo stretchy="false" form="postfix">)</mml:mo></mml:mrow></mml:math> from the path integral of the effective theory on the Coulomb branch. The holomorphic kernel of this equation, which receives contributions only from the instantons, is not modular but ‘mock modular’. The partition function has correct modular properties expected from S <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:mi>S</mml:mi></mml:math> -duality only after including the anomalous nonholomorphic boundary contributions from anti-instantons. Using M-theory duality, we relate this phenomenon to the holomorphic anomaly of the elliptic genus of a two-dimensional noncompact sigma model and compute it independently in two dimensions. The anomaly both in four and in two dimensions can be traced to a topological term in the effective action of six-dimensional (2,0) <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:mrow><mml:mo stretchy="false" form="prefix">(</mml:mo><mml:mn>2</mml:mn><mml:mo>,</mml:mo><mml:mn>0</mml:mn><mml:mo stretchy="false" form="postfix">)</mml:mo></mml:mrow></mml:math> theory on the tensor branch. We consider generalizations to other manifolds and other gauge groups to show that mock modularity is generic and essential for exhibiting duality when the relevant field space is noncompact.