Axiomatic Explanations for Visual Search, Retrieval, and Similarity\n Learning

Visual search, recommendation, and contrastive similarity learning power\ntechnologies that impact billions of users worldwide. Modern model\narchitectures can be complex and difficult to interpret, and there are several\ncompeting techniques one can use to explain a search engine's behavior. We show\nthat the theory of fair credit assignment provides a $\\textit{unique}$\naxiomatic solution that generalizes several existing recommendation- and\nmetric-explainability techniques in the literature. Using this formalism, we\nshow when existing approaches violate "fairness" and derive methods that\nsidestep these shortcomings and naturally handle counterfactual information.\nMore specifically, we show existing approaches implicitly approximate\nsecond-order Shapley-Taylor indices and extend CAM, GradCAM, LIME, SHAP, SBSM,\nand other methods to search engines. These extensions can extract pairwise\ncorrespondences between images from trained $\\textit{opaque-box}$ models. We\nalso introduce a fast kernel-based method for estimating Shapley-Taylor indices\nthat require orders of magnitude fewer function evaluations to converge.\nFinally, we show that these game-theoretic measures yield more consistent\nexplanations for image similarity architectures.\n