Side-Tuning: A Baseline for Network Adaptation via Additive Side\n Networks

When training a neural network for a desired task, one may prefer to adapt a\npre-trained network rather than starting from randomly initialized weights.\nAdaptation can be useful in cases when training data is scarce, when a single\nlearner needs to perform multiple tasks, or when one wishes to encode priors in\nthe network. The most commonly employed approaches for network adaptation are\nfine-tuning and using the pre-trained network as a fixed feature extractor,\namong others.\n In this paper, we propose a straightforward alternative: side-tuning.\nSide-tuning adapts a pre-trained network by training a lightweight "side"\nnetwork that is fused with the (unchanged) pre-trained network via summation.\nThis simple method works as well as or better than existing solutions and it\nresolves some of the basic issues with fine-tuning, fixed features, and other\ncommon approaches. In particular, side-tuning is less prone to overfitting, is\nasymptotically consistent, and does not suffer from catastrophic forgetting in\nincremental learning. We demonstrate the performance of side-tuning under a\ndiverse set of scenarios, including incremental learning (iCIFAR, iTaskonomy),\nreinforcement learning, imitation learning (visual navigation in Habitat), NLP\nquestion-answering (SQuAD v2), and single-task transfer learning (Taskonomy),\nwith consistently promising results.\n