Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few\n Examples

Few-shot classification refers to learning a classifier for new classes given\nonly a few examples. While a plethora of models have emerged to tackle it, we\nfind the procedure and datasets that are used to assess their progress lacking.\nTo address this limitation, we propose Meta-Dataset: a new benchmark for\ntraining and evaluating models that is large-scale, consists of diverse\ndatasets, and presents more realistic tasks. We experiment with popular\nbaselines and meta-learners on Meta-Dataset, along with a competitive method\nthat we propose. We analyze performance as a function of various\ncharacteristics of test tasks and examine the models' ability to leverage\ndiverse training sources for improving their generalization. We also propose a\nnew set of baselines for quantifying the benefit of meta-learning in\nMeta-Dataset. Our extensive experimentation has uncovered important research\nchallenges and we hope to inspire work in these directions.\n