This series encourages proposals on cutting-edge science, technology and best practices including (but not limited to) the following topics.



  • Novel or atypical challenge protocols, particularly relating to gaming and education.
  • Novel or atypical challenge protocols to tackle complex tasks with very large datasets, multi-modal data, and data streams.
  • Methods and metrics of entry evaluation, quantitative and qualitative challenges.
  • Methods of data collection, “ground-truthing”, and preparation including bifurcation/anonymization, data generating models.
  • Teaching challenge organization.
  • Hackatons and on-site challenges.
  • Challenge indexing and retrieval, challenge recommenders.



  • Societal of psychological studies of theories about gaming and education.
  • Experimental design, size data set, data split, error bounds, statistical significance, violation of typical assumptions (e.g. i.i.d. data).
  • Game theory applied to the analysis of challenge participation, competition and collaboration among participants.
  • Diagnosis of data sanity, artifacts in data, data leakage.



  • Re-usable challenge platforms, innovative software environments.
  • Linking data and software repositories to challenges.
  • Security/privacy, intellectual property, licenses.
  • Cheating prevention and remedies.
  • Issues raised by requiring code submission.
  • Challenges requiring user interaction with the platform (active learning, reinforcement learning).
  • Dissemination, fact sheets, proceedings, crowdsourced papers, indexing post-challenge publications.
  • Long term impact, on-going benchmarks, metrics of impact.
  • Participant rewards, stimulation of participation, advertising, sponsors.
  • Profiling participants, improving participant professional and social benefits.




  • Challenges as an educational tool.
  • Where to venture next: opportunities for challenge organizers to organize challenges in new domains with high societal impact.
  • Successful challenge leading to significant breakthrough or improvement over the state-of-the-art or unexpected interesting results.
  • Rigorous study of the impact of challenges, analyzing topics and tasks lending themselves to high impact machine learning challenges.
  • Challenges organized or supported by Government agencies, funding opportunities.