EUCogII-Euron Workshop:
Challenges, Good Experimental Methodology & Benchmarking

08.04.2011, 13:00-18:00

euRobotics Forum & EURON/EUROP Annual Meeting

April 6-8, 2011 in Västerås, Sweden


Fabio Bonsignorio
Vincent C. Müller

Motivation and objectives:

This workshop will try to shed light on cognitive systems and robotics research in term of challenges, experiments and performance measurement. The purpose is to advance research and allow to pass from a pre-paradigmatic to a mature scientific phase by means of a more rigorous scientific methodology.

As the complexity of current robotic and embodied intelligent systems grows,and cognitive sciences progress, it is more and more necessary to define proper experimental approaches and benchmarking procedures.On the one hand, reliable benchmarks are called for in order to allow the comparison of the many research results in cognitive systems and robotics research, so that their industrial application is eventually possible. On the other hand, if cognitive sciences and robotics aim to be regarded as serious science, or modern engineering, it seems necessary to be able to verify by what degree new procedures and algorithms proposed constitute a real advancement (and can be used in new applications). Both replication of results and benchmarking are needed to foster a cumulative advancement of our knowledge of natural and artificial cognitive agents and even to correctly appreciate disruptive innovation and to give solid ground to theoretical questions (e.g.: "Embodiment fad or future?", the subject of the next EUCogII meeting). Should we take inspiration from experimental practice in disciplines such as biology or medicine?

In order to address these needs the European Robotics Network of Excellence (EURON) has funded a Special Interest Group on Good Experimental Methodology and Benchmarking. Recently EuCognition organized a workshop in Rapperswil with the aim to formulate a basic document outlining a) list of challenges, b) systematization of challenges and c) explanations for some of the challenges formulated. A draft of this propopsal will be discussed at the workshop. Its starting points were that a cognitive system successfully pursues goals; it is flexible (adaptive) and autonomous - challenges can thus be defined as degree of success towards this aim. The set of challenges should:

  • provide a long-term vision and fruitful orientation for present work
  • be as theory and strategy neutral as possible
  • not be domain specific, not be oriented towards toy problems or scenarios
  • be systematic
  • be measurable

Invited Speakers:

Anne Bajart (EU Commission, DG INFSO, Luxembourg)
Fabio Bonsignorio (U Carlos III, Madrid & Heron Robots)
Massimo Caccia (CNR, Italy)
Manja Lohse (Bielefeld University)
Matteo Matteucci (Polimi, Milano)
Giorgio Metta (IIT, Genoa)
Vincent C. Müller (Anatolia, Thessaloniki)

Provisional Schedule
Good Experimental Methodology and Benchmarking
A. Bajart: Benchmarking activities in FP7 Challenge 2 - Cognitive Systems and Robotics
13:20 F. Bonsignorio: Ongoing Activities on Replicable Robotics Research and Benchmarking
13:40 G Metta: Common Platforms and Benchmarks for Cognitive systems
14:00 M Matteucci: Rawseeds and SLAM benchmarking
14:20 M. Lohse: Challenges in the Evaluation of Interactive Cognitive Systems
14:40 M. Caccia: Benchmarking in Marine Robotics
15:00-15:30 Coffee Break
15:30-18:00 Experimental method, Benchmarking and Systematic Challenges
15:30 V. Müller: Presentation of the EUCogII Rapperswil workshop on 'challenges for artificial cognitive systems'
15:50 Open Discussion
17:30 A manifesto early draft

Approach and expected results

Format: The workshop will consist of presentations interleaved with a significant amount of additional time for discussions between the presentations and at the end of the full single track session.

How can participants contribute to, and prepare for, the workshop?

By checking these links:

Planned follow up

Selected contributions may be invited to submit to one or more journal special issue in both cognitive science and robotics communities. A 'manifesto' summarizing background theoretical and experimental methodology challenges in cognitive systems and robotics.