Challenges, Good Experimental Methodology & Benchmarking
euRobotics Forum & EURON/EUROP Annual Meeting
April 6-8, 2011 in Västerås, Sweden
Motivation and objectives:
This workshop will try to shed light on cognitive systems and robotics research in term of challenges, experiments and performance measurement. The purpose is to advance research and allow to pass from a pre-paradigmatic to a mature scientific phase by means of a more rigorous scientific methodology.
As the complexity of current robotic and embodied intelligent systems grows,and cognitive sciences progress, it is more and more necessary to define proper experimental approaches and benchmarking procedures.On the one hand, reliable benchmarks are called for in order to allow the comparison of the many research results in cognitive systems and robotics research, so that their industrial application is eventually possible. On the other hand, if cognitive sciences and robotics aim to be regarded as serious science, or modern engineering, it seems necessary to be able to verify by what degree new procedures and algorithms proposed constitute a real advancement (and can be used in new applications). Both replication of results and benchmarking are needed to foster a cumulative advancement of our knowledge of natural and artificial cognitive agents and even to correctly appreciate disruptive innovation and to give solid ground to theoretical questions (e.g.: "Embodiment fad or future?", the subject of the next EUCogII meeting). Should we take inspiration from experimental practice in disciplines such as biology or medicine?
In order to address these needs the European Robotics Network of Excellence (EURON) has funded a Special Interest Group on Good Experimental Methodology and Benchmarking. Recently EuCognition organized a workshop in Rapperswil with the aim to formulate a basic document outlining a) list of challenges, b) systematization of challenges and c) explanations for some of the challenges formulated. A draft of this propopsal will be discussed at the workshop. Its starting points were that a cognitive system successfully pursues goals; it is flexible (adaptive) and autonomous - challenges can thus be defined as degree of success towards this aim. The set of challenges should:
- provide a long-term vision and fruitful orientation for present work
- be as theory and strategy neutral as possible
- not be domain specific, not be oriented towards toy problems or scenarios
- be systematic
- be measurable
Invited Speakers:Anne Bajart (EU Commission, DG INFSO, Luxembourg)
Fabio Bonsignorio (U Carlos III, Madrid & Heron Robots)
Massimo Caccia (CNR, Italy)
Manja Lohse (Bielefeld University)
Matteo Matteucci (Polimi, Milano)
Giorgio Metta (IIT, Genoa)
Vincent C. Müller (Anatolia, Thessaloniki)
||Good Experimental Methodology and Benchmarking|
||A. Bajart: Benchmarking activities in FP7 Challenge 2 - Cognitive Systems and Robotics|
|13:20||F. Bonsignorio: Ongoing Activities on Replicable Robotics Research and Benchmarking|
|13:40||G Metta: Common Platforms and Benchmarks for Cognitive systems|
|14:00||M Matteucci: Rawseeds and SLAM benchmarking|
|14:20||M. Lohse: Challenges in the Evaluation of Interactive Cognitive Systems|
|14:40||M. Caccia: Benchmarking in Marine Robotics|
|15:30-18:00||Experimental method, Benchmarking and Systematic Challenges|
|15:30||V. Müller: Presentation of the EUCogII Rapperswil workshop on 'challenges for artificial cognitive systems'|
|17:30||A manifesto early draft|
Approach and expected results
Format: The workshop will consist of presentations interleaved with a significant amount of additional time for discussions between the presentations and at the end of the full single track session.
How can participants contribute to, and prepare for, the workshop?
By checking these links:
Planned follow up
Selected contributions may be invited to submit to one or more journal special issue in both cognitive science and robotics communities. A 'manifesto' summarizing background theoretical and experimental methodology challenges in cognitive systems and robotics.