WORKSHOP PROGRAMME

RMIT Building 80 (Floor 05, Room 12)

445 Swanston Street, Melbourne, Victoria, 3000
Venue Info - Google map

CALL FOR PAPERS

The 2nd international workshop on evaluating general-purpose AI (EGPAI2017) will be held in conjunction with IJCAI 2017 in Melbourne, Australia (August 20, 2017). Take a look at the 1st edition EGPAI 2016)

Up to now, most AI systems are tested on specific tasks. However, to be considered truly intelligent, a system should exhibit enough flexibility to find a diversity of solutions for a range of tasks, some of which may not be known until after the system is deployed. Very recently there has been a large number of events, challenges and platforms that are giving a new perspective to how AI can be evaluated, such as the Arcade Learning Environment video games, the Video Game Definition Language (VGDL), OpenAI Gym, Microsoft Malmo, OpenAI Universe, Facebook TorchCarft, Facebook CommAI-env, GoodAI school, Google DeepMind Lab, etc. This workshop will welcome formalisations, methodologies and testbenches for evaluating the numerous aspects of this type of general AI systems. More specifically, we are interested in theoretical or experimental research focused on the development of concepts, tools and clear metrics to characterise and measure the intelligence, and other cognitive abilities, of general AI agents. Furthermore, EGPAI2017 will participate in the IJCAI2017 special theme on AI & Autonomy. Therefore, the workshop will welcome papers on the evaluation of autonomous agents of any kind, such as robots, software agents, artificial life agents, and any sort of autonomous systems capable of operating in long-term, real-world scenarios. There will be a panel dealing with this topic.

We are interested in questions such as: Can the various tasks and benchmarks in AI provide a general basis for evaluation and comparison of a broad range of such systems?, Can there be a theory of tasks, or cognitive abilities, that enables a more direct comparison and characterisation of AI systems? How much does the specificity of an AI agent relate to how fast it can achieve acceptable performance?, How does the structure of a cognitive system relate to how easy or difficult a task - or various classes of tasks - are for it to perform and learn?

TOPICS

We welcome regular papers, short papers, demo papers about benchmarks or tools, and position papers, and encourage discussions over a broad list of topics (not exhaustive):

• Analysis and comparisons of AI benchmarks and competitions. Lessons learnt.
• Proposals for new general tasks, evaluation environments, workbenches and general AI development platforms.
• Theoretical or experimental accounts of the space of tasks, abilities and their dependencies.
• Evaluation of development in robotics and other autonomous agents, and cumulative learning in general learning systems.
• Tasks and methods for evaluating: transfer learning, cognitive growth, structural self-modification and self-programming.
• Evaluation of social, verbal and other general abilities in multi-agent systems, video games and artificial social ecosystems.
• Evaluation of autonomous systems: cognitive architectures and multi-agent systems versus general components: machine learning techniques, SAT solvers, planners, etc.
• Unified theories for evaluating intelligence and other cognitive abilities, independently of the kind of subject (humans, animals or machines): universal psychometrics.
• Analysis of reward aggregation and utility functions, environment properties (Markov, ergodic, etc.) in the characterisation of reinforcement learning tasks.
• Methods supporting automatic generation of tasks and problems with systematically introduced variations.
• Better understanding of the characterisation of task requirements and difficulty (energy, time, trials needed..), beyond algorithmic complexity.
• Evaluation of AI systems using generalised cognitive tests for humans. Computer models taking IQ tests. Psychometric AI.
• Adaptation of evaluation tools from comparative psychology and psychometrics to AI: Item Response Theory (IRT), adaptive testing, hierarchical factor analysis.
• Evaluation methods for multiresolutional perception in AI systems and agents.

We are planning to have a demo session which will present real platforms and ways to evaluate AI systems for several tasks in these platforms. The discussion session will include a panel and a more open discussion about the research challenges around the workshop topics, continuation of the workshop, future initiatives, etc.

KEYNOTE TALKS

DAVID L. DOWE

David Dowe Associate Professor, Clayton School of Information Technology, Monash University. He works primarily in Minimum Message Length (MML) - a unifying tool in machine learning which combines Bayesianism, (algorithmic) information theory and Kolmogorov complexity. Some of the many areas in which he has applied MML include statistical inference (and model selection and point estimation), prediction, machine learning, econometrics (including time series and panel data), proofs of financial market inefficiency, knowledge discovery, data mining, theories of (quantifying) intelligence and new forms of (universal) intelligence test (for biological and non-biological agents), philosophy of science, the problem of induction, bioinformatics, linguistics (evolutionary [tree] models), image analysis, etc.

Talk: Information Theory

JAN FEYEREISL

Jan Feyereisl Executive Director of the AI Roadmap Institute and a Senior Research Scientist at GoodAI. He oversees the institute's mission to accelerate the search for safe human-level artificial intelligence by encouraging, studying, mapping and comparing roadmaps towards this goal. As a scientist at GoodAI, together with his colleagues, he is developing general artificial intelligence "as fast as possible" to help humanity and understand the universe.

Talk: General AI Challenge - Round One: Gradual Learning

RYOTA KANAI

Ryota Kanai is the founder & CEO of ARAYA Brain Imaging. He is well-known as one of the world-leading researchers in structural and functional neuroimage analysis. After graduating from Department of Biophysics, Kyoto University, Ryota earned his PhD in Experimental Psychology from Universiteit Utrecht (The Netherlands). He worked as a postdoc in Caltech (with Prof. Shin Shimojo), University College London (with Prof. Vincent Walsh and Prof. Geraint Rees), and started his own laboratory in University of Sussex.

Talk: Towards Concious AI

GENERAL AI CHALLENGE SPECIAL TRACK

Registered participants for the General AI Challenge are welcome to submit a short summary (2-4 pages) explaining their approach for solving the challenge and their experience so far (at the moment of submission). The deadline is June 27th, 2017 (note that the deadline for this special track is different from the main EGPAI2017 submission deadline (May 23rd, 2017). EGPAI2017 will host a special session for this track, reporting on the state of the General AI Challenge, including an invited talk given from the challenge organisers (GoodAI) and a few selected short reports from the participants. Only registered participants for the challenge can submit to this special track. This must be done through the EGPAI2017 submission platform, but the title of the paper must start with the following text: "General AI Challenge Participant: ". Submission for the special track is compatible with the submission of other regular papers for EGPAI2017 (under its general deadline and instructions).

The papers for this special track will be lightly reviewed by the EGPAI organisers in coordination with the GoodAI people. Accepted papers of this track will be presented during the workshop and will appear in the workshop proceedings. Papers can focus on the learning methods the participants are using for training their agents but we especially welcome those submissions that touch upon evaluation-related issues (quantitative, validation of agents ability to learn gradually or qualitative, interpretability, white-box analysis of agents) and the main obstacles of the General AI challenge (non-stationarity, catastrophic forgetting, sample complexity, limited training data, generality, etc.).

SUBMISSION DETAILS

• We solicit submissions (full or short papers) including: original research contributions, applications and experiences, surveys, comparisons, and state-of-the-art reports, tool or demo papers, position papers related to the topics mentioned above and work in progress papers.
• Submitted papers must be formatted according to the camera-ready style for IJCAI 2017, and submitted electronically in PDF format through Easychair
• Papers are allowed a maximum six (6) pages, excluded references. References can take up to one page. Formatting Guidelines, LaTeX Styles and Word Template can be downloaded from here.
• Authorship is not anonymous (single-blind review). Papers will be reviewed by the program committee.
• The designated author will be notified by email about acceptance or rejection by TBA. Details of the reviewing process will be posted on the EGPAI 2017 website.

PROGRAM COMMITTEE

Name	Affiliation
Marco Baroni	Facebook AI Research
Jordi Bieger	CADIA, Reykjavik University
Angelo Cangelosi	Plymouth University
Emmanuel Dupoux	EHESS
Helgi P. Helgason	Activity Stream
Katja Hofmann	Microsoft Research
Sean B. Holden	Cambridge University
Estevam R. Hruschka	Carnegie Mellon University
Armand Joulin	Facebook AI Research
Jan Koutnik	IDSIA
Edward Keedwell	Exeter University
Tomas Mikolov	Facebook AI Research
Frans A. Oliehoek	University of Amsterdam
Ricardo B.C. Prudencio	Uni. Fed. de Pernambuco
Ute Schmid	Bamberg University
Bas Steunebrink	IDSIA
Pei Wang	Temple University