Second Workshop on
ROC Analysis in ML
Bonn, Germany,  11 August, 2005

hold within ICML’2005, the 22nd International Conference on Machine Learning

Followed by ROCML'06.

Brief description

Receiver Operating Characteristic Analysis (ROC Analysis) is related in a direct and natural way to cost/benefit analysis of diagnostic decision making. Widely used in medicine for many decades, it has been introduced relatively recently in machine learning. In this context, ROC analysis provides tools to select possibly optimal models and to discard suboptimal ones independently from (and prior to specifying) the cost context or the class distribution. Furthermore, the Area Under the ROC Curve (AUC) has been shown to be a better evaluation measure than accuracy in contexts with variable misclassification costs and/or imbalanced datasets. AUC is also the standard measure when using classifiers to rank examples, and, hence, is used in applications where ranking is crucial, such as campaign design, model combination, collaboration strategies, and co-learning.

Nevertheless, there are many open questions and some limitations that hamper a broader use and applicability of ROC analysis. Its use in data mining and machine learning is still below its full potential. An important limitation of ROC analysis, despite some recent progress, is its possible but difficult extension for more than two classes.

This workshop follows up a first workshop (ROCAI'04) held within ECAI-2004. The main goal of this first workshop was to foster the cross-fertilisation of ideas and applications of ROC analysis with related areas in artificial intelligence and to gather points of views from broad AI fields. This second workshop will focus on the point of view of machine learning, in particular on some issues raised during the first workshop, e.g. ROC analysis software repository, multiclass extension, statistical analysis.

Organisation

We encourage submissions on hot topics raised during the first edition, e.g. ROC analysis software repository, multiclass extension, statistical analysis (cf. ROCAI'04). To promote a workshop atmosphere the program committee will select a few topics and relevant papers. The program and accepted papers will be published on the workshop website before the call for participation where we will encourage people to read them in order to focus on the discussion during the workshop. The authors of accepted papers will be asked to prepare the discussion rather than to detail their paper, by summarising their contributions, on the one hand, and by comparing them to the other points of view presented in their session, on the other hand.

Accepted Papers 

List of accepted papers

Schedule

Invited Talk

Important Dates

• Deadline for the title and abstract:       1 April 2005
  
• Deadline for the workshop papers:     7 April 2005  
• Notification of acceptance:               29 April 2005 
• Camera Ready copies:                     27 May 2005   
• Workshop dates:                             11 August 2005
  

• Hendrik Blockeel, K.U.Leuven, Belgium.
• Stephan Dreiseitl FHS Hagenberg, Austria.
  
• Richard M. Everson, University of Exeter, UK.
• Cèsar Ferri, Technical University of Valencia, Spain.
• Jonathan E. Fieldsend, University of Exeter, UK.
• Peter Flach, University of Bristol, UK.
• Johannes Fürnkranz, TU Darmstadt, Germany .
• José Hernández-Orallo, Technical University of Valencia, Spain.
• Rob Holte, University of Alberta, Canada.
• Nicolas Lachiche, University of Strasbourg, France.
• Sofus A. Macskassy, New York University, USA.
• Alain Rakotomamonjy, Insa de Rouen, France.
• Francesco Tortorella, University of Cassino, Italy.

Workshop Organizing Committee

• Cèsar Ferri, Technical University of Valencia, Spain.
• Nicolas Lachiche, University of Strasbourg, France (contact person).
• Sofus A. Macskassy,  New York University, USA.
  
• Alain Rakotomamonjy, INSA de Rouen, France
  

Submission Guidelines

Potential participants are invited to submit papers according to one of the following formats:

• Position papers : short paper (2-4 pages)  specifically written for the purposes of this workshop that highlight open problems and/or new applications of ROC analysis.
• Research papers : these papers (up to 8 pages) present technical contributions on ROC analysis in Machine Learning and should follow ICML main conference standard.

Authors should submit their papers electronically (PDF or PS format) to the contact person (lachiche@lsiit.u-strasbg.fr) It is recommended to submit papers using the final camera-ready ICML 2005 conference paper style, including author names.

Some relevants links (introduction to ROC analysis, software, etc.)

• The ROC Convex Hull: software and papers, mantained by Tom Fawcett.
• Receiver Operating Characteristic (ROC) Literature Research: ROC papers and articles, mantained by Kelly Zou.
  
• The Magnificent ROC: A nice introduction to ROC Analysis.
• ROC Analysis Software.
• JROCFITi: A ROC Analysis Applet.
• ROCOn: A tool for visualising ROC graphs.
• Very introductory slides on ROC Analysis.
• Tutorial on "The Many Faces of ROC Analysis in Machine Learning" by Peter Flach
• ROCR: An R package on ROC Analysis

Supported by