First Workshop on
ROC Analysis in AI
Valencia, Spain, 22 August, 2004
hold within ECAI’2004, the European Conference on Artificial Intelligence.
Receiver Operating Characteristic Analysis (ROC Analysis) is related in a direct and natural way to cost/benefit analysis of diagnostic decision making. Widely used in medicine for many decades, it has been introduced relatively recently in some areas of artificial intelligence: machine learning, multiagent systems, intelligent decision support and expert systems. In this context, ROC analysis provides tools to select possibly optimal models and to discard suboptimal ones independently from (and prior to specifying) the cost context or the class distribution. Furthermore, the Area Under the ROC Curve (AUC) has been shown to be a better evaluation measure than accuracy in contexts with variable misclassification costs and/or imbalanced datasets. AUC is also the standard measure when using classifiers to rank examples, and, hence, is used in applications where ranking is crucial, such as campaign design, model combination, collaboration strategies, and co-learning.
there are many open questions and some limitations that hamper a
and applicability of ROC analysis. Its connections with other
measures is not yet clarified completely, its incorporation in decision
and expert systems technology just envisaged, its use for improving the
decisions of (communities of) intelligent agents unexplored, and its
data mining still below its full potential. Among the limitations of
analysis, an important one, despite some recent progress, is its
difficult extension for more than two classes.
One of the main goals of the workshop is to foster the cross-fertilisation of ideas and applications with related areas in artificial intelligence. Consequently, the presentations and discussions will be open to a broad list of topics (not exhaustive):
This is a one-day workshop (see schedule),
sessions consisting of short paper presentations, devoting an important share of time to informal
interaction between the participants. The workshop will be closed with
discussion about more promising open problems and research areas of ROC
analysis, continuation of the workshop, future related events, etc.
Papers will be selected by the program committee according to the quality of the submission and its relevance to the workshop topics. All accepted papers will be gathered in printed form and distributed to registered attendees as workshop notes.
The publication of a selected set of papers for a special volume or a journal issue is considered, but this will depend on the success and overall results of the workshop.
List of accepted papers
Workshop Organizing Committee
Potential participants are invited to submit short papers (no longer than 10 pages), which may be in the form of a technical paper, a position paper (e.g. highlighting open problems or new applications of ROC analysis), an overview of their research or a software demonstration.
Authors should submit their papers electronically (PDF or PS format) to the contact person (firstname.lastname@example.org). It is recommended to submit papers using the final camera-ready ECAI 2004 conference paper style, including author names.
Those wishing to attend the meeting without submitting a paper should send a short statement of interest to the contact person describing their work or interest in the area.
Some relevants links (introduction to ROC analysis, software, etc.)