TAC 2008: Call for Participation

Text Analysis Conference (TAC)
http://www.nist.gov/tac/

Track Evaluations: February - October 2008
Workshop: November 17-18, 2008

Conducted by:
National Institute of Standards and Technology (NIST)

With support from:
Intelligence Advanced Research Projects Activity (IARPA)

The Text Analysis Conference (TAC) is a new series of evaluation
workshops organized to encourage research in Natural Language
Processing and related applications, by providing a large test
collection, common evaluation procedures, and a forum for
organizations to share their results. TAC grew out of the Document
Understanding Conference (DUC) and the Question Answering Track of the
Text REtrieval Conference (TREC). TAC comprises a set of tasks known
as "tracks," each of which focuses on a particular subproblem of NLP.

You are invited to participate in TAC 2008. NIST will provide test
data for each track, and track participants will run their NLP systems
on the data and return their results to NIST for evaluation.
Organizations may choose to participate in any or all of the tracks.
The annual conference culminates in a November workshop at NIST in
Gaithersburg, Maryland. Dissemination of TAC work and results other
than in the (publicly available) conference proceedings is welcomed,
but the conditions of participation specifically preclude any
advertising claims based on TAC results. All results submitted to
NIST are published in the proceedings and are archived on the TAC web
site.

TASK DESCRIPTION

TAC 2008 will have three tracks: Question Answering, Recognizing
Textual Entailment, and Summarization. Each track is a continuation
of an evaluation series previously organized under different
frameworks, though specific tasks in a track may differ from previous
years.

Below is a brief summary of the tracks. The exact definition of the
tasks to be performed in each TAC 2008 track is still being
formulated. Track discussion takes place on the track mailing list.
To be added to a track mailing list, follow the instructions given in
the track web page for contacting the mailing list. For questions
about the track, send mail to the track coordinator (or post the
question to the track mailing list once you join).

Question Answering: The goal of the QA Track is to develop systems
that search large document collections to retrieve precise answers to
questions, rather than entire documents.
Track coordinator: Hoa Trang Dang, hoa.dang@nist.gov
Track web page: http://www.nist.gov/tac/tracks/2008/qa/
Mailing list: trec-qa@nist.gov

Recognizing Textual Entailment: The goal of the RTE Track is to
develop systems that recognize when one piece of text entails another.
Track coordinator: Danilo Giampiccolo, giampiccolo@celct.it
Track web page: http://www.nist.gov/tac/tracks/2008/rte/
Mailing list: rte@nist.gov

Summarization: The goal of the Summarization Track is to develop
systems that produce short, coherent summaries of text.
Track coordinator: Hoa Trang Dang, hoa.dang@nist.gov
Track web page: http://www.nist.gov/tac/tracks/2008/summarization/
Mailing list: duc_list@nist.gov

CONFERENCE FORMAT

TAC 2008 culminates in a November workshop that is used as a forum
both for presentation of results (including failure analyses and
system comparisons), and for more lengthy system presentations
describing techniques used, experiments run on the data, and other
issues of interest to researchers in NLP. Track participants
interested in giving a presentation during the workshop should submit
a 500-word abstract in September describing the experiments they
performed. As there is a limited amount of time for oral
presentations, the TAC advisory committee will use the abstracts to
determine which participants are asked to speak and which will present
in a poster session.

DATA

For most tracks, training and test materials are available from NIST;
some tracks may use special collections that are available from other
organizations for a fee. Many of the existing TREC/DUC data
(documents, topics, and judgments) are available for training purposes
and may also be used in some of the tracks.

All documents are typical of those seen in a real-world situation
(i.e., there will not be arcane vocabulary, but there may be missing
pieces of text or typographical errors). For most tracks, the
judgments against which each system's output will be scored will be
made by experienced assessors based on manual analysis and the pooled
output of all track participants.

REGISTRATION FOR TAC 2008 TRACKS

Organizations wishing to participate in any of the TAC 2008 tracks
should register online by March 4, 2008 at:
http://www.nist.gov/tac/tracks/2008/track-app.html

Registration for a track does not commit you to participating in the
track, but is helpful to know for planning. Late registration will be
permitted only if resources allow. Any questions about conference
participation should be sent to the TAC project manager:
tac-manager@nist.gov

SCHEDULE

March 4 -- Deadline for registration for track participation
Beginning March 4 -- Submit signed User Agreements to NIST
mid June - early September -- Deadlines for results submission for most
tracks
September 16 (estimated) -- Deadline for workshop presentation proposals
September 16 (estimated) -- Deadline for proposals for future tracks/tasks
By early October -- Release of individual evaluated results to participants
mid October -- Deadline for participants' notebook papers
November 17-18 -- TAC 2008 workshop in Gaithersburg, Maryland, USA
early February 2009 -- Deadline for participants' final proceedings papers

ADVISORY COMMITTEE

John Burger, MITRE
John Conroy, IDA/CCS
Ido Dagan, Bar Ilan University
Hoa Trang Dang, NIST (chair)
Maarten de Rijke, University of Amsterdam
Bonnie Dorr, University of Maryland
Donna Harman, NIST
Andy Hickl, Language Computer Corporation
Ed Hovy, ISI/USC
Boris Katz, Massachusetts Institute of Technology
Bernardo Magnini, ITC-irst
Kathy McKeown, Columbia University
Ani Nenkova, University of Pennsylvania
Drago Radev, University of Michigan
Lucy Vanderwende, Microsoft Research
Ellen Voorhees, NIST
Ralph Weischedel, BBN