\documentclass[a4paper,10pt]{article}
\usepackage[utf8x]{inputenc}
\usepackage{authblk}
\usepackage{fixltx2e}
\usepackage{url}
\setlength\parindent{0pt}

%opening

\title{ParCor 1.0: Pronoun Coreference Annotation Guidelines}
\author[1]{Liane~Guillou}
\author[2]{Christian~Hardmeier}
\author[2]{Aaron~Smith}
\author[2]{J\"{o}rg~Tiedemann}
\author[1]{Bonnie~Webber}
\affil[1]{University of Edinburgh}
\affil[2]{University of Uppsala}

\renewcommand\Authands{ and }

\begin{document}

\maketitle

\tableofcontents

\section{Note}
These guidelines were presented to the English and German annotators who worked on the annotation of the EU Bookshop and TED Talks texts in the ParCor 1.0 corpus. The document is split into three main sections: General guidelines (applicable to both text genres) and additional guidelines specific to the annotation of the EU Bookshop and TED Talks portions of the corpus respectively.


\section{Pre-populated Markables}
In order to assist the annotation process pre-populated MMAX-2 \textit{markables} are provided as a starting point to the manual annotation. These markables represent:
\begin{itemize}
 \item Pronouns: These will first appear as bold blue text with a magenta background and the background colour will disappear once you have determined a new \textit{type} for them -- e.g. anaphoric, pleonastic or event.
 \item Noun Phrases (NPs): A set of potential antecedents for pronouns. These will appear as normal blue text (as for any other markable in MMAX-2).
\end{itemize}

Please note the markables were produced by automated tools and may not be 100\% accurate. You should look for errors such as:
\begin{itemize}
 \item Pronouns that have not been identified (and therefore are not labelled as markables)
 \item Words that have been mis-labelled as pronouns
 \item Potential pronoun antecedents (the set of NPs) which may be missing and therefore need to be added (manually), or ones where their span may be too large or small and therefore needs adjusting (manually)
\end{itemize}


\section{General Guidelines: What to Include}
We wish to construct links between pronouns and their antecedent(s). These linked pronoun-antecedent pairs should exclude events (and references to the events) and pleonastic/dummy pronouns (see Section \ref{WhatToExclude} on what to exclude).

The following set of guidelines has been condensed from the MUC-7 guidelines\footnote{\url{http://www.itl.nist.gov/iaui/894.02/related_projects/muc/proceedings/co_task.html}}.

Pronoun \textit{forms} to be annotated, include:
\begin{itemize}
 \item Personal: First, second and third-person
 \item Possessive
 \item Demonstrative 
 \item Relative
 \item Reflexive (TED Talks only)
 \item Pronominal adverbs (EU Bookshop only)
 \item Generic
\end{itemize}

The possessive forms of pronouns used as determiners are markable. Thus in:

\begin{quote}
The company and [\textbf{its chairperson}]
\end{quote}

there are two potentially markable elements: \textbf{its} and the entire NP, \textbf{its chairperson}.

First, second, and third-person pronouns are all markable, so in:
\begin{quote}
``There is no business reason for [\textbf{my}] departure'', [\textbf{he}] added.
\end{quote}

\textbf{my} and \textbf{he} should be marked as coreferential. 


\subsection{Anaphoric and Cataphoric Pronouns}
We are interested in marking pronouns (e.g. he, she, it, they, us,...) and their \textit{antecedent} (the thing that the pronoun refers to). For example, in the following example:

\begin{quote}
[\textbf{Alan Turing}]\textsubscript{1} was born at Paddington, London. [\textbf{His}]\textsubscript{1} father, [\textbf{Julius Mathison Turing}]\textsubscript{2}, was a British member of the Indian Civil Service and [\textbf{he}]\textsubscript{2} was often abroad...
\end{quote}

the pronoun \textbf{His} refers to \textbf{Alan Turing} - in other words the antecedent of \textbf{His} is \textbf{Alan Turing}· The pronoun \textbf{he} refers to \textbf{Julius Mathison Turing}, not to \textbf{Alan Turing}.\\

In some cases a pronoun can refer to more than one entity. Consider the following example in which the pronoun \textbf{They}, refers to two people: \textbf{John} and \textbf{Mary}, captured in the single conjoined NP antecedent \textbf{John and Mary}:

\begin{quote}
[\textbf{John and Mary}] went to the cinema. [\textbf{They}] saw a film about penguins.
\end{quote}

When a pronoun appears after its antecedent/referent in a text we call this \textit{anaphora} (the relationship is \textit{anaphoric}). The pronouns in the above examples are anaphoric. When a pronoun appears before its referent in a text we call this \textit{cataphora} (the relationship is \textit{cataphoric}). The pronoun \textbf{she} in the example below is cataphoric:

\begin{quote}
If [\textbf{she}] is in town, [\textbf{Mary}] can join us for dinner.
\end{quote}

We are only interested in cataphoric relations in which the pronoun and its referent occur in the same sentence. Also, consider the following rule for deciding if a pronoun is anaphoric/cataphoric: if the pronoun can be marked as anaphoric, mark it as such. If no possible antecedent appears before the pronoun, then consider linking it as cataphoric. (We will use the term \textit{antecedent} to refer to the NP that either a cataphoric or anaphoric pronoun refers to.)


\subsection{Speaker/Addressee Reference Pronouns}
These are defined as:
\begin{itemize}
 \item \textit{Addressee reference}: Where the pronoun primarily refers to the addressee (person being addressed)
 \item \textit{Speaker reference}: Where the pronoun primarily refers to the speaker or may not include the addressee
\end{itemize}

As a guideline:
\begin{itemize}
 \item First-person pronouns normally refer to the speaker, in the case of the singular (e.g. the English ``I''), or to the speaker and others, in the case of the plural (e.g. the English ``we'')
 \item Second-person pronouns normally refer to the person or persons being addressed (e.g. the English ``you''); in the plural they may also refer to the person or persons being addressed together with third parties
 \item Third-person pronouns normally refer to third parties other than the speaker or the person being addressed (e.g. the English ``he'', ``she'', ``it'', ``they'')
 \item Plural pronouns like ``us'', ``you'' and ``we'' may be more difficult. ``we'', ``us'' and ``our'' will most likely be speaker reference, and instances of ``you'' and ``your'' will likely be addressee reference
 \item Be aware that it may be difficult to distinguish between \textit{speaker reference} and \textit{addressee reference} in some cases
\end{itemize}


\subsection{Pleonastic Pronouns}
\label{PleonasticPronouns}
These are pronouns that do not actually refer to an entity. In other words, the pronoun could not be replaced with an NP as with a regular pronoun. Often a \textit{subject} is required by syntax i.e. something is required in that position. In some cases there will not be a \textit{subject} so a ``dummy'' pronoun is required to fill the gap. For example in the following sentences the pronoun \textbf{it} does not refer to anything but is included as something is required by the syntax of the language in the subject position:

\begin{itemize}
 \item \textbf{It} is raining
 \item \textbf{It} is well known that apples taste different from oranges
\end{itemize}

\textbf{It} is commonly used as a pleonastic pronoun in English. Other pronouns such as \textbf{they} and \textbf{you} may also be used in cases where they do not refer to a specific entity:
\begin{itemize}
 \item In this country, if \textbf{you} own a house \textbf{you} have to pay taxes
 \item \textbf{They} say you should never mix business with pleasure
\end{itemize}

In the case of \textit{pleonastic} pronouns we wish to make a partial annotation: Marking the pronoun as \textit{pleonastic}, but not linking it to anything (because it does not refer to anything).


\subsection{Identifying the Antecedent(s)}
Once an \textit{anaphoric} or \textit{cataphoric} pronoun has been identified a pronoun, its \textit{antecedent} needs to be determined. There are several cases. The pronoun may refer to:

\begin{itemize}
 \item An entity (represented by a noun or NP)
 \item An event (see Section \ref{Events})
 \item Nothing (see Section \ref{PleonasticPronouns})
 \item It may be possible to tell that a pronoun is anaphoric, but there is no specific antecedent in the text. For example the pronoun \textbf{these} in ``Access to 0800 numbers...\textbf{these} calls'' (see Section \ref{NoSpecificAntecedent})
 \item A word may have been marked as a pronoun in error (i.e. the automated pre-processing pipeline made an incorrect choice)
\end{itemize}

In order to identify what a pronoun refers to, the pronoun itself should be used as a starting point. Look back earlier in the text (working backwards sentence by sentence) until the nearest non-pronominal antecedent is identified. For example, in:

\begin{quote}
The details of [\textbf{Miyamoto Musashi}]'s early life are difficult to verify. [\textbf{Musashi}] simply states in Gorin no Sho that [\textbf{he}] was born in Harima Province 
\end{quote}

the pronoun \textbf{he} should be linked to \textbf{Musashi}, the nearest antecedent, and not to \textbf{Miyamoto Musashi} which appears earlier in the text.


\subsection{Special Case: Pronoun has Multiple Antecedents}
In cases like:
\begin{quote}
[\textbf{John}] likes documentaries. [\textbf{Mary}] likes films about animals. The last time [\textbf{they}] went to the cinema [\textbf{they}] compromised and saw a film about penguins.
\end{quote}

\textbf{They} refers to both \textbf{John} and \textbf{Mary}, who are mentioned in separate sentences so there is no NP span that covers both \textbf{John} and \textbf{Mary}. In cases like these, if all of the antecedents can be identified and it is clear from the texts what the antecedent are, the pronoun should be linked to each of the separate antecedent ``parts''. It is important to ensure that all ``parts'' are linked. 

It is important to first ensure that no NP exists that covers all parts of the antecedent.


\subsection{Special Case: They}
When the pronoun \textbf{they} is used to refer anaphorically to a collective noun (such as \textbf{the government}), it should be considered a plural pronoun and marked as such.


\subsection{Special Case: No Specific Antecedent}
\label{NoSpecificAntecedent}
For example, in the sentence:

\begin{quote}
There's a study called the streaming trials. \textbf{They} took 100 people and split them into two groups
\end{quote}

There is no antecedent in the text tow which the pronoun \textbf{They} may be linked. In this case, the pronoun should be marked as ``anaphoric but no specific antecedent''.


\subsection{Special Case: ``he or she'', ``him or her'', ``his or her'' and ``his or hers''}
English lacks ungendered person pronouns, and the former solution of just using ``male pronouns'' (e.g. ``he'') is now considered bad form. Therefore, you may come across instances of ``he or she'' in a text. For example:

\begin{quote}
If your child is thinking about a gap year, [\textbf{he or she}] can get good advice from this website.
\end{quote}

In such cases, \textbf{he or she} should be considered a single unit (or markable), just as if it had been written ``s/he'' (which is a common alternative). This solution will also make the phrase easier to resolve, since it can only be linked to a non-specific antecedent.

The same applies to instances of ``him or her'', ``his or her'' and ``his or hers''.


\subsection{Special Case: ``s/he''}
Treat this as a complete unit (or markable) and as a pronoun.


\subsection{Special case: The Pronoun Refers to a Modifier}
In some cases, the pronoun may refer to a modifier in an NP. Consider the following example:

\begin{quote}
The unionists used to be [\textbf{EU supporters}], but now they are questioning how [\textbf{it}] has developed...
\end{quote}

Here, the pronoun \textbf{it} cannot be linked to the complete NP \textbf{EU supporters}, but \textbf{it} can be linked to \textbf{EU} (the modifier). If none of the automatically generated markables are suitable, the span of an existing markable should be adjusted or a new markable created. The resulting markable may or may not be an NP.

However, with compounds like EU-supporters, these exist as a single unit and cannot be split any further (i.e. it is not possible to construct a markable that covers only the \textbf{EU} part). In such cases, it is necessary to search for a stand-alone instance of \textbf{EU} earlier in the text and link the pronoun \textbf{it} to that instance (assuming one can be found).


\subsection{How Much of a Markable to Annotate}
A markable is any pronoun, noun or NP that will be ``marked'' because it forms part of pronoun-antecedent pair, or a pronoun for which there is no antecedent to be marked. For pronouns, the markable will be a single word. For a pronoun's antecedent(s), the markable will be a noun or an NP. For noun or NP markables, the following rules apply. The markable must:

\begin{itemize}
 \item Contain the head (main) noun
 \begin{itemize}
  \item E.g. \textbf{task} is the head in \textbf{the coreference task}
  \item If the head is name then the entire name (not just a part of it) should be marked. E.g. \textbf{Frederick F. Fernwhistle Jr.} in \textbf{the Honorable Frederick F. Fernwhistle Jr.}
 \end{itemize}
 \item Also include all text which may be considered a modifier of the NP
 \begin{itemize}
  \item E.g. \textbf{the Honorable Frederick F. Fernwhistle Jr.}
  \item E.g. \textbf{Mr. Holland}
  \item E.g. \textbf{the coreference task} (where \textbf{task} is the head) -- this provides information about what the task is and separates it from \textbf{other coreference tasks}, \textbf{the scheduling task}, etc.
  \item E.g. \textbf{the big black dog} (where \textbf{dog} is the head)
  \item Determiners such as \textbf{the} should be included
 \end{itemize}
\end{itemize}

N.B. The automatically generated set of markables may contain NPs that have incorrect spans. The spans may therefore require manual adjustment.


\subsection{Relationships Between Markables}
For a pronoun and its antecedent(s), the relationship between the elements is termed as \textit{anaphoric/cataphoric}. For \textit{pleonastic} and \textit{event} pronouns there will not be a link to an antecedent markable.


\section{General Guidelines: What to Exclude}
\label{WhatToExclude}

\subsection{The Events in Event Reference}
\label{Events}
The events in event reference --- where pronouns are used to refer to an event that has happened or will happen, should not be marked. Event pronouns can refer back to whole sections of text or concepts evoked by the text. For example in:

\begin{quote}
Ted [\textbf{arrived late}]. [\textbf{This}] annoyed Mary.
\end{quote}

\textbf{This} refers to the event \textbf{arrived late}.

Another example:

\begin{quote}
Vulnerable consumers in particular might need [\textbf{specific support}] to enable them to finance necessary investments to reduce energy consumption. [\textbf{This}] task...
\end{quote}

Using deictics that vaguely refer to what the speaker is talking about (as in the above example) is bad writing, but examples like this exist in some of the texts. Here \textbf{this} should be treated as an instance of event reference.

In general, events should be easy to identify as they should contain verbs. As with the annotation of \textit{pleonastic} pronouns a partial annotation is required: The pronoun is marked as \textit{event}, but is not linked to the event itself.

Identifying pronouns that refer to events can be difficult, therefore the following simple rule is proposed:
\begin{itemize} 
 \item \textit{English:} Try replacing the pronoun with a period and then start a new sentence \textit{or} test if you can replace an instance of \textbf{which} with \textbf{this}
 \item \textit{German:} Try replacing the pronoun with a period and then start a new sentence with \textbf{das}
\end{itemize}

If the resulting ``new text'' reads OK, then it is likely that the pronoun refers to an event. As an example of how this test would work, consider the following sentence:

\begin{quote}
Ted arrived late, [\textbf{which}] annoyed Mary.
\end{quote}

Question: Is \textbf{which}" an event pronoun?

Replace the pronoun \textbf{which} with a period and start the new sentence with \textbf{This}:

\begin{quote}
Ted arrived late[\textbf{. This}] annoyed Mary.
\end{quote}

Result: Mark \textbf{which} as an event pronoun as the ``test'' passed.

If two pronouns refer to the same event, each should be marked as an \textit{event} pronoun (as opposed to marking the second as anaphoric to the first) and the two instances linked together.


\section{Special Instructions for the Annotation of Written Text: EU Bookshop}
The following instructions are specific to the annotation of written text and should be used when annotating the EU Bookshop documents.


\subsection{Reflexive Pronouns}
Reflexive Pronouns should not be marked.

In cases like ``the man himself'' we do not treat \textbf{himself} as a pronoun. Instead it should be considered an NP (the markable span can be amended in MMAX-2) if it has been automatically marked as a pronoun in error.


\subsection{Indefinite Pronouns}
An indefinite pronoun is a pronoun that refers to one or more unspecified beings, objects, or places. For example:

\begin{quote}
[\textbf{Anyone}] can see that she was looking for trouble.
\end{quote}

Here, \textbf{Anyone} is an indefinite pronoun as it does not refer to a specific person or group of people.

Indefinite pronouns should be marked as \textit{pronoun}, to indicate that they have been ``seen'' in the text. As they will be marked as instances of the type \textit{pronoun}, they will not be linked to anything, nor will any other features be recorded.

\subsection{Numbers/Quantifiers Used as Pronouns}
When deciding whether to link a pronoun to an \textit{antecedent}, the following rules apply:

\begin{itemize}
 \item many of \textbf{them} ...: \textbf{them} should be linked to its antecedent
 \item \textbf{one} of the fast growing economies: \textbf{one} should be marked as a pronoun but not linked to anything
 \item \textbf{others} ...: \textbf{others} is anaphoric and has an antecedent, but it is not coreferent with its antecedent. It should be marked as a pronoun but not linked to anything
 \item \textbf{both}: This is anaphoric, either to two individuals or two events or  situations. If \textbf{both} here is a \textit{bare} pronoun, it should be  marked and linked. If it has a head (as in ``both boys''), then it should be marked as a pronoun but not linked to anything
 \item \textbf{each}: This is anaphoric to a set. If \textbf{each} here is a \textit{bare} pronoun, it should be marked and linked. If it has a head (as in ``each boy''), then it should be marked as a pronoun but not linked to anything
\end{itemize}


\subsection{Pronominal Adverbs}
Pronominal Adverbs are a type of adverb occurring in both English and German (although they appear to be used more frequently in the German texts). They are formed by replacing a preposition and a pronoun. We wish to annotate these.

For example:
\begin{itemize}
 \item For that → therefore
 \item In that → therein
 \item By this → hereby
 \item To this → hereto
 \item In which → wherein
\end{itemize}

\subsection{Pronouns Within Quoted Text}
Identifying whether a first-person or second-person pronoun within quoted text can become difficult. Furthermore, the focus is on translating coreference in \textit{normal} text, not \textit{quoted} text. We therefore simplify the annotation task using the following rules:

\begin{itemize}
 \item First and second-person pronouns within quoted text should simply be marked as instances of type \textit{pronoun}
 \item Third-person personal pronouns should be marked as normal 
 \item In some cases, the text may read like an interview (with questions and answers) but with no quotes. In this case, the text is not to be treated as quoted text. Speaker/addressee reference pronouns should be annotated as normal.
\end{itemize}


\subsection{Difficult Choices: Deciding Between Anaphoric or Event Categories}
In some scenarios is it possible to read the text in more than one way and both readings appear to be equally likely. For example, it may be possible to mark the pronoun as either \textit{event reference} (referring to a phrase with a verb) or \textit{anaphoric} (referring to an NP), i.e. it is ambiguous. As an example, consider:

\begin{quote}
In the framework of the North Seas Countries' Offshore Grid Initiative, ENTSO-E is already conducting grid studies for northwestern Europe with a 2030 horizon. [\textbf{This}] should feed into ENTSO-E’s work for a modular development plan of a pan-European electricity highways system up to 2050. 
\end{quote}
                   
In this example, the pronoun \textbf{This} could refer to:
\begin{itemize}
 \item North Seas Countries' Offshore Grid Initiative (NP)
 \item conducting grid studies for northwestern Europe with a 2030 horizon (Verb Phrase)
\end{itemize}

In scenarios such as these, if multiple labels would be possible, select \textit{anaphoric} and link the pronoun to the NP. This will provide more information when the data is used for training translation systems.

If it is \textit{impossible} to tell what the pronoun refers to or if the text is very poorly written, the pronoun may be marked as \textit{Not sure. Help!}. This will help to identify those scenarios that are very difficult for humans (and therefore even more difficult for machines) to determine.


\section{Special Instructions for the Annotation of Spoken Text: TED Talks}
The following instructions are specific to the annotation of transcribed spoken text and should be used when annotating the TED Talks documents.

\subsection{Reflexive Pronouns}
Reflexive pronouns should be annotated in English and German.

For English:
\begin{itemize}
 \item Exclude instances of \textbf{myself} from the annotation as it is a singular first-person pronoun
\end{itemize}

For German:
\begin{itemize}
 \item Include instances of \textbf{mich} even though it is a singular first-person pronoun as it can be reflexive \textit{or} personal and it is important to make the distinction
 \item The pronouns \textbf{mich}, \textbf{dich}, \textbf{uns} and \textbf{euch} can all be used as either personal or reflexive pronouns. Mark whether they are personal or reflexive
\end{itemize}


\subsection{First-person Pronouns}
Singular first-person pronouns (I, me, etc.) do not need to be marked as they can be recovered automatically.


\subsection{Speaker Reference}
For pronouns that fall into the \textit{speaker reference} category (used for all instances of \textbf{we}), the audience should be recorded. There is an \textit{audience} attribute (in MMAX-2), which can be set as either:

\begin{itemize}
 \item \textit{Exclusive we}, meaning the speaker and his/her clique but not the audience
 \item \textit{Co-present we}, meaning the speaker plus everyone physically present in the room
 \item \textit{All-inclusive we}, incorporating everything else
\end{itemize}


\subsection{Addressee Reference}
For pronouns that fall into the \textit{addressee reference} category, the audience should be recorded. There is an \textit{audience} attribute (in MMAX-2), which can be set as either:

\begin{itemize}
 \item \textit{Deictic}, meaning that the speaker is really referring to the audience or a specific person
 \item \textit{Generic}, as in phrases such as: In England, if [\textbf{you}] own a house [\textbf{you}] have to pay taxes 
\end{itemize}

When a speaker uses deictic \textbf{you}, talking to the whole audience, it should always be marked as plural, even in cases like ``Imagine \textbf{you}'re walking alone in the woods'', where there is clearly a singular sense to the word.

For generic cases of \textbf{you}, it is not necessary to make a singular vs. plural distinction.

N.B. \textbf{you} should not be labelled as as pleonastic


\subsection{Pronouns Within Quoted Text}
These pronouns should be annotated strictly from the point of view of the quoted speaker, not of the speaker who quotes the utterance. In particular, this means:

\begin{itemize}
 \item First-person pronouns are always speaker reference
 \item Second-person pronouns are always addressee reference
 \item A coreference relation is never marked between a first-person or second-person pronoun inside quoted speech with a pronoun outside the quoted speech passage (as in ` He said, ``I do.'' ', where \textbf{he} and \textbf{I} could arguably be marked as coreferent)
\end{itemize}

In examples such as:

\begin{quote}
I said, ``[\textbf{Miguel}], what makes your fish taste so good?'' [\textbf{He}] pointed at the algae. 
\end{quote}

Do not link the pronoun \textbf{He} (outside of the quote) to \textbf{Miguel} (inside of the quote). Instead, look for an earlier instance of the entity (i.e. Miguel) in the text that does not appear in quotes and link the pronoun (i.e. \textbf{He}) to that instance.


\subsection{Extra-Textual Reference}
For cases where the speaker refers to something such as a slide or prop, the pronoun should be marked as \textit{extra-textual}. Two pronouns referring to the same object should both be marked as \textit{extra-textual} and linked together as co-referents.

The \textit{extra-textual} category can also be used within quoted text when a third-person is referred to such as the \textbf{he} in:
\begin{quote}
People when they see me say ``[\textbf{he}]'s a bit weird''
\end{quote}

N.B. This is rarely required


\subsection{No Explicit Antecedent}
In cases like:

\begin{quote}
There's a study called the streaming trials. [\textbf{They}] took 100 people and split them into two groups
\end{quote}

where there is no explicit antecedent for \textbf{They}, the pronoun should be marked as \textit{anaphoric} and the \textit{no explicit antecedent} sub-category should also be selected. 

Do not mark \textbf{They} as pleonastic.


\subsection{Split Antecedent}
This should be marked if the pronoun has multiple antecedents. All components of the antecedent should be linked to the pronoun directly, and not to each other.


\subsection{Simple Antecedent}
For all cases except where there is \textit{no specific antecedent} or there is a \textit{split reference}.


\subsection{Indefinite Pronouns, Pronominal Adverbs and Numbers/Quantifiers Used as Pronouns}
Instances of these pronouns should not be marked.


\end{document}
