% Copyright © 2004-2009 Adam Przepiórkowski
%
% This program is free software; you can redistribute it and/or
% modify it under the terms of the GNU General Public License
% as published by the Free Software Foundation; either version 2
% of the License, or (at your option) any later version.
%
% This program is distributed in the hope that it will be useful,
% but WITHOUT ANY WARRANTY; without even the implied warranty of
% MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
% GNU General Public License for more details.
%
% You should have received a copy of the GNU General Public License
% along with this program; if not, write to the Free Software
% Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
% MA 02110-1301, USA.

\author{Adam Przepiórkowski}

\section{Segmentation}
\label{sec:segmentation}

Tags are assigned to segments (tokens, roughly -- words).  Segments are not
longer than orthographic words (`from space to space'), but sometimes
segments are shorter than orthographic words:

\begin{itemize}

\item Agglutinative forms of the lexeme \textsc{być} `to be' are separate
segments, so the following words consist of two segments each:
\emph{[łgał][eś]} `lied-you', \emph{[długo][śmy]} `long time-we',
\emph{[tak][em]} `so-I'.

\item Also particles \emph{by} (subjunctive particle), \emph{-ż(e)}
(emphatic particle) and \emph{-li} (question particle) are considered to be
separate segments, so the following words consist of a number of segments:
\emph{[przyszedł][by]} `come-would', \emph{[napisała][by][m]}
`write-would-I', \emph{[chodź][że]} `come-Emph',
\emph{[potrzebował][że][by][ś]} `need-Emph-would-you', \emph{[znasz][li]}
`know-Q'.

\item The post-prepositional weak pronominal form \emph{-ń}, as in
\emph{[do][ń]} `to-him' or \emph{[ze][ń]} `with-him', is also a separate
segment.

\item Some words containing the hyphen are also split into segments, namely:

\begin{itemize}

\item words such as \emph{[polsko][-][niemiecki]} `Polish-German',

\item double names, e.g., \emph{[Kowalska][-][Nowakowska]}.

\end{itemize}

On the other hand, inflected acronyms such as \emph{PRL-u} are not
split into smaller segments.

\item Sentence-final words containing word-final full stops, e.g.,
abbreviations such as \emph{itp.} `etc.', ordinal numbers written in digits,
and initials, are also split into smaller segments, e.g.: \emph{[itp][.]},
\emph{[George] [W][.]}, etc.  The reason for that comes from the double role
of the full stop in such cases: it is a part of the word and at the same
time it plays the role of a sentence-final punctuation mark.  When such
words do not occur in sentence-final positions, they are considered to be
single segments.

\end{itemize}

The segmentation principles given above lead to the segmentation
of~\ref{seg1}. (translated into English in~\ref{seg1a}.) that is presented
in~\ref{seg2}.

\begin{enumerate}

\item \label{seg1} Pojechalibyśmy z~Janem M. Rokitą i~Janem
Nowakiem-Jeziorańskim na sesję polsko-amerykańską, gdyby nas zaprosił George
W. Byłaby to nasza już 2. doń podróż od czasów PRL-u, a~może i~3., czy
nawet~4.

\item \label{seg1a} `We would go with Jan M. Rokita and Jan
Nowak-Jeziorański to the Polish-American session, if we were invited by
George W.  That would already be our 2nd trip to him since the times of PRL,
and perhaps 3rd, or even 4th.'

\item \label{seg2} \emph{[Pojechali][by][śmy]
    [z]~[Janem] [M.] [Rokitą] [i]~[Janem]
    [Nowakiem][-][Jeziorańskim] [na] [sesję]
    [polsko][-][amerykańską][,] [gdyby] [nas]
    [zaprosił] [George] [W][.]
    [Była][by] [to] [nasza] [już] [2.]
    [do][ń] [podróż] [od] [czasów]
    [PRL-u][,] [a]~[może] [i] [3.][,]
    [czy] [nawet] [4][.]}
\end{enumerate}
