% Poliqarp statistical extension - documentation
% Copyright © 2006-2009 Aleksander Buczyński
%
% This program is free software; you can redistribute it and/or
% modify it under the terms of the GNU General Public License
% as published by the Free Software Foundation; either version 2
% of the License, or (at your option) any later version.
%
% This program is distributed in the hope that it will be useful,
% but WITHOUT ANY WARRANTY; without even the implied warranty of
% MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
% GNU General Public License for more details.
%
% You should have received a copy of the GNU General Public License
% along with this program; if not, write to the Free Software
% Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
% MA 02110-1301, USA.

\author{Aleksander Buczyński}

\section{Statistical queries}

\emph{Statistical queries are supported only by the graphical version of
Poliqarp. They are not available in the web interface.}

Originally, Poliqarp was designed as a concordancer, responding to
every query with a list of matches with contexts of selected
width. This provides the user with examples of usage of specific
constructions, but one can imagine many corpora problems in the case
of which browsing through hundreds of occurrences is neither convenient
nor efficient. The statistical extension (currently available only for the
stand-alone version of Poliqarp) introduces the possibility to easily find
answers to questions like:
\begin{itemize}
\item what is the frequency distribution of a given word's forms?
\item what parts of speech may occur after a given word?
\item what verbs are used most often in a given style?
\end{itemize}
The extension also provides several statistical measures for
collocation detection, and for investigating correlations between
individual attributes.

\subsection{Syntax synopsis}

A statistical query has the following syntax (square brackets denote
optional parts):

\texttt{<pattern> group by <attr list> \emph{[; <attr list>]} [interp <method>] [sort <order>] [min <cmin>] [count <nmax>]}

\noindent where:

\texttt{<pattern>} is a Poliqarp query as described in previous chapter;
only segment sequences matching \texttt{<pattern>} will be taken into account in the statistics;

\texttt{<attr list>} is a list of attribute specifications (for
example \texttt{base} or \texttt{2.case}), separated by commas; each
attribute specification consists of an optional segment specification
(for example \texttt{2.} or \texttt{-1.}), and an obligatory attribute
name (for example \texttt{base} or \texttt{case}); see \ref{multiword} for details;

\texttt{<method>} is an interpretation selection method
(\texttt{random} or \texttt{combine}), as described in \ref{ambiguities};

\texttt{<order>} is a sorting order, as described in \ref{sort}
(simple queries) and \ref{colfun} (queries with partial grouping);

\texttt{<cmin>} is a minimum frequency threshold; only results which
occurred at least \texttt{<cmin>} times in the matches should be displayed;

\texttt{<nmax>} is a number (or \texttt{all}) of samples selected from the results to create the statistics; see \ref{count} for explanation.

\subsection{Simple statistical queries}
\label{basic}

The pattern matches can be grouped according to a set of segment
attributes, specified in grouping rules.  The simplest grouping rule
consists of one attribute name.  For example, to find the frequencies
of the forms of the word \emph{woda} (water), one could write:

\texttt{[base=woda] group by orth}

The results of the query is a table.  Each different value of the
specified attribute (\texttt{orth}) encountered in the matches of the
first part of the query (\texttt{[base=woda]}) corresponds to one row
in the results.  Each row displays a value of the specified attribute,
and the number of matches that contain this particular value.

It is possible to include more attributes in grouping rules, separated
by commas, e.g.:

\texttt{[base=woda] group by orth, number, case}

In the results of this query, each row corresponds to a unique combination of values of the specified attributes (\emph{woda sg nom}, \emph{wody pl nom}, \emph{wody sg gen}, etc.)  This takes into account a distinction between homonymous forms (\emph{wody} may be \emph{sg gen}, \emph{pl nom}, etc.)

\subsection{Multiword patterns}
\label{multiword}

Patterns can return matches longer than a single segment. To
specify the segment whose attribute will be used for grouping, one
should add the segment number (with a dot) before the name of the
attribute. For example, to find all verbs occurring immediately after
the word \emph{woda} (water), one can type:

\texttt{[base=woda][pos=verb] group by 2.base}

Specification \texttt{2.base} refers to the base form of the second
segment of the match (the \emph{verb} after \emph{woda}).  Note that
subsequent numbers refer to the subsequent segments of the match, not
segment specifications in the query. For example, to find the
frequency distribution of three subsequent adverbs, the user should
type (\texttt{1.} can be skipped):

\texttt{[pos=adv]\lbr{}3\rbr{} group by 1.base, 2.base, 3.base}

To make it possible to address segments in matches of possibly
variable length, negative numbers can be used as segment
specifications.  Such numbers mean counting from the end of the match.
For example, to allow an optional adverb between \emph{woda} and the
verb, the query should be modified as follows:

\texttt{[base=woda][pos=adv]?[pos=verb] group by -1.base}

Specification \texttt{-1.base} specifies the base form of last segment
of the result. Similarly, \texttt{-2.} would refer to the second last,
\texttt{-3.} --- third last, etc.

\subsection{Ambiguities}
\label{ambiguities}

By default, one random interpretation of each segment is chosen for
grouping. But if \texttt{interp combine} is added after an attribute
specification, the value of the attribute will be calculated as a
concatenation of all the unique values of the attribute in all the
interpretations, separated by a vertical bar.  For example, to find
all possible interpretations' combination for word forms that may be a
form of the verb \emph{mieć}, one could write:

\texttt{[base\textasciitilde mieć] group by base interp combine}

The results of such query will include word pairs like
\emph{mama/mieć} (mom/to have), \emph{mieć/mienie} (to
have/property), \emph{maić/mieć} (to decorate with leaves and
flowers/to have), \emph{mielić/mieć} (to grind/to have),
etc.  The results will vary significantly depending on the
value of the ``Show only disambiguated results'' option in Poliqarp
configuration; if it is checked, the interpretations discarded by the
tagger will not appear in the results.

\subsection{Sample size}
\label{count}

To return results quickly, the statistics are by default calculated on basis of 1000 randomly choosen samples from the pattern matches.  If one needs a more precise number, the sample size can be adjusted with the keyword \texttt{count}, for example:

\texttt{[base=woda] group by orth count 5000}

To count all pattern matches simply use \texttt{count all}, for example:

\texttt{[base=woda] group by orth count all}

Note: as for version 1.2 there is a hardcoded limit of 500\,000 matches.  If you really need a bigger sample, you can split your query into a few more specific ones (for example divide it by \texttt{meta style}).

\subsection{Results sorting and selection}
\label{sort}

Results can be sorted in alphabetic order (\texttt{sort a fronte}) or
according to their frequency (\texttt{sort by freq}). If partial
grouping is used, the results can be also sorted according to a
collocation function --- see \ref{colfun} for details.

The results selection is now limited to a frequency threshold
(\texttt{min n}).

\subsection{Partial grouping}
\label{partial_grouping}

It is quite easy to find most frequent bigrams using only the basic
syntax:

\texttt{[][] group by 1.base, 2.base sort by freq}

However, such results are often insufficient. For example, in
collocation detection, not only the bigram frequency, but also the
frequencies of its constituents have to be taken into
account. Therefore, a special separator --- semicolon ``;'' --- has
been introduced, which makes it possible to split the grouping rules
into two parts.  For example:

\texttt{[][] group by 1.base; 2.base sort by freq}

will cause the program to group the results by: \texttt{1.base} (the
part before the semicolon); \texttt{2.base} (the part after the
semicolon); \texttt{1.base, 2.base} (both).  For each line of the
results of the last grouping, the results of the partial groupings
should also be displayed.  The \texttt{sort} and \texttt{min}
modifiers are always applied to the last grouping.

Each of the grouping parts may include more than one attribute, but
the grouping may have no more than two parts. In other words: the
grouping rules can include any number of commas, but not more than one
semicolon.

The syntax does not necessarily have to be used for bigrams.
Different grouping parts may even include references to the same
segment, for example:

\texttt{[base=woda] group by number; case}

\subsection{Collocation functions}
\label{colfun}

A few dependency measures for statistical detection of collocations
have been added as possible sorting parameters (for example
\texttt{sort by cp} or \texttt{sort by dice}) in queries with partial
grouping (see \ref{partial_grouping}).  All the currently implemented measures are functions of the
following parameters:

$c(w_1)$ --- number of occurrences of $w_1$, where $w_1$ is the combination of values of the attributes defined by the first part of
  the grouping rules;

$c(w_2)$ --- number of occurrences of $w_2$, where $w_2$ is the combination of values of the attributes defined by the second part
  of the grouping rules;

$c(w_1w_2)$ --- number of occurrences of the combination of $w_1$ and $w_2$.

The functions are:
\begin{itemize}
\item \texttt{cp} --- conditional probability
\[\mathrm{cp}(w_1,w_2) = \frac{c(w_1w_2)}{c(w_1)}\]
\item \texttt{scp} --- symmetric conditional probability
\[\mathrm{scp}(w_1,w_2) = \frac{c(w_1w_2)^2}{c(w_1)c(w_2)}\]
\item \texttt{maxcp} --- maximum conditional probability
\[\mathrm{maxcp}(w_1,w_2) = \max\left(\frac{c(w_1w_2)}{c(w_1)}, \frac{c(w_1w_2)}{c(w_2)}\right)\]
\item \texttt{dice} --- Dice's formula
\[\mathrm{dice}(w_1,w_2) = \frac{2 c(w_1w_2)}{c(w_1)+c(w_2)}\]
\end{itemize}

Because dependency measures in fact prefer rare bigrams, a minimum frequency threshold is recommended, for example:

\texttt{[pos!=interp]\lbr{}2\rbr{} group by base; 2.base sort by scp min 2}

An alternative approach is to add some frequency bias to the dependency
test value, for example:

\texttt{[pos!=interp]\lbr{}2\rbr{} group by base; 2.base sort by scp bias 0.5}

\texttt{bias $b$} means ``before sorting, multiply the function results
by power $b$ of frequency'':

\[x \: \mathrm{bias} \: b = x \: c(w_1w_2)^b\]

For example:

\[\mathrm{scp} \: \mathrm{bias} \: 0.5 = \frac{c(w_1w_2)^{2,5}}{c(w_1)c(w_2)}\]

Of course, \texttt{bias} and \texttt{min} keywords can be
combined, for example:

\texttt{[]\lbr{}2\rbr{} group by base; 2.base sort by scp bias 0.5 min 2}

For consistency, the minimum threshold is still applied to the bare frequency, not the biased collocation function.

\end{document}
