Repository navigation
Expand file tree
/
Copy pathchanges.tex
More file actions
2304 lines (1931 loc) · 106 KB
/
Copy pathchanges.tex
File metadata and controls
2304 lines (1931 loc) · 106 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
% changes.tex
%
% diana, Oct 2002
%
% $Id: changes.tex,v 1.78 2006/10/23 12:56:36 ian Exp $
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\chapt[chap:changes]{Change Log}
\markboth{Change Log}{Change Log}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
This \chapthing\ lists major changes to GATE (currently Developer and Embedded
only) in roughly chronological order by
release. Changes in the documentation are also referenced here.
%NEW CHANGES SHOULD BE ADDED IN `recent-changes.tex' NOT HERE!!!
\input{recent-changes}
\rcSect[8.4.1]{Version 8.4.1 (June 2017)}
This is a minor release that fixes one rarely encountered but serious bug with
the handling of CDATA sections within the text content of GATE XML format
documents. CDATA has always been handled correctly in annotation and document
feature values, this bug would only affect a small number of documents where
the text contains many less-than signs (\verb!<<<!) \emph{and} few annotations.
In particular, \emph{annotated} documents that have been processed using the
GATE tokeniser are extremely unlikely to be affected as each less-than sign is
treated as a separate \texttt{Token} annotation.
This release also includes one small improvement to the Twitter hashtag
tokeniser so it recognises the names of some political parties when they occur
within hashtags such as \verb!#VoteLabour!.
\rcSect[8.4]{Version 8.4 (February 2017)}
GATE Developer and Embedded 8.4 is mainly a bug fix release, with a small
number of critical fixes compared to version 8.3. This will be the final
major release of GATE before major re-structuring of the codebase and the plugin
system for version 8.5.
\begin{itemize}
\item Fixed an issue which had prevented the use of Java 8 lambda expressions
in the RHS of JAPE rules, even when running on Java 8.
\item Removed OpenCalais and Zemanta plugins as the web services they depend on
have changed and the plugins no longer work.
\item Fixed a bug that could cause the searchable datastore GUI to freeze.
\item Fixes to the TermRaider and Hindi sample applications
\end{itemize}
\rcSubsect{Java compatibility}
For GATE 8.4 we recommend the use of the latest Java 8 from Oracle. If you are
still restricted to Java 7, most components will still work with the exception
of the Stanford CoreNLP tools and the TwitIE application (which uses the
Stanford POS tagger). Future versions of GATE will require Java 8 as a
minimum.
\rcSect[8.3]{Version 8.3 (January 2017)}
GATE Developer and Embedded 8.3 is mainly a bug fix release, with several
critical fixes and functionality improvements.
\begin{itemize}
\item JAPE grammars can now match and create annotation types and features
with spaces or punctuation in their names, by using double quotes around
the type or feature name (e.g. \verb!{"w:p"}!).
\item Fixed a regression in 8.2 that meant saved application states and
``export for GATE Cloud'' packages created on Windows would not load on other
platforms.
\item Fixed a bug in the Stanford CoreNLP plugin which would sometimes fail
when GATE is installed in a directory whose path contains spaces.
\item Various improvements to the Twitter normaliser and emoticon finder.
\item Improvements to the \verb!Lang_French! and \verb!Lang_German! components.
Further improvements will follow in the next release.
\item Improvement to the \verb!Crowd_Sourcing! plugin to allow a default
option to be specified for clasification jobs.
\item Fixed Java version detection in the Windows EXE launcher.
\item Detection of document format using clues from the content is now much
more efficient.
\item Fixed some GUI deadlocks in the searchable data store GUI and the plugin
manager.
\item Fixed a long-standing bug in the regex sentence splitter for documents
with long sequences of blank lines.
\item Removed Minipar parser plugin as the data files on which it depends are
no longer available for download, and the \verb!Tagger_NormaGene! plugin as
the service it relies on is no longer online.
\end{itemize}
Plus the usual suite of miscellaneous smaller bug fixes.
\rcSubsect{Java compatibility}
For GATE 8.3 we recommend the use of the latest Java 8 from Oracle. If you are
still restricted to Java 7, most components will still work with the exception
of the Stanford CoreNLP tools and the TwitIE application (which uses the
Stanford POS tagger). Future versions of GATE will require Java 8 as a
minimum.
\rcSect[8.2]{Version 8.2 (May 2016)}
GATE Developer and Embedded 8.2 is mainly a bug fix release -- there are a few
new plugins but the emphasis is on bug fixing and library updates.
\begin{itemize}
\item New tools for temporal expression and event detection, including a wrapper
for the HeidelTime tagger (section~\ref{sec:misc-creole:gate-time}).
\item New language plugins for Danish and Welsh named entitiy recognition.
\item Performance improvements in the ANNIE NER system, in particular to deal
better with hyphenated names and titles.
\item Improvements to TermRaider to support GATE documents that contain many
independent sections (e.g. web forums, lists of tweets).
\item Bug fixes in the handling of Twitter JSON data -- GATE now has full
round-trip support for Twitter JSON, tweets can be loaded, annotated, and
saved back to the same format accurately. The JSON format parser has been
separated from the rest of the Twitter plugin, making it easier to add JSON
support to non-TwitIE applications.
\item Updated dependencies -- the \verb!Stanford_CoreNLP! plugin now uses
version 3.6.0 of Stanford CoreNLP, and the Groovy plugin uses Groovy version
2.4.4
\item GCP input and output handlers added to the \verb!Format_CSV! plugin.
\end{itemize}
Plus the usual suite of miscellaneous smaller bug fixes.
\rcSubsect{Java compatibility}
For GATE 8.2 we recommend the use of the latest Java 8 from Oracle. If you are
still restricted to Java 7, most components will still work with the exception
of the Stanford CoreNLP tools. Future versions of GATE will require Java 8 as
a minimum.
\rcSect[8.1]{Version 8.1 (June 2015)}
\rcSubsect{New plugins and significant new features}
\begin{itemize}
\item Integration of the Stanford NER tools -- all the Stanford tools in GATE
have been brought together under a single \texttt{Stanford\_CoreNLP} plugin.
\item Improved crowdsourcing tools (chapter~\ref{chap:crowd}), including tools
to perform automatic adjudication of multiply-annotated documents.
\item Parsers for new document formats, including the \emph{DataSift} format
(section~\ref{sec:creole:datasift}) for social media data, and an improved
Twitter JSON parser (section~\ref{sec:social:twitter:format}) which can import
the standoff annotations Twitter themselves provide (hashtags, etc.).
\item Improved support for data \emph{export}, making it easy for plugins to
add their own export formats accessible through the GUI and the API. New
exporters are provided for the Twitter JSON format
(section~\ref{sec:social:twitter:export}) and a more configurable inline
XML format.
\item A new plugin for simplifying sentences using linguistic rules and other
information (section~\ref{sec:misc-creole:linguistic-simplifier}), contributed
by the \htlink{https://www.forgetit-project.eu}{ForgetIT project}.
\end{itemize}
\rcSubsect{Library updates and bugfixes}
\begin{itemize}
\item Apache Tika (for parsing PDF, MS Word, etc.) updated to version 1.7.
\item ASM (for processing CREOLE metadata) updated to version 5.0.3. This
allows the use of Java 8 language features such as lambdas in third-party
plugins, though GATE itself remains compatible with Java 7.
\item Stanford CoreNLP tools updated to version 3.4 (the latest version
that is compatible with Java 7).
\item MetaMap libraries (for UMLS) updated to version 2014.
\item Better support in the GATE Unicode Tokeniser for scripts that use
supplementary characters beyond the basic 16 bit range.
\item Bugfixes in the segment processing PR.
\end{itemize}
\rcSubsect{Tools for developers}
Three new tools have been added to the \verb!Developer_Tools! plugin:
\begin{itemize}
\item Menu options to produce Java heap dumps to aid debugging.
\item Menu option to dynamically increase the Log4J logging verbosity
at runtime.
\item A tool that attempts to unload all plugins that are loaded but not
currently in use.
\end{itemize}
Other changes that benefit developers include:
\begin{itemize}
\item New helper methods in the \verb!gate.Utils! class.
\item The \verb!gate.util.ProcessManager! API has been extended to allow an
external process to be kept running. This is especially useful for running
some external tools which have very long starup times yet can be reused
across documents.
\item Some changes in the management of classloaders that should reduce the
potential for deadlocks.
\end{itemize}
\ldots and as always, a range of smaller improvements and bug fixes.
\rcSect[8.0]{Version 8.0 (May 2014)}
GATE 8.0 is a major release which brings some major new features, many new and
updated plugins, and significant under-the-bonnet changes to GATE Embedded.
\rcSubsect{Major changes}
\textbf{Java 7 required}
GATE 8.0 requires \textbf{Java 7} or later to run.
\textbf{Tools for Twitter}
A new ``Twitter'' plugin provides tools dedicated to Twitter data:
\begin{itemize}
\item format parsers to handle Tweets in the JSON formats produced by the
Twitter APIs
\item Twitter-specific components such as a tokeniser and POS tagger
\item the \emph{TwitIE} named entity annotation pipeline.
\end{itemize}
See section~\ref{sec:social:twitter} for full details.
\textbf{ANNIE Refreshed}
The ANNIE named entity annotation pipeline which has been the mainstay of many
GATE applications for many years has been brought up to date, with new
gazetteers and improved JAPE grammars giving improved precision and recall on
common test corpora.
\textbf{Tools for Crowd Sourcing}
A new \verb!Crowd_Sourcing! plugin provides facilities to support generation of
manually annotated corpora via the CrowdFlower crowdsourcing
platform\footnote{\url{http://www.crowdflower.com}}. The plugin provides
support for two different kinds of tasks, general entity annotation (e.g.
determining which words in a given sentence are person names) and entity
linking (e.g. for ontology-based annotation, where the spans of the entities
are known but not which particular ontology instance each annotation
corresponds to). Using crowdsourcing you can generate multiply-annotated gold
standard corpora rapidly and at relatively low cost. For full details see
chapter~\ref{chap:crowd}.
\rcSubsect{Other new and improved plugins}
\begin{itemize}
\item New language plugins to support \emph{Russian} and \emph{Bulgarian}
\item Integration of the Stanford POS Tagger
(section~\ref{sec:misc:creole:stanford}), which is used by TwitIE
\item A document normalizer plugin, predominantly to normalize punctuation such
as Microsoft Word ``ssec:creole:datasiftmart quotes'' (see
section~\ref{sec:misc-creole:doc-normalizer})
\item Wrappers for the \emph{AlchemyAPI} keyword and entity extraction services
(in the \verb!AlchemyAPI! plugin)
\item Wrapper for the \emph{TextRazor} annotation service (see
section~\ref{sec:misc-creole:textrazor}).
\item New document format parser to populate a GATE corpus from one or more CSV
files (see section~\ref{sec:creole:csv}).
\item Support for loading and saving GATE XML files in the binary
\emph{FastInfoset} format (see section~\ref{sec:creole:fastinfoset}).
\item Various improvements to the \verb!Learning! plugin, in particular to
support numeric and boolean features
\item Improvements to the \emph{TermRaider} term extraction plugin (see
section~\ref{sec:creole:termraider})
\item The OntoRoot Gazetteer (in the \verb!Gazetteer_Ontology_Based! plugin)
now supports tokenisers and POS taggers other than the default ANNIE PR
types, making it possible to use other preprocessing tools for non-English
data
\item Further improvements to the classloading model to better isolate plugins
from one another.
\item A new \verb!enableDebugging! runtime parameter for JAPE grammars will add
additional features to every generated annotation detailing which rule was
responsible for creating the annotation.
\end{itemize}
\rcSubsect{Bug fixes and other improvements}
\begin{itemize}
\item The annotation schema LR type is now available by default without the
need to load any plugins. The schemas that were previously loaded by default
by the ANNIE plugin must now be loaded explicitly if you require them
(section~\ref{sec:developer:schemaannotationeditor}). Annotation schemas now
support the \verb!include! element, so multiple schemas can be loaded by
loading a single master file.
\item The segment processing PR
(section~\ref{sec:alignment:segment-processing}) now preserves annotation
IDs, allowing ID-sensitive tools such as coreference to work properly.
\end{itemize}
\rcSubsect{For developers}
Changes of note for users of the GATE Embedded APIs include:
\begin{itemize}
\item A new data model to represent relations between annotations, see
section~\ref{sec:api:relations} for details. The \verb!Coref_Tools! plugin
has been retrofitted to use this new model to represent coreference
chains. Relation information is preserved when saving documents as GATE XML,
but note that such documents will not be compatible with older versions of
GATE.
\item A new ``resource helper'' mechanism allows plugins to contribute
additional actions to existing resource types, both in the Developer GUI
(section~\ref{sec:creole-model:tools:resourcehelpers}) and in the Embedded API
(section~\ref{sec:api:resourcehelpers})
\item A new class \verb!gate.corpora.DocumentJsonUtils! provides methods to
export a GATE document in a JSON format compatible with that used by Twitter.
See the JavaDoc documentation for details.
\item Many deprecated classes, fields and methods have been removed. If you
were previously calling any of these deprecated APIs you will need to update
your code accordingly. Also some classes in the GATE core that were only
used by one plugin have been moved into the respective plugin's source tree.
In particular, Java RHS actions in JAPE rules no longer provide the
long-deprecated \verb!annotations! variable -- use \verb!inputAS! or
\verb!outputAS! as appropriate.
\item Many library dependencies have been updated to more recent versions.
\item The GATE APIs make much wider use of generics than previously -- many
places in the code that previously used raw types are now properly generic
\item A new \verb!Developer_Tools! plugin
(section~\ref{sec:misc-creole:dev-tools}) provides utilities to assist in
debugging applications in GATE Developer.
\end{itemize}
If you are working on the core GATE source code, note that:
\begin{itemize}
\item the source tree has been split into ``main'' and ``test'', isolating the
test classes from the rest of the source
\item each plugin is now a separate Eclipse ``project'', and the main project
is just the core sources, which makes it easier to control dependencies among
the different parts
\item dependencies are no longer checked in to subversion, instead they are
fetched at build time from the Maven central repository by Apache Ivy.
\end{itemize}
\rcSect[7.1]{Version 7.1 (November 2012)}
\rcSubsect{New plugins}
The \emph{TermRaider} plugin (see Section~\ref{sec:creole:termraider})
provides a toolkit and sample application for term extraction.
Two new plugins, \emph{Tagger\_Zemanta} (since removed) and
\emph{Tagger\_Lupedia} provide PRs that wrap online annotation services provided
by Zemanta and Ontotext.
A new plugin named \emph{Coref\_Tools} includes a framework for fast
co-reference processing, and one PR that performs orthographical co-reference in
the style of the ANNIE Orthomatcher. See Section~\ref{sec:creole:coref-tools}
for full details.
A new \emph{Configurable Exporter} PR in the Tools plugin, allowing annotations
and features to be exported in formats specified by the user (e.g. for use with
external machine learning tools). See Section~\ref{sec:misc-creole:confexport}
for details.
Support for reading a number of new document formats has also been added:
\begin{itemize}
\item \emph{PubMed and the Cochrane Library} formats (see
Section~\ref{sec:creole:pubmed}).
\item \emph{CoNLL ``IOB''} format (see Section~\ref{sec:corpora:conll}).
\item\emph{MediaWiki} markup, both plain text and XML dump files such as those
from Wikipedia (see Section~\ref{sec:creole:mediawiki}).
\end{itemize}
In addition, ``ready-made applications'' have been added to many existing
plugins (notably the \emph{Lang\_*} non-English language plugins) to make it
easier to experiment with their PRs.
\rcSubsect{Library updates}
Updated the Stanford Parser plugin (see Section~\ref{sec:parsers:stanford}) to
version 2.0.4 of the parser itself, and added run-time parameters to the PR to
control the parser's dependency options.
The Measurement and Number taggers have been upgraded to use JAPE+ instead of
JAPE. This should result in faster processing, and also allows for more memory
efficient duplication of PR instances, i.e. when a pool of applications is
created.
The OpenNLP plugin has been completely revised to use Apache OpenNLP 1.5.2 and
the corresponding set of models. See Section~\ref{sec:misc-creole:opennlp} for
details.
The native launcher for GATE on Mac OS X now works with Oracle Java 7 as well
as Apple Java 6.
\rcSubsect{GATE Embedded API changes}
Some of the most significant changes in this version are ``under the bonnet''
in GATE Embedded:
\begin{itemize}
\item The class loading architecture underlying the loading of plugins and the
generation of code from JAPE grammars has been re-worked. The new version
allows for the complete unloading of plugins and for better memory handling of
generated classes. Different plugins can now also use different versions of the
same 3rd party libraries. There have also been a number of changes to the way
plugins are (un)loaded which should provide for more consistent behaviour.
\item The GATE XML format has been updated to handle more value types
(essentially every data type supported by XStream
(\url{http://xstream.codehaus.org/faq.html}) should be usable as feature name
or value. Files in the new format can be opened without error by older GATE
versions, but the data for the previously-unsupported types will be
interpreted as a String, containing an XML fragment.
\item The PRs defined in the ANNIE plugin are now described by annotations on
the Java classes rather than explicitly inside creole.xml. The main reason for
this change is to enable the definitions to be inherited to any subclasses of
these PRs. Creating an empty subclass is a common way of providing a PR with a
different set of default parameters (this is used extensively in the language
plugins to provide custom gazetteers and named entity transducers). This has
the added benefit of ensuring that new features also automatically percolate
down to these subclasses. If you have developed your own PR that extends one
of the ANNIE ones you may find it has acquired new parameters that were not
there previously, you may need to use the \verb!@HiddenCreoleParameter!
annotation to suppress them.
\item The corpus parameter of LanguageAnalyser (an interface most, if not all,
PRs implement) is now annotated as \verb|@Optional| as most implementations
do not actually require the parameter to be set.
\item When saving an application the plugins are now saved in the same order in
which they were originally loaded into GATE. This ensures that dependencies
between plugins are correctly maintained when applications are restored.
\item API support for working with relations between annotations was added. See
Section~\ref{sec:api:relations} for more details.
\item The method of populating a corpus from a single file has been updated to
allow any mime type to be used when creating the new documents.
\end{itemize}
And numerous smaller bug fixes and performance improvements\ldots
\rcSect[7.0]{Version 7.0 (February 2012)}
\rcSubsect{Major new features}
The CREOLE Plugin Manager has been completely re-written and now includes
support for installing new plugins from remote update sites. See
section~\ref{sec:developer:installplugins} for more details. In addition,
plugins can now contribute additional ``ready-made applications'' to the GATE
Developer menus alongside the standard applications (ANNIE, etc.). Details can
be found in section~\ref{sec:development:readymade}.
A new plugin named \texttt{JAPE\_Plus} has been added. It contains a new JAPE
execution engine that includes various optimisations and should be significantly
faster than the standard engine. \texttt{JAPE\_Plus} has not yet been
comprehensively tested, so it should be considered {\em beta} software, and used
with caution. See Section~\ref{sec:jape:plus} for more details.
A new Java-based launcher has been implemented which now replaces the use of
Apache ANT for starting-up GATE Developer. The GATE Developer application now
behaves in a more natural way in dock-based desktop environments such as Mac OS
X and Ubuntu Unity.
Improved the support for processing biomedical text by adding new PRs to
incorporate the following tools: AbGene, the NormaGene tagger, the GENIA
sentence splitter, MutationFinder and the Penn BioTagger (contains a tokenizer
and three taggers for gene, malignancy and variation). For full details of
these new resources see section~\ref{sec:domain-creole:biomed}.
The Flexible Gazetteer PR has been rewritten to provide a better and faster
implementation. The two parameters \texttt{inputAnnotationSetName} and
\texttt{outputAnnotationSetName} have been renamed to \texttt{inputASName} and
\texttt{outputASName}, however old applications with the old parameters
should still work. Please see Section \ref{sec:gazetteers:flexgazetteer} for
more details.
\rcSubsect{Removal of deprecated functionality}
Various components were removed in this release as they have been unsupported
and deprecated in previous releases:
\begin{itemize}
\item the GATE Unicode Kit (GUK), which has been superseded by improved native
support for localisation in the various target operating systems. If you
still require GUK it is available as a separate software project at
\url{http://gate.svn.sourceforge.net/viewvc/gate/guk/trunk}.
\item the database-backed datastore implementation.
\item the plugins Jape\_Compiler (superseded by JAPE\_Plus) and
Ontology\_OWLIM2.
\end{itemize}
In addition the Web\_Search\_Google, Web\_Search\_Yahoo and
Web\_Translate\_Google plugins have been removed as the underlying web services
on which they depend are no longer available. Documentation for obsolete
plugins can be found in appendix~\ref{chap:obsolete-plugins}, and if you
require any of them for your application please see
\verb|plugins/Obsolete/README.TXT| in the GATE Developer distribution.
\rcSubsect{Other enhancements and bug fixes}
CREOLE plugins can now use \htlink{http://ant.apache.org/ivy/}{Apache Ivy} to
include third-party dependencies. See section~\ref{sec:creole-model:config} for details.
The Default ANNIE Gazetteer now allows a user to specify different annotation
types to be used for annotating entries from different lists. For example, a
user may want to find city names mentioned in a gazetteer list (e.g. city.lst)
and annotate the matching strings as City. Please see
section~\ref{sec:annie:gazetteer} for more details.
The Segment Processing PR has two additional run-time parameters called
\texttt{segmentAnnotationFeatureName} and
\texttt{segmentAnnotationFeatureValue}. These features allow users to specify
a constraint on feature name and feature value. If user has provided values
for these parameters, only the annotations with the specified feature name and
feature value are processed with the Segment Processing PR. Also, the
parameter \texttt{controller} has been renamed to \texttt{analyser} which means
the Segment Processing PR can now also run an individual PR on the specified
segments\footnote{Existing saved applications using the \texttt{controller}
parameter will still work \emph{provided} the controller in question implements
the \texttt{LanguageAnalyser} interface. The \texttt{CorpusController}
implementations supplied as standard with GATE all implement this interface.}.
See \ref{sec:alignment:segment-processing} for more information on
section-by-section processing.
The Hash Gazetteer (section~\ref{sec:gazetteers:hash}) now properly supports
the \texttt{caseSensitive} parameter (previously the parameter could be set but
had no effect).
The Document Reset PR (Section~\ref{sec:misc-creole:reset}) now defaults to
keeping the Key set as well as Original markups. This makes working with
pre-annotated gold standard document less dangerous (assuming you put the gold
standard annotations in a set called Key).
Updated Stanford Parser plugin (see Section~\ref{sec:parsers:stanford}) to
version~1.6.8.
The TextCat based Language Identification PR now supports generating new
language fingerprints. See
section~\ref{sec:misc-creole:language-identification} for full details.
Added support for reading XCAS and XMI-format documents created by UIMA. See
section~\ref{sec:corpora:uima} for details.
Various improvements to the GATE Developer GUI:
\begin{itemize}
\item added support in the document editor to switch the principal text
orientation, to better support documents written in right-to-left languages
such as Arabic, Hebrew or Urdu (section~\ref{sec:developer:documents}).
\item added new mouse shortcuts to the Annotation Stack view in the document
editor to speed up the curation process
(section~\ref{sec:developer:annotationsstackview}).
\item the document editor layout is now saved to the user preferences file,
gate.xml. It means that you can give this file to a new user so s/he will
have a preconfigured document editor
(section~\ref{sec:developer:documents}).
\item the script behind an instance of the Groovy Scripting PR
(section~\ref{sec:api:groovy:pr}) can now be edited from within GATE
Developer through a new visual resource which supports syntax highlighting.
\end{itemize}
The rule and phase names are now accessible in a JAPE Java RHS by the
\verb=ruleName()= and \verb=phaseName()= methods and the name of the
JAPE processing resource executing the JAPE transducer is accessible
through the action context \verb=getPRName()= method.
See section~\ref{sec:jape:javarhsoverview}.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\rcSect[6.1]{Version 6.1 (April 2011)}
\rcSubsect{New CREOLE Plugins}
\textbf{Tagger\_Numbers} to annotate many kinds of numbers in documents and
determine their numeric values. The tagger can annotate numbers expressed in
many forms including Arabic and Roman numerals, words (in English, French,
German and Spanish) and scientific notation (4.3e6 = 4300000). See
section~\ref{sec:misc-creole:numbers} for full details.
\textbf{Tagger\_Measurements} to annotate many different forms of measurement
expressions (``5.5 metres'', ``1 minute 30 seconds'', ``10 to 15 pounds'',
etc.) along with their normalized values in SI units. See
section~\ref{sec:misc-creole:measurements} for full details.
\textbf{Tagger\_Boilerpipe}, which contains a
boilerpipe\footnote{\htlinkplain{http://code.google.com/p/boilerpipe/}} based
PR for performing content detection. See
section~\ref{sec:misc-creole:boilerpipe} for full details.
\textbf{Tagger\_DateNormalizer} to annotate and normalize dates within a
document. See section~\ref{sec:misc-creole:datenormalizer} for full details.
\textbf{Schema\_Tools} providing a ``Schema Enforcer'' PR that can be used to
create a clean output annotation set based on a set of annotation schemas. See
section~\ref{sec:misc-creole:schemaenforcer} for full details.
\textbf{Teamware\_Tools} providing a new PR called QA Summariser for Teamware.
When documents are annotated using GATE Teamware, this PR can be used for
generating a summary of agreements among annotators. See
section~\ref{sec:eval:qaForTW} for full details.
\textbf{Tagger\_MetaMap} has been rewritten to make use of the new MetaMap Java
API features. There are numerous performance enhancements and bug fixes
detailed in section~\ref{sec:misc-creole:metamap}. Note that this version of
the plugin is \emph{not} compatible with the version provided in GATE 6.0,
though this earlier version is still available in the Obsolete directory if
required.
\rcSubsect{Other new features and improvements}
Added support for handling controller events to JAPE by making it possible
to define \texttt{ControllerStarted}, \texttt{ControllerFinished}, and
\texttt{ControllerAborted} code blocks in a JAPE file (see
section~\ref{sec:jape:javarhsoverview}).
JAPE Java right-hand-side code can now access an \texttt{ActionContext} object
through the predefined field \texttt{ctx} which allows access to the corpus LR
and the transducer PR and their features (see
section~\ref{sec:jape:javarhsoverview}).
Three new optional attributes can be specified in {\tt<GATECONFIG>}
element of {\tt gate.xml} or local configuration file:
\begin{itemize}
\item \textbf{addNamespaceFeatures} - set to ``true'' to deserialize namespace prefix and URI information as features.
\item \textbf{namespaceURI} - The feature name to use that will hold the namespace URI of the element, e.g. ``namespace''
\item \textbf{namespacePrefix} - The feature name to use that will hold the namespace prefix of the element, e.g. ``prefix''
\end{itemize}
Setting these attributes will alter GATE's default namespace deserialization
behaviour to remove the namespace prefix and add it as a feature, along with
the namespace URI. This allows namespace-prefixed elements in the
\textbf{Original markups} annotation set to be matched with JAPE expressions,
and also allows namespace scope to be added to new annotations when serialized
to XML. See \ref{sec:corpora:input} for details.
Searchable Serial Datastores (Lucene-based) are now portable and can be moved
across different systems. Also, several GUI improvements have been made to ease
the creation of Lucene datastores. See chapter~\ref{chap:annic} for details.
The populate method that allowed populating corpus from a trecweb file has been
made more generic to accept a tag. The method extracts content between the start
and end of this tag to create new documents. In GATE Developer, right-clicking
on an instance of the Corpus and choosing the option ``Populate from Single
Concatenated File" allows users to populate the corpus using this functionality.
See Section \ref{sec:api:corpora} for more details.
Fixed a regression in the JAPE parser that prevented the use of RHS macros that
refer to a LHS label (named blocks \verb|:label { ... }| and assignments
\verb|:label.Type = {}|
Enhanced the Groovy scriptable controller with some features inspired by the
realtime controller, in particular the ability to ignore exceptions thrown by
PRs and the ability to limit the running time of certain PRs. See
section~\ref{sec:api:groovy:controller} for details.
The Ontology and Gazetteer\_LKB plugins have been upgraded to use Sesame 3.2.3
and OWLIM 3.5.
The Websphinx Crawler PR has new runtime
parameters for controlling the maximum page size and spoofing the user-agent.
A few bug fixes and improvements to the ``recover'' logic of the
\texttt{packagegapp} Ant task (see section~\ref{sec:ant:packagegapp}).
\ldots and many other smaller bugfixes.
\textbf{Note: As of version 6.1, GATE Developer and Embedded require Java 6 or
later and will no longer run on Java 5}. If you require Java 5 compatibility
you should use GATE 6.0.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\rcSect[6.0]{Version 6.0 (November 2010)}\ifnested\label{subsec:changes:6.0b1}\else\label{sec:changes:6.0b1}\fi
\rcSubsect{Major new features}
Added an annotation tool for the document editor: the Relation Annotation
Tool (RAT). It is designed to annotate a document with ontology instances
and to create relations between annotations with ontology object
properties. It is close and compatible with the Ontology Annotation Tool
(OAT) but focus on relations between annotations. See
section~\ref{sec:ontologies:rat} for details.
Added a new \emph{scriptable controller} to the Groovy plugin, whose execution
strategy is controlled by a simple Groovy DSL. This supports more powerful
conditional execution than is possible with the standard conditional
controllers (for example, based on the presence or absence of a particular
annotation, or a combination of several document feature values), rich flow
control using Groovy loops, etc. See
section~\ref{sec:api:groovy:controller} for details.
A new version of Alignment Editor has been added to the GATE distribution. It
consists of several new features such as the new alignment viewer, ability to
create alignment tasks and store in xml files, three different views to align
the text (links view and matrix view - suitable for character, word and phrase
alignments, parallel view - suitable for sentence or long text alignment), an
alignment exporter and many more. See chapter \ref{chap:alignment} for more
information.
MetaMap, from the National Library of Medicine (NLM), maps biomedical text to
the \textbf{UMLS Metathesaurus} and allows Metathesaurus concepts to be
discovered in a text corpus. The Tagger\_MetaMap plugin for GATE wraps the
MetaMap Java API client to allow GATE to communicate with a remote (or local)
MetaMap PrologBeans \textbf{mmserver} and MetaMap distribution. This allows the
content of specified annotations (or the entire document content) to be
processed by MetaMap and the results converted to GATE annotations and features.
See section~\ref{sec:misc-creole:metamap} for details.
A new plugin called Web\_Translate\_Google has been added with a PR called
Google Translator PR in it. It allows users to translate text using the
Google translation services. See section \ref{sec:misc-creole:google-translate}
for more information.
New Gazetteer Editor for ANNIE Gazetteer that can be used instead of
Gaze. It uses tables instead of text area to display the gazetteer
definition and lists, allows sorting on any column, filtering of the lists,
reloading a list, etc. See section~\ref{sec:gazetteers:anniegazeditor}.
\rcSubsect{Breaking changes}
This release contains a few small changes that are not backwards-compatible:
\begin{itemize}
\item Changed the semantics of the ontology-aware matching mode in JAPE to take
account of the default namespace in an ontology. Now {\tt class} feature
values that are not complete URIs will be treated as naming classes within the
default namespace of the target ontology only, and not (as previously) any
class whose URI ends with the specified name. This is more consistent with the
way OWL normally works, as well as being much more efficient to execute. See
section~\ref{sec:ontologies:ontology-aware-jape} for more details.
\item Updated the WordNet plugin to support more recent releases of WordNet
than 1.6. The format of the configuration file has changed, if you are using
the previous WordNet 1.6 support you will need to update your configuration.
See section~\ref{sec:misc-creole:wn} for details.
\item The deprecated Tagger\_TreeTagger plugin has been removed, applications
that used it will need to be updated to use the Tagger\_Framework plugin
instead. See section~\ref{sec:parsers:taggerframework} for details of how to
do this.
\end{itemize}
\rcSubsect{Other new features and bugfixes}
The concept of {\it templates} has been introduced to JAPE. This is a way to
declare named ``variables'' in a JAPE grammar that can contain placeholders
that are filled in when the template is referenced. See
section~\ref{sec:jape:templates} for full details.
Added a JAPE operator to get the string covered by a left-hand-side label and
assign it to a feature of a new annotation on the right hand side (see
section~\ref{sec:jape:metaproperties}).
Added a new API to the CREOLE registry to permit plugins that live
entirely on the classpath. {\tt CreoleRegister.registerComponent} instructs
the registry to scan a single java Class for annotations, adding it to the
set of registered plugins. See section~\ref{sec:api:plugins} for details.
Maven artifacts for GATE are now published to the central Maven
repository. See section~\ref{sec:gettingstarted:maven} for details.
Bugfix: {\tt DocumentImpl} no longer changes its {\tt stringContent} parameter
value whenever the document's content changes. Among other things, this means
that saved application states will no longer contain the full text of the
documents in their corpus, and documents containing XML or HTML tags that were
originally created from string content (rather than a URL) can now safely be
stored in saved application states and the GATE Developer saved session.
A processing resource called Quality Assurance PR has been added in the Tools
plugin. The PR wraps the functionality of the Quality Assurance Tool
(section \ref{sec:eval:corpusqualityassurance}).
A new section for using the Corpus Quality Assurance from GATE Embedded has
been written. See section~\ref{sec:eval:corpusqualityassurance}.
The Generic Tagger PR (in the Tagger\_Framework plugin) now allows more
flexible specification of the input to the tagger, and is no longer limited to
passing just the ``string'' feature from the input annotations. See
section~\ref{sec:parsers:taggerframework} for details.
Added new parameters and options to the LingPipe Language Identifier PR.
(section~\ref{sec:misc-creole:lingpipe:langid}), and corrected the
documentation for the LingPipe POS Tagger
(section~\ref{sec:misc-creole:lingpipe:postagger}).
In the document editor, fixed several exceptions to make editing text with
annotations highlighted working. So you should now be able to edit the text
and the annotations should behave correctly that is to say move, expand or
disappear according to the text insertions and deletions.
Options for document editor: read-only and insert append/prepend have been
moved from the options dialogue to the document editor toolbar at the top
right on the triangle icon that display a menu with the options. See
section~\ref{sec:developer:documents}.
Added new parameters and options to the Crawl PR and document features to its
output.
Fixed a bug where ontology-aware JAPE rules worked correctly when the target
annotation's class was a subclass of the class specified in the rule, but
failed when the two class names matched exactly.
Improved support for conditional pipelines containing non-LanguageAnalyser
processing resources.
Added the current {\tt Corpus} to the script binding for the Groovy Script PR,
allowing a Groovy script to access and set corpus-level features. Also added
callbacks that a Groovy script can implement to do additional pre- or
post-processing before the first and after the last document in a corpus. See
section~\ref{sec:api:groovy} for details.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\sect[sec:changes:5.2.1]{Version 5.2.1 (May 2010)}
This is a bugfix release to resolve several bugs that were reported shortly
after the release of version 5.2:
\begin{itemize}
\item Fixed some bugs with the automatic ``create instance'' feature in OAT
(the ontology annotation tool) when used with the new {\tt Ontology} plugin.
\item Added validation to datatype property values of the \emph{date},
\emph{time} and \emph{datetime} types.
\item Fixed a bug with Gazetteer\_LKB that prevented it working when the
{\tt dictionaryPath} contained spaces.
\item Added a utility class to handle common cases of encoding URIs for use in
ontologies, and fixed the example code to show how to make use of this. See
chapter~\ref{chap:ontologies} for details.
\item The annotation set transfer PR now copies the feature map of each
annotation it transfers, rather than re-using the same FeatureMap (this means
that when used to copy annotations rather than move them, the copied
annotation is independent from the original and modifying the features of one
does not modify the other). See section~\ref{sec:misc-creole:ast} for
details.
\item The Log4J log files are now created by default in the {\tt .gate}
directory under the user's home directory, rather than being created in the
current directory when GATE starts, to be more friendly when GATE is
installed in a shared location where the user does not have write permission.
\end{itemize}
This release also fixes some shortcomings in the Groovy support added by 5.2,
in particular:
\begin{itemize}
\item The {\tt corpora} variable in the console now includes persistent corpora
(loaded from a datastore) as well as transient corpora.
\item The subscript notation for annotation sets works with long values as well
as ints, so \verb|someAS[annotation.start()..annotation.end()]| works as
expected.
\end{itemize}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\sect[sec:changes:5.2]{Version 5.2 (April 2010)}
\subsect{JAPE and JAPE-related}
Introduced a utility class
\htlink{http://gate.ac.uk/gate/doc/javadoc/gate/Utils.html}{gate.Utils}
containing static utility methods for frequently-used idioms such as getting
the string covered by an annotation, finding the start and end offsets of
annotations and sets, etc. This class is particularly useful on the right hand
side of JAPE rules (section~\ref{sec:jape:javarhsoverview}).
Added type parameters to the \verb|bindings| map available on the RHS of JAPE
rules, so you can now do \verb|AnnotationSet as = bindings.get("label")|
without a cast (see section~\ref{sec:jape:javarhsoverview}).
Fixed a bug with JAPE's handling of features called ``class'' in
non-ontology-aware mode. Previously JAPE would always match such features
using an equality test, even if a different operator was used in the grammar,
i.e. \verb|{SomeType.class != "foo"}| was matched as
\verb|{SomeType.class == "foo"}|. The correct operator is now used. Note that
this does not affect the ontology-aware behaviour: when an ontology parameter
is specified, ``class'' features are always matched using ontology subsumption.
Custom JAPE operators and annotation accessors can now be loaded from plugins
as well as from the \texttt{lib} directory (see
section~\ref{sec:jape:customoperators}).
\subsect{Other Changes}
Added a mechanism to allow plugins to contribute menu items to the ``Tools''
menu in GATE Developer. See section~\ref{sec:creole-model:tools} for details.
Enhanced Groovy support in GATE: the Groovy console and Groovy Script PR (in
the Groovy plugin) now import many GATE classes by default, and a number of
utility methods are mixed in to some of the core GATE API classes to make them
more natural to use in Groovy. See section~\ref{sec:api:groovy} for details.
Modified the batch learning PR (in the \texttt{Learning} plugin) to make it
safe to use several instances in {\it APPLICATION} mode with the same
configuration file and the same learned model at the same time (e.g. in a
multithreaded application). The other modes (including training and
evaluation) are unchanged, and thus are still \emph{not} safe for use in this
way. Also fixed a bug that prevented APPLICATION mode from working anywhere
other than as the last PR in a pipeline when running over a corpus in a
datastore.
Introduced a simple way to create duplicate copies of an existing resource
instance, with a way for individual resource types to override the default
duplication algorithm if they know a better way to deal with duplicating
themselves. See section~\ref{sec:api:duplicate}.
Enhanced the Spring support in GATE to provide easy access to the new
duplication API, and to simplify the configuration of the built-in Spring
pooling mechanisms when writing multi-threaded Spring-based applications. See
section~\ref{sec:api:spring}.
The GAPP packager Ant task now respects the ordering of mapping hints, with
earlier hints taking precedence over later ones (see
section~\ref{sec:ant:packagegapp:hints}).
Bug fix in the UIMA plugin from Roland Cornelissen - \texttt{AnalysisEnginePR}
now properly shuts down the wrapped \texttt{AnalysisEngine} when the PR is
deleted.
Patch from Matt Nathan to allow several instances of a gazetteer PR in an
embedded application to share a single copy of their internal data structures,
saving considerable memory compared to loading several complete copies of the
same gazetteer lists (see section~\ref{sec:gazetteers:shared}).
In the corpus quality assurance, measures for classification tasks have been
added. You can also now set the beta for the fscore. This tool has been
optimised to work with datastores so that it doesn't need to read all the
documents before comparing them.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\sect[5.1]{Version 5.1 (December 2009)}
\label{sec:changes:5.1b1}\label{sec:changes:5.1b2}\htcode{<A name="sec:changes:5.1b1"></A><A name="sec:changes:5.1b2"></A>}
Version 5.1 is a major increment with lots of new features and integration of
a number of important systems from 3rd parties (e.g. LingPipe, OpenNLP,
OpenCalais, a revised UIMA connector). We've stuck with the 5 series (instead
of jumping to 6.0) because the core remains stable and backwards compatible.
Other highlights include:
\begin{itemize}
\item
an entirely new ontology API from Johann Petrak of OFAI (the old one is still
available but as a plugin)
\item
new benchmarking facilities for JAPE from Andrew Borthwick and colleagues at
Intelius
\item
new quality assurance tools from Thomas Heitz and colleagues at Ontotext and
Sheffield
\item
a generic tagger integration framework from Ren\'e Witte of Concordia University
\item
several new code contributions from Ontotext, including a large
knowledge-based gazetteer and various plugin wrappers from Marin Nozchev,
Georgi Georgiev and colleagues
\item
a revised and reordered user guide, amalgamated with the programmers' guide
and other materials
\item
Groovy support, application composition, section-by-section processing and
lots of other bits and pieces
\end{itemize}
\subsect{New Features}
\subsubsect{LingPipe Support}
LingPipe is a suite of Java libraries for the linguistic analysis of human
language. We have provided a plugin called `LingPipe' with wrappers for some
of the resources available in the LingPipe library. For more details, see the
section \ref{sec:misc-creole:lingpipe}.
\subsubsect{OpenNLP Support}
OpenNLP provides tools for sentence detection, tokenization, pos-tagging,
chunking and parsing, named-entity detection, and coreference. The tools use
Maximum Entropy modelling. We have provided a plugin called `OpenNLP' with
wrappers for some of the resources available in the OpenNLP Tools library. For
more details, see section \ref{sec:misc-creole:opennlp}.
\subsubsect{OpenCalais Support}
We added a new PR called `OpenCalais PR'. This will process a document
through the OpenCalais service, and add OpenCalais entity annotations to the
document. (This plugin was subsequently removed in GATE 8.4)
\subsubsect{Ontology API}
The ontology API (package \verb!gate.creole.ontology! has been changed, the
existing ontology implementation based on Sesame1 and OWLIM2
(package \verb!gate.creole.ontology.owlim!) has been moved into
the plugin \verb!Ontology_OWLIM2!. An upgraded implementation
based on Sesame2 and OWLIM3 that also provides a number of new
features has been added as plugin \verb!Ontology!.
\subsubsect{Benchmarking Improvements}
A number of improvements to the benchmarking support in GATE. JAPE transducers
now log the time spent in individual phases of a multi-phase grammar and by
individual rules within each phase. Other PRs that use JAPE grammars
internally (the pronominal coreferencer, English tokeniser) log the time taken
by their internal transducers. A reporting tool, called `Profiling Reports'
under the `Tools' menu makes summary information easily available. For more
details, see chapter~\ref{chap:profiling}.
\subsubsect{GUI improvements}
To deal with quality assurance of annotations, one component has been
updated and two new components have been added. The annotation diff tool has
a new mode to copy annotations to a consensus set, see
section~\ref{sec:eval:adiff}. An annotation stack view has been added in the
document editor and it allows to copy annotations to a consensus set, see
section~\ref{sec:developer:annotationsstackview}. A corpus view has been
added for all corpus to get statistics like precision, recall and F-measure,
see section~\ref{sec:eval:corpusqualityassurance}.
An annotation stack view has been added in the document editor to make