Package weka.core.tokenizers

Class TweetNLPTokenizer

java.lang.Object

weka.core.tokenizers.Tokenizer

weka.core.tokenizers.TweetNLPTokenizer

All Implemented Interfaces:

java.io.Serializable, java.util.Enumeration<java.lang.String>, OptionHandler, RevisionHandler

public class TweetNLPTokenizer
extends Tokenizer

A Twitter-specific tokenizer based on the CMU TwitterNLP library: http://www.cs.cmu.edu/~ark/TweetNLP/ BibTeX:

 @InProceedings{twitterNLP,
        Title                    = {Part-of-speech tagging for twitter: Annotation, features, and experiments},
        Author                   = {Gimpel, Kevin and Schneider, Nathan and O'Connor, Brendan and Das, Dipanjan and Mills, Daniel and Eisenstein, Jacob and Heilman, Michael and Yogatama, Dani and Flanigan, Jeffrey and Smith, Noah A},
        Booktitle                = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2},
        Year                     = {2011},
        Organization             = {Association for Computational Linguistics},
        Pages                    = {42--47}
        }
 

Version:

$Revision: 1 $

Author:

Felipe Bravo-Marquez (fbravoma@waikato.ac.nz)

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor	Description
TweetNLPTokenizer()

Method Summary

Modifier and Type	Method	Description
java.lang.String	getRevision()	Returns the revision string.
TechnicalInformation	getTechnicalInformation()	Returns an instance of a TechnicalInformation object, containing detailed information about the technical background of this class, e.g., paper reference or book this class is based on.
java.lang.String	globalInfo()	Returns a string describing this tokenizer.
boolean	hasMoreElements()	Tests if this enumeration contains more elements.
static void	main(java.lang.String[] args)	Runs the tokenizer with the given options and strings to tokenize.
java.lang.String	nextElement()	Returns the next element of this enumeration if this enumeration object has at least one more element to provide.
void	tokenize(java.lang.String s)	Sets the string to tokenize.

Methods inherited from interface java.util.Enumeration

asIterator

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from class weka.core.tokenizers.Tokenizer

getOptions, listOptions, runTokenizer, setOptions, tokenize

Constructor Detail

TweetNLPTokenizer

public TweetNLPTokenizer()

Method Detail

globalInfo

public java.lang.String globalInfo()

Returns a string describing this tokenizer.

Specified by:

globalInfo in class Tokenizer

Returns:

a description of the filter suitable for displaying in the explorer/experimenter gui

getTechnicalInformation

public TechnicalInformation getTechnicalInformation()

Returns an instance of a TechnicalInformation object, containing detailed information about the technical background of this class, e.g., paper reference or book this class is based on.

Returns:

the technical information about this class

hasMoreElements

public boolean hasMoreElements()

Tests if this enumeration contains more elements.

Specified by:

hasMoreElements in interface java.util.Enumeration<java.lang.String>

Specified by:

hasMoreElements in class Tokenizer

Returns:

true if and only if this enumeration object contains at least one more element to provide; false otherwise.

nextElement

public java.lang.String nextElement()

Returns the next element of this enumeration if this enumeration object has at least one more element to provide.

Specified by:

nextElement in interface java.util.Enumeration<java.lang.String>

Specified by:

nextElement in class Tokenizer

Returns:

the next element of this enumeration.

tokenize

public void tokenize(java.lang.String s)

Sets the string to tokenize. Tokenization happens immediately.

Specified by:

tokenize in class Tokenizer

Parameters:

s - the string to tokenize

getRevision

public java.lang.String getRevision()

Returns the revision string.

Returns:

the revision

main

public static void main(java.lang.String[] args)

Runs the tokenizer with the given options and strings to tokenize. The tokens are printed to stdout.

Parameters:

args - the commandline options and strings to tokenize