--print_tokenized_lines

Switch

--print_tokenized_lines

Description

Prints tokenized version of messages to lines.

Argument and Default Value

You must supply an output file name.

Details

Looks for the table TABLENAME_tok, where TABLENAME is specified by -t. Each line of the output file contains the message id, lanugage, and tokens. Example:

# Sample message from tokenized input table: # ["is", "worth", "it", "just", "follow", "your", "heart", "its", "never", "wrong", ":", "-", "rrb", "-"] # Output line: # 128675651556356096 en is worth it just follow your heart its never wrong : - rrb -

Other Switches

Required Switches: -d, -t Optional Switches: --feat_whitelist Example Commands ================

# General command
dlatkInterface.py -d DATABASE -t TABLE --print_tokenized_lines OUTPUTFILE_NAME

# Example, using the packaged dla_tutorial dataset (see tut_dla)
# searches for the table msgs_tok (built by --add_tokenized)
# outputs the file msgs_tokenized.txt
dlatkInterface.py -d dla_tutorial -t msgs --add_tokenized
dlatkInterface.py -d dla_tutorial -t msgs --print_tokenized_lines msgs_tokenized.txt