--add_char_ngrams

Switch

--add_char_ngrams

Description

Extracts character n-grams (rather than word n-grams) from a message table.

Argument and Default Value

No argument. Boolean switch, default: False. Uses -n for the n-gram size(s), and can be combined with --sqrt/--gzip_csv, with or without --use_collocs.

Details

Verified table naming: -n 3 creates feat$3Cgram$<corptable>$<correl_field> (the C marks it as a character n-gram table, distinguishing it from the word-level feat$3gram$...).

Known issue — broken in SQLite mode. FeatureExtractor.addCharNGramTable() calls mm.disableTableKeys(...) directly — a raw mysqlmethods call issuing MySQL's ALTER TABLE ... DISABLE KEYS syntax — instead of going through the SQLite-aware self.data_engine.disable_table_keys() that --add_ngrams's own addNGramTable() uses (which has no such call at all). Verified: running --add_char_ngrams against a SQLite corpus creates the (empty) destination table, then fails with sqlite3.OperationalError: near "DISABLE": syntax error before any rows are inserted. Works against MySQL.

Other Switches

Required Switches:

• -d, -g, -t

Optional Switches:

• -n

• --sqrt, --gzip_csv

• --use_collocs

Example Commands

Using the packaged dla_tutorial dataset against MySQL (see Differential Language Analysis (DLA) Tutorial) — broken against SQLite, see Known issue above:

dlatkInterface.py -d dla_tutorial -t msgs -c user_id --add_char_ngrams -n 3