--tokenizer_model

Switch

--tokenizer_model NAME

Description

Tokenizer model to use when extracting features with --bert_model/--emb_model, when it differs from fwflag_embedding_model (or when the model isn't in DLATK's built-in shorthand list — see --emb_class).

Argument and Default Value

A Hugging Face tokenizer name. Default: None.

Details

Verified end-to-end together with --emb_class auto: needed for any embedding model name not in DLATK's internal shorthand list — without it, tokenizer lookup fails with KeyError.

Other Switches

Required Switches:

• --bert_model or --emb_model

Optional Switches:

• --emb_class

Example Commands

dlatkInterface.py -d dla_tutorial -t msgs -c user_id --add_bert \
--embedding_model prajjwal1/bert-tiny --emb_class auto --tokenizer_model prajjwal1/bert-tiny