--topic_dupe_filter

Switch

--topic_dupe_filter

Description

Removes topics not passing a duplicate-word filter directly from the correlation results, before any tagcloud/wordcloud output is generated — topics whose top words overlap too much with a more strongly-correlated topic are dropped from the correlation matrix entirely.

Argument and Default Value

No argument. Boolean switch, default: False.

Details

Verified end-to-end: runs oa.topicDupeFilterCorrels(correls, topiclexicon) right after --correlate computes results, inside the main correlate/tagcloud code path. This is a separate mechanism from --tagcloud_filter/--no_tagcloud_filter (which control the duplicateFilter parameter of the tagcloud-writing step itself, default True) — the two can be used independently or together.

Other Switches

Required Switches:

--correlate (implicitly enabled by --topic_tagcloud etc.)
--topic_lexicon

Example Commands

Using the packaged dla_tutorial dataset (see Differential Language Analysis (DLA) Tutorial):

dlatkInterface.py -d dla_tutorial -t msgs -c user_id -f 'feat$cat_met_a30_2000_cp_w$msgs$user_id$16to16' \
--outcome_table blog_outcomes --outcomes age --topic_tagcloud --topic_lexicon met_a30_2000_freq_t50ll \
--topic_dupe_filter --output_name ./topic_output