In the Margins Main Menu Welcome The In the Margins home page Lexomics The starting point for the Lexomics path Manual Start page for the Lexos Manual Topics Explore this path to learn about the Lexomic methods Glossary Glossary of terms used in Lexos and In the Margins Bibliography Beginning of bibliography path Lexos Install Guide Install Guide

Advanced Options

5371

Tokenize

By default Lexos splits strings of text into tokens every time it encounters a space character. For Western languages, this means that each token generally corresponds to a word. Click the by Characters radio button to treat every character as a separate token.

If you wish to use n-grams (by Tokens), increase the 1-gram incrementer to 2, 3, 4, etc. For example, "the dog ran" would produce the 1-gram tokens the, dog, ran., the 2-grams the dog, dog ran, and so on. Alternately, 2-grams (by Characters) would create tokens:  th, he, e , and so on.

Note that increasing the n-gram size may produce a larger DTM, and the table will thus take longer to build.

Culling Options

"Culling Options" is a generic term we use for methods of decreasing the number of terms used to generate the DTM based on statistical criteria (as opposed to something like applying a stopword list in Scrubber). Lexos offers three different methods:

1. Most Frequent Words: This method takes a slice of the DTM containing only the top-N most frequently occurring terms. The default setting is 100, meaning "use" only the top-100 most frequently occurring terms.
2. Culling: This method builds the DTM using only terms that occur in at least N documents. The default setting is 1. To cull words in the DTM to only include terms that appear at least once in all the active documents, set the value to the total number of active documents. (Note: you can quickly determine the number of active documents by panning over the folder icon in the upper-right hand corner; for example, to use only those terms that appear at least once in your ten active documents, you could set the option to be: Must be in 10 documents.
 

Normalize

By default, Lexos displays the frequency of the occurrence of terms in your documents as a proportion of the entire text. If you wish to see the actual number of occurrences, click the Raw Counts radio button. You may also attempt to take into account differences in the lengths of your documents by calculating their Term Frequency-Inverse Document Frequency (TF-IDF). Lexos offers three different methods of calculating TF-IDF based on Euclidean Distance, Manhattan Distance, or without using a distance metric (Norm: None). For further discussion on these options, see the topics article on [TF-IDF](http://scalar.usc.edu/works/lexos/tf-idf).

Assign Temporary Labels

Lexos automatically uses the label in the "Document Name" column in the Manage tool as the document label. However, you may change the label used in your table by entering a new value for it in the forms displayed in Assign Temporary Labels. This is particularly useful if you want to save different labels when you download your DTM. Keep in mind that whatever labels you set will be applied in all other Lexos tools that use the Advanced Options. However, the original document name in Manage will not be affected. After assigning temporary labels in Tokenizer, click the Regenerate Table button to rebuild the table with the new labels.