This site requires Javascript to be turned on. Please enable Javascript and reload the page.

In the Margins Main Menu Welcome The In the Margins home page Lexomics The starting point for the Lexomics path Manual Start page for the Lexos Manual Topics Explore this path to learn about the Lexomic methods Glossary Glossary of terms used in Lexos and In the Margins Bibliography Beginning of bibliography path Lexos Install Guide Install Guide

The Scrub Tool

5371

The Scrubbing tool allows you to make document-wide edits, such as the removal of punctuation, digits, and capital letters as well as how to handle hyphens and possessive apostrophes. Additionally, Scrub allows you to input a list of stopwords, lemmas, consolidations (character replacements), and special characters, each of which will be explained in more detail below.

Using Lexos
The first part of the scrubbing process involves several simple options which effect the entire document. In almost all cases, their names provide ample evidence of their functionality, so the important thing to remember is that they will take effect in each of your selected (active) documents or document segments.

Remove punctuation:
All unicode characters have an associated set of metadata for classifying its "type" of character. If this option is selected, any unicode character in each of the active texts that has a Punctuation Character Property (begins with a 'P') or a Symbol Character Property (begins with 'S') is removed. The specific Punctuation and Symbols that are removed are listed below:

Punctuation
Pc	Connector punctuation
Pd	Dash punctuation
Pe	Close punctuation
Pf	Final punctuation
Pi	Initial punctuation
Po	Other punctuation
Ps	Open punctuation
S	Symbol
Sc	Currency symbol
Sk	Modifier symbol
Sm	Mathematical symbol
So	Other symbol

If Remove Punctuation is selected, two additional options are presented:
Keep Hyphens - If this option is selected, any character in a set of hyphen characters are replaced with the (ASCII) hyphen-minus character and then the hyphen-minus characters remain in the text. A hyphen character is any of the following unicode hyphen characters: [u'\u058A', u'\u05BE', u'\u2010', u'\u2011', u'\u2012', u'\u2013', u'\u2014', u'\u2015', u'\uFE58', u'\uFE63', u'\uFF0D ].

Make lowercase:

Contents of this path:

Lemmas