This site requires Javascript to be turned on. Please enable Javascript and reload the page.

In the Margins Main Menu Welcome The In the Margins home page Lexomics The starting point for the Lexomics path Manual Start page for the Lexos Manual Topics Explore this path to learn about the Lexomic methods Glossary Glossary of terms used in Lexos and In the Margins Bibliography Beginning of bibliography path Lexos Install Guide Install Guide

Scrubbing

5371

Scrubbing topics go here. (this is not organized)

0. Gutenberg files
Documents from gutenberg.org containing "bioler plate" materials about the text at the beginning and end of

1. Make lowercase:

Remove punctuations
All unicode characters have an associated set of metadata for classifying its "type" of character. If this option is selected, any unicode character in each of the active texts that has a Punctuation Character Property (begins with a 'P') or a Symbol Character Property (begins with 'S') is removed. The specific Punctuation and Symbols that are removed are listed below:

Punctuation
Pc	Connector punctuation
Pd	Dash punctuation
Pe	Close punctuation
Pf	Final punctuation
Pi	Initial punctuation
Po	Other punctuation
Ps	Open punctuation
S	Symbol
Sc	Currency symbol
Sk	Modifier symbol
Sm	Mathematical symbol
So	Other symbol

If Remove Punctuation is selected, two additional options are presented:
Keep Hyphens - If this option is selected, any character in a set of hyphen characters are replaced with the (ASCII) hyphen-minus character and then the hyphen-minus characters remain in the text. A hyphen character is any of the following unicode hyphen characters: [u'\u058A', u'\u05BE', u'\u2010', u'\u2011', u'\u2012', u'\u2013', u'\u2014', u'\u2015', u'\uFE58', u'\uFE63', u'\uFF0D ].

This page has paths:

Pre-Processing Scott Kleinman
Contents of this path:

Contents of this path:

Special Characters and Non-Roman Writing Systems