Module textual

Functions

BasicTokenizer(s, minlen=1)
Split on any non-word letter.

source code

WordRegexTokenizer(s)
Find 3 or more letter words.

source code

WordNumberRegexTokenizer(s)
Find 4 or more letter words or numbers/currencies.

source code

Vectorize(s, af, tokenizer=<function WordRegexTokenizer at 0x101a98c08>)
Create an AtomVector from a string.

source code

Variables

wordsplitter_rex = re.compile(r'\W+')

word_regex = re.compile(r'\b[a-z]{3,}\b')

word_number_regex = re.compile(r'\b[a-z][a-z0-9]{3,}\b|(\$|\b)...

__package__ = 'mekano'

Function Details

Split on any non-word letter.

Words need not start with [a-z]

Find 3 or more letter words.

Words must start with [a-z]

Find 4 or more letter words or numbers/currencies.

Words must start with [a-z]

Create an AtomVector from a string.

Tokenizes string 's' using tokenizer, creating atoms using AtomFactory 'af'.

Variables Details

Value:

re.compile(r'\b[a-z][a-z0-9]{3,}\b|(\$|\b)[0-9]+(,[0-9]{3})*(\.[0-9]+)
?\b')