Replace Pioneer Home   All Examples   Free Download

 New request --free  RSS: Replace Pioneer Examples

1352.Text file parser -- How split words from file where all words are joined without spaces?

User: Aaron -- 2016-03-14          << 1351  1353 >>
Hits: 853
Type: Text file parser   
Search all Text file parser examples
Description:
Requirement to split words from concatenated text file where all words are joined
without spaces. This would likely require the use of a dictionary file containing
common English words sorted by frequency. Here's an example:
https://github.com/first20hours/google-10000-english

Script needs to process all punctuation marks, new line breaks, and treat Upper and lower case words as unique. Words not included in dictionary file should be treated as new words.
Input Sample:
Itwasthebestoftimes,itwastheworstoftimes,itwastheageofwisdom,itwastheageoffoolishness, 

itwastheepochofbelief,itwastheepochofincredulity,itwastheseasonoflight,itwastheseasonofDarkness,etc.
Output Sample:
It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, 

it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, etc.
Answer:
Hint: You need to Download and install "Replace Pioneer" on windows platform to finish following steps.
To split by priority of frequency is not a best approach. Here we try to sort the words by the length, the longer words has higher priority.
1. ctrl-o open text file
2. ctrl-h open 'replace' dialogue:
* set 'search for pattern' to: 
* set 'replace with pattern' to: 
* click 'advanced tab':
* set 'run following at the beginning of replace' to:

* set 'run following for each matched unit' to:


3. click 'replace', done.

Note:
(1) you need to put file in d:\test\google-10000-english.txt
(2) some of the words like 'epoch', 'foolishness' does not exist in google-10000-english.txt, which will cause some problem. You can add them manually
(3) even if all words exit in dictionary, still there will be some pro

Screenshot 1:  Replace_Window


Screenshot 2:  Replace_Advanced_Window


Similar Examples:
How to split a CSV file with Header according to value of  column B? (50%)
How to split a CSV file with Header according to value of a column? (50%)
How to delete all the lines without ceramics in text file? (50%)
How to split a text file according to first word in each line? (50%)
How to count and sort the frequency of all words appeared in many text files? (50%)
How to split a csv file according to the value of column A? (49%)
How to extract column X from csv file where X is decided by column A? (49%)
How to replace words in a text file only if they are whole words? (49%)

Check Demo of Text file parser
Keywords:
best  epoch  google  not included  requirement  line breaks  unctuation  punctuation  lower case  not include  text sort words by frequency  parser for a english text file  sort file by line length  sort the line by length in text file  split even  sort line length  sort words by frequency  sort text file by line length