•     Data generation

1. process the record (e.g. remove spaces, insert delimiters, insert end of record delimiter) and save it to a string
2. randomly remove delimiters and save it to another string s2

•     HMM model construction

1. convert s2 into integer array a1

• mapping:
  
• COL_SEP ',' => 1
• REC_SEP '#' => 2
• A-Z  => 3-28
• a-z => 29-54
• 0-9 => 55-64
• OTHERS =>65
  

1. use Viterbi algorithm to find most probable states in array a2

•     Result evaluation

1. Use a2 and s2 to clean the data to produce string s3
2. compare the string s3 and the one obtained in 1 to see if the cleaning is done correctly