Split text in Python in order to experiment with tokenization for a future post. In this post I experiment with splitting text in Python in order to experiment on how to tokenize text. The idea is to ...

prepro_split_tokenize will conbine both training and validation data, and randomly the dataset into train, val and test. It will also tokenize the captions using jiebe. prepro_labels.py will map all ...