First section
This is an example article
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
Second section
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
subsection
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
Another section
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
istanbul, not constantinople, been a long time gone, old constantinople, why did constantinople get the works? nobody knows but the turksssssss......hahahaha xD lol
FYI, no, not on drugs, just bored...
token_list_main = list("""abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890-_=+;:\"\'—’.,<>/?`“”~!@#$%^&*(){}[]| \\""")
token_list = ['
', '', '', '\n', '\r', '\t', ''] + token_list_main
token_dict = {x: i for i, x in enumerate(token_list)}
reverse_token_dict = {i: x for i, x in enumerate(token_list)}
json.dump([token_dict, reverse_token_dict], open('tokenizer.json', 'w'))
token_list_main = list("""abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890-_=+;:\"\'—’.,<>/?`“”~!@#$%^&*(){}[]| \\""")
token_list = ['', '', '', '\n', '\r', '\t', ''] + token_list_main
token_dict = {x: i for i, x in enumerate(token_list)}
reverse_token_dict = {i: x for i, x in enumerate(token_list)}
json.dump([token_dict, reverse_token_dict], open('tokenizer.json', 'w'))
token_list_main = list("""abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890-_=+;:\"\'—’.,<>/?`“”~!@#$%^&*(){}[]| \\""")
token_list = ['', '', '', '\n', '\r', '\t', ''] + token_list_main
token_dict = {x: i for i, x in enumerate(token_list)}
reverse_token_dict = {i: x for i, x in enumerate(token_list)}
json.dump([token_dict, reverse_token_dict], open('tokenizer.json', 'w'))
token_list_main = list("""abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890-_=+;:\"\'—’.,<>/?`“”~!@#$%^&*(){}[]| \\""")
token_list = ['', '', '', '\n', '\r', '\t', ''] + token_list_main
token_dict = {x: i for i, x in enumerate(token_list)}
reverse_token_dict = {i: x for i, x in enumerate(token_list)}
json.dump([token_dict, reverse_token_dict], open('tokenizer.json', 'w'))
token_list_main = list("""abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890-_=+;:\"\'—’.,<>/?`“”~!@#$%^&*(){}[]| \\""")
token_list = ['', '', '', '\n', '\r', '\t', ''] + token_list_main
token_dict = {x: i for i, x in enumerate(token_list)}
reverse_token_dict = {i: x for i, x in enumerate(token_list)}
json.dump([token_dict, reverse_token_dict], open('tokenizer.json', 'w'))
token_list_main = list("""abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890-_=+;:\"\'—’.,<>/?`“”~!@#$%^&*(){}[]| \\""")
token_list = ['', '', '', '\n', '\r', '\t', ''] + token_list_main
token_dict = {x: i for i, x in enumerate(token_list)}
reverse_token_dict = {i: x for i, x in enumerate(token_list)}
json.dump([token_dict, reverse_token_dict], open('tokenizer.json', 'w'))
token_list_main = list("""abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890-_=+;:\"\'—’.,<>/?`“”~!@#$%^&*(){}[]| \\""")
token_list = ['', '', '', '\n', '\r', '\t', ''] + token_list_main
token_dict = {x: i for i, x in enumerate(token_list)}
reverse_token_dict = {i: x for i, x in enumerate(token_list)}
json.dump([token_dict, reverse_token_dict], open('tokenizer.json', 'w'))
``````````
hello world