# tok 0.2.1

- Fixed tiny documentation issue for normalizers.

# tok 0.2.0

- Updated upstream tokenizers to 0.20.3
- Update extendr-api to 0.8.1

# tok 0.1.5

- add message translation in FR (#19, @cregouby)

# tok 0.1.4

- Updated libR-sys to fix mac oldrel notes. (#18)
- Report rust version during compilation.

# tok 0.1.3

# tok 0.1.2

- Added support for authentication. (#7)
- Added support for training tokenizers from scratch. (#8)
- Added support for serializing tokenizers. (#8)
- Implemented support for configuring padding and truncation. (#9)
- Added support for decoders, post_processors and normalizers. (#10)
- Added support for `get_vocab_size()`. (#10)
- Updated extendr version to avoid non-API function calls. (#11)

# tok 0.1.1

- Dependencies are now vendored to avoid versioning problems on CRAN. (#5)

# tok 0.1.0

* Initial CRAN submission.
