2cbe1733c8
## Summary The tokenizer was split into a forward and a backwards tokenizer. The backwards tokenizer uses the same names as the forwards ones (e.g. `next_token`). The backwards tokenizer gets the comment ranges that we already built to skip comments. --------- Co-authored-by: Micha Reiser <micha@reiser.io>
11 lines
185 B
Rust
11 lines
185 B
Rust
mod comment_ranges;
|
|
mod cursor;
|
|
pub mod textwrap;
|
|
mod tokenizer;
|
|
mod whitespace;
|
|
|
|
pub use comment_ranges::CommentRanges;
|
|
pub use cursor::*;
|
|
pub use tokenizer::*;
|
|
pub use whitespace::*;
|