Handle t-string prefixes in SimpleTokenizer (#20578)
The simple tokenizer is meant to skip strings, but it was recording a `Name` token for t-strings (from the `t`). This PR fixes that.
This commit is contained in:
@@ -599,6 +599,16 @@ impl<'a> SimpleTokenizer<'a> {
|
||||
| "rb"
|
||||
| "rf"
|
||||
| "u"
|
||||
| "T"
|
||||
| "TR"
|
||||
| "Tr"
|
||||
| "RT"
|
||||
| "Rt"
|
||||
| "t"
|
||||
| "tR"
|
||||
| "tr"
|
||||
| "rT"
|
||||
| "rt"
|
||||
)
|
||||
{
|
||||
self.bogus = true;
|
||||
|
||||
Reference in New Issue
Block a user