Dhruv Manilawala
68a8978454
Consider line continuation character for re-lexing ( #12008 )
...
## Summary
This PR fixes a bug where the re-lexing logic didn't consider the line
continuation character being present before the newline character. This
meant that the lexer was being moved back to the newline character which
is actually ignored via `\`.
Considering the following code:
```py
f'middle {'string':\
'format spec'}
```
The old token stream is:
```
...
Colon 18..19
FStringMiddle 19..29 (flags = F_STRING)
Newline 20..21
Indent 21..29
String 29..42
Rbrace 42..43
...
```
Notice how the ranges are overlapping between the `FStringMiddle` token
and the tokens emitted after moving the lexer backwards.
After this fix, the new token stream which is without moving the lexer
backwards in this scenario:
```
FStringStart 0..2 (flags = F_STRING)
FStringMiddle 2..9 (flags = F_STRING)
Lbrace 9..10
String 10..18
Colon 18..19
FStringMiddle 19..29 (flags = F_STRING)
FStringEnd 29..30 (flags = F_STRING)
Name 30..36
Name 37..41
Unknown 41..44
Newline 44..45
```
fixes : #12004
## Test Plan
Add test cases and update the snapshots.
2024-06-25 02:13:54 +00:00
..
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-06-03 18:23:50 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-06-03 18:23:50 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-19 16:46:15 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-06-17 06:47:00 +00:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-06-17 06:47:00 +00:00
2024-06-17 06:47:00 +00:00
2024-06-17 06:47:00 +00:00
2024-06-17 06:47:00 +00:00
2024-05-17 10:26:48 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-06-17 06:47:00 +00:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-06-17 06:47:00 +00:00
2024-06-17 06:47:00 +00:00
2024-06-17 06:47:00 +00:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-05-07 11:46:10 +00:00
2024-05-07 11:46:10 +00:00
2024-05-07 11:46:10 +00:00
2024-05-07 11:46:10 +00:00
2024-06-17 06:47:00 +00:00
2024-05-07 11:46:10 +00:00
2024-05-07 11:46:10 +00:00
2024-05-07 11:46:10 +00:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-06-17 06:47:00 +00:00
2024-04-18 17:57:39 +05:30
2024-04-23 04:19:05 +00:00
2024-04-23 04:19:05 +00:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-06-17 06:47:00 +00:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-06-17 06:47:00 +00:00
2024-04-18 17:57:39 +05:30
2024-04-23 04:19:05 +00:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-06-17 06:47:00 +00:00
2024-05-07 11:46:10 +00:00
2024-04-23 04:19:05 +00:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-06-12 13:57:35 +05:30
2024-06-18 12:14:41 +05:30
2024-06-20 16:27:36 +05:30
2024-04-18 17:57:39 +05:30
2024-04-23 04:19:05 +00:00
2024-04-23 04:19:05 +00:00
2024-04-23 04:19:05 +00:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-26 20:38:35 -06:00
2024-04-26 20:38:35 -06:00
2024-04-26 20:38:35 -06:00
2024-06-17 06:47:00 +00:00
2024-04-26 20:38:35 -06:00
2024-04-26 20:38:35 -06:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-06-17 06:47:00 +00:00
2024-06-17 06:47:00 +00:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-06-17 06:47:00 +00:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-06-17 06:47:00 +00:00
2024-06-17 06:47:00 +00:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-05-07 11:46:10 +00:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-06-03 18:23:50 +05:30
2024-06-03 18:23:50 +05:30
2024-06-03 18:23:50 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-06-03 18:23:50 +05:30
2024-06-03 18:23:50 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-19 11:36:26 +00:00
2024-04-18 17:57:39 +05:30
2024-06-17 06:47:00 +00:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-29 10:36:15 +00:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-06-17 06:47:00 +00:00
2024-06-17 06:47:00 +00:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-06-17 06:47:00 +00:00
2024-06-17 06:47:00 +00:00
2024-06-20 16:27:36 +05:30
2024-06-25 02:13:54 +00:00
2024-06-25 02:13:54 +00:00
2024-06-20 16:27:36 +05:30
2024-06-20 16:27:36 +05:30
2024-06-20 16:27:36 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-06-17 06:47:00 +00:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-05-07 11:46:10 +00:00
2024-05-07 11:46:10 +00:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-06-17 06:47:00 +00:00
2024-06-03 18:23:50 +05:30
2024-04-18 17:57:39 +05:30
2024-05-07 11:46:10 +00:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-06-17 06:47:00 +00:00
2024-04-18 17:57:39 +05:30
2024-06-06 08:59:56 +00:00
2024-06-06 08:59:56 +00:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-26 09:47:29 +02:00
2024-04-26 09:47:29 +02:00
2024-04-26 09:47:29 +02:00
2024-04-26 09:47:29 +02:00
2024-04-26 09:47:29 +02:00
2024-04-26 09:47:29 +02:00
2024-04-26 09:47:29 +02:00
2024-04-26 09:47:29 +02:00
2024-04-26 09:47:29 +02:00
2024-04-26 09:47:29 +02:00
2024-06-18 12:14:41 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-06-06 08:59:56 +00:00
2024-06-17 06:47:00 +00:00
2024-04-18 21:39:30 +05:30
2024-04-18 14:30:15 +00:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-06-17 06:47:00 +00:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-06-03 18:23:50 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-05-07 11:46:10 +00:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-05-07 11:46:10 +00:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-05-07 11:46:10 +00:00
2024-05-07 11:46:10 +00:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-05-07 11:46:10 +00:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-05-07 11:46:10 +00:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-05-07 11:46:10 +00:00
2024-05-07 11:46:10 +00:00
2024-04-23 04:19:05 +00:00
2024-04-26 20:38:35 -06:00
2024-06-03 18:23:50 +05:30
2024-04-26 20:38:35 -06:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-06-03 18:23:50 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-06-03 18:23:50 +05:30
2024-04-18 17:57:39 +05:30
2024-06-03 18:23:50 +05:30
2024-06-03 18:23:50 +05:30
2024-06-03 18:23:50 +05:30
2024-06-03 18:23:50 +05:30
2024-06-03 18:23:50 +05:30
2024-06-03 18:23:50 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-26 20:38:35 -06:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-26 09:47:29 +02:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-26 20:38:35 -06:00
2024-04-26 09:47:29 +02:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-05-07 11:46:10 +00:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-26 09:47:29 +02:00
2024-04-18 17:57:39 +05:30
2024-04-18 17:57:39 +05:30
2024-04-26 09:47:29 +02:00
2024-04-26 09:47:29 +02:00
2024-04-26 09:47:29 +02:00