## Summary
This is an error message only change to lead an implementor of a new
rule that has an unformatted or invalid bad example to the
right code.
## Test Plan
n/a
## Summary
Noticed in #5954: we walk _past_ the root rather than stopping _at_ the
root when attempting to traverse along the parent path. It's effectively
an off-by-one bug.
## Summary
Updated doc comment for `call_datetime_without_tzinfo.rs`. Online docs
also benefit from this update.
## Test Plan
Checked docs via
[mkdocs](389fe13c93/CONTRIBUTING.md (L267-L296))
## Summary
Check for unused private `TypeVar`. See [original
implementation](2a86db8271/pyi.py (L1958)).
```
$ flake8 --select Y018 crates/ruff/resources/test/fixtures/flake8_pyi/PYI018.pyi
crates/ruff/resources/test/fixtures/flake8_pyi/PYI018.pyi:4:1: Y018 TypeVar "_T" is not used
crates/ruff/resources/test/fixtures/flake8_pyi/PYI018.pyi:5:1: Y018 TypeVar "_P" is not used
```
```
$ ./target/debug/ruff --select PYI018 crates/ruff/resources/test/fixtures/flake8_pyi/PYI018.pyi --no-cache
crates/ruff/resources/test/fixtures/flake8_pyi/PYI018.pyi:4:1: PYI018 TypeVar `_T` is never used
crates/ruff/resources/test/fixtures/flake8_pyi/PYI018.pyi:5:1: PYI018 TypeVar `_P` is never used
Found 2 errors.
```
In the file `unused_private_type_declaration.rs`, I'm planning to add
other rules that are similar to `PYI018` like the `PYI046`, `PYI047` and
`PYI049`.
ref #848
## Test Plan
Snapshots and manual runs of flake8.
## Summary
In Ruff only a subset of rules are enabled by default. This change
change aims to clarify that when adding a new rule, you must explicitly
use the `--select name_of_rule` command to ensure the rule gets
executed.
This was talked about on Discord a while back.
## Test Plan
Checked docs via mkdocs
This PR introduces two new nodes for the parser to recognize the
`MagicCommand` tokens:
* `StmtLineMagic` for statement position i.e., magic commands on their
own line:
```python
%matplotlib inline
!pwd
```
* `ExprLineMagic` for expression position i.e., magic commands in an
assignment statement:
```python
# Only `?` and `!` are valid in this position
dir = !pwd
```
Both nodes are identical in their structure as in it contains the magic
kind and the command value as `String`.
Emit `MagicCommand` token when it is the assignment value[^1] i.e., on
the right side of an assignment statement.
Examples:
```python
pwd = !pwd
foo = %timeit a = b
bar = %timeit a % 3
baz = %matplotlib \
inline"
```
[^1]: Only `%` and `!` are valid in that position, other magic kinds are
not valid
Previously, empty lambda arguments (e.g. `lambda: 1`) would get the
range of the entire expression, which leads to incorrect comment
placement. Now empty lambda arguments get an empty range between the
`lambda` and the `:` tokens.
The old if layout couldn't differentiate between an else block with a
single if statement and an elif statement. Additionally we getting rid
of the recursion in favor of a single if struct with a vec of elif/else
children. This is accompanied by a big refactoring in ruff which removes
a bunch of TODOs and false negatives.
Lex Jupyter line magic with `Mode::Jupyter`
This PR adds a new token `MagicCommand`[^1] which the lexer will
recognize when in `Mode::Jupyter`. The rules for the lexer is as
follows:
1. Given that we are at the start of line, skip the indentation and look
for [characters that represent the start of a magic
command](635815e8f1/IPython/core/inputtransformer2.py (L335-L346)),
determine the magic kind and capture all the characters following it as
the command string.
2. If the command extends multiple lines, the lexer will skip the line
continuation character (`\`) but only if it's followed by a newline
(`\n` or `\r`). The reason to skip this only in case of newline is
because they can occur in the command string which we should not skip:
```rust
// Skip this backslash
// v
// !pwd \
// && ls -a | sed 's/^/\\ /'
// ^^
// Don't skip these backslashes
```
3. The parser, when in `Mode::Jupyter`, will filter these tokens before
the parsing begins. There is a small caveat when the magic command is
indented. In the following example, when the parser filters out magic
command, it'll throw an indentation error:
```python
for i in range(5):
!ls
# What the parser will see
for i in range(5):
```
[^1]: I would prefer to have some other name as this not only represent
a line magic (`%`) but also shell command (`!`), help command (`?`) and
others. In original implementation, it's named as ["IPython
Syntax"](635815e8f1/IPython/core/inputtransformer2.py (L332))
Extends #95Closes#82
Adds parsing of new `type` soft keyword for defining type aliases.
Supports type alias statements as defined in PEP 695 e.g.
```python
type IntOrStr = int | str
type ListOrSet[T] = list[T] | set[T]
type AnimalOrVegetable = Animal | "Vegetable"
type RecursiveList[T] = T | list[RecursiveList[T]]
```
All type parameter kinds are supported as in #95.
Builds on soft keyword abstractions introduced in https://github.com/RustPython/RustPython/pull/4519
This removes the ASDL code generation in favor of handwriting the AST.
The motivations for moving away from the ASDL are:
* CPython compatibility is no longer a goal
* The ASDL grammar isn't as expressive as we would like
* The codegen scripts have a high complexity which makes extensions time
consuming
* We don't make heavy use of code generation (compared to e.g.
RustPython that generates Pyo3 bindings, a fold implementation etc).
We may want to revisit a grammar based code generation in the future,
e.g. by using [ungrammar](https://github.com/rust-analyzer/ungrammar)
This adds the missing implementation of `Ranged` for `TextRange` itself
```rust
impl Ranged for TextRange {
fn range(&self) -> TextRange {
*self
}
}
```
This allows e.g. using `has_comments` with arbitrary ranges instead of
just a node.
It also adds .venv to the .gitignore
In the example below, `arg` is `&Expr`, so `&Ranged`, but `entries()`
want a `T: Ranged`. This adds the missing bridge impl.
```rust
let all_args = format_with(|f| {
f.join_comma_separated()
.entries(
// We have the parentheses from the call so the arguments never need any
args.iter()
.map(|arg| (arg, arg.format().with_options(Parenthesize::Never))),
)
.nodes(keywords.iter())
.finish()
});
```
## Summary
This PR adds `TextRange` to `Identifier`. Right now, the AST only
includes ranges for identifiers in certain cases (`Expr::Name`,
`Keyword`, etc.), namely when the identifier comprises an entire AST
node. In Ruff, we do additional ad-hoc lexing to extract identifiers
from source code.
One frequent example: given a function `def f(): ...`, we lex to find
the range of `f`, for use in diagnostics.
Another: `except ValueError as e`, for which the AST doesn't include a
range for `e`.
Note that, as an optimization, we avoid storing the `TextRange` for
`Expr::Name`, since it's already included.
* Move `range` from `Attributed` to `Node`s
* No Attributed + custom for Range PoC
* Generate all located variants, generate enum implementations
* Implement `Copy` on simple enums
* Move `Suite` to `ranged` and `located`
* Update tests
---------
Co-authored-by: Jeong YunWon <jeong@youknowone.org>
- Split parser core and compiler core. Fix#14
- AST int type to `u32`
- Updated asdl_rs.py and update_asdl.sh fix#6
- Use `ruff_python_ast::SourceLocation` for Python source location. Deleted our own Location.
- Renamed ast::Located to ast::Attributed to distinguish terms for TextSize and SourceLocation
- `ast::<Node>`s for TextSize located ast. `ast::located::<Node>` for Python source located ast.
- And also strictly renaming `located` to refer only python location related interfaces.
- `SourceLocator` to convert locations.
- New `source-code` features of to disable python locations when unnecessary.
- Also including fully merging https://github.com/astral-sh/RustPython/pull/4closes#9
Since we parse an `elif:` block as an `If` node, its location should
include its `orelse` node like it would for an `if:` block.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
This token is completely ignored by the parser, but it's useful for
other users of the lexer, such as the Ruff linter. For example, the
token is helpful for a "trailing comma" lint.
The same idea exists in Python's `tokenize` module - there is a NEWLINE
token (logical newline), and a NL token (non-logical newline).
Fixes#4385.
docs.rs failed to build the documentation of the recently released
rustpython-parser 0.2.0 because the build.rs script couldn't write the
parser.rs file because docs.rs builds the documentation in a sandbox
with a read-only filesystem.
This commit fixes this by writing the parser.rs file to the cargo output
directory instead, as recommended by the docs.rs documentation.[1]
Fixes#4436.
[1]: https://docs.rs/about/builds#read-only-directories
Because the ‘with’ item grammar disallows named expressions, CPython
parses ‘with (a := 0, b := 1):’ as a tuple rather than two ‘with’
items.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
Since the upstream grammar for this is not LR(1), we abuse LALRPOP
macros and the Into/TryInto traits to build a cover grammar that
converts to either tuples or `with` items after additional validation.
It’s annoying and ugly, but something like this is basically our only
option short of switching to a more powerful parser algorithm.
Fixes#4145.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
-`crates/ruff_textwrap`: library crate to indent and dedent Python source code.
-`crates/ruff_python_parser`: library crate containing the Python parser.
-`crates/ruff_wasm`: library crate for exposing Ruff as a WebAssembly module. Powers the
[Ruff Playground](https://play.ruff.rs/).
@@ -224,9 +223,12 @@ Once you've completed the code for the rule itself, you can define tests with th
For example, if you're adding a new rule named `E402`, you would run:
```shell
cargo run -p ruff_cli -- check crates/ruff/resources/test/fixtures/pycodestyle/E402.py --no-cache
cargo run -p ruff_cli -- check crates/ruff/resources/test/fixtures/pycodestyle/E402.py --no-cache --select E402
```
**Note:** Only a subset of rules are enabled by default. When testing a new rule, ensure that
you activate it by adding `--select ${rule_code}` to the command.
1. Add the test to the relevant `crates/ruff/src/rules/[linter]/mod.rs` file. If you're contributing
a rule to a pre-existing set, you should be able to find a similar example to pattern-match
against. If you're adding a new linter, you'll need to create a new `mod.rs` file (see,
@@ -313,7 +315,7 @@ even patch releases may contain [non-backwards-compatible changes](https://semve
1. Run the release workflow with the version number (without starting `v`) as input. Make sure
main has your merged PR as last commit
1. The release workflow will do the following:
1. Build all the assets. If this fails (even though we tested in step 4), we haven’t tagged or
1. Build all the assets. If this fails (even though we tested in step 4), we haven't tagged or
uploaded anything, you can restart after pushing a fix.
1. Upload to PyPI.
1. Create and push the Git tag (as extracted from `pyproject.toml`). We create the Git tag only
@@ -726,9 +728,9 @@ diagnostics, then our current compilation pipeline proceeds as follows:
1. **File discovery**: Given paths like `foo/`, locate all Python files in any specified subdirectories, taking into account our hierarchical settings system and any `exclude` options.
1. **Package resolution**: Determine the “package root” for every file by traversing over its parent directories and looking for `__init__.py` files.
1. **Package resolution**: Determine the "package root" for every file by traversing over its parent directories and looking for `__init__.py` files.
1. **Cache initialization**: For every “package root”, initialize an empty cache.
1. **Cache initialization**: For every "package root", initialize an empty cache.
1. **Analysis**: For every file, in parallel:
@@ -736,7 +738,7 @@ diagnostics, then our current compilation pipeline proceeds as follows:
1. **Tokenization**: Run the lexer over the file to generate a token stream.
1. **Indexing**: Extract metadata from the token stream, such as: comment ranges, `# noqa` locations, `# isort: off` locations, “doc lines”, etc.
1. **Indexing**: Extract metadata from the token stream, such as: comment ranges, `# noqa` locations, `# isort: off` locations, "doc lines", etc.
1. **Token-based rule evaluation**: Run any lint rules that are based on the contents of the token stream (e.g., commented-out code).
@@ -746,9 +748,9 @@ diagnostics, then our current compilation pipeline proceeds as follows:
1. **Parsing**: Run the parser over the token stream to produce an AST. (This consumes the token stream, so anything that relies on the token stream needs to happen before parsing.)
1. **AST-based rule evaluation**: Run any lint rules that are based on the AST. This includes the vast majority of lint rules. As part of this step, we also build the semantic model for the current file as we traverse over the AST. Some lint rules are evaluated eagerly, as we iterate over the AST, while others are evaluated in a deferred manner (e.g., unused imports, since we can’t determine whether an import is unused until we’ve finished analyzing the entire file), after we’ve finished the initial traversal.
1. **AST-based rule evaluation**: Run any lint rules that are based on the AST. This includes the vast majority of lint rules. As part of this step, we also build the semantic model for the current file as we traverse over the AST. Some lint rules are evaluated eagerly, as we iterate over the AST, while others are evaluated in a deferred manner (e.g., unused imports, since we can't determine whether an import is unused until we've finished analyzing the entire file), after we've finished the initial traversal.
1. **Import-based rule evaluation**: Run any lint rules that are based on the module’s imports (e.g., import sorting). These could, in theory, be included in the AST-based rule evaluation phase — they’re just separated for simplicity.
1. **Import-based rule evaluation**: Run any lint rules that are based on the module's imports (e.g., import sorting). These could, in theory, be included in the AST-based rule evaluation phase — they're just separated for simplicity.
1. **Physical line-based rule evaluation**: Run any lint rules that are based on physical lines (e.g., line-length).
@@ -757,3 +759,116 @@ diagnostics, then our current compilation pipeline proceeds as follows:
1. **Cache write**: Write the generated diagnostics to the package cache using the file as a key.
1. **Reporting**: Print diagnostics in the specified format (text, JSON, etc.), to the specified output channel (stdout, a file, etc.).
### Import Categorization
To understand Ruff's import categorization system, we first need to define two concepts:
- "Project root": The directory containing the `pyproject.toml`, `ruff.toml`, or `.ruff.toml` file,
discovered by identifying the "closest" such directory for each Python file. (If you're running
via `ruff --config /path/to/pyproject.toml`, then the current working directory is used as the
"project root".)
- "Package root": The top-most directory defining the Python package that includes a given Python
file. To find the package root for a given Python file, traverse up its parent directories until
you reach a parent directory that doesn't contain an `__init__.py` file (and isn't marked as
a [namespace package](https://beta.ruff.rs/docs/settings/#namespace-packages)); take the directory
just before that, i.e., the first directory in the package.
For example, given:
```text
my_project
├── pyproject.toml
└── src
└── foo
├── __init__.py
└── bar
├── __init__.py
└── baz.py
```
Then when analyzing `baz.py`, the project root would be the top-level directory (`./my_project`),
and the package root would be `./my_project/src/foo`.
#### Project root
The project root does not have a significant impact beyond that all relative paths within the loaded
configuration file are resolved relative to the project root.
For example, to indicate that `bar` above is a namespace package (it isn't, but let's run with it),
the `pyproject.toml` would list `namespace-packages = ["./src/bar"]`, which would resolve
to `my_project/src/bar`.
The same logic applies when providing a configuration file via `--config`. In that case, the
_current working directory_ is used as the project root, and so all paths in that configuration file
are resolved relative to the current working directory. (As a general rule, we want to avoid relying
on the current working directory as much as possible, to ensure that Ruff exhibits the same behavior
regardless of where and how you invoke it — but that's hard to avoid in this case.)
Additionally, if a `pyproject.toml` file _extends_ another configuration file, Ruff will still use
the directory containing that `pyproject.toml` file as the project root. For example, if
`./my_project/pyproject.toml` contains:
```toml
[tool.ruff]
extend = "/path/to/pyproject.toml"
```
Then Ruff will use `./my_project` as the project root, even though the configuration file extends
`/path/to/pyproject.toml`. As such, if the configuration file at `/path/to/pyproject.toml` contains
any relative paths, they will be resolved relative to `./my_project`.
If a project uses nested configuration files, then Ruff would detect multiple project roots, one for
each configuration file.
#### Package root
The package root is used to determine a file's "module path". Consider, again, `baz.py`. In that
case, `./my_project/src/foo` was identified as the package root, so the module path for `baz.py`
would resolve to `foo.bar.baz` — as computed by taking the relative path from the package root
(inclusive of the root itself). The module path can be thought of as "the path you would use to
import the module" (e.g., `import foo.bar.baz`).
The package root and module path are used to, e.g., convert relative to absolute imports, and for
import categorization, as described below.
#### Import categorization
When sorting and formatting import blocks, Ruff categorizes every import into one of five
categories:
1. **"Future"**: the import is a `__future__` import. That's easy: just look at the name of the
imported module!
1. **"Standard library"**: the import comes from the Python standard library (e.g., `import os`).
This is easy too: we include a list of all known standard library modules in Ruff itself, so it's
a simple lookup.
1. **"Local folder"**: the import is a relative import (e.g., `from .foo import bar`). This is easy
too: just check if the import includes a `level` (i.e., a dot-prefix).
1. **"First party"**: the import is part of the current project. (More on this below.)
1. **"Third party"**: everything else.
The real challenge lies in determining whether an import is first-party — everything else is either
trivial, or (as in the case of third-party) merely defined as "not first-party".
There are three ways in which an import can be categorized as "first-party":
1. **Explicit settings**: the import is marked as such via the `known-first-party` setting. (This
should generally be seen as an escape hatch.)
1. **Same-package**: the imported module is in the same package as the current file. This gets back
to the importance of the "package root" and the file's "module path". Imagine that we're
analyzing `baz.py` above. If `baz.py` contains any imports that appear to come from the `foo`
package (e.g., `from foo import bar` or `import foo.bar`), they'll be classified as first-party
automatically. This check is as simple as comparing the first segment of the current file's
module path to the first segment of the import.
1. **Source roots**: Ruff supports a `[src](https://beta.ruff.rs/docs/settings/#src)` setting, which
sets the directories to scan when identifying first-party imports. The algorithm is
straightforward: given an import, like `import foo`, iterate over the directories enumerated in
the `src` setting and, for each directory, check for the existence of a subdirectory `foo` or a
file `foo.py`.
By default, `src` is set to the project root. In the above example, we'd want to set
`src = ["./src"]` to ensure that we locate `./my_project/src/foo` and thus categorize `import foo`
as first-party in `baz.py`. In practice, for this limited example, setting `src = ["./src"]` is
unnecessary, as all imports within `./my_project/src/foo` would be categorized as first-party via
the same-package heuristic; but your project contains multiple packages, you'll want to set `src`
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.