Try to reduce false positives more

Co-authored-by: Micha Reiser <micha@reiser.io>

.pre-commit-config.yaml

@@ -6,6 +6,8 @@ exclude: |
     crates/red_knot_workspace/resources/.*|
     crates/ruff_linter/resources/.*|
     crates/ruff_linter/src/rules/.*/snapshots/.*|
+    crates/ruff_notebook/resources/.*|
+    crates/ruff_server/resources/.*|
     crates/ruff/resources/.*|
     crates/ruff_python_formatter/resources/.*|
     crates/ruff_python_formatter/tests/snapshots/.*|
@@ -15,7 +17,7 @@ exclude: |
 
 repos:
   - repo: https://github.com/abravalheri/validate-pyproject
-    rev: v0.18
+    rev: v0.19
     hooks:
       - id: validate-pyproject
 
@@ -57,7 +59,7 @@ repos:
         pass_filenames: false # This makes it a lot faster
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.5.7
+    rev: v0.6.1
     hooks:
       - id: ruff-format
       - id: ruff

CHANGELOG.md

@@ -1,5 +1,39 @@
 # Changelog
 
+## 0.6.2
+
+### Preview features
+
+- \[`flake8-simplify`\] Extend `open-file-with-context-handler` to work with other standard-library IO modules (`SIM115`) ([#12959](https://github.com/astral-sh/ruff/pull/12959))
+- \[`ruff`\] Avoid `unused-async` for functions with FastAPI route decorator (`RUF029`) ([#12938](https://github.com/astral-sh/ruff/pull/12938))
+- \[`ruff`\] Ignore `fstring-missing-syntax` (`RUF027`) for `fastAPI` paths ([#12939](https://github.com/astral-sh/ruff/pull/12939))
+- \[`ruff`\] Implement check for Decimal called with a float literal (RUF032) ([#12909](https://github.com/astral-sh/ruff/pull/12909))
+
+### Rule changes
+
+- \[`flake8-bugbear`\] Update diagnostic message when expression is at the end of function (`B015`) ([#12944](https://github.com/astral-sh/ruff/pull/12944))
+- \[`flake8-pyi`\] Skip type annotations in `string-or-bytes-too-long` (`PYI053`) ([#13002](https://github.com/astral-sh/ruff/pull/13002))
+- \[`flake8-type-checking`\] Always recognise relative imports as first-party ([#12994](https://github.com/astral-sh/ruff/pull/12994))
+- \[`flake8-unused-arguments`\] Ignore unused arguments on stub functions (`ARG001`) ([#12966](https://github.com/astral-sh/ruff/pull/12966))
+- \[`pylint`\] Ignore augmented assignment for `self-cls-assignment` (`PLW0642`) ([#12957](https://github.com/astral-sh/ruff/pull/12957))
+
+### Server
+
+- Show full context in error log messages ([#13029](https://github.com/astral-sh/ruff/pull/13029))
+
+### Bug fixes
+
+- \[`pep8-naming`\] Don't flag `from` imports following conventional import names (`N817`) ([#12946](https://github.com/astral-sh/ruff/pull/12946))
+- \[`pylint`\] - Allow `__new__` methods to have `cls` as their first argument even if decorated with `@staticmethod` for `bad-staticmethod-argument` (`PLW0211`) ([#12958](https://github.com/astral-sh/ruff/pull/12958))
+
+### Documentation
+
+- Add `hyperfine` installation instructions; update `hyperfine` code samples ([#13034](https://github.com/astral-sh/ruff/pull/13034))
+- Expand note to use Ruff with other language server in Kate ([#12806](https://github.com/astral-sh/ruff/pull/12806))
+- Update example for `PT001` as per the new default behavior ([#13019](https://github.com/astral-sh/ruff/pull/13019))
+- \[`perflint`\] Improve docs for `try-except-in-loop` (`PERF203`) ([#12947](https://github.com/astral-sh/ruff/pull/12947))
+- \[`pydocstyle`\] Add reference to `lint.pydocstyle.ignore-decorators` setting to rule docs ([#12996](https://github.com/astral-sh/ruff/pull/12996))
+
 ## 0.6.1
 
 This is a hotfix release to address an issue with `ruff-pre-commit`. In v0.6,

CONTRIBUTING.md

@@ -2,35 +2,6 @@
 
 Welcome! We're happy to have you here. Thank you in advance for your contribution to Ruff.
 
-- [The Basics](#the-basics)
-    - [Prerequisites](#prerequisites)
-    - [Development](#development)
-    - [Project Structure](#project-structure)
-    - [Example: Adding a new lint rule](#example-adding-a-new-lint-rule)
-        - [Rule naming convention](#rule-naming-convention)
-        - [Rule testing: fixtures and snapshots](#rule-testing-fixtures-and-snapshots)
-    - [Example: Adding a new configuration option](#example-adding-a-new-configuration-option)
-- [MkDocs](#mkdocs)
-- [Release Process](#release-process)
-    - [Creating a new release](#creating-a-new-release)
-- [Ecosystem CI](#ecosystem-ci)
-- [Benchmarking and Profiling](#benchmarking-and-profiling)
-    - [CPython Benchmark](#cpython-benchmark)
-    - [Microbenchmarks](#microbenchmarks)
-        - [Benchmark-driven Development](#benchmark-driven-development)
-        - [PR Summary](#pr-summary)
-        - [Tips](#tips)
-    - [Profiling Projects](#profiling-projects)
-        - [Linux](#linux)
-        - [Mac](#mac)
-- [`cargo dev`](#cargo-dev)
-- [Subsystems](#subsystems)
-    - [Compilation Pipeline](#compilation-pipeline)
-    - [Import Categorization](#import-categorization)
-        - [Project root](#project-root)
-        - [Package root](#package-root)
-        - [Import categorization](#import-categorization-1)
-
 ## The Basics
 
 Ruff welcomes contributions in the form of pull requests.
@@ -333,22 +304,34 @@ even patch releases may contain [non-backwards-compatible changes](https://semve
 ### Creating a new release
 
 1. Install `uv`: `curl -LsSf https://astral.sh/uv/install.sh | sh`
+
 1. Run `./scripts/release.sh`; this command will:
+
     - Generate a temporary virtual environment with `rooster`
     - Generate a changelog entry in `CHANGELOG.md`
     - Update versions in `pyproject.toml` and `Cargo.toml`
     - Update references to versions in the `README.md` and documentation
     - Display contributors for the release
+
 1. The changelog should then be editorialized for consistency
+
     - Often labels will be missing from pull requests they will need to be manually organized into the proper section
     - Changes should be edited to be user-facing descriptions, avoiding internal details
+
 1. Highlight any breaking changes in `BREAKING_CHANGES.md`
+
 1. Run `cargo check`. This should update the lock file with new versions.
+
 1. Create a pull request with the changelog and version updates
+
 1. Merge the PR
+
 1. Run the [release workflow](https://github.com/astral-sh/ruff/actions/workflows/release.yml) with:
+
     - The new version number (without starting `v`)
+
 1. The release workflow will do the following:
+
     1. Build all the assets. If this fails (even though we tested in step 4), we haven't tagged or
         uploaded anything, you can restart after pushing a fix. If you just need to rerun the build,
         make sure you're [re-running all the failed
@@ -359,14 +342,25 @@ even patch releases may contain [non-backwards-compatible changes](https://semve
     1. Attach artifacts to draft GitHub release
     1. Trigger downstream repositories. This can fail non-catastrophically, as we can run any
         downstream jobs manually if needed.
+
 1. Verify the GitHub release:
+
     1. The Changelog should match the content of `CHANGELOG.md`
     1. Append the contributors from the `scripts/release.sh` script
+
 1. If needed, [update the schemastore](https://github.com/astral-sh/ruff/blob/main/scripts/update_schemastore.py).
+
     1. One can determine if an update is needed when
         `git diff old-version-tag new-version-tag -- ruff.schema.json` returns a non-empty diff.
     1. Once run successfully, you should follow the link in the output to create a PR.
-1. If needed, update the `ruff-lsp` and `ruff-vscode` repositories.
+
+1. If needed, update the [`ruff-lsp`](https://github.com/astral-sh/ruff-lsp) and
+    [`ruff-vscode`](https://github.com/astral-sh/ruff-vscode) repositories and follow
+    the release instructions in those repositories. `ruff-lsp` should always be updated
+    before `ruff-vscode`.
+
+    This step is generally not required for a patch release, but should always be done
+    for a minor release.
 
 ## Ecosystem CI
 
@@ -389,7 +383,7 @@ We have several ways of benchmarking and profiling Ruff:
 - Microbenchmarks which run the linter or the formatter on individual files. These run on pull requests.
 - Profiling the linter on either the microbenchmarks or entire projects
 
-> \[!NOTE\]
+> **Note**
 > When running benchmarks, ensure that your CPU is otherwise idle (e.g., close any background
 > applications, like web browsers). You may also want to switch your CPU to a "performance"
 > mode, if it exists, especially when benchmarking short-lived processes.
@@ -403,12 +397,18 @@ which makes it a good target for benchmarking.
 git clone --branch 3.10 https://github.com/python/cpython.git crates/ruff_linter/resources/test/cpython
 ```
 
+Install `hyperfine`:
+
+```shell
+cargo install hyperfine
+```
+
 To benchmark the release build:
 
 ```shell
 cargo build --release && hyperfine --warmup 10 \
-  "./target/release/ruff ./crates/ruff_linter/resources/test/cpython/ --no-cache -e" \
-  "./target/release/ruff ./crates/ruff_linter/resources/test/cpython/ -e"
+  "./target/release/ruff check ./crates/ruff_linter/resources/test/cpython/ --no-cache -e" \
+  "./target/release/ruff check ./crates/ruff_linter/resources/test/cpython/ -e"
 
 Benchmark 1: ./target/release/ruff ./crates/ruff_linter/resources/test/cpython/ --no-cache
   Time (mean ± σ):     293.8 ms ±   3.2 ms    [User: 2384.6 ms, System: 90.3 ms]
@@ -427,7 +427,7 @@ To benchmark against the ecosystem's existing tools:
 
 ```shell
 hyperfine --ignore-failure --warmup 5 \
-  "./target/release/ruff ./crates/ruff_linter/resources/test/cpython/ --no-cache" \
+  "./target/release/ruff check ./crates/ruff_linter/resources/test/cpython/ --no-cache" \
   "pyflakes crates/ruff_linter/resources/test/cpython" \
   "autoflake --recursive --expand-star-imports --remove-all-unused-imports --remove-unused-variables --remove-duplicate-keys resources/test/cpython" \
   "pycodestyle crates/ruff_linter/resources/test/cpython" \
@@ -473,7 +473,7 @@ To benchmark a subset of rules, e.g. `LineTooLong` and `DocLineTooLong`:
 
 ```shell
 cargo build --release && hyperfine --warmup 10 \
-  "./target/release/ruff ./crates/ruff_linter/resources/test/cpython/ --no-cache -e --select W505,E501"
+  "./target/release/ruff check ./crates/ruff_linter/resources/test/cpython/ --no-cache -e --select W505,E501"
 ```
 
 You can run `poetry install` from `./scripts/benchmarks` to create a working environment for the

Cargo.lock

@@ -1926,6 +1926,7 @@ dependencies = [
  "ruff_db",
  "ruff_index",
  "ruff_python_ast",
+ "ruff_python_literal",
  "ruff_python_parser",
  "ruff_python_stdlib",
  "ruff_source_file",
@@ -1935,6 +1936,7 @@ dependencies = [
  "smallvec",
  "static_assertions",
  "tempfile",
+ "thiserror",
  "tracing",
  "walkdir",
  "zip",
@@ -1950,10 +1952,8 @@ dependencies = [
  "libc",
  "lsp-server",
  "lsp-types",
- "red_knot_python_semantic",
  "red_knot_workspace",
  "ruff_db",
- "ruff_linter",
  "ruff_notebook",
  "ruff_python_ast",
  "ruff_source_file",
@@ -1996,7 +1996,6 @@ dependencies = [
  "ruff_text_size",
  "rustc-hash 2.0.0",
  "salsa",
- "thiserror",
  "tracing",
 ]
 
@@ -2090,7 +2089,7 @@ dependencies = [
 
 [[package]]
 name = "ruff"
-version = "0.6.1"
+version = "0.6.2"
 dependencies = [
  "anyhow",
  "argfile",
@@ -2282,7 +2281,7 @@ dependencies = [
 
 [[package]]
 name = "ruff_linter"
-version = "0.6.1"
+version = "0.6.2"
 dependencies = [
  "aho-corasick",
  "annotate-snippets 0.9.2",
@@ -2602,7 +2601,7 @@ dependencies = [
 
 [[package]]
 name = "ruff_wasm"
-version = "0.6.1"
+version = "0.6.2"
 dependencies = [
  "console_error_panic_hook",
  "console_log",
@@ -2742,7 +2741,7 @@ checksum = "e86697c916019a8588c99b5fac3cead74ec0b4b819707a682fd4d23fa0ce1ba1"
 [[package]]
 name = "salsa"
 version = "0.18.0"
-source = "git+https://github.com/MichaReiser/salsa.git?tag=red-knot-0.0.1#ece083e15b79f155f9e4368ec1318cec9a08d88b"
+source = "git+https://github.com/salsa-rs/salsa.git?rev=f608ff8b24f07706492027199f51132244034f29#f608ff8b24f07706492027199f51132244034f29"
 dependencies = [
  "append-only-vec",
  "arc-swap",
@@ -2762,12 +2761,12 @@ dependencies = [
 [[package]]
 name = "salsa-macro-rules"
 version = "0.1.0"
-source = "git+https://github.com/MichaReiser/salsa.git?tag=red-knot-0.0.1#ece083e15b79f155f9e4368ec1318cec9a08d88b"
+source = "git+https://github.com/salsa-rs/salsa.git?rev=f608ff8b24f07706492027199f51132244034f29#f608ff8b24f07706492027199f51132244034f29"
 
 [[package]]
 name = "salsa-macros"
 version = "0.18.0"
-source = "git+https://github.com/MichaReiser/salsa.git?tag=red-knot-0.0.1#ece083e15b79f155f9e4368ec1318cec9a08d88b"
+source = "git+https://github.com/salsa-rs/salsa.git?rev=f608ff8b24f07706492027199f51132244034f29#f608ff8b24f07706492027199f51132244034f29"
 dependencies = [
  "heck",
  "proc-macro2",

Cargo.toml

@@ -108,7 +108,7 @@ rand = { version = "0.8.5" }
 rayon = { version = "1.10.0" }
 regex = { version = "1.10.2" }
 rustc-hash = { version = "2.0.0" }
-salsa = { git = "https://github.com/MichaReiser/salsa.git", tag = "red-knot-0.0.1" }
+salsa = { git = "https://github.com/salsa-rs/salsa.git", rev = "f608ff8b24f07706492027199f51132244034f29" }
 schemars = { version = "0.8.16" }
 seahash = { version = "4.1.0" }
 serde = { version = "1.0.197", features = ["derive"] }

README.md

@@ -136,8 +136,8 @@ curl -LsSf https://astral.sh/ruff/install.sh | sh
 powershell -c "irm https://astral.sh/ruff/install.ps1 | iex"
 
 # For a specific version.
-curl -LsSf https://astral.sh/ruff/0.6.1/install.sh | sh
-powershell -c "irm https://astral.sh/ruff/0.6.1/install.ps1 | iex"
+curl -LsSf https://astral.sh/ruff/0.6.2/install.sh | sh
+powershell -c "irm https://astral.sh/ruff/0.6.2/install.ps1 | iex"
 ```
 
 You can also install Ruff via [Homebrew](https://formulae.brew.sh/formula/ruff), [Conda](https://anaconda.org/conda-forge/ruff),
@@ -170,7 +170,7 @@ Ruff can also be used as a [pre-commit](https://pre-commit.com/) hook via [`ruff
 ```yaml
 - repo: https://github.com/astral-sh/ruff-pre-commit
   # Ruff version.
-  rev: v0.6.1
+  rev: v0.6.2
   hooks:
     # Run the linter.
     - id: ruff

crates/red_knot/src/logging.rs

@@ -5,8 +5,8 @@ use colored::Colorize;
 use std::fmt;
 use std::fs::File;
 use std::io::BufWriter;
-use tracing::log::LevelFilter;
 use tracing::{Event, Subscriber};
+use tracing_subscriber::filter::LevelFilter;
 use tracing_subscriber::fmt::format::Writer;
 use tracing_subscriber::fmt::{FmtContext, FormatEvent, FormatFields};
 use tracing_subscriber::registry::LookupSpan;
@@ -60,10 +60,10 @@ pub(crate) enum VerbosityLevel {
 impl VerbosityLevel {
     const fn level_filter(self) -> LevelFilter {
         match self {
-            VerbosityLevel::Default => LevelFilter::Warn,
-            VerbosityLevel::Verbose => LevelFilter::Info,
-            VerbosityLevel::ExtraVerbose => LevelFilter::Debug,
-            VerbosityLevel::Trace => LevelFilter::Trace,
+            VerbosityLevel::Default => LevelFilter::WARN,
+            VerbosityLevel::Verbose => LevelFilter::INFO,
+            VerbosityLevel::ExtraVerbose => LevelFilter::DEBUG,
+            VerbosityLevel::Trace => LevelFilter::TRACE,
         }
     }
 
@@ -88,7 +88,7 @@ pub(crate) fn setup_tracing(level: VerbosityLevel) -> anyhow::Result<TracingGuar
         match level {
             VerbosityLevel::Default => {
                 // Show warning traces
-                EnvFilter::default().add_directive(tracing::level_filters::LevelFilter::WARN.into())
+                EnvFilter::default().add_directive(LevelFilter::WARN.into())
             }
             level => {
                 let level_filter = level.level_filter();

crates/red_knot/src/main.rs

@@ -7,12 +7,12 @@ use colored::Colorize;
 use crossbeam::channel as crossbeam_channel;
 use salsa::plumbing::ZalsaDatabase;
 
-use red_knot_python_semantic::{ProgramSettings, SearchPathSettings};
+use red_knot_python_semantic::SitePackages;
 use red_knot_server::run_server;
 use red_knot_workspace::db::RootDatabase;
-use red_knot_workspace::site_packages::VirtualEnvironment;
 use red_knot_workspace::watch;
 use red_knot_workspace::watch::WorkspaceWatcher;
+use red_knot_workspace::workspace::settings::Configuration;
 use red_knot_workspace::workspace::WorkspaceMetadata;
 use ruff_db::system::{OsSystem, System, SystemPath, SystemPathBuf};
 use target_version::TargetVersion;
@@ -65,15 +65,14 @@ to resolve type information for the project's third-party dependencies.",
         value_name = "PATH",
         help = "Additional path to use as a module-resolution source (can be passed multiple times)"
     )]
-    extra_search_path: Vec<SystemPathBuf>,
+    extra_search_path: Option<Vec<SystemPathBuf>>,
 
     #[arg(
         long,
         help = "Python version to assume when resolving types",
-        default_value_t = TargetVersion::default(),
-        value_name="VERSION")
-    ]
-    target_version: TargetVersion,
+        value_name = "VERSION"
+    )]
+    target_version: Option<TargetVersion>,
 
     #[clap(flatten)]
     verbosity: Verbosity,
@@ -86,6 +85,36 @@ to resolve type information for the project's third-party dependencies.",
     watch: bool,
 }
 
+impl Args {
+    fn to_configuration(&self, cli_cwd: &SystemPath) -> Configuration {
+        let mut configuration = Configuration::default();
+
+        if let Some(target_version) = self.target_version {
+            configuration.target_version = Some(target_version.into());
+        }
+
+        if let Some(venv_path) = &self.venv_path {
+            configuration.search_paths.site_packages = Some(SitePackages::Derived {
+                venv_path: SystemPath::absolute(venv_path, cli_cwd),
+            });
+        }
+
+        if let Some(custom_typeshed_dir) = &self.custom_typeshed_dir {
+            configuration.search_paths.custom_typeshed =
+                Some(SystemPath::absolute(custom_typeshed_dir, cli_cwd));
+        }
+
+        if let Some(extra_search_paths) = &self.extra_search_path {
+            configuration.search_paths.extra_paths = extra_search_paths
+                .iter()
+                .map(|path| Some(SystemPath::absolute(path, cli_cwd)))
+                .collect();
+        }
+
+        configuration
+    }
+}
+
 #[derive(Debug, clap::Subcommand)]
 pub enum Command {
     /// Start the language server
@@ -115,22 +144,13 @@ pub fn main() -> ExitStatus {
 }
 
 fn run() -> anyhow::Result<ExitStatus> {
-    let Args {
-        command,
-        current_directory,
-        custom_typeshed_dir,
-        extra_search_path: extra_paths,
-        venv_path,
-        target_version,
-        verbosity,
-        watch,
-    } = Args::parse_from(std::env::args().collect::<Vec<_>>());
+    let args = Args::parse_from(std::env::args().collect::<Vec<_>>());
 
-    if matches!(command, Some(Command::Server)) {
+    if matches!(args.command, Some(Command::Server)) {
         return run_server().map(|()| ExitStatus::Success);
     }
 
-    let verbosity = verbosity.level();
+    let verbosity = args.verbosity.level();
     countme::enable(verbosity.is_trace());
     let _guard = setup_tracing(verbosity)?;
 
@@ -146,10 +166,12 @@ fn run() -> anyhow::Result<ExitStatus> {
             })?
     };
 
-    let cwd = current_directory
+    let cwd = args
+        .current_directory
+        .as_ref()
         .map(|cwd| {
             if cwd.as_std_path().is_dir() {
-                Ok(SystemPath::absolute(&cwd, &cli_base_path))
+                Ok(SystemPath::absolute(cwd, &cli_base_path))
             } else {
                 Err(anyhow!(
                     "Provided current-directory path '{cwd}' is not a directory."
@@ -160,33 +182,18 @@ fn run() -> anyhow::Result<ExitStatus> {
         .unwrap_or_else(|| cli_base_path.clone());
 
     let system = OsSystem::new(cwd.clone());
-    let workspace_metadata = WorkspaceMetadata::from_path(system.current_directory(), &system)?;
-
-    // TODO: Verify the remaining search path settings eagerly.
-    let site_packages = venv_path
-        .map(|path| {
-            VirtualEnvironment::new(path, &OsSystem::new(cli_base_path))
-                .and_then(|venv| venv.site_packages_directories(&system))
-        })
-        .transpose()?
-        .unwrap_or_default();
-
-    // TODO: Respect the settings from the workspace metadata. when resolving the program settings.
-    let program_settings = ProgramSettings {
-        target_version: target_version.into(),
-        search_paths: SearchPathSettings {
-            extra_paths,
-            src_root: workspace_metadata.root().to_path_buf(),
-            custom_typeshed: custom_typeshed_dir,
-            site_packages,
-        },
-    };
+    let cli_configuration = args.to_configuration(&cwd);
+    let workspace_metadata = WorkspaceMetadata::from_path(
+        system.current_directory(),
+        &system,
+        Some(cli_configuration.clone()),
+    )?;
 
     // TODO: Use the `program_settings` to compute the key for the database's persistent
     //   cache and load the cache if it exists.
-    let mut db = RootDatabase::new(workspace_metadata, program_settings, system)?;
+    let mut db = RootDatabase::new(workspace_metadata, system)?;
 
-    let (main_loop, main_loop_cancellation_token) = MainLoop::new();
+    let (main_loop, main_loop_cancellation_token) = MainLoop::new(cli_configuration);
 
     // Listen to Ctrl+C and abort the watch mode.
     let main_loop_cancellation_token = Mutex::new(Some(main_loop_cancellation_token));
@@ -198,7 +205,7 @@ fn run() -> anyhow::Result<ExitStatus> {
         }
     })?;
 
-    let exit_status = if watch {
+    let exit_status = if args.watch {
         main_loop.watch(&mut db)?
     } else {
         main_loop.run(&mut db)
@@ -238,10 +245,12 @@ struct MainLoop {
 
     /// The file system watcher, if running in watch mode.
     watcher: Option<WorkspaceWatcher>,
+
+    cli_configuration: Configuration,
 }
 
 impl MainLoop {
-    fn new() -> (Self, MainLoopCancellationToken) {
+    fn new(cli_configuration: Configuration) -> (Self, MainLoopCancellationToken) {
         let (sender, receiver) = crossbeam_channel::bounded(10);
 
         (
@@ -249,6 +258,7 @@ impl MainLoop {
                 sender: sender.clone(),
                 receiver,
                 watcher: None,
+                cli_configuration,
             },
             MainLoopCancellationToken { sender },
         )
@@ -331,7 +341,7 @@ impl MainLoop {
                 MainLoopMessage::ApplyChanges(changes) => {
                     revision += 1;
                     // Automatically cancels any pending queries and waits for them to complete.
-                    db.apply_changes(changes);
+                    db.apply_changes(changes, Some(&self.cli_configuration));
                     if let Some(watcher) = self.watcher.as_mut() {
                         watcher.update(db);
                     }

crates/red_knot/tests/file_watching.rs

@@ -5,12 +5,11 @@ use std::time::Duration;
 
 use anyhow::{anyhow, Context};
 
-use red_knot_python_semantic::{
-    resolve_module, ModuleName, Program, ProgramSettings, PythonVersion, SearchPathSettings,
-};
+use red_knot_python_semantic::{resolve_module, ModuleName, Program, PythonVersion, SitePackages};
 use red_knot_workspace::db::RootDatabase;
 use red_knot_workspace::watch;
 use red_knot_workspace::watch::{directory_watcher, WorkspaceWatcher};
+use red_knot_workspace::workspace::settings::{Configuration, SearchPathConfiguration};
 use red_knot_workspace::workspace::WorkspaceMetadata;
 use ruff_db::files::{system_path_to_file, File, FileError};
 use ruff_db::source::source_text;
@@ -25,7 +24,7 @@ struct TestCase {
     /// We need to hold on to it in the test case or the temp files get deleted.
     _temp_dir: tempfile::TempDir,
     root_dir: SystemPathBuf,
-    search_path_settings: SearchPathSettings,
+    configuration: Configuration,
 }
 
 impl TestCase {
@@ -41,10 +40,6 @@ impl TestCase {
         &self.db
     }
 
-    fn db_mut(&mut self) -> &mut RootDatabase {
-        &mut self.db
-    }
-
     fn stop_watch(&mut self) -> Vec<watch::ChangeEvent> {
         self.try_stop_watch(Duration::from_secs(10))
             .expect("Expected watch changes but observed none.")
@@ -105,16 +100,20 @@ impl TestCase {
         Some(all_events)
     }
 
+    fn apply_changes(&mut self, changes: Vec<watch::ChangeEvent>) {
+        self.db.apply_changes(changes, Some(&self.configuration));
+    }
+
     fn update_search_path_settings(
         &mut self,
-        f: impl FnOnce(&SearchPathSettings) -> SearchPathSettings,
+        configuration: SearchPathConfiguration,
     ) -> anyhow::Result<()> {
         let program = Program::get(self.db());
 
-        let new_settings = f(&self.search_path_settings);
+        self.configuration.search_paths = configuration.clone();
+        let new_settings = configuration.into_settings(self.db.workspace().root(&self.db));
 
-        program.update_search_paths(&mut self.db, new_settings.clone())?;
-        self.search_path_settings = new_settings;
+        program.update_search_paths(&mut self.db, &new_settings)?;
 
         if let Some(watcher) = &mut self.watcher {
             watcher.update(&self.db);
@@ -127,7 +126,6 @@ impl TestCase {
     fn collect_package_files(&self, path: &SystemPath) -> Vec<File> {
         let package = self.db().workspace().package(self.db(), path).unwrap();
         let files = package.files(self.db());
-        let files = files.read();
         let mut collected: Vec<_> = files.into_iter().collect();
         collected.sort_unstable_by_key(|file| file.path(self.db()).as_system_path().unwrap());
         collected
@@ -180,17 +178,14 @@ fn setup<F>(setup_files: F) -> anyhow::Result<TestCase>
 where
     F: SetupFiles,
 {
-    setup_with_search_paths(setup_files, |_root, workspace_path| SearchPathSettings {
-        extra_paths: vec![],
-        src_root: workspace_path.to_path_buf(),
-        custom_typeshed: None,
-        site_packages: vec![],
+    setup_with_search_paths(setup_files, |_root, _workspace_path| {
+        SearchPathConfiguration::default()
     })
 }
 
 fn setup_with_search_paths<F>(
     setup_files: F,
-    create_search_paths: impl FnOnce(&SystemPath, &SystemPath) -> SearchPathSettings,
+    create_search_paths: impl FnOnce(&SystemPath, &SystemPath) -> SearchPathConfiguration,
 ) -> anyhow::Result<TestCase>
 where
     F: SetupFiles,
@@ -222,25 +217,34 @@ where
 
     let system = OsSystem::new(&workspace_path);
 
-    let workspace = WorkspaceMetadata::from_path(&workspace_path, &system)?;
-    let search_path_settings = create_search_paths(&root_path, workspace.root());
+    let search_paths = create_search_paths(&root_path, &workspace_path);
 
-    for path in search_path_settings
+    for path in search_paths
         .extra_paths
         .iter()
-        .chain(search_path_settings.site_packages.iter())
-        .chain(search_path_settings.custom_typeshed.iter())
+        .flatten()
+        .chain(search_paths.custom_typeshed.iter())
+        .chain(search_paths.site_packages.iter().flat_map(|site_packages| {
+            if let SitePackages::Known(path) = site_packages {
+                path.as_slice()
+            } else {
+                &[]
+            }
+        }))
     {
         std::fs::create_dir_all(path.as_std_path())
             .with_context(|| format!("Failed to create search path '{path}'"))?;
     }
 
-    let settings = ProgramSettings {
-        target_version: PythonVersion::default(),
-        search_paths: search_path_settings.clone(),
+    let configuration = Configuration {
+        target_version: Some(PythonVersion::PY312),
+        search_paths,
     };
 
-    let db = RootDatabase::new(workspace, settings, system)?;
+    let workspace =
+        WorkspaceMetadata::from_path(&workspace_path, &system, Some(configuration.clone()))?;
+
+    let db = RootDatabase::new(workspace, system)?;
 
     let (sender, receiver) = crossbeam::channel::unbounded();
     let watcher = directory_watcher(move |events| sender.send(events).unwrap())
@@ -255,7 +259,7 @@ where
         watcher: Some(watcher),
         _temp_dir: temp_dir,
         root_dir: root_path,
-        search_path_settings,
+        configuration,
     };
 
     // Sometimes the file watcher reports changes for events that happened before the watcher was started.
@@ -308,7 +312,7 @@ fn new_file() -> anyhow::Result<()> {
 
     let changes = case.stop_watch();
 
-    case.db_mut().apply_changes(changes);
+    case.apply_changes(changes);
 
     let foo = case.system_file(&foo_path).expect("foo.py to exist.");
 
@@ -331,7 +335,7 @@ fn new_ignored_file() -> anyhow::Result<()> {
 
     let changes = case.stop_watch();
 
-    case.db_mut().apply_changes(changes);
+    case.apply_changes(changes);
 
     assert!(case.system_file(&foo_path).is_ok());
     assert_eq!(&case.collect_package_files(&bar_path), &[bar_file]);
@@ -355,7 +359,7 @@ fn changed_file() -> anyhow::Result<()> {
 
     assert!(!changes.is_empty());
 
-    case.db_mut().apply_changes(changes);
+    case.apply_changes(changes);
 
     assert_eq!(source_text(case.db(), foo).as_str(), "print('Version 2')");
     assert_eq!(&case.collect_package_files(&foo_path), &[foo]);
@@ -378,7 +382,7 @@ fn deleted_file() -> anyhow::Result<()> {
 
     let changes = case.stop_watch();
 
-    case.db_mut().apply_changes(changes);
+    case.apply_changes(changes);
 
     assert!(!foo.exists(case.db()));
     assert_eq!(&case.collect_package_files(&foo_path), &[] as &[File]);
@@ -410,7 +414,7 @@ fn move_file_to_trash() -> anyhow::Result<()> {
 
     let changes = case.stop_watch();
 
-    case.db_mut().apply_changes(changes);
+    case.apply_changes(changes);
 
     assert!(!foo.exists(case.db()));
     assert_eq!(&case.collect_package_files(&foo_path), &[] as &[File]);
@@ -442,7 +446,7 @@ fn move_file_to_workspace() -> anyhow::Result<()> {
 
     let changes = case.stop_watch();
 
-    case.db_mut().apply_changes(changes);
+    case.apply_changes(changes);
 
     let foo_in_workspace = case.system_file(&foo_in_workspace_path)?;
 
@@ -470,7 +474,7 @@ fn rename_file() -> anyhow::Result<()> {
 
     let changes = case.stop_watch();
 
-    case.db_mut().apply_changes(changes);
+    case.apply_changes(changes);
 
     assert!(!foo.exists(case.db()));
 
@@ -511,7 +515,7 @@ fn directory_moved_to_workspace() -> anyhow::Result<()> {
 
     let changes = case.stop_watch();
 
-    case.db_mut().apply_changes(changes);
+    case.apply_changes(changes);
 
     let init_file = case
         .system_file(sub_new_path.join("__init__.py"))
@@ -562,7 +566,7 @@ fn directory_moved_to_trash() -> anyhow::Result<()> {
 
     let changes = case.stop_watch();
 
-    case.db_mut().apply_changes(changes);
+    case.apply_changes(changes);
 
     // `import sub.a` should no longer resolve
     assert!(resolve_module(case.db().upcast(), ModuleName::new_static("sub.a").unwrap()).is_none());
@@ -616,7 +620,7 @@ fn directory_renamed() -> anyhow::Result<()> {
 
     let changes = case.stop_watch();
 
-    case.db_mut().apply_changes(changes);
+    case.apply_changes(changes);
 
     // `import sub.a` should no longer resolve
     assert!(resolve_module(case.db().upcast(), ModuleName::new_static("sub.a").unwrap()).is_none());
@@ -681,7 +685,7 @@ fn directory_deleted() -> anyhow::Result<()> {
 
     let changes = case.stop_watch();
 
-    case.db_mut().apply_changes(changes);
+    case.apply_changes(changes);
 
     // `import sub.a` should no longer resolve
     assert!(resolve_module(case.db().upcast(), ModuleName::new_static("sub.a").unwrap()).is_none());
@@ -695,15 +699,13 @@ fn directory_deleted() -> anyhow::Result<()> {
 
 #[test]
 fn search_path() -> anyhow::Result<()> {
-    let mut case =
-        setup_with_search_paths([("bar.py", "import sub.a")], |root_path, workspace_path| {
-            SearchPathSettings {
-                extra_paths: vec![],
-                src_root: workspace_path.to_path_buf(),
-                custom_typeshed: None,
-                site_packages: vec![root_path.join("site_packages")],
-            }
-        })?;
+    let mut case = setup_with_search_paths(
+        [("bar.py", "import sub.a")],
+        |root_path, _workspace_path| SearchPathConfiguration {
+            site_packages: Some(SitePackages::Known(vec![root_path.join("site_packages")])),
+            ..SearchPathConfiguration::default()
+        },
+    )?;
 
     let site_packages = case.root_path().join("site_packages");
 
@@ -716,7 +718,7 @@ fn search_path() -> anyhow::Result<()> {
 
     let changes = case.stop_watch();
 
-    case.db_mut().apply_changes(changes);
+    case.apply_changes(changes);
 
     assert!(resolve_module(case.db().upcast(), ModuleName::new_static("a").unwrap()).is_some());
     assert_eq!(
@@ -737,9 +739,9 @@ fn add_search_path() -> anyhow::Result<()> {
     assert!(resolve_module(case.db().upcast(), ModuleName::new_static("a").unwrap()).is_none());
 
     // Register site-packages as a search path.
-    case.update_search_path_settings(|settings| SearchPathSettings {
-        site_packages: vec![site_packages.clone()],
-        ..settings.clone()
+    case.update_search_path_settings(SearchPathConfiguration {
+        site_packages: Some(SitePackages::Known(vec![site_packages.clone()])),
+        ..SearchPathConfiguration::default()
     })
     .expect("Search path settings to be valid");
 
@@ -747,7 +749,7 @@ fn add_search_path() -> anyhow::Result<()> {
 
     let changes = case.stop_watch();
 
-    case.db_mut().apply_changes(changes);
+    case.apply_changes(changes);
 
     assert!(resolve_module(case.db().upcast(), ModuleName::new_static("a").unwrap()).is_some());
 
@@ -756,21 +758,19 @@ fn add_search_path() -> anyhow::Result<()> {
 
 #[test]
 fn remove_search_path() -> anyhow::Result<()> {
-    let mut case =
-        setup_with_search_paths([("bar.py", "import sub.a")], |root_path, workspace_path| {
-            SearchPathSettings {
-                extra_paths: vec![],
-                src_root: workspace_path.to_path_buf(),
-                custom_typeshed: None,
-                site_packages: vec![root_path.join("site_packages")],
-            }
-        })?;
+    let mut case = setup_with_search_paths(
+        [("bar.py", "import sub.a")],
+        |root_path, _workspace_path| SearchPathConfiguration {
+            site_packages: Some(SitePackages::Known(vec![root_path.join("site_packages")])),
+            ..SearchPathConfiguration::default()
+        },
+    )?;
 
     // Remove site packages from the search path settings.
     let site_packages = case.root_path().join("site_packages");
-    case.update_search_path_settings(|settings| SearchPathSettings {
-        site_packages: vec![],
-        ..settings.clone()
+    case.update_search_path_settings(SearchPathConfiguration {
+        site_packages: None,
+        ..SearchPathConfiguration::default()
     })
     .expect("Search path settings to be valid");
 
@@ -783,6 +783,48 @@ fn remove_search_path() -> anyhow::Result<()> {
     Ok(())
 }
 
+#[test]
+fn changed_versions_file() -> anyhow::Result<()> {
+    let mut case = setup_with_search_paths(
+        |root_path: &SystemPath, workspace_path: &SystemPath| {
+            std::fs::write(workspace_path.join("bar.py").as_std_path(), "import sub.a")?;
+            std::fs::create_dir_all(root_path.join("typeshed/stdlib").as_std_path())?;
+            std::fs::write(root_path.join("typeshed/stdlib/VERSIONS").as_std_path(), "")?;
+            std::fs::write(
+                root_path.join("typeshed/stdlib/os.pyi").as_std_path(),
+                "# not important",
+            )?;
+
+            Ok(())
+        },
+        |root_path, _workspace_path| SearchPathConfiguration {
+            custom_typeshed: Some(root_path.join("typeshed")),
+            ..SearchPathConfiguration::default()
+        },
+    )?;
+
+    // Unset the custom typeshed directory.
+    assert_eq!(
+        resolve_module(case.db(), ModuleName::new("os").unwrap()),
+        None
+    );
+
+    std::fs::write(
+        case.root_path()
+            .join("typeshed/stdlib/VERSIONS")
+            .as_std_path(),
+        "os: 3.0-",
+    )?;
+
+    let changes = case.stop_watch();
+
+    case.apply_changes(changes);
+
+    assert!(resolve_module(case.db(), ModuleName::new("os").unwrap()).is_some());
+
+    Ok(())
+}
+
 /// Watch a workspace that contains two files where one file is a hardlink to another.
 ///
 /// Setup:
@@ -829,7 +871,7 @@ fn hard_links_in_workspace() -> anyhow::Result<()> {
 
     let changes = case.stop_watch();
 
-    case.db_mut().apply_changes(changes);
+    case.apply_changes(changes);
 
     assert_eq!(source_text(case.db(), foo).as_str(), "print('Version 2')");
 
@@ -900,7 +942,7 @@ fn hard_links_to_target_outside_workspace() -> anyhow::Result<()> {
 
     let changes = case.stop_watch();
 
-    case.db_mut().apply_changes(changes);
+    case.apply_changes(changes);
 
     assert_eq!(source_text(case.db(), bar).as_str(), "print('Version 2')");
 
@@ -939,7 +981,7 @@ mod unix {
 
         let changes = case.stop_watch();
 
-        case.db_mut().apply_changes(changes);
+        case.apply_changes(changes);
 
         assert_eq!(
             foo.permissions(case.db()),
@@ -1024,7 +1066,7 @@ mod unix {
 
         let changes = case.take_watch_changes();
 
-        case.db_mut().apply_changes(changes);
+        case.apply_changes(changes);
 
         assert_eq!(
             source_text(case.db(), baz.file()).as_str(),
@@ -1037,7 +1079,7 @@ mod unix {
 
         let changes = case.stop_watch();
 
-        case.db_mut().apply_changes(changes);
+        case.apply_changes(changes);
 
         assert_eq!(
             source_text(case.db(), baz.file()).as_str(),
@@ -1108,7 +1150,7 @@ mod unix {
 
         let changes = case.stop_watch();
 
-        case.db_mut().apply_changes(changes);
+        case.apply_changes(changes);
 
         // The file watcher is guaranteed to emit one event for the changed file, but it isn't specified
         // if the event is emitted for the "original" or linked path because both paths are watched.
@@ -1177,11 +1219,11 @@ mod unix {
 
                 Ok(())
             },
-            |_root, workspace| SearchPathSettings {
-                extra_paths: vec![],
-                src_root: workspace.to_path_buf(),
-                custom_typeshed: None,
-                site_packages: vec![workspace.join(".venv/lib/python3.12/site-packages")],
+            |_root, workspace| SearchPathConfiguration {
+                site_packages: Some(SitePackages::Known(vec![
+                    workspace.join(".venv/lib/python3.12/site-packages")
+                ])),
+                ..SearchPathConfiguration::default()
             },
         )?;
 
@@ -1216,7 +1258,7 @@ mod unix {
 
         let changes = case.stop_watch();
 
-        case.db_mut().apply_changes(changes);
+        case.apply_changes(changes);
 
         assert_eq!(
             source_text(case.db(), baz_original_file).as_str(),

crates/red_knot_python_semantic/Cargo.toml

@@ -17,6 +17,7 @@ ruff_python_ast = { workspace = true }
 ruff_python_stdlib = { workspace = true }
 ruff_source_file = { workspace = true }
 ruff_text_size = { workspace = true }
+ruff_python_literal = { workspace = true }
 
 anyhow = { workspace = true }
 bitflags = { workspace = true }
@@ -26,6 +27,7 @@ countme = { workspace = true }
 once_cell = { workspace = true }
 ordermap = { workspace = true }
 salsa = { workspace = true }
+thiserror = { workspace = true }
 tracing = { workspace = true }
 rustc-hash = { workspace = true }
 hashbrown = { workspace = true }

crates/red_knot_python_semantic/src/db.rs

@@ -1,15 +1,18 @@
+use ruff_db::files::File;
 use ruff_db::{Db as SourceDb, Upcast};
 
 /// Database giving access to semantic information about a Python program.
 #[salsa::db]
-pub trait Db: SourceDb + Upcast<dyn SourceDb> {}
+pub trait Db: SourceDb + Upcast<dyn SourceDb> {
+    fn is_file_open(&self, file: File) -> bool;
+}
 
 #[cfg(test)]
 pub(crate) mod tests {
     use std::sync::Arc;
 
     use crate::module_resolver::vendored_typeshed_stubs;
-    use ruff_db::files::Files;
+    use ruff_db::files::{File, Files};
     use ruff_db::system::{DbWithTestSystem, System, TestSystem};
     use ruff_db::vendored::VendoredFileSystem;
     use ruff_db::{Db as SourceDb, Upcast};
@@ -91,7 +94,11 @@ pub(crate) mod tests {
     }
 
     #[salsa::db]
-    impl Db for TestDb {}
+    impl Db for TestDb {
+        fn is_file_open(&self, file: File) -> bool {
+            !file.path(self).is_vendored_path()
+        }
+    }
 
     #[salsa::db]
     impl salsa::Database for TestDb {

crates/red_knot_python_semantic/src/lib.rs

@@ -5,7 +5,7 @@ use rustc_hash::FxHasher;
 pub use db::Db;
 pub use module_name::ModuleName;
 pub use module_resolver::{resolve_module, system_module_search_paths, vendored_typeshed_stubs};
-pub use program::{Program, ProgramSettings, SearchPathSettings};
+pub use program::{Program, ProgramSettings, SearchPathSettings, SitePackages};
 pub use python_version::PythonVersion;
 pub use semantic_model::{HasTy, SemanticModel};
 
@@ -19,6 +19,7 @@ mod program;
 mod python_version;
 pub mod semantic_index;
 mod semantic_model;
+pub(crate) mod site_packages;
 pub mod types;
 
 type FxOrderSet<V> = ordermap::set::OrderSet<V, BuildHasherDefault<FxHasher>>;

crates/red_knot_python_semantic/src/module_resolver/mod.rs

@@ -13,7 +13,6 @@ use resolver::SearchPathIterator;
 mod module;
 mod path;
 mod resolver;
-mod state;
 mod typeshed;
 
 #[cfg(test)]

crates/red_knot_python_semantic/src/module_resolver/path.rs

@@ -9,11 +9,11 @@ use ruff_db::files::{system_path_to_file, vendored_path_to_file, File, FileError
 use ruff_db::system::{System, SystemPath, SystemPathBuf};
 use ruff_db::vendored::{VendoredPath, VendoredPathBuf};
 
+use super::typeshed::{typeshed_versions, TypeshedVersionsParseError, TypeshedVersionsQueryResult};
 use crate::db::Db;
 use crate::module_name::ModuleName;
-
-use super::state::ResolverState;
-use super::typeshed::{TypeshedVersionsParseError, TypeshedVersionsQueryResult};
+use crate::module_resolver::resolver::ResolverContext;
+use crate::site_packages::SitePackagesDiscoveryError;
 
 /// A path that points to a Python module.
 ///
@@ -60,7 +60,7 @@ impl ModulePath {
     }
 
     #[must_use]
-    pub(crate) fn is_directory(&self, resolver: &ResolverState) -> bool {
+    pub(super) fn is_directory(&self, resolver: &ResolverContext) -> bool {
         let ModulePath {
             search_path,
             relative_path,
@@ -74,7 +74,7 @@ impl ModulePath {
                     == Err(FileError::IsADirectory)
             }
             SearchPathInner::StandardLibraryCustom(stdlib_root) => {
-                match query_stdlib_version(Some(stdlib_root), relative_path, resolver) {
+                match query_stdlib_version(relative_path, resolver) {
                     TypeshedVersionsQueryResult::DoesNotExist => false,
                     TypeshedVersionsQueryResult::Exists
                     | TypeshedVersionsQueryResult::MaybeExists => {
@@ -84,7 +84,7 @@ impl ModulePath {
                 }
             }
             SearchPathInner::StandardLibraryVendored(stdlib_root) => {
-                match query_stdlib_version(None, relative_path, resolver) {
+                match query_stdlib_version(relative_path, resolver) {
                     TypeshedVersionsQueryResult::DoesNotExist => false,
                     TypeshedVersionsQueryResult::Exists
                     | TypeshedVersionsQueryResult::MaybeExists => resolver
@@ -96,7 +96,7 @@ impl ModulePath {
     }
 
     #[must_use]
-    pub(crate) fn is_regular_package(&self, resolver: &ResolverState) -> bool {
+    pub(super) fn is_regular_package(&self, resolver: &ResolverContext) -> bool {
         let ModulePath {
             search_path,
             relative_path,
@@ -113,7 +113,7 @@ impl ModulePath {
                         .is_ok()
             }
             SearchPathInner::StandardLibraryCustom(search_path) => {
-                match query_stdlib_version(Some(search_path), relative_path, resolver) {
+                match query_stdlib_version(relative_path, resolver) {
                     TypeshedVersionsQueryResult::DoesNotExist => false,
                     TypeshedVersionsQueryResult::Exists
                     | TypeshedVersionsQueryResult::MaybeExists => system_path_to_file(
@@ -124,7 +124,7 @@ impl ModulePath {
                 }
             }
             SearchPathInner::StandardLibraryVendored(search_path) => {
-                match query_stdlib_version(None, relative_path, resolver) {
+                match query_stdlib_version(relative_path, resolver) {
                     TypeshedVersionsQueryResult::DoesNotExist => false,
                     TypeshedVersionsQueryResult::Exists
                     | TypeshedVersionsQueryResult::MaybeExists => resolver
@@ -136,7 +136,7 @@ impl ModulePath {
     }
 
     #[must_use]
-    pub(crate) fn to_file(&self, resolver: &ResolverState) -> Option<File> {
+    pub(super) fn to_file(&self, resolver: &ResolverContext) -> Option<File> {
         let db = resolver.db.upcast();
         let ModulePath {
             search_path,
@@ -150,7 +150,7 @@ impl ModulePath {
                 system_path_to_file(db, search_path.join(relative_path)).ok()
             }
             SearchPathInner::StandardLibraryCustom(stdlib_root) => {
-                match query_stdlib_version(Some(stdlib_root), relative_path, resolver) {
+                match query_stdlib_version(relative_path, resolver) {
                     TypeshedVersionsQueryResult::DoesNotExist => None,
                     TypeshedVersionsQueryResult::Exists
                     | TypeshedVersionsQueryResult::MaybeExists => {
@@ -159,7 +159,7 @@ impl ModulePath {
                 }
             }
             SearchPathInner::StandardLibraryVendored(stdlib_root) => {
-                match query_stdlib_version(None, relative_path, resolver) {
+                match query_stdlib_version(relative_path, resolver) {
                     TypeshedVersionsQueryResult::DoesNotExist => None,
                     TypeshedVersionsQueryResult::Exists
                     | TypeshedVersionsQueryResult::MaybeExists => {
@@ -273,19 +273,15 @@ fn stdlib_path_to_module_name(relative_path: &Utf8Path) -> Option<ModuleName> {
 
 #[must_use]
 fn query_stdlib_version(
-    custom_stdlib_root: Option<&SystemPath>,
     relative_path: &Utf8Path,
-    resolver: &ResolverState,
+    context: &ResolverContext,
 ) -> TypeshedVersionsQueryResult {
     let Some(module_name) = stdlib_path_to_module_name(relative_path) else {
         return TypeshedVersionsQueryResult::DoesNotExist;
     };
-    let ResolverState {
-        db,
-        typeshed_versions,
-        target_version,
-    } = resolver;
-    typeshed_versions.query_module(*db, &module_name, custom_stdlib_root, *target_version)
+    let ResolverContext { db, target_version } = context;
+
+    typeshed_versions(*db).query_module(&module_name, *target_version)
 }
 
 /// Enumeration describing the various ways in which validation of a search path might fail.
@@ -293,7 +289,7 @@ fn query_stdlib_version(
 /// If validation fails for a search path derived from the user settings,
 /// a message must be displayed to the user,
 /// as type checking cannot be done reliably in these circumstances.
-#[derive(Debug, PartialEq, Eq)]
+#[derive(Debug)]
 pub(crate) enum SearchPathValidationError {
     /// The path provided by the user was not a directory
     NotADirectory(SystemPathBuf),
@@ -304,18 +300,20 @@ pub(crate) enum SearchPathValidationError {
     NoStdlibSubdirectory(SystemPathBuf),
 
     /// The typeshed path provided by the user is a directory,
-    /// but no `stdlib/VERSIONS` file exists.
+    /// but `stdlib/VERSIONS` could not be read.
     /// (This is only relevant for stdlib search paths.)
-    NoVersionsFile(SystemPathBuf),
-
-    /// `stdlib/VERSIONS` is a directory.
-    /// (This is only relevant for stdlib search paths.)
-    VersionsIsADirectory(SystemPathBuf),
+    FailedToReadVersionsFile {
+        path: SystemPathBuf,
+        error: std::io::Error,
+    },
 
     /// The path provided by the user is a directory,
     /// and a `stdlib/VERSIONS` file exists, but it fails to parse.
     /// (This is only relevant for stdlib search paths.)
     VersionsParseError(TypeshedVersionsParseError),
+
+    /// Failed to discover the site-packages for the configured virtual environment.
+    SitePackagesDiscovery(SitePackagesDiscoveryError),
 }
 
 impl fmt::Display for SearchPathValidationError {
@@ -325,9 +323,16 @@ impl fmt::Display for SearchPathValidationError {
             Self::NoStdlibSubdirectory(path) => {
                 write!(f, "The directory at {path} has no `stdlib/` subdirectory")
             }
-            Self::NoVersionsFile(path) => write!(f, "Expected a file at {path}/stdlib/VERSIONS"),
-            Self::VersionsIsADirectory(path) => write!(f, "{path}/stdlib/VERSIONS is a directory."),
+            Self::FailedToReadVersionsFile { path, error } => {
+                write!(
+                    f,
+                    "Failed to read the custom typeshed versions file '{path}': {error}"
+                )
+            }
             Self::VersionsParseError(underlying_error) => underlying_error.fmt(f),
+            SearchPathValidationError::SitePackagesDiscovery(error) => {
+                write!(f, "Failed to discover the site-packages directory: {error}")
+            }
         }
     }
 }
@@ -342,6 +347,18 @@ impl std::error::Error for SearchPathValidationError {
     }
 }
 
+impl From<TypeshedVersionsParseError> for SearchPathValidationError {
+    fn from(value: TypeshedVersionsParseError) -> Self {
+        Self::VersionsParseError(value)
+    }
+}
+
+impl From<SitePackagesDiscoveryError> for SearchPathValidationError {
+    fn from(value: SitePackagesDiscoveryError) -> Self {
+        Self::SitePackagesDiscovery(value)
+    }
+}
+
 type SearchPathResult<T> = Result<T, SearchPathValidationError>;
 
 #[derive(Debug, Clone, PartialEq, Eq, Hash)]
@@ -384,11 +401,10 @@ pub(crate) struct SearchPath(Arc<SearchPathInner>);
 
 impl SearchPath {
     fn directory_path(system: &dyn System, root: SystemPathBuf) -> SearchPathResult<SystemPathBuf> {
-        let canonicalized = system.canonicalize_path(&root).unwrap_or(root);
-        if system.is_directory(&canonicalized) {
-            Ok(canonicalized)
+        if system.is_directory(&root) {
+            Ok(root)
         } else {
-            Err(SearchPathValidationError::NotADirectory(canonicalized))
+            Err(SearchPathValidationError::NotADirectory(root))
         }
     }
 
@@ -407,32 +423,22 @@ impl SearchPath {
     }
 
     /// Create a new standard-library search path pointing to a custom directory on disk
-    pub(crate) fn custom_stdlib(db: &dyn Db, typeshed: SystemPathBuf) -> SearchPathResult<Self> {
+    pub(crate) fn custom_stdlib(db: &dyn Db, typeshed: &SystemPath) -> SearchPathResult<Self> {
         let system = db.system();
-        if !system.is_directory(&typeshed) {
+        if !system.is_directory(typeshed) {
             return Err(SearchPathValidationError::NotADirectory(
                 typeshed.to_path_buf(),
             ));
         }
+
         let stdlib =
             Self::directory_path(system, typeshed.join("stdlib")).map_err(|err| match err {
-                SearchPathValidationError::NotADirectory(path) => {
-                    SearchPathValidationError::NoStdlibSubdirectory(path)
+                SearchPathValidationError::NotADirectory(_) => {
+                    SearchPathValidationError::NoStdlibSubdirectory(typeshed.to_path_buf())
                 }
                 err => err,
             })?;
-        let typeshed_versions =
-            system_path_to_file(db.upcast(), stdlib.join("VERSIONS")).map_err(|err| match err {
-                FileError::NotFound => SearchPathValidationError::NoVersionsFile(typeshed),
-                FileError::IsADirectory => {
-                    SearchPathValidationError::VersionsIsADirectory(typeshed)
-                }
-            })?;
-        super::typeshed::parse_typeshed_versions(db, typeshed_versions)
-            .as_ref()
-            .map_err(|validation_error| {
-                SearchPathValidationError::VersionsParseError(validation_error.clone())
-            })?;
+
         Ok(Self(Arc::new(SearchPathInner::StandardLibraryCustom(
             stdlib,
         ))))
@@ -623,11 +629,11 @@ mod tests {
     use ruff_db::Db;
 
     use crate::db::tests::TestDb;
-
-    use super::*;
     use crate::module_resolver::testing::{FileSpec, MockedTypeshed, TestCase, TestCaseBuilder};
     use crate::python_version::PythonVersion;
 
+    use super::*;
+
     impl ModulePath {
         #[must_use]
         fn join(&self, component: &str) -> ModulePath {
@@ -638,15 +644,6 @@ mod tests {
     }
 
     impl SearchPath {
-        #[must_use]
-        pub(crate) fn is_stdlib_search_path(&self) -> bool {
-            matches!(
-                &*self.0,
-                SearchPathInner::StandardLibraryCustom(_)
-                    | SearchPathInner::StandardLibraryVendored(_)
-            )
-        }
-
         fn join(&self, component: &str) -> ModulePath {
             self.to_module_path().join(component)
         }
@@ -661,7 +658,7 @@ mod tests {
             .build();
 
         assert_eq!(
-            SearchPath::custom_stdlib(&db, stdlib.parent().unwrap().to_path_buf())
+            SearchPath::custom_stdlib(&db, stdlib.parent().unwrap())
                 .unwrap()
                 .to_module_path()
                 .with_py_extension(),
@@ -669,7 +666,7 @@ mod tests {
         );
 
         assert_eq!(
-            &SearchPath::custom_stdlib(&db, stdlib.parent().unwrap().to_path_buf())
+            &SearchPath::custom_stdlib(&db, stdlib.parent().unwrap())
                 .unwrap()
                 .join("foo")
                 .with_pyi_extension(),
@@ -780,7 +777,7 @@ mod tests {
         let TestCase { db, stdlib, .. } = TestCaseBuilder::new()
             .with_custom_typeshed(MockedTypeshed::default())
             .build();
-        SearchPath::custom_stdlib(&db, stdlib.parent().unwrap().to_path_buf())
+        SearchPath::custom_stdlib(&db, stdlib.parent().unwrap())
             .unwrap()
             .to_module_path()
             .push("bar.py");
@@ -792,7 +789,7 @@ mod tests {
         let TestCase { db, stdlib, .. } = TestCaseBuilder::new()
             .with_custom_typeshed(MockedTypeshed::default())
             .build();
-        SearchPath::custom_stdlib(&db, stdlib.parent().unwrap().to_path_buf())
+        SearchPath::custom_stdlib(&db, stdlib.parent().unwrap())
             .unwrap()
             .to_module_path()
             .push("bar.rs");
@@ -824,7 +821,7 @@ mod tests {
             .with_custom_typeshed(MockedTypeshed::default())
             .build();
 
-        let root = SearchPath::custom_stdlib(&db, stdlib.parent().unwrap().to_path_buf()).unwrap();
+        let root = SearchPath::custom_stdlib(&db, stdlib.parent().unwrap()).unwrap();
 
         // Must have a `.pyi` extension or no extension:
         let bad_absolute_path = SystemPath::new("foo/stdlib/x.py");
@@ -872,8 +869,7 @@ mod tests {
             .with_custom_typeshed(typeshed)
             .with_target_version(target_version)
             .build();
-        let stdlib =
-            SearchPath::custom_stdlib(&db, stdlib.parent().unwrap().to_path_buf()).unwrap();
+        let stdlib = SearchPath::custom_stdlib(&db, stdlib.parent().unwrap()).unwrap();
         (db, stdlib)
     }
 
@@ -898,7 +894,7 @@ mod tests {
         };
 
         let (db, stdlib_path) = py38_typeshed_test_case(TYPESHED);
-        let resolver = ResolverState::new(&db, PythonVersion::PY38);
+        let resolver = ResolverContext::new(&db, PythonVersion::PY38);
 
         let asyncio_regular_package = stdlib_path.join("asyncio");
         assert!(asyncio_regular_package.is_directory(&resolver));
@@ -926,7 +922,7 @@ mod tests {
         };
 
         let (db, stdlib_path) = py38_typeshed_test_case(TYPESHED);
-        let resolver = ResolverState::new(&db, PythonVersion::PY38);
+        let resolver = ResolverContext::new(&db, PythonVersion::PY38);
 
         let xml_namespace_package = stdlib_path.join("xml");
         assert!(xml_namespace_package.is_directory(&resolver));
@@ -948,7 +944,7 @@ mod tests {
         };
 
         let (db, stdlib_path) = py38_typeshed_test_case(TYPESHED);
-        let resolver = ResolverState::new(&db, PythonVersion::PY38);
+        let resolver = ResolverContext::new(&db, PythonVersion::PY38);
 
         let functools_module = stdlib_path.join("functools.pyi");
         assert!(functools_module.to_file(&resolver).is_some());
@@ -964,7 +960,7 @@ mod tests {
         };
 
         let (db, stdlib_path) = py38_typeshed_test_case(TYPESHED);
-        let resolver = ResolverState::new(&db, PythonVersion::PY38);
+        let resolver = ResolverContext::new(&db, PythonVersion::PY38);
 
         let collections_regular_package = stdlib_path.join("collections");
         assert_eq!(collections_regular_package.to_file(&resolver), None);
@@ -980,7 +976,7 @@ mod tests {
         };
 
         let (db, stdlib_path) = py38_typeshed_test_case(TYPESHED);
-        let resolver = ResolverState::new(&db, PythonVersion::PY38);
+        let resolver = ResolverContext::new(&db, PythonVersion::PY38);
 
         let importlib_namespace_package = stdlib_path.join("importlib");
         assert_eq!(importlib_namespace_package.to_file(&resolver), None);
@@ -1001,7 +997,7 @@ mod tests {
         };
 
         let (db, stdlib_path) = py38_typeshed_test_case(TYPESHED);
-        let resolver = ResolverState::new(&db, PythonVersion::PY38);
+        let resolver = ResolverContext::new(&db, PythonVersion::PY38);
 
         let non_existent = stdlib_path.join("doesnt_even_exist");
         assert_eq!(non_existent.to_file(&resolver), None);
@@ -1029,7 +1025,7 @@ mod tests {
         };
 
         let (db, stdlib_path) = py39_typeshed_test_case(TYPESHED);
-        let resolver = ResolverState::new(&db, PythonVersion::PY39);
+        let resolver = ResolverContext::new(&db, PythonVersion::PY39);
 
         // Since we've set the target version to Py39,
         // `collections` should now exist as a directory, according to VERSIONS...
@@ -1058,7 +1054,7 @@ mod tests {
         };
 
         let (db, stdlib_path) = py39_typeshed_test_case(TYPESHED);
-        let resolver = ResolverState::new(&db, PythonVersion::PY39);
+        let resolver = ResolverContext::new(&db, PythonVersion::PY39);
 
         // The `importlib` directory now also exists
         let importlib_namespace_package = stdlib_path.join("importlib");
@@ -1082,7 +1078,7 @@ mod tests {
         };
 
         let (db, stdlib_path) = py39_typeshed_test_case(TYPESHED);
-        let resolver = ResolverState::new(&db, PythonVersion::PY39);
+        let resolver = ResolverContext::new(&db, PythonVersion::PY39);
 
         // The `xml` package no longer exists on py39:
         let xml_namespace_package = stdlib_path.join("xml");

crates/red_knot_python_semantic/src/module_resolver/resolver.rs

@@ -1,19 +1,19 @@
+use rustc_hash::{FxBuildHasher, FxHashSet};
 use std::borrow::Cow;
 use std::iter::FusedIterator;
-
-use rustc_hash::{FxBuildHasher, FxHashSet};
+use std::ops::Deref;
 
 use ruff_db::files::{File, FilePath, FileRootKind};
-use ruff_db::system::{DirectoryEntry, SystemPath, SystemPathBuf};
-use ruff_db::vendored::VendoredPath;
-
-use crate::db::Db;
-use crate::module_name::ModuleName;
-use crate::{Program, SearchPathSettings};
+use ruff_db::system::{DirectoryEntry, System, SystemPath, SystemPathBuf};
+use ruff_db::vendored::{VendoredFileSystem, VendoredPath};
 
 use super::module::{Module, ModuleKind};
 use super::path::{ModulePath, SearchPath, SearchPathValidationError};
-use super::state::ResolverState;
+use crate::db::Db;
+use crate::module_name::ModuleName;
+use crate::module_resolver::typeshed::{vendored_typeshed_versions, TypeshedVersions};
+use crate::site_packages::VirtualEnvironment;
+use crate::{Program, PythonVersion, SearchPathSettings, SitePackages};
 
 /// Resolves a module name to a module.
 pub fn resolve_module(db: &dyn Db, module_name: ModuleName) -> Option<Module> {
@@ -41,7 +41,7 @@ pub(crate) fn resolve_module_query<'db>(
 
     let module = Module::new(name.clone(), kind, search_path, module_file);
 
-    tracing::debug!(
+    tracing::trace!(
         "Resolved module '{name}' to '{path}'.",
         path = module_file.path(db)
     );
@@ -122,7 +122,7 @@ pub(crate) fn search_paths(db: &dyn Db) -> SearchPathIterator {
     Program::get(db).search_paths(db).iter(db)
 }
 
-#[derive(Debug, PartialEq, Eq, Default)]
+#[derive(Debug, PartialEq, Eq)]
 pub(crate) struct SearchPaths {
     /// Search paths that have been statically determined purely from reading Ruff's configuration settings.
     /// These shouldn't ever change unless the config settings themselves change.
@@ -135,6 +135,8 @@ pub(crate) struct SearchPaths {
     /// in terms of module-resolution priority until we've discovered the editable installs
     /// for the first `site-packages` path
     site_packages: Vec<SearchPath>,
+
+    typeshed_versions: ResolvedTypeshedVersions,
 }
 
 impl SearchPaths {
@@ -146,8 +148,14 @@ impl SearchPaths {
     /// [module resolution order]: https://typing.readthedocs.io/en/latest/spec/distributing.html#import-resolution-ordering
     pub(crate) fn from_settings(
         db: &dyn Db,
-        settings: SearchPathSettings,
+        settings: &SearchPathSettings,
     ) -> Result<Self, SearchPathValidationError> {
+        fn canonicalize(path: &SystemPath, system: &dyn System) -> SystemPathBuf {
+            system
+                .canonicalize_path(path)
+                .unwrap_or_else(|_| path.to_path_buf())
+        }
+
         let SearchPathSettings {
             extra_paths,
             src_root,
@@ -161,45 +169,65 @@ impl SearchPaths {
         let mut static_paths = vec![];
 
         for path in extra_paths {
-            tracing::debug!("Adding static extra search-path '{path}'");
+            let path = canonicalize(path, system);
+            files.try_add_root(db.upcast(), &path, FileRootKind::LibrarySearchPath);
+            tracing::debug!("Adding extra search-path '{path}'");
 
-            let search_path = SearchPath::extra(system, path)?;
-            files.try_add_root(
-                db.upcast(),
-                search_path.as_system_path().unwrap(),
-                FileRootKind::LibrarySearchPath,
-            );
-            static_paths.push(search_path);
+            static_paths.push(SearchPath::extra(system, path)?);
         }
 
-        tracing::debug!("Adding static search path '{src_root}'");
-        static_paths.push(SearchPath::first_party(system, src_root)?);
+        tracing::debug!("Adding first-party search path '{src_root}'");
+        static_paths.push(SearchPath::first_party(system, src_root.to_path_buf())?);
 
-        static_paths.push(if let Some(custom_typeshed) = custom_typeshed {
-            tracing::debug!("Adding static custom-sdtlib search-path '{custom_typeshed}'");
+        let (typeshed_versions, stdlib_path) = if let Some(custom_typeshed) = custom_typeshed {
+            let custom_typeshed = canonicalize(custom_typeshed, system);
+            tracing::debug!("Adding custom-stdlib search path '{custom_typeshed}'");
 
-            let search_path = SearchPath::custom_stdlib(db, custom_typeshed)?;
             files.try_add_root(
                 db.upcast(),
-                search_path.as_system_path().unwrap(),
+                &custom_typeshed,
                 FileRootKind::LibrarySearchPath,
             );
-            search_path
+
+            let versions_path = custom_typeshed.join("stdlib/VERSIONS");
+
+            let versions_content = system.read_to_string(&versions_path).map_err(|error| {
+                SearchPathValidationError::FailedToReadVersionsFile {
+                    path: versions_path,
+                    error,
+                }
+            })?;
+
+            let parsed: TypeshedVersions = versions_content.parse()?;
+
+            let search_path = SearchPath::custom_stdlib(db, &custom_typeshed)?;
+
+            (ResolvedTypeshedVersions::Custom(parsed), search_path)
         } else {
-            SearchPath::vendored_stdlib()
-        });
+            tracing::debug!("Using vendored stdlib");
+            (
+                ResolvedTypeshedVersions::Vendored(vendored_typeshed_versions()),
+                SearchPath::vendored_stdlib(),
+            )
+        };
+
+        static_paths.push(stdlib_path);
+
+        let site_packages_paths = match site_packages_paths {
+            SitePackages::Derived { venv_path } => VirtualEnvironment::new(venv_path, system)
+                .and_then(|venv| venv.site_packages_directories(system))?,
+            SitePackages::Known(paths) => paths
+                .iter()
+                .map(|path| canonicalize(path, system))
+                .collect(),
+        };
 
         let mut site_packages: Vec<_> = Vec::with_capacity(site_packages_paths.len());
 
         for path in site_packages_paths {
-            tracing::debug!("Adding site-package path '{path}'");
-            let search_path = SearchPath::site_packages(system, path)?;
-            files.try_add_root(
-                db.upcast(),
-                search_path.as_system_path().unwrap(),
-                FileRootKind::LibrarySearchPath,
-            );
-            site_packages.push(search_path);
+            tracing::debug!("Adding site-packages search path '{path}'");
+            files.try_add_root(db.upcast(), &path, FileRootKind::LibrarySearchPath);
+            site_packages.push(SearchPath::site_packages(system, path)?);
         }
 
         // TODO vendor typeshed's third-party stubs as well as the stdlib and fallback to them as a final step
@@ -224,16 +252,48 @@ impl SearchPaths {
         Ok(SearchPaths {
             static_paths,
             site_packages,
+            typeshed_versions,
         })
     }
 
-    pub(crate) fn iter<'a>(&'a self, db: &'a dyn Db) -> SearchPathIterator<'a> {
+    pub(super) fn iter<'a>(&'a self, db: &'a dyn Db) -> SearchPathIterator<'a> {
         SearchPathIterator {
             db,
             static_paths: self.static_paths.iter(),
             dynamic_paths: None,
         }
     }
+
+    pub(crate) fn custom_stdlib(&self) -> Option<&SystemPath> {
+        self.static_paths.iter().find_map(|search_path| {
+            if search_path.is_standard_library() {
+                search_path.as_system_path()
+            } else {
+                None
+            }
+        })
+    }
+
+    pub(super) fn typeshed_versions(&self) -> &TypeshedVersions {
+        &self.typeshed_versions
+    }
+}
+
+#[derive(Debug, PartialEq, Eq)]
+enum ResolvedTypeshedVersions {
+    Vendored(&'static TypeshedVersions),
+    Custom(TypeshedVersions),
+}
+
+impl Deref for ResolvedTypeshedVersions {
+    type Target = TypeshedVersions;
+
+    fn deref(&self) -> &Self::Target {
+        match self {
+            ResolvedTypeshedVersions::Vendored(versions) => versions,
+            ResolvedTypeshedVersions::Custom(versions) => versions,
+        }
+    }
 }
 
 /// Collect all dynamic search paths. For each `site-packages` path:
@@ -251,6 +311,7 @@ pub(crate) fn dynamic_resolution_paths(db: &dyn Db) -> Vec<SearchPath> {
     let SearchPaths {
         static_paths,
         site_packages,
+        typeshed_versions: _,
     } = Program::get(db).search_paths(db);
 
     let mut dynamic_paths = Vec::new();
@@ -315,12 +376,16 @@ pub(crate) fn dynamic_resolution_paths(db: &dyn Db) -> Vec<SearchPath> {
         let installations = all_pth_files.iter().flat_map(PthFile::items);
 
         for installation in installations {
+            let installation = system
+                .canonicalize_path(&installation)
+                .unwrap_or(installation);
+
             if existing_paths.insert(Cow::Owned(installation.clone())) {
-                match SearchPath::editable(system, installation) {
+                match SearchPath::editable(system, installation.clone()) {
                     Ok(search_path) => {
                         tracing::debug!(
                             "Adding editable installation to module resolution path {path}",
-                            path = search_path.as_system_path().unwrap()
+                            path = installation
                         );
                         dynamic_paths.push(search_path);
                     }
@@ -482,7 +547,7 @@ struct ModuleNameIngredient<'db> {
 fn resolve_name(db: &dyn Db, name: &ModuleName) -> Option<(SearchPath, File, ModuleKind)> {
     let program = Program::get(db);
     let target_version = program.target_version(db);
-    let resolver_state = ResolverState::new(db, target_version);
+    let resolver_state = ResolverContext::new(db, target_version);
     let is_builtin_module =
         ruff_python_stdlib::sys::is_builtin_module(target_version.minor, name.as_str());
 
@@ -545,7 +610,7 @@ fn resolve_name(db: &dyn Db, name: &ModuleName) -> Option<(SearchPath, File, Mod
 fn resolve_package<'a, 'db, I>(
     module_search_path: &SearchPath,
     components: I,
-    resolver_state: &ResolverState<'db>,
+    resolver_state: &ResolverContext<'db>,
 ) -> Result<ResolvedPackage, PackageKind>
 where
     I: Iterator<Item = &'a str>,
@@ -627,6 +692,21 @@ impl PackageKind {
     }
 }
 
+pub(super) struct ResolverContext<'db> {
+    pub(super) db: &'db dyn Db,
+    pub(super) target_version: PythonVersion,
+}
+
+impl<'db> ResolverContext<'db> {
+    pub(super) fn new(db: &'db dyn Db, target_version: PythonVersion) -> Self {
+        Self { db, target_version }
+    }
+
+    pub(super) fn vendored(&self) -> &VendoredFileSystem {
+        self.db.vendored()
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use ruff_db::files::{system_path_to_file, File, FilePath};
@@ -781,7 +861,7 @@ mod tests {
                 "Search path for {module_name} was unexpectedly {search_path:?}"
             );
             assert!(
-                search_path.is_stdlib_search_path(),
+                search_path.is_standard_library(),
                 "Expected a stdlib search path, but got {search_path:?}"
             );
         }
@@ -877,7 +957,7 @@ mod tests {
                 "Search path for {module_name} was unexpectedly {search_path:?}"
             );
             assert!(
-                search_path.is_stdlib_search_path(),
+                search_path.is_standard_library(),
                 "Expected a stdlib search path, but got {search_path:?}"
             );
         }
@@ -1194,13 +1274,13 @@ mod tests {
 
         Program::from_settings(
             &db,
-            ProgramSettings {
+            &ProgramSettings {
                 target_version: PythonVersion::PY38,
                 search_paths: SearchPathSettings {
                     extra_paths: vec![],
                     src_root: src.clone(),
                     custom_typeshed: Some(custom_typeshed.clone()),
-                    site_packages: vec![site_packages],
+                    site_packages: SitePackages::Known(vec![site_packages]),
                 },
             },
         )
@@ -1699,13 +1779,16 @@ not_a_directory
 
         Program::from_settings(
             &db,
-            ProgramSettings {
+            &ProgramSettings {
                 target_version: PythonVersion::default(),
                 search_paths: SearchPathSettings {
                     extra_paths: vec![],
                     src_root: SystemPathBuf::from("/src"),
                     custom_typeshed: None,
-                    site_packages: vec![venv_site_packages, system_site_packages],
+                    site_packages: SitePackages::Known(vec![
+                        venv_site_packages,
+                        system_site_packages,
+                    ]),
                 },
             },
         )

crates/red_knot_python_semantic/src/module_resolver/state.rs

@@ -1,25 +0,0 @@
-use ruff_db::vendored::VendoredFileSystem;
-
-use super::typeshed::LazyTypeshedVersions;
-use crate::db::Db;
-use crate::python_version::PythonVersion;
-
-pub(crate) struct ResolverState<'db> {
-    pub(crate) db: &'db dyn Db,
-    pub(crate) typeshed_versions: LazyTypeshedVersions<'db>,
-    pub(crate) target_version: PythonVersion,
-}
-
-impl<'db> ResolverState<'db> {
-    pub(crate) fn new(db: &'db dyn Db, target_version: PythonVersion) -> Self {
-        Self {
-            db,
-            typeshed_versions: LazyTypeshedVersions::new(),
-            target_version,
-        }
-    }
-
-    pub(crate) fn vendored(&self) -> &VendoredFileSystem {
-        self.db.vendored()
-    }
-}

crates/red_knot_python_semantic/src/module_resolver/testing.rs

@@ -4,7 +4,7 @@ use ruff_db::vendored::VendoredPathBuf;
 use crate::db::tests::TestDb;
 use crate::program::{Program, SearchPathSettings};
 use crate::python_version::PythonVersion;
-use crate::ProgramSettings;
+use crate::{ProgramSettings, SitePackages};
 
 /// A test case for the module resolver.
 ///
@@ -179,6 +179,7 @@ impl TestCaseBuilder<UnspecifiedTypeshed> {
             first_party_files,
             site_packages_files,
         } = self;
+
         TestCaseBuilder {
             typeshed_option: typeshed,
             target_version,
@@ -195,6 +196,7 @@ impl TestCaseBuilder<UnspecifiedTypeshed> {
             site_packages,
             target_version,
         } = self.with_custom_typeshed(MockedTypeshed::default()).build();
+
         TestCase {
             db,
             src,
@@ -223,13 +225,13 @@ impl TestCaseBuilder<MockedTypeshed> {
 
         Program::from_settings(
             &db,
-            ProgramSettings {
+            &ProgramSettings {
                 target_version,
                 search_paths: SearchPathSettings {
                     extra_paths: vec![],
                     src_root: src.clone(),
                     custom_typeshed: Some(typeshed.clone()),
-                    site_packages: vec![site_packages.clone()],
+                    site_packages: SitePackages::Known(vec![site_packages.clone()]),
                 },
             },
         )
@@ -279,13 +281,11 @@ impl TestCaseBuilder<VendoredTypeshed> {
 
         Program::from_settings(
             &db,
-            ProgramSettings {
+            &ProgramSettings {
                 target_version,
                 search_paths: SearchPathSettings {
-                    extra_paths: vec![],
-                    src_root: src.clone(),
-                    custom_typeshed: None,
-                    site_packages: vec![site_packages.clone()],
+                    site_packages: SitePackages::Known(vec![site_packages.clone()]),
+                    ..SearchPathSettings::new(src.clone())
                 },
             },
         )

crates/red_knot_python_semantic/src/module_resolver/typeshed/mod.rs

@@ -1,6 +1,6 @@
 pub use self::vendored::vendored_typeshed_stubs;
 pub(super) use self::versions::{
-    parse_typeshed_versions, LazyTypeshedVersions, TypeshedVersionsParseError,
+    typeshed_versions, vendored_typeshed_versions, TypeshedVersions, TypeshedVersionsParseError,
     TypeshedVersionsQueryResult,
 };
 

crates/red_knot_python_semantic/src/module_resolver/typeshed/versions.rs

@@ -1,4 +1,3 @@
-use std::cell::OnceCell;
 use std::collections::BTreeMap;
 use std::fmt;
 use std::num::{NonZeroU16, NonZeroUsize};
@@ -6,78 +5,12 @@ use std::ops::{RangeFrom, RangeInclusive};
 use std::str::FromStr;
 
 use once_cell::sync::Lazy;
-use ruff_db::system::SystemPath;
 use rustc_hash::FxHashMap;
 
-use ruff_db::files::{system_path_to_file, File};
-
 use super::vendored::vendored_typeshed_stubs;
 use crate::db::Db;
 use crate::module_name::ModuleName;
-use crate::python_version::PythonVersion;
-
-#[derive(Debug)]
-pub(crate) struct LazyTypeshedVersions<'db>(OnceCell<&'db TypeshedVersions>);
-
-impl<'db> LazyTypeshedVersions<'db> {
-    #[must_use]
-    pub(crate) fn new() -> Self {
-        Self(OnceCell::new())
-    }
-
-    /// Query whether a module exists at runtime in the stdlib on a certain Python version.
-    ///
-    /// Simply probing whether a file exists in typeshed is insufficient for this question,
-    /// as a module in the stdlib may have been added in Python 3.10, but the typeshed stub
-    /// will still be available (either in a custom typeshed dir or in our vendored copy)
-    /// even if the user specified Python 3.8 as the target version.
-    ///
-    /// For top-level modules and packages, the VERSIONS file can always provide an unambiguous answer
-    /// as to whether the module exists on the specified target version. However, VERSIONS does not
-    /// provide comprehensive information on all submodules, meaning that this method sometimes
-    /// returns [`TypeshedVersionsQueryResult::MaybeExists`].
-    /// See [`TypeshedVersionsQueryResult`] for more details.
-    #[must_use]
-    pub(crate) fn query_module(
-        &self,
-        db: &'db dyn Db,
-        module: &ModuleName,
-        stdlib_root: Option<&SystemPath>,
-        target_version: PythonVersion,
-    ) -> TypeshedVersionsQueryResult {
-        let versions = self.0.get_or_init(|| {
-            let versions_path = if let Some(system_path) = stdlib_root {
-                system_path.join("VERSIONS")
-            } else {
-                return &VENDORED_VERSIONS;
-            };
-            let Ok(versions_file) = system_path_to_file(db.upcast(), &versions_path) else {
-                todo!(
-                    "Still need to figure out how to handle VERSIONS files being deleted \
-                    from custom typeshed directories! Expected a file to exist at {versions_path}"
-                )
-            };
-            // TODO(Alex/Micha): If VERSIONS is invalid,
-            // this should invalidate not just the specific module resolution we're currently attempting,
-            // but all type inference that depends on any standard-library types.
-            // Unwrapping here is not correct...
-            parse_typeshed_versions(db, versions_file).as_ref().unwrap()
-        });
-        versions.query_module(module, target_version)
-    }
-}
-
-#[salsa::tracked(return_ref)]
-pub(crate) fn parse_typeshed_versions(
-    db: &dyn Db,
-    versions_file: File,
-) -> Result<TypeshedVersions, TypeshedVersionsParseError> {
-    // TODO: Handle IO errors
-    let file_content = versions_file
-        .read_to_string(db.upcast())
-        .unwrap_or_default();
-    file_content.parse()
-}
+use crate::{Program, PythonVersion};
 
 static VENDORED_VERSIONS: Lazy<TypeshedVersions> = Lazy::new(|| {
     TypeshedVersions::from_str(
@@ -88,6 +21,14 @@ static VENDORED_VERSIONS: Lazy<TypeshedVersions> = Lazy::new(|| {
     .unwrap()
 });
 
+pub(crate) fn vendored_typeshed_versions() -> &'static TypeshedVersions {
+    &VENDORED_VERSIONS
+}
+
+pub(crate) fn typeshed_versions(db: &dyn Db) -> &TypeshedVersions {
+    Program::get(db).search_paths(db).typeshed_versions()
+}
+
 #[derive(Debug, PartialEq, Eq, Clone)]
 pub(crate) struct TypeshedVersionsParseError {
     line_number: Option<NonZeroU16>,
@@ -174,7 +115,7 @@ impl TypeshedVersions {
     }
 
     #[must_use]
-    fn query_module(
+    pub(in crate::module_resolver) fn query_module(
         &self,
         module: &ModuleName,
         target_version: PythonVersion,
@@ -204,7 +145,7 @@ impl TypeshedVersions {
     }
 }
 
-/// Possible answers [`LazyTypeshedVersions::query_module()`] could give to the question:
+/// Possible answers [`TypeshedVersions::query_module()`] could give to the question:
 /// "Does this module exist in the stdlib at runtime on a certain target version?"
 #[derive(Debug, Copy, PartialEq, Eq, Clone, Hash)]
 pub(crate) enum TypeshedVersionsQueryResult {

crates/red_knot_python_semantic/src/program.rs

@@ -3,7 +3,7 @@ use anyhow::Context;
 use salsa::Durability;
 use salsa::Setter;
 
-use ruff_db::system::SystemPathBuf;
+use ruff_db::system::{SystemPath, SystemPathBuf};
 
 use crate::module_resolver::SearchPaths;
 use crate::Db;
@@ -12,33 +12,31 @@ use crate::Db;
 pub struct Program {
     pub target_version: PythonVersion,
 
-    #[default]
     #[return_ref]
     pub(crate) search_paths: SearchPaths,
 }
 
 impl Program {
-    pub fn from_settings(db: &dyn Db, settings: ProgramSettings) -> anyhow::Result<Self> {
+    pub fn from_settings(db: &dyn Db, settings: &ProgramSettings) -> anyhow::Result<Self> {
         let ProgramSettings {
             target_version,
             search_paths,
         } = settings;
 
-        tracing::info!("Target version: {target_version}");
+        tracing::info!("Target version: Python {target_version}");
 
         let search_paths = SearchPaths::from_settings(db, search_paths)
             .with_context(|| "Invalid search path settings")?;
 
-        Ok(Program::builder(settings.target_version)
+        Ok(Program::builder(settings.target_version, search_paths)
             .durability(Durability::HIGH)
-            .search_paths(search_paths)
             .new(db))
     }
 
     pub fn update_search_paths(
-        &self,
+        self,
         db: &mut dyn Db,
-        search_path_settings: SearchPathSettings,
+        search_path_settings: &SearchPathSettings,
     ) -> anyhow::Result<()> {
         let search_paths = SearchPaths::from_settings(db, search_path_settings)?;
 
@@ -49,16 +47,20 @@ impl Program {
 
         Ok(())
     }
+
+    pub fn custom_stdlib_search_path(self, db: &dyn Db) -> Option<&SystemPath> {
+        self.search_paths(db).custom_stdlib()
+    }
 }
 
-#[derive(Debug, Eq, PartialEq)]
+#[derive(Clone, Debug, Eq, PartialEq)]
 pub struct ProgramSettings {
     pub target_version: PythonVersion,
     pub search_paths: SearchPathSettings,
 }
 
 /// Configures the search paths for module resolution.
-#[derive(Eq, PartialEq, Debug, Clone, Default)]
+#[derive(Eq, PartialEq, Debug, Clone)]
 pub struct SearchPathSettings {
     /// List of user-provided paths that should take first priority in the module resolution.
     /// Examples in other type checkers are mypy's MYPYPATH environment variable,
@@ -74,5 +76,25 @@ pub struct SearchPathSettings {
     pub custom_typeshed: Option<SystemPathBuf>,
 
     /// The path to the user's `site-packages` directory, where third-party packages from ``PyPI`` are installed.
-    pub site_packages: Vec<SystemPathBuf>,
+    pub site_packages: SitePackages,
+}
+
+impl SearchPathSettings {
+    pub fn new(src_root: SystemPathBuf) -> Self {
+        Self {
+            src_root,
+            extra_paths: vec![],
+            custom_typeshed: None,
+            site_packages: SitePackages::Known(vec![]),
+        }
+    }
+}
+
+#[derive(Debug, Clone, Eq, PartialEq)]
+pub enum SitePackages {
+    Derived {
+        venv_path: SystemPathBuf,
+    },
+    /// Resolved site packages paths
+    Known(Vec<SystemPathBuf>),
 }

crates/red_knot_python_semantic/src/semantic_index.rs

@@ -154,6 +154,10 @@ impl<'db> SemanticIndex<'db> {
         &self.scopes[id]
     }
 
+    pub(crate) fn scope_ids(&self) -> impl Iterator<Item = ScopeId> {
+        self.scope_ids_by_scope.iter().copied()
+    }
+
     /// Returns the id of the parent scope.
     pub(crate) fn parent_scope_id(&self, scope_id: FileScopeId) -> Option<FileScopeId> {
         let scope = self.scope(scope_id);
@@ -463,6 +467,25 @@ mod tests {
         ));
     }
 
+    #[test]
+    fn augmented_assignment() {
+        let TestCase { db, file } = test_case("x += 1");
+        let scope = global_scope(&db, file);
+        let global_table = symbol_table(&db, scope);
+
+        assert_eq!(names(&global_table), vec!["x"]);
+
+        let use_def = use_def_map(&db, scope);
+        let definition = use_def
+            .first_public_definition(global_table.symbol_id_by_name("x").unwrap())
+            .unwrap();
+
+        assert!(matches!(
+            definition.node(&db),
+            DefinitionKind::AugmentedAssignment(_)
+        ));
+    }
+
     #[test]
     fn class_scope() {
         let TestCase { db, file } = test_case(
@@ -767,6 +790,56 @@ def f(a: str, /, b: str, c: int = 1, *args, d: int = 2, **kwargs):
         assert_eq!(names(&inner_comprehension_symbol_table), vec!["x"]);
     }
 
+    #[test]
+    fn with_item_definition() {
+        let TestCase { db, file } = test_case(
+            "
+with item1 as x, item2 as y:
+    pass
+",
+        );
+
+        let index = semantic_index(&db, file);
+        let global_table = index.symbol_table(FileScopeId::global());
+
+        assert_eq!(names(&global_table), vec!["item1", "x", "item2", "y"]);
+
+        let use_def = index.use_def_map(FileScopeId::global());
+        for name in ["x", "y"] {
+            let Some(definition) = use_def.first_public_definition(
+                global_table.symbol_id_by_name(name).expect("symbol exists"),
+            ) else {
+                panic!("Expected with item definition for {name}");
+            };
+            assert!(matches!(definition.node(&db), DefinitionKind::WithItem(_)));
+        }
+    }
+
+    #[test]
+    fn with_item_unpacked_definition() {
+        let TestCase { db, file } = test_case(
+            "
+with context() as (x, y):
+    pass
+",
+        );
+
+        let index = semantic_index(&db, file);
+        let global_table = index.symbol_table(FileScopeId::global());
+
+        assert_eq!(names(&global_table), vec!["context", "x", "y"]);
+
+        let use_def = index.use_def_map(FileScopeId::global());
+        for name in ["x", "y"] {
+            let Some(definition) = use_def.first_public_definition(
+                global_table.symbol_id_by_name(name).expect("symbol exists"),
+            ) else {
+                panic!("Expected with item definition for {name}");
+            };
+            assert!(matches!(definition.node(&db), DefinitionKind::WithItem(_)));
+        }
+    }
+
     #[test]
     fn dupes() {
         let TestCase { db, file } = test_case(
@@ -998,4 +1071,28 @@ def x():
             vec!["bar", "foo", "Test", "<module>"]
         );
     }
+
+    #[test]
+    fn match_stmt_symbols() {
+        let TestCase { db, file } = test_case(
+            "
+match subject:
+    case a: ...
+    case [b, c, *d]: ...
+    case e as f: ...
+    case {'x': g, **h}: ...
+    case Foo(i, z=j): ...
+    case k | l: ...
+    case _: ...
+",
+        );
+
+        let global_table = symbol_table(&db, global_scope(&db, file));
+
+        assert!(global_table.symbol_by_name("Foo").unwrap().is_used());
+        assert_eq!(
+            names(&global_table),
+            vec!["subject", "a", "b", "c", "d", "f", "e", "h", "g", "Foo", "i", "j", "k", "l"]
+        );
+    }
 }

crates/red_knot_python_semantic/src/semantic_index/builder.rs

@@ -7,7 +7,7 @@ use ruff_db::parsed::ParsedModule;
 use ruff_index::IndexVec;
 use ruff_python_ast as ast;
 use ruff_python_ast::name::Name;
-use ruff_python_ast::visitor::{walk_expr, walk_stmt, Visitor};
+use ruff_python_ast::visitor::{walk_expr, walk_pattern, walk_stmt, Visitor};
 use ruff_python_ast::AnyParameterRef;
 
 use crate::ast_node_ref::AstNodeRef;
@@ -26,6 +26,8 @@ use crate::semantic_index::use_def::{FlowSnapshot, UseDefMapBuilder};
 use crate::semantic_index::SemanticIndex;
 use crate::Db;
 
+use super::definition::WithItemDefinitionNodeRef;
+
 pub(super) struct SemanticIndexBuilder<'db> {
     // Builder state
     db: &'db dyn Db,
@@ -495,6 +497,20 @@ where
                 self.visit_expr(&node.target);
                 self.current_assignment = None;
             }
+            ast::Stmt::AugAssign(
+                aug_assign @ ast::StmtAugAssign {
+                    range: _,
+                    target,
+                    op: _,
+                    value,
+                },
+            ) => {
+                debug_assert!(self.current_assignment.is_none());
+                self.visit_expr(value);
+                self.current_assignment = Some(aug_assign.into());
+                self.visit_expr(target);
+                self.current_assignment = None;
+            }
             ast::Stmt::If(node) => {
                 self.visit_expr(&node.test);
                 let pre_if = self.flow_snapshot();
@@ -547,6 +563,18 @@ where
                     self.flow_merge(break_state);
                 }
             }
+            ast::Stmt::With(ast::StmtWith { items, body, .. }) => {
+                for item in items {
+                    self.visit_expr(&item.context_expr);
+                    if let Some(optional_vars) = item.optional_vars.as_deref() {
+                        self.add_standalone_expression(&item.context_expr);
+                        self.current_assignment = Some(item.into());
+                        self.visit_expr(optional_vars);
+                        self.current_assignment = None;
+                    }
+                }
+                self.visit_body(body);
+            }
             ast::Stmt::Break(_) => {
                 self.loop_break_states.push(self.flow_snapshot());
             }
@@ -563,12 +591,21 @@ where
 
         match expr {
             ast::Expr::Name(name_node @ ast::ExprName { id, ctx, .. }) => {
-                let flags = match ctx {
+                let mut flags = match ctx {
                     ast::ExprContext::Load => SymbolFlags::IS_USED,
                     ast::ExprContext::Store => SymbolFlags::IS_DEFINED,
                     ast::ExprContext::Del => SymbolFlags::IS_DEFINED,
                     ast::ExprContext::Invalid => SymbolFlags::empty(),
                 };
+                if matches!(
+                    self.current_assignment,
+                    Some(CurrentAssignment::AugAssign(_))
+                ) && !ctx.is_invalid()
+                {
+                    // For augmented assignment, the target expression is also used, so we should
+                    // record that as a use.
+                    flags |= SymbolFlags::IS_USED;
+                }
                 let symbol = self.add_or_update_symbol(id.clone(), flags);
                 if flags.contains(SymbolFlags::IS_DEFINED) {
                     match self.current_assignment {
@@ -584,6 +621,9 @@ where
                         Some(CurrentAssignment::AnnAssign(ann_assign)) => {
                             self.add_definition(symbol, ann_assign);
                         }
+                        Some(CurrentAssignment::AugAssign(aug_assign)) => {
+                            self.add_definition(symbol, aug_assign);
+                        }
                         Some(CurrentAssignment::Named(named)) => {
                             // TODO(dhruvmanila): If the current scope is a comprehension, then the
                             // named expression is implicitly nonlocal. This is yet to be
@@ -596,6 +636,15 @@ where
                                 ComprehensionDefinitionNodeRef { node, first },
                             );
                         }
+                        Some(CurrentAssignment::WithItem(with_item)) => {
+                            self.add_definition(
+                                symbol,
+                                WithItemDefinitionNodeRef {
+                                    node: with_item,
+                                    target: name_node,
+                                },
+                            );
+                        }
                         None => {}
                     }
                 }
@@ -609,11 +658,11 @@ where
             }
             ast::Expr::Named(node) => {
                 debug_assert!(self.current_assignment.is_none());
-                self.current_assignment = Some(node.into());
                 // TODO walrus in comprehensions is implicitly nonlocal
+                self.visit_expr(&node.value);
+                self.current_assignment = Some(node.into());
                 self.visit_expr(&node.target);
                 self.current_assignment = None;
-                self.visit_expr(&node.value);
             }
             ast::Expr::Lambda(lambda) => {
                 if let Some(parameters) = &lambda.parameters {
@@ -721,17 +770,38 @@ where
             self.visit_parameter(parameter);
         }
     }
+
+    fn visit_pattern(&mut self, pattern: &'ast ast::Pattern) {
+        if let ast::Pattern::MatchAs(ast::PatternMatchAs {
+            name: Some(name), ..
+        })
+        | ast::Pattern::MatchStar(ast::PatternMatchStar {
+            name: Some(name),
+            range: _,
+        })
+        | ast::Pattern::MatchMapping(ast::PatternMatchMapping {
+            rest: Some(name), ..
+        }) = pattern
+        {
+            // TODO(dhruvmanila): Add definition
+            self.add_or_update_symbol(name.id.clone(), SymbolFlags::IS_DEFINED);
+        }
+
+        walk_pattern(self, pattern);
+    }
 }
 
 #[derive(Copy, Clone, Debug)]
 enum CurrentAssignment<'a> {
     Assign(&'a ast::StmtAssign),
     AnnAssign(&'a ast::StmtAnnAssign),
+    AugAssign(&'a ast::StmtAugAssign),
     Named(&'a ast::ExprNamed),
     Comprehension {
         node: &'a ast::Comprehension,
         first: bool,
     },
+    WithItem(&'a ast::WithItem),
 }
 
 impl<'a> From<&'a ast::StmtAssign> for CurrentAssignment<'a> {
@@ -746,8 +816,20 @@ impl<'a> From<&'a ast::StmtAnnAssign> for CurrentAssignment<'a> {
     }
 }
 
+impl<'a> From<&'a ast::StmtAugAssign> for CurrentAssignment<'a> {
+    fn from(value: &'a ast::StmtAugAssign) -> Self {
+        Self::AugAssign(value)
+    }
+}
+
 impl<'a> From<&'a ast::ExprNamed> for CurrentAssignment<'a> {
     fn from(value: &'a ast::ExprNamed) -> Self {
         Self::Named(value)
     }
 }
+
+impl<'a> From<&'a ast::WithItem> for CurrentAssignment<'a> {
+    fn from(value: &'a ast::WithItem) -> Self {
+        Self::WithItem(value)
+    }
+}

crates/red_knot_python_semantic/src/semantic_index/definition.rs

@@ -44,8 +44,10 @@ pub(crate) enum DefinitionNodeRef<'a> {
     NamedExpression(&'a ast::ExprNamed),
     Assignment(AssignmentDefinitionNodeRef<'a>),
     AnnotatedAssignment(&'a ast::StmtAnnAssign),
+    AugmentedAssignment(&'a ast::StmtAugAssign),
     Comprehension(ComprehensionDefinitionNodeRef<'a>),
     Parameter(ast::AnyParameterRef<'a>),
+    WithItem(WithItemDefinitionNodeRef<'a>),
 }
 
 impl<'a> From<&'a ast::StmtFunctionDef> for DefinitionNodeRef<'a> {
@@ -72,6 +74,12 @@ impl<'a> From<&'a ast::StmtAnnAssign> for DefinitionNodeRef<'a> {
     }
 }
 
+impl<'a> From<&'a ast::StmtAugAssign> for DefinitionNodeRef<'a> {
+    fn from(node: &'a ast::StmtAugAssign) -> Self {
+        Self::AugmentedAssignment(node)
+    }
+}
+
 impl<'a> From<&'a ast::Alias> for DefinitionNodeRef<'a> {
     fn from(node_ref: &'a ast::Alias) -> Self {
         Self::Import(node_ref)
@@ -90,6 +98,12 @@ impl<'a> From<AssignmentDefinitionNodeRef<'a>> for DefinitionNodeRef<'a> {
     }
 }
 
+impl<'a> From<WithItemDefinitionNodeRef<'a>> for DefinitionNodeRef<'a> {
+    fn from(node_ref: WithItemDefinitionNodeRef<'a>) -> Self {
+        Self::WithItem(node_ref)
+    }
+}
+
 impl<'a> From<ComprehensionDefinitionNodeRef<'a>> for DefinitionNodeRef<'a> {
     fn from(node: ComprehensionDefinitionNodeRef<'a>) -> Self {
         Self::Comprehension(node)
@@ -114,6 +128,12 @@ pub(crate) struct AssignmentDefinitionNodeRef<'a> {
     pub(crate) target: &'a ast::ExprName,
 }
 
+#[derive(Copy, Clone, Debug)]
+pub(crate) struct WithItemDefinitionNodeRef<'a> {
+    pub(crate) node: &'a ast::WithItem,
+    pub(crate) target: &'a ast::ExprName,
+}
+
 #[derive(Copy, Clone, Debug)]
 pub(crate) struct ComprehensionDefinitionNodeRef<'a> {
     pub(crate) node: &'a ast::Comprehension,
@@ -151,6 +171,9 @@ impl DefinitionNodeRef<'_> {
             DefinitionNodeRef::AnnotatedAssignment(assign) => {
                 DefinitionKind::AnnotatedAssignment(AstNodeRef::new(parsed, assign))
             }
+            DefinitionNodeRef::AugmentedAssignment(augmented_assignment) => {
+                DefinitionKind::AugmentedAssignment(AstNodeRef::new(parsed, augmented_assignment))
+            }
             DefinitionNodeRef::Comprehension(ComprehensionDefinitionNodeRef { node, first }) => {
                 DefinitionKind::Comprehension(ComprehensionDefinitionKind {
                     node: AstNodeRef::new(parsed, node),
@@ -165,6 +188,12 @@ impl DefinitionNodeRef<'_> {
                     DefinitionKind::ParameterWithDefault(AstNodeRef::new(parsed, parameter))
                 }
             },
+            DefinitionNodeRef::WithItem(WithItemDefinitionNodeRef { node, target }) => {
+                DefinitionKind::WithItem(WithItemDefinitionKind {
+                    node: AstNodeRef::new(parsed.clone(), node),
+                    target: AstNodeRef::new(parsed, target),
+                })
+            }
         }
     }
 
@@ -182,11 +211,13 @@ impl DefinitionNodeRef<'_> {
                 target,
             }) => target.into(),
             Self::AnnotatedAssignment(node) => node.into(),
+            Self::AugmentedAssignment(node) => node.into(),
             Self::Comprehension(ComprehensionDefinitionNodeRef { node, first: _ }) => node.into(),
             Self::Parameter(node) => match node {
                 ast::AnyParameterRef::Variadic(parameter) => parameter.into(),
                 ast::AnyParameterRef::NonVariadic(parameter) => parameter.into(),
             },
+            Self::WithItem(WithItemDefinitionNodeRef { node: _, target }) => target.into(),
         }
     }
 }
@@ -200,9 +231,11 @@ pub enum DefinitionKind {
     NamedExpression(AstNodeRef<ast::ExprNamed>),
     Assignment(AssignmentDefinitionKind),
     AnnotatedAssignment(AstNodeRef<ast::StmtAnnAssign>),
+    AugmentedAssignment(AstNodeRef<ast::StmtAugAssign>),
     Comprehension(ComprehensionDefinitionKind),
     Parameter(AstNodeRef<ast::Parameter>),
     ParameterWithDefault(AstNodeRef<ast::ParameterWithDefault>),
+    WithItem(WithItemDefinitionKind),
 }
 
 #[derive(Clone, Debug)]
@@ -238,7 +271,6 @@ impl ImportFromDefinitionKind {
 }
 
 #[derive(Clone, Debug)]
-#[allow(dead_code)]
 pub struct AssignmentDefinitionKind {
     assignment: AstNodeRef<ast::StmtAssign>,
     target: AstNodeRef<ast::ExprName>,
@@ -254,6 +286,22 @@ impl AssignmentDefinitionKind {
     }
 }
 
+#[derive(Clone, Debug)]
+pub struct WithItemDefinitionKind {
+    node: AstNodeRef<ast::WithItem>,
+    target: AstNodeRef<ast::ExprName>,
+}
+
+impl WithItemDefinitionKind {
+    pub(crate) fn node(&self) -> &ast::WithItem {
+        self.node.node()
+    }
+
+    pub(crate) fn target(&self) -> &ast::ExprName {
+        self.target.node()
+    }
+}
+
 #[derive(Copy, Clone, Eq, PartialEq, Hash, Debug)]
 pub(crate) struct DefinitionNodeKey(NodeKey);
 
@@ -293,6 +341,12 @@ impl From<&ast::StmtAnnAssign> for DefinitionNodeKey {
     }
 }
 
+impl From<&ast::StmtAugAssign> for DefinitionNodeKey {
+    fn from(node: &ast::StmtAugAssign) -> Self {
+        Self(NodeKey::from_node(node))
+    }
+}
+
 impl From<&ast::Comprehension> for DefinitionNodeKey {
     fn from(node: &ast::Comprehension) -> Self {
         Self(NodeKey::from_node(node))

crates/red_knot_python_semantic/src/semantic_model.rs

@@ -184,14 +184,9 @@ mod tests {
 
         Program::from_settings(
             &db,
-            ProgramSettings {
+            &ProgramSettings {
                 target_version: PythonVersion::default(),
-                search_paths: SearchPathSettings {
-                    extra_paths: vec![],
-                    src_root: SystemPathBuf::from("/src"),
-                    site_packages: vec![],
-                    custom_typeshed: None,
-                },
+                search_paths: SearchPathSettings::new(SystemPathBuf::from("/src")),
             },
         )?;
 

crates/red_knot_python_semantic/src/site_packages.rs

@@ -13,9 +13,10 @@ use std::io;
 use std::num::NonZeroUsize;
 use std::ops::Deref;
 
-use red_knot_python_semantic::PythonVersion;
 use ruff_db::system::{System, SystemPath, SystemPathBuf};
 
+use crate::PythonVersion;
+
 type SitePackagesDiscoveryResult<T> = Result<T, SitePackagesDiscoveryError>;
 
 /// Abstraction for a Python virtual environment.
@@ -24,7 +25,7 @@ type SitePackagesDiscoveryResult<T> = Result<T, SitePackagesDiscoveryError>;
 /// The format of this file is not defined anywhere, and exactly which keys are present
 /// depends on the tool that was used to create the virtual environment.
 #[derive(Debug)]
-pub struct VirtualEnvironment {
+pub(crate) struct VirtualEnvironment {
     venv_path: SysPrefixPath,
     base_executable_home_path: PythonHomePath,
     include_system_site_packages: bool,
@@ -41,7 +42,7 @@ pub struct VirtualEnvironment {
 }
 
 impl VirtualEnvironment {
-    pub fn new(
+    pub(crate) fn new(
         path: impl AsRef<SystemPath>,
         system: &dyn System,
     ) -> SitePackagesDiscoveryResult<Self> {
@@ -55,7 +56,7 @@ impl VirtualEnvironment {
 
         let venv_path = SysPrefixPath::new(path, system)?;
         let pyvenv_cfg_path = venv_path.join("pyvenv.cfg");
-        tracing::debug!("Attempting to parse virtual environment metadata at {pyvenv_cfg_path}");
+        tracing::debug!("Attempting to parse virtual environment metadata at '{pyvenv_cfg_path}'");
 
         let pyvenv_cfg = system
             .read_to_string(&pyvenv_cfg_path)
@@ -157,7 +158,7 @@ impl VirtualEnvironment {
     /// Return a list of `site-packages` directories that are available from this virtual environment
     ///
     /// See the documentation for `site_packages_dir_from_sys_prefix` for more details.
-    pub fn site_packages_directories(
+    pub(crate) fn site_packages_directories(
         &self,
         system: &dyn System,
     ) -> SitePackagesDiscoveryResult<Vec<SystemPathBuf>> {
@@ -191,7 +192,7 @@ impl VirtualEnvironment {
             } else {
                 tracing::warn!(
                     "Failed to resolve `sys.prefix` of the system Python installation \
-from the `home` value in the `pyvenv.cfg` file at {}. \
+from the `home` value in the `pyvenv.cfg` file at '{}'. \
 System site-packages will not be used for module resolution.",
                     venv_path.join("pyvenv.cfg")
                 );
@@ -204,7 +205,7 @@ System site-packages will not be used for module resolution.",
 }
 
 #[derive(Debug, thiserror::Error)]
-pub enum SitePackagesDiscoveryError {
+pub(crate) enum SitePackagesDiscoveryError {
     #[error("Invalid --venv-path argument: {0} could not be canonicalized")]
     VenvDirCanonicalizationError(SystemPathBuf, #[source] io::Error),
     #[error("Invalid --venv-path argument: {0} does not point to a directory on disk")]
@@ -221,7 +222,7 @@ pub enum SitePackagesDiscoveryError {
 
 /// The various ways in which parsing a `pyvenv.cfg` file could fail
 #[derive(Debug)]
-pub enum PyvenvCfgParseErrorKind {
+pub(crate) enum PyvenvCfgParseErrorKind {
     TooManyEquals { line_number: NonZeroUsize },
     MalformedKeyValuePair { line_number: NonZeroUsize },
     NoHomeKey,
@@ -370,7 +371,7 @@ fn site_packages_directory_from_sys_prefix(
 ///
 /// [`sys.prefix`]: https://docs.python.org/3/library/sys.html#sys.prefix
 #[derive(Debug, PartialEq, Eq, Clone)]
-pub struct SysPrefixPath(SystemPathBuf);
+pub(crate) struct SysPrefixPath(SystemPathBuf);
 
 impl SysPrefixPath {
     fn new(
@@ -425,7 +426,7 @@ impl Deref for SysPrefixPath {
 
 impl fmt::Display for SysPrefixPath {
     fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
-        write!(f, "`sys.prefix` path {}", self.0)
+        write!(f, "`sys.prefix` path '{}'", self.0)
     }
 }
 
@@ -482,7 +483,7 @@ impl Deref for PythonHomePath {
 
 impl fmt::Display for PythonHomePath {
     fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
-        write!(f, "`home` location {}", self.0)
+        write!(f, "`home` location '{}'", self.0)
     }
 }
 

crates/red_knot_python_semantic/src/types.rs

@@ -5,21 +5,37 @@ use crate::builtins::builtins_scope;
 use crate::semantic_index::definition::Definition;
 use crate::semantic_index::symbol::{ScopeId, ScopedSymbolId};
 use crate::semantic_index::{
-    global_scope, symbol_table, use_def_map, DefinitionWithConstraints,
+    global_scope, semantic_index, symbol_table, use_def_map, DefinitionWithConstraints,
     DefinitionWithConstraintsIterator,
 };
 use crate::types::narrow::narrowing_constraint;
 use crate::{Db, FxOrderSet};
 
+pub(crate) use self::builder::{IntersectionBuilder, UnionBuilder};
+pub(crate) use self::diagnostic::TypeCheckDiagnostics;
+pub(crate) use self::infer::{
+    infer_definition_types, infer_expression_types, infer_scope_types, TypeInference,
+};
+
 mod builder;
+mod diagnostic;
 mod display;
 mod infer;
 mod narrow;
 
-pub(crate) use self::builder::{IntersectionBuilder, UnionBuilder};
-pub(crate) use self::infer::{
-    infer_definition_types, infer_expression_types, infer_scope_types, TypeInference,
-};
+pub fn check_types(db: &dyn Db, file: File) -> TypeCheckDiagnostics {
+    let _span = tracing::trace_span!("check_types", file=?file.path(db)).entered();
+
+    let index = semantic_index(db, file);
+    let mut diagnostics = TypeCheckDiagnostics::new();
+
+    for scope_id in index.scope_ids() {
+        let result = infer_scope_types(db, scope_id);
+        diagnostics.extend(result.diagnostics());
+    }
+
+    diagnostics
+}
 
 /// Infer the public type of a symbol (its type as seen from outside its scope).
 pub(crate) fn symbol_ty<'db>(
@@ -165,6 +181,8 @@ pub enum Type<'db> {
     IntLiteral(i64),
     /// A boolean literal, either `True` or `False`.
     BooleanLiteral(bool),
+    /// A bytes literal
+    BytesLiteral(BytesLiteralType<'db>),
     // TODO protocols, callable types, overloads, generics, type vars
 }
 
@@ -206,20 +224,42 @@ impl<'db> Type<'db> {
         }
     }
 
+    /// Resolve a member access of a type.
+    ///
+    /// For example, if `foo` is `Type::Instance(<Bar>)`,
+    /// `foo.member(&db, "baz")` returns the type of `baz` attributes
+    /// as accessed from instances of the `Bar` class.
+    ///
+    /// TODO: use of this method currently requires manually checking
+    /// whether the returned type is `Unknown`/`Unbound`
+    /// (or a union with `Unknown`/`Unbound`) in many places.
+    /// Ideally we'd use a more type-safe pattern, such as returning
+    /// an `Option` or a `Result` from this method, which would force
+    /// us to explicitly consider whether to handle an error or propagate
+    /// it up the call stack.
     #[must_use]
     pub fn member(&self, db: &'db dyn Db, name: &Name) -> Type<'db> {
         match self {
             Type::Any => Type::Any,
-            Type::Never => todo!("attribute lookup on Never type"),
+            Type::Never => {
+                // TODO: attribute lookup on Never type
+                Type::Unknown
+            }
             Type::Unknown => Type::Unknown,
             Type::Unbound => Type::Unbound,
-            Type::None => todo!("attribute lookup on None type"),
-            Type::Function(_) => todo!("attribute lookup on Function type"),
+            Type::None => {
+                // TODO: attribute lookup on None type
+                Type::Unknown
+            }
+            Type::Function(_) => {
+                // TODO: attribute lookup on function type
+                Type::Unknown
+            }
             Type::Module(file) => global_symbol_ty_by_name(db, *file, name),
             Type::Class(class) => class.class_member(db, name),
             Type::Instance(_) => {
                 // TODO MRO? get_own_instance_member, get_instance_member
-                todo!("attribute lookup on Instance type")
+                Type::Unknown
             }
             Type::Union(union) => union
                 .elements(db)
@@ -231,13 +271,17 @@ impl<'db> Type<'db> {
             Type::Intersection(_) => {
                 // TODO perform the get_member on each type in the intersection
                 // TODO return the intersection of those results
-                todo!("attribute lookup on Intersection type")
+                Type::Unknown
             }
             Type::IntLiteral(_) => {
                 // TODO raise error
                 Type::Unknown
             }
             Type::BooleanLiteral(_) => Type::Unknown,
+            Type::BytesLiteral(_) => {
+                // TODO defer to Type::Instance(<bytes from typeshed>).member
+                Type::Unknown
+            }
         }
     }
 
@@ -333,3 +377,116 @@ pub struct IntersectionType<'db> {
     /// directly in intersections rather than as a separate type.
     negative: FxOrderSet<Type<'db>>,
 }
+
+#[salsa::interned]
+pub struct BytesLiteralType<'db> {
+    #[return_ref]
+    value: Box<[u8]>,
+}
+
+#[cfg(test)]
+mod tests {
+    use anyhow::Context;
+
+    use ruff_db::files::system_path_to_file;
+    use ruff_db::system::{DbWithTestSystem, SystemPathBuf};
+
+    use crate::db::tests::TestDb;
+    use crate::{Program, ProgramSettings, PythonVersion, SearchPathSettings};
+
+    use super::TypeCheckDiagnostics;
+
+    fn setup_db() -> TestDb {
+        let db = TestDb::new();
+        db.memory_file_system()
+            .create_directory_all("/src")
+            .unwrap();
+
+        Program::from_settings(
+            &db,
+            &ProgramSettings {
+                target_version: PythonVersion::default(),
+                search_paths: SearchPathSettings::new(SystemPathBuf::from("/src")),
+            },
+        )
+        .expect("Valid search path settings");
+
+        db
+    }
+
+    fn assert_diagnostic_messages(diagnostics: &TypeCheckDiagnostics, expected: &[&str]) {
+        let messages: Vec<&str> = diagnostics
+            .iter()
+            .map(|diagnostic| diagnostic.message())
+            .collect();
+        assert_eq!(&messages, expected);
+    }
+
+    #[test]
+    fn unresolved_import_statement() -> anyhow::Result<()> {
+        let mut db = setup_db();
+
+        db.write_file("src/foo.py", "import bar\n")
+            .context("Failed to write foo.py")?;
+
+        let foo = system_path_to_file(&db, "src/foo.py").context("Failed to resolve foo.py")?;
+
+        let diagnostics = super::check_types(&db, foo);
+        assert_diagnostic_messages(&diagnostics, &["Import 'bar' could not be resolved."]);
+
+        Ok(())
+    }
+
+    #[test]
+    fn unresolved_import_from_statement() {
+        let mut db = setup_db();
+
+        db.write_file("src/foo.py", "from bar import baz\n")
+            .unwrap();
+        let foo = system_path_to_file(&db, "src/foo.py").unwrap();
+        let diagnostics = super::check_types(&db, foo);
+        assert_diagnostic_messages(&diagnostics, &["Import 'bar' could not be resolved."]);
+    }
+
+    #[test]
+    fn unresolved_import_from_resolved_module() {
+        let mut db = setup_db();
+
+        db.write_files([("/src/a.py", ""), ("/src/b.py", "from a import thing")])
+            .unwrap();
+
+        let b_file = system_path_to_file(&db, "/src/b.py").unwrap();
+        let b_file_diagnostics = super::check_types(&db, b_file);
+        assert_diagnostic_messages(
+            &b_file_diagnostics,
+            &["Could not resolve import of 'thing' from 'a'"],
+        );
+    }
+
+    #[ignore = "\
+A spurious second 'Unresolved import' diagnostic message is emitted on `b.py`, \
+despite the symbol existing in the symbol table for `a.py`"]
+    #[test]
+    fn resolved_import_of_symbol_from_unresolved_import() {
+        let mut db = setup_db();
+
+        db.write_files([
+            ("/src/a.py", "import foo as foo"),
+            ("/src/b.py", "from a import foo"),
+        ])
+        .unwrap();
+
+        let a_file = system_path_to_file(&db, "/src/a.py").unwrap();
+        let a_file_diagnostics = super::check_types(&db, a_file);
+        assert_diagnostic_messages(
+            &a_file_diagnostics,
+            &["Import 'foo' could not be resolved."],
+        );
+
+        // Importing the unresolved import into a second first-party file should not trigger
+        // an additional "unresolved import" violation
+        let b_file = system_path_to_file(&db, "/src/b.py").unwrap();
+        let b_file_diagnostics = super::check_types(&db, b_file);
+        assert_eq!(&*b_file_diagnostics, &[]);
+    }
+}

crates/red_knot_python_semantic/src/types/diagnostic.rs

@@ -0,0 +1,111 @@
+use ruff_db::files::File;
+use ruff_text_size::{Ranged, TextRange};
+use std::fmt::Formatter;
+use std::ops::Deref;
+use std::sync::Arc;
+
+#[derive(Debug, Eq, PartialEq)]
+pub struct TypeCheckDiagnostic {
+    // TODO: Don't use string keys for rules
+    pub(super) rule: String,
+    pub(super) message: String,
+    pub(super) range: TextRange,
+    pub(super) file: File,
+}
+
+impl TypeCheckDiagnostic {
+    pub fn rule(&self) -> &str {
+        &self.rule
+    }
+
+    pub fn message(&self) -> &str {
+        &self.message
+    }
+
+    pub fn file(&self) -> File {
+        self.file
+    }
+}
+
+impl Ranged for TypeCheckDiagnostic {
+    fn range(&self) -> TextRange {
+        self.range
+    }
+}
+
+/// A collection of type check diagnostics.
+///
+/// The diagnostics are wrapped in an `Arc` because they need to be cloned multiple times
+/// when going from `infer_expression` to `check_file`. We could consider
+/// making [`TypeCheckDiagnostic`] a Salsa struct to have them Arena-allocated (once the Tables refactor is done).
+/// Using Salsa struct does have the downside that it leaks the Salsa dependency into diagnostics and
+/// each Salsa-struct comes with an overhead.
+#[derive(Default, Eq, PartialEq)]
+pub struct TypeCheckDiagnostics {
+    inner: Vec<std::sync::Arc<TypeCheckDiagnostic>>,
+}
+
+impl TypeCheckDiagnostics {
+    pub fn new() -> Self {
+        Self { inner: Vec::new() }
+    }
+
+    pub(super) fn push(&mut self, diagnostic: TypeCheckDiagnostic) {
+        self.inner.push(Arc::new(diagnostic));
+    }
+
+    pub(crate) fn shrink_to_fit(&mut self) {
+        self.inner.shrink_to_fit();
+    }
+}
+
+impl Extend<TypeCheckDiagnostic> for TypeCheckDiagnostics {
+    fn extend<T: IntoIterator<Item = TypeCheckDiagnostic>>(&mut self, iter: T) {
+        self.inner.extend(iter.into_iter().map(std::sync::Arc::new));
+    }
+}
+
+impl Extend<std::sync::Arc<TypeCheckDiagnostic>> for TypeCheckDiagnostics {
+    fn extend<T: IntoIterator<Item = Arc<TypeCheckDiagnostic>>>(&mut self, iter: T) {
+        self.inner.extend(iter);
+    }
+}
+
+impl<'a> Extend<&'a std::sync::Arc<TypeCheckDiagnostic>> for TypeCheckDiagnostics {
+    fn extend<T: IntoIterator<Item = &'a Arc<TypeCheckDiagnostic>>>(&mut self, iter: T) {
+        self.inner
+            .extend(iter.into_iter().map(std::sync::Arc::clone));
+    }
+}
+
+impl std::fmt::Debug for TypeCheckDiagnostics {
+    fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+        self.inner.fmt(f)
+    }
+}
+
+impl Deref for TypeCheckDiagnostics {
+    type Target = [std::sync::Arc<TypeCheckDiagnostic>];
+
+    fn deref(&self) -> &Self::Target {
+        &self.inner
+    }
+}
+
+impl IntoIterator for TypeCheckDiagnostics {
+    type Item = Arc<TypeCheckDiagnostic>;
+    type IntoIter = std::vec::IntoIter<std::sync::Arc<TypeCheckDiagnostic>>;
+
+    fn into_iter(self) -> Self::IntoIter {
+        self.inner.into_iter()
+    }
+}
+
+impl<'a> IntoIterator for &'a TypeCheckDiagnostics {
+    type Item = &'a Arc<TypeCheckDiagnostic>;
+    type IntoIter = std::slice::Iter<'a, std::sync::Arc<TypeCheckDiagnostic>>;
+
+    fn into_iter(self) -> Self::IntoIter {
+        self.inner.iter()
+    }
+}

crates/red_knot_python_semantic/src/types/display.rs

@@ -2,6 +2,9 @@
 
 use std::fmt::{Display, Formatter};
 
+use ruff_python_ast::str::Quote;
+use ruff_python_literal::escape::AsciiEscape;
+
 use crate::types::{IntersectionType, Type, UnionType};
 use crate::Db;
 
@@ -38,6 +41,14 @@ impl Display for DisplayType<'_> {
             Type::BooleanLiteral(boolean) => {
                 write!(f, "Literal[{}]", if *boolean { "True" } else { "False" })
             }
+            Type::BytesLiteral(bytes) => {
+                let escape =
+                    AsciiEscape::with_preferred_quote(bytes.value(self.db).as_ref(), Quote::Double);
+
+                f.write_str("Literal[")?;
+                escape.bytes_repr().write(f)?;
+                f.write_str("]")
+            }
         }
     }
 }

crates/red_knot_python_semantic/src/types/infer.rs

@@ -29,7 +29,8 @@ use salsa::plumbing::AsId;
 use ruff_db::files::File;
 use ruff_db::parsed::parsed_module;
 use ruff_python_ast as ast;
-use ruff_python_ast::{Expr, ExprContext};
+use ruff_python_ast::{AnyNodeRef, ExprContext};
+use ruff_text_size::Ranged;
 
 use crate::builtins::builtins_scope;
 use crate::module_name::ModuleName;
@@ -40,9 +41,10 @@ use crate::semantic_index::expression::Expression;
 use crate::semantic_index::semantic_index;
 use crate::semantic_index::symbol::{FileScopeId, NodeWithScopeKind, NodeWithScopeRef, ScopeId};
 use crate::semantic_index::SemanticIndex;
+use crate::types::diagnostic::{TypeCheckDiagnostic, TypeCheckDiagnostics};
 use crate::types::{
-    builtins_symbol_ty_by_name, definitions_ty, global_symbol_ty_by_name, ClassType, FunctionType,
-    Name, Type, UnionBuilder,
+    builtins_symbol_ty_by_name, definitions_ty, global_symbol_ty_by_name, BytesLiteralType,
+    ClassType, FunctionType, Name, Type, UnionBuilder,
 };
 use crate::Db;
 
@@ -123,13 +125,16 @@ pub(crate) enum InferenceRegion<'db> {
 }
 
 /// The inferred types for a single region.
-#[derive(Debug, Eq, PartialEq, Default, Clone)]
+#[derive(Debug, Eq, PartialEq, Default)]
 pub(crate) struct TypeInference<'db> {
     /// The types of every expression in this region.
     expressions: FxHashMap<ScopedExpressionId, Type<'db>>,
 
     /// The types of every definition in this region.
     definitions: FxHashMap<Definition<'db>, Type<'db>>,
+
+    /// The diagnostics for this region.
+    diagnostics: TypeCheckDiagnostics,
 }
 
 impl<'db> TypeInference<'db> {
@@ -142,9 +147,14 @@ impl<'db> TypeInference<'db> {
         self.definitions[&definition]
     }
 
+    pub(crate) fn diagnostics(&self) -> &[std::sync::Arc<TypeCheckDiagnostic>] {
+        &self.diagnostics
+    }
+
     fn shrink_to_fit(&mut self) {
         self.expressions.shrink_to_fit();
         self.definitions.shrink_to_fit();
+        self.diagnostics.shrink_to_fit();
     }
 }
 
@@ -235,6 +245,7 @@ impl<'db> TypeInferenceBuilder<'db> {
     fn extend(&mut self, inference: &TypeInference<'db>) {
         self.types.definitions.extend(inference.definitions.iter());
         self.types.expressions.extend(inference.expressions.iter());
+        self.types.diagnostics.extend(&inference.diagnostics);
     }
 
     /// Infers types in the given [`InferenceRegion`].
@@ -303,6 +314,9 @@ impl<'db> TypeInferenceBuilder<'db> {
             DefinitionKind::AnnotatedAssignment(annotated_assignment) => {
                 self.infer_annotated_assignment_definition(annotated_assignment.node(), definition);
             }
+            DefinitionKind::AugmentedAssignment(augmented_assignment) => {
+                self.infer_augment_assignment_definition(augmented_assignment.node(), definition);
+            }
             DefinitionKind::NamedExpression(named_expression) => {
                 self.infer_named_expression_definition(named_expression.node(), definition);
             }
@@ -319,6 +333,9 @@ impl<'db> TypeInferenceBuilder<'db> {
             DefinitionKind::ParameterWithDefault(parameter_with_default) => {
                 self.infer_parameter_with_default_definition(parameter_with_default, definition);
             }
+            DefinitionKind::WithItem(with_item) => {
+                self.infer_with_item_definition(with_item.target(), with_item.node(), definition);
+            }
         }
     }
 
@@ -604,13 +621,42 @@ impl<'db> TypeInferenceBuilder<'db> {
         } = with_statement;
 
         for item in items {
-            self.infer_expression(&item.context_expr);
-            self.infer_optional_expression(item.optional_vars.as_deref());
+            match item.optional_vars.as_deref() {
+                Some(ast::Expr::Name(name)) => {
+                    self.infer_definition(name);
+                }
+                _ => {
+                    // TODO infer definitions in unpacking assignment
+                    self.infer_expression(&item.context_expr);
+                }
+            }
         }
 
         self.infer_body(body);
     }
 
+    fn infer_with_item_definition(
+        &mut self,
+        target: &ast::ExprName,
+        with_item: &ast::WithItem,
+        definition: Definition<'db>,
+    ) {
+        let expression = self.index.expression(&with_item.context_expr);
+        let result = infer_expression_types(self.db, expression);
+        self.extend(result);
+
+        // TODO(dhruvmanila): The correct type inference here is the return type of the __enter__
+        // method of the context manager.
+        let context_expr_ty = self
+            .types
+            .expression_ty(with_item.context_expr.scoped_ast_id(self.db, self.scope));
+
+        self.types
+            .expressions
+            .insert(target.scoped_ast_id(self.db, self.scope), context_expr_ty);
+        self.types.definitions.insert(definition, context_expr_ty);
+    }
+
     fn infer_match_statement(&mut self, match_statement: &ast::StmtMatch) {
         let ast::StmtMatch {
             range: _,
@@ -763,15 +809,35 @@ impl<'db> TypeInferenceBuilder<'db> {
     }
 
     fn infer_augmented_assignment_statement(&mut self, assignment: &ast::StmtAugAssign) {
-        // TODO this should be a Definition
+        if assignment.target.is_name_expr() {
+            self.infer_definition(assignment);
+        } else {
+            // TODO currently we don't consider assignments to non-Names to be Definitions
+            self.infer_augment_assignment(assignment);
+        }
+    }
+
+    fn infer_augment_assignment_definition(
+        &mut self,
+        assignment: &ast::StmtAugAssign,
+        definition: Definition<'db>,
+    ) {
+        let target_ty = self.infer_augment_assignment(assignment);
+        self.types.definitions.insert(definition, target_ty);
+    }
+
+    fn infer_augment_assignment(&mut self, assignment: &ast::StmtAugAssign) -> Type<'db> {
         let ast::StmtAugAssign {
             range: _,
             target,
             op: _,
             value,
         } = assignment;
-        self.infer_expression(target);
         self.infer_expression(value);
+        self.infer_expression(target);
+
+        // TODO(dhruvmanila): Resolve the target type using the value type and the operator
+        Type::Unknown
     }
 
     fn infer_type_alias_statement(&mut self, type_alias_statement: &ast::StmtTypeAlias) {
@@ -832,7 +898,26 @@ impl<'db> TypeInferenceBuilder<'db> {
             asname: _,
         } = alias;
 
-        let module_ty = self.module_ty_from_name(ModuleName::new(name));
+        let module_ty = ModuleName::new(name)
+            .ok_or(ModuleResolutionError::InvalidSyntax)
+            .and_then(|module_name| self.module_ty_from_name(module_name));
+
+        let module_ty = match module_ty {
+            Ok(ty) => ty,
+            Err(ModuleResolutionError::InvalidSyntax) => {
+                tracing::debug!("Failed to resolve import due to invalid syntax");
+                Type::Unknown
+            }
+            Err(ModuleResolutionError::UnresolvedModule) => {
+                self.add_diagnostic(
+                    AnyNodeRef::Alias(alias),
+                    "unresolved-import",
+                    format_args!("Import '{name}' could not be resolved."),
+                );
+                Type::Unknown
+            }
+        };
+
         self.types.definitions.insert(definition, module_ty);
     }
 
@@ -880,10 +965,18 @@ impl<'db> TypeInferenceBuilder<'db> {
     /// - `tail` is the relative module name stripped of all leading dots:
     ///   - `from .foo import bar` => `tail == "foo"`
     ///   - `from ..foo.bar import baz` => `tail == "foo.bar"`
-    fn relative_module_name(&self, tail: Option<&str>, level: NonZeroU32) -> Option<ModuleName> {
+    fn relative_module_name(
+        &self,
+        tail: Option<&str>,
+        level: NonZeroU32,
+    ) -> Result<ModuleName, ModuleResolutionError> {
         let Some(module) = file_to_module(self.db, self.file) else {
-            tracing::debug!("Failed to resolve file {:?} to a module", self.file);
-            return None;
+            tracing::debug!(
+                "Relative module resolution '{}' failed; could not resolve file '{}' to a module",
+                format_import_from_module(level.get(), tail),
+                self.file.path(self.db)
+            );
+            return Err(ModuleResolutionError::UnresolvedModule);
         };
         let mut level = level.get();
         if module.kind().is_package() {
@@ -891,17 +984,19 @@ impl<'db> TypeInferenceBuilder<'db> {
         }
         let mut module_name = module.name().to_owned();
         for _ in 0..level {
-            module_name = module_name.parent()?;
+            module_name = module_name
+                .parent()
+                .ok_or(ModuleResolutionError::UnresolvedModule)?;
         }
         if let Some(tail) = tail {
             if let Some(valid_tail) = ModuleName::new(tail) {
                 module_name.extend(&valid_tail);
             } else {
-                tracing::debug!("Failed to resolve relative import due to invalid syntax");
-                return None;
+                tracing::debug!("Relative module resolution failed: invalid syntax");
+                return Err(ModuleResolutionError::InvalidSyntax);
             }
         }
-        Some(module_name)
+        Ok(module_name)
     }
 
     fn infer_import_from_definition(
@@ -921,16 +1016,27 @@ impl<'db> TypeInferenceBuilder<'db> {
         // `follow_nonexistent_import_bare_to_module()`.
         let ast::StmtImportFrom { module, level, .. } = import_from;
         tracing::trace!("Resolving imported object {alias:?} from statement {import_from:?}");
+        let module = module.as_deref();
         let module_name = if let Some(level) = NonZeroU32::new(*level) {
-            self.relative_module_name(module.as_deref(), level)
+            tracing::trace!(
+                "Resolving imported object '{}' from module '{}' relative to file '{}'",
+                alias.name,
+                format_import_from_module(level.get(), module),
+                self.file.path(self.db),
+            );
+            self.relative_module_name(module, level)
         } else {
-            let module_name = module
-                .as_ref()
-                .expect("Non-relative import should always have a non-None `module`!");
-            ModuleName::new(module_name)
+            tracing::trace!(
+                "Resolving imported object '{}' from module '{}'",
+                alias.name,
+                format_import_from_module(*level, module),
+            );
+            module
+                .and_then(ModuleName::new)
+                .ok_or(ModuleResolutionError::InvalidSyntax)
         };
 
-        let module_ty = self.module_ty_from_name(module_name);
+        let module_ty = module_name.and_then(|module_name| self.module_ty_from_name(module_name));
 
         let ast::Alias {
             range: _,
@@ -943,11 +1049,34 @@ impl<'db> TypeInferenceBuilder<'db> {
         // the runtime error will occur immediately (rather than when the symbol is *used*,
         // as would be the case for a symbol with type `Unbound`), so it's appropriate to
         // think of the type of the imported symbol as `Unknown` rather than `Unbound`
-        let ty = module_ty
+        let member_ty = module_ty
+            .unwrap_or(Type::Unbound)
             .member(self.db, &Name::new(&name.id))
             .replace_unbound_with(self.db, Type::Unknown);
 
-        self.types.definitions.insert(definition, ty);
+        if matches!(module_ty, Err(ModuleResolutionError::UnresolvedModule)) {
+            self.add_diagnostic(
+                AnyNodeRef::StmtImportFrom(import_from),
+                "unresolved-import",
+                format_args!(
+                    "Import '{}{}' could not be resolved.",
+                    ".".repeat(*level as usize),
+                    module.unwrap_or_default()
+                ),
+            );
+        } else if module_ty.is_ok() && member_ty.is_unknown() {
+            self.add_diagnostic(
+                AnyNodeRef::Alias(alias),
+                "unresolved-import",
+                format_args!(
+                    "Could not resolve import of '{name}' from '{}{}'",
+                    ".".repeat(*level as usize),
+                    module.unwrap_or_default()
+                ),
+            );
+        }
+
+        self.types.definitions.insert(definition, member_ty);
     }
 
     fn infer_return_statement(&mut self, ret: &ast::StmtReturn) {
@@ -961,10 +1090,13 @@ impl<'db> TypeInferenceBuilder<'db> {
         }
     }
 
-    fn module_ty_from_name(&self, module_name: Option<ModuleName>) -> Type<'db> {
-        module_name
-            .and_then(|module_name| resolve_module(self.db, module_name))
-            .map_or(Type::Unknown, |module| Type::Module(module.file()))
+    fn module_ty_from_name(
+        &self,
+        module_name: ModuleName,
+    ) -> Result<Type<'db>, ModuleResolutionError> {
+        resolve_module(self.db, module_name)
+            .map(|module| Type::Module(module.file()))
+            .ok_or(ModuleResolutionError::UnresolvedModule)
     }
 
     fn infer_decorator(&mut self, decorator: &ast::Decorator) -> Type<'db> {
@@ -1036,7 +1168,7 @@ impl<'db> TypeInferenceBuilder<'db> {
             ast::Expr::Yield(yield_expression) => self.infer_yield_expression(yield_expression),
             ast::Expr::YieldFrom(yield_from) => self.infer_yield_from_expression(yield_from),
             ast::Expr::Await(await_expression) => self.infer_await_expression(await_expression),
-            Expr::IpyEscapeCommand(_) => todo!("Implement Ipy escape command support"),
+            ast::Expr::IpyEscapeCommand(_) => todo!("Implement Ipy escape command support"),
         };
 
         let expr_id = expression.scoped_ast_id(self.db, self.scope);
@@ -1074,9 +1206,12 @@ impl<'db> TypeInferenceBuilder<'db> {
     }
 
     #[allow(clippy::unused_self)]
-    fn infer_bytes_literal_expression(&mut self, _literal: &ast::ExprBytesLiteral) -> Type<'db> {
-        // TODO
-        Type::Unknown
+    fn infer_bytes_literal_expression(&mut self, literal: &ast::ExprBytesLiteral) -> Type<'db> {
+        // TODO: ignoring r/R prefixes for now, should normalize bytes values
+        Type::BytesLiteral(BytesLiteralType::new(
+            self.db,
+            literal.value.bytes().collect(),
+        ))
     }
 
     fn infer_fstring_expression(&mut self, fstring: &ast::ExprFString) -> Type<'db> {
@@ -1094,21 +1229,7 @@ impl<'db> TypeInferenceBuilder<'db> {
                         flags: _,
                     } = fstring;
                     for element in elements {
-                        match element {
-                            ast::FStringElement::Literal(_) => {
-                                // TODO string literal type
-                            }
-                            ast::FStringElement::Expression(expr_element) => {
-                                let ast::FStringExpressionElement {
-                                    range: _,
-                                    expression,
-                                    debug_text: _,
-                                    conversion: _,
-                                    format_spec: _,
-                                } = expr_element;
-                                self.infer_expression(expression);
-                            }
-                        }
+                        self.infer_fstring_element(element);
                     }
                 }
             }
@@ -1118,6 +1239,30 @@ impl<'db> TypeInferenceBuilder<'db> {
         Type::Unknown
     }
 
+    fn infer_fstring_element(&mut self, element: &ast::FStringElement) {
+        match element {
+            ast::FStringElement::Literal(_) => {
+                // TODO string literal type
+            }
+            ast::FStringElement::Expression(expr_element) => {
+                let ast::FStringExpressionElement {
+                    range: _,
+                    expression,
+                    debug_text: _,
+                    conversion: _,
+                    format_spec,
+                } = expr_element;
+                self.infer_expression(expression);
+
+                if let Some(format_spec) = format_spec {
+                    for spec_element in &format_spec.elements {
+                        self.infer_fstring_element(spec_element);
+                    }
+                }
+            }
+        }
+    }
+
     #[allow(clippy::unused_self)]
     fn infer_ellipsis_literal_expression(
         &mut self,
@@ -1542,6 +1687,7 @@ impl<'db> TypeInferenceBuilder<'db> {
         let left_ty = self.infer_expression(left);
         let right_ty = self.infer_expression(right);
 
+        // TODO flatten the matches by matching on (left_ty, right_ty, op)
         match left_ty {
             Type::Any => Type::Any,
             Type::Unknown => Type::Unknown,
@@ -1580,6 +1726,22 @@ impl<'db> TypeInferenceBuilder<'db> {
                     _ => Type::Unknown, // TODO
                 }
             }
+            Type::BytesLiteral(lhs) => {
+                match right_ty {
+                    Type::BytesLiteral(rhs) => {
+                        match op {
+                            ast::Operator::Add => Type::BytesLiteral(BytesLiteralType::new(
+                                self.db,
+                                [lhs.value(self.db).as_ref(), rhs.value(self.db).as_ref()]
+                                    .concat()
+                                    .into_boxed_slice(),
+                            )),
+                            _ => Type::Unknown, // TODO
+                        }
+                    }
+                    _ => Type::Unknown, // TODO
+                }
+            }
             _ => Type::Unknown, // TODO
         }
     }
@@ -1683,6 +1845,28 @@ impl<'db> TypeInferenceBuilder<'db> {
         }
     }
 
+    /// Adds a new diagnostic.
+    ///
+    /// The diagnostic does not get added if the rule isn't enabled for this file.
+    fn add_diagnostic(&mut self, node: AnyNodeRef, rule: &str, message: std::fmt::Arguments) {
+        if !self.db.is_file_open(self.file) {
+            return;
+        }
+
+        // TODO: Don't emit the diagnostic if:
+        // * The enclosing node contains any syntax errors
+        // * The rule is disabled for this file. We probably want to introduce a new query that
+        //   returns a rule selector for a given file that respects the package's settings,
+        //   any global pragma comments in the file, and any per-file-ignores.
+
+        self.types.diagnostics.push(TypeCheckDiagnostic {
+            file: self.file,
+            rule: rule.to_string(),
+            message: message.to_string(),
+            range: node.range(),
+        });
+    }
+
     pub(super) fn finish(mut self) -> TypeInference<'db> {
         self.infer_region();
         self.types.shrink_to_fit();
@@ -1690,6 +1874,20 @@ impl<'db> TypeInferenceBuilder<'db> {
     }
 }
 
+fn format_import_from_module(level: u32, module: Option<&str>) -> String {
+    format!(
+        "{}{}",
+        ".".repeat(level as usize),
+        module.unwrap_or_default()
+    )
+}
+
+#[derive(Debug, Copy, Clone, PartialEq, Eq)]
+enum ModuleResolutionError {
+    InvalidSyntax,
+    UnresolvedModule,
+}
+
 #[cfg(test)]
 mod tests {
     use anyhow::Context;
@@ -1720,14 +1918,9 @@ mod tests {
 
         Program::from_settings(
             &db,
-            ProgramSettings {
+            &ProgramSettings {
                 target_version: PythonVersion::default(),
-                search_paths: SearchPathSettings {
-                    extra_paths: Vec::new(),
-                    src_root,
-                    site_packages: vec![],
-                    custom_typeshed: None,
-                },
+                search_paths: SearchPathSettings::new(src_root),
             },
         )
         .expect("Valid search path settings");
@@ -1747,13 +1940,11 @@ mod tests {
 
         Program::from_settings(
             &db,
-            ProgramSettings {
+            &ProgramSettings {
                 target_version: PythonVersion::default(),
                 search_paths: SearchPathSettings {
-                    extra_paths: Vec::new(),
-                    src_root,
-                    site_packages: vec![],
                     custom_typeshed: Some(SystemPathBuf::from(typeshed)),
+                    ..SearchPathSettings::new(src_root)
                 },
             },
         )
@@ -1943,6 +2134,16 @@ mod tests {
         Ok(())
     }
 
+    #[test]
+    fn from_import_with_no_module_name() -> anyhow::Result<()> {
+        // This test checks that invalid syntax in a `StmtImportFrom` node
+        // leads to the type being inferred as `Unknown`
+        let mut db = setup_db();
+        db.write_file("src/foo.py", "from import bar")?;
+        assert_public_ty(&db, "src/foo.py", "bar", "Unknown");
+        Ok(())
+    }
+
     #[test]
     fn resolve_base_class_by_name() -> anyhow::Result<()> {
         let mut db = setup_db();
@@ -2054,6 +2255,28 @@ mod tests {
         Ok(())
     }
 
+    #[test]
+    fn bytes_type() -> anyhow::Result<()> {
+        let mut db = setup_db();
+
+        db.write_dedented(
+            "src/a.py",
+            "
+            w = b'red' b'knot'
+            x = b'hello'
+            y = b'world' + b'!'
+            z = b'\\xff\\x00'
+            ",
+        )?;
+
+        assert_public_ty(&db, "src/a.py", "w", "Literal[b\"redknot\"]");
+        assert_public_ty(&db, "src/a.py", "x", "Literal[b\"hello\"]");
+        assert_public_ty(&db, "src/a.py", "y", "Literal[b\"world!\"]");
+        assert_public_ty(&db, "src/a.py", "z", "Literal[b\"\\xff\\x00\"]");
+
+        Ok(())
+    }
+
     #[test]
     fn resolve_union() -> anyhow::Result<()> {
         let mut db = setup_db();
@@ -2109,6 +2332,23 @@ mod tests {
         Ok(())
     }
 
+    #[test]
+    fn walrus_self_plus_one() -> anyhow::Result<()> {
+        let mut db = setup_db();
+
+        db.write_dedented(
+            "src/a.py",
+            "
+            x = 0
+            (x := x + 1)
+            ",
+        )?;
+
+        assert_public_ty(&db, "src/a.py", "x", "Literal[1]");
+
+        Ok(())
+    }
+
     #[test]
     fn ifexpr() -> anyhow::Result<()> {
         let mut db = setup_db();

crates/red_knot_server/Cargo.toml

@@ -11,10 +11,8 @@ repository = { workspace = true }
 license = { workspace = true }
 
 [dependencies]
-red_knot_python_semantic = { workspace = true }
 red_knot_workspace = { workspace = true }
 ruff_db = { workspace = true }
-ruff_linter = { workspace = true }
 ruff_notebook = { workspace = true }
 ruff_python_ast = { workspace = true }
 ruff_source_file = { workspace = true }

crates/red_knot_server/src/server.rs

@@ -3,11 +3,11 @@
 use std::num::NonZeroUsize;
 use std::panic::PanicInfo;
 
-use lsp_server as lsp;
-use lsp_types as types;
+use lsp_server::Message;
 use lsp_types::{
-    ClientCapabilities, DiagnosticOptions, NotebookCellSelector, NotebookDocumentSyncOptions,
-    NotebookSelector, TextDocumentSyncCapability, TextDocumentSyncOptions,
+    ClientCapabilities, DiagnosticOptions, DiagnosticServerCapabilities, MessageType,
+    ServerCapabilities, TextDocumentSyncCapability, TextDocumentSyncKind, TextDocumentSyncOptions,
+    Url,
 };
 
 use self::connection::{Connection, ConnectionInitializer};
@@ -74,7 +74,7 @@ impl Server {
             init_params.client_info.as_ref(),
         );
 
-        let mut workspace_for_url = |url: lsp_types::Url| {
+        let mut workspace_for_url = |url: Url| {
             let Some(workspace_settings) = workspace_settings.as_mut() else {
                 return (url, ClientSettings::default());
             };
@@ -93,13 +93,18 @@ impl Server {
             }).collect())
             .or_else(|| {
                 tracing::warn!("No workspace(s) were provided during initialization. Using the current working directory as a default workspace...");
-                let uri = types::Url::from_file_path(std::env::current_dir().ok()?).ok()?;
+                let uri = Url::from_file_path(std::env::current_dir().ok()?).ok()?;
                 Some(vec![workspace_for_url(uri)])
             })
             .ok_or_else(|| {
                 anyhow::anyhow!("Failed to get the current working directory while creating a default workspace.")
             })?;
 
+        if workspaces.len() > 1 {
+            // TODO(dhruvmanila): Support multi-root workspaces
+            anyhow::bail!("Multi-root workspaces are not supported yet");
+        }
+
         Ok(Self {
             connection,
             worker_threads,
@@ -149,7 +154,7 @@ impl Server {
             try_show_message(
                 "The Ruff language server exited with a panic. See the logs for more details."
                     .to_string(),
-                lsp_types::MessageType::ERROR,
+                MessageType::ERROR,
             )
             .ok();
         }));
@@ -182,9 +187,9 @@ impl Server {
                 break;
             }
             let task = match msg {
-                lsp::Message::Request(req) => api::request(req),
-                lsp::Message::Notification(notification) => api::notification(notification),
-                lsp::Message::Response(response) => scheduler.response(response),
+                Message::Request(req) => api::request(req),
+                Message::Notification(notification) => api::notification(notification),
+                Message::Response(response) => scheduler.response(response),
             };
             scheduler.dispatch(task);
         }
@@ -206,28 +211,17 @@ impl Server {
             .unwrap_or_default()
     }
 
-    fn server_capabilities(position_encoding: PositionEncoding) -> types::ServerCapabilities {
-        types::ServerCapabilities {
+    fn server_capabilities(position_encoding: PositionEncoding) -> ServerCapabilities {
+        ServerCapabilities {
             position_encoding: Some(position_encoding.into()),
-            diagnostic_provider: Some(types::DiagnosticServerCapabilities::Options(
-                DiagnosticOptions {
-                    identifier: Some(crate::DIAGNOSTIC_NAME.into()),
-                    ..Default::default()
-                },
-            )),
-            notebook_document_sync: Some(types::OneOf::Left(NotebookDocumentSyncOptions {
-                save: Some(false),
-                notebook_selector: [NotebookSelector::ByCells {
-                    notebook: None,
-                    cells: vec![NotebookCellSelector {
-                        language: "python".to_string(),
-                    }],
-                }]
-                .to_vec(),
+            diagnostic_provider: Some(DiagnosticServerCapabilities::Options(DiagnosticOptions {
+                identifier: Some(crate::DIAGNOSTIC_NAME.into()),
+                ..Default::default()
             })),
             text_document_sync: Some(TextDocumentSyncCapability::Options(
                 TextDocumentSyncOptions {
                     open_close: Some(true),
+                    change: Some(TextDocumentSyncKind::INCREMENTAL),
                     ..Default::default()
                 },
             )),

crates/red_knot_server/src/server/api.rs

@@ -1,13 +1,15 @@
-use crate::{server::schedule::Task, session::Session, system::url_to_system_path};
 use lsp_server as server;
 
+use crate::server::schedule::Task;
+use crate::session::Session;
+use crate::system::{url_to_any_system_path, AnySystemPath};
+
 mod diagnostics;
 mod notifications;
 mod requests;
 mod traits;
 
 use notifications as notification;
-use red_knot_workspace::db::RootDatabase;
 use requests as request;
 
 use self::traits::{NotificationHandler, RequestHandler};
@@ -43,6 +45,7 @@ pub(super) fn notification<'a>(notif: server::Notification) -> Task<'a> {
     match notif.method.as_str() {
         notification::DidCloseTextDocumentHandler::METHOD => local_notification_task::<notification::DidCloseTextDocumentHandler>(notif),
         notification::DidOpenTextDocumentHandler::METHOD => local_notification_task::<notification::DidOpenTextDocumentHandler>(notif),
+        notification::DidChangeTextDocumentHandler::METHOD => local_notification_task::<notification::DidChangeTextDocumentHandler>(notif),
         notification::DidOpenNotebookHandler::METHOD => {
             local_notification_task::<notification::DidOpenNotebookHandler>(notif)
         }
@@ -82,12 +85,18 @@ fn background_request_task<'a, R: traits::BackgroundDocumentRequestHandler>(
     Ok(Task::background(schedule, move |session: &Session| {
         let url = R::document_url(&params).into_owned();
 
-        let Ok(path) = url_to_system_path(&url) else {
+        let Ok(path) = url_to_any_system_path(&url) else {
             return Box::new(|_, _| {});
         };
-        let db = session
-            .workspace_db_for_path(path.as_std_path())
-            .map(RootDatabase::snapshot);
+        let db = match path {
+            AnySystemPath::System(path) => {
+                match session.workspace_db_for_path(path.as_std_path()) {
+                    Some(db) => db.snapshot(),
+                    None => session.default_workspace_db().snapshot(),
+                }
+            }
+            AnySystemPath::SystemVirtual(_) => session.default_workspace_db().snapshot(),
+        };
 
         let Some(snapshot) = session.take_snapshot(url) else {
             return Box::new(|_, _| {});

crates/red_knot_server/src/server/api/notifications.rs

@@ -1,9 +1,11 @@
+mod did_change;
 mod did_close;
 mod did_close_notebook;
 mod did_open;
 mod did_open_notebook;
 mod set_trace;
 
+pub(super) use did_change::DidChangeTextDocumentHandler;
 pub(super) use did_close::DidCloseTextDocumentHandler;
 pub(super) use did_close_notebook::DidCloseNotebookHandler;
 pub(super) use did_open::DidOpenTextDocumentHandler;

crates/red_knot_server/src/server/api/notifications/did_change.rs

@@ -0,0 +1,55 @@
+use lsp_server::ErrorCode;
+use lsp_types::notification::DidChangeTextDocument;
+use lsp_types::DidChangeTextDocumentParams;
+
+use red_knot_workspace::watch::ChangeEvent;
+
+use crate::server::api::traits::{NotificationHandler, SyncNotificationHandler};
+use crate::server::api::LSPResult;
+use crate::server::client::{Notifier, Requester};
+use crate::server::Result;
+use crate::session::Session;
+use crate::system::{url_to_any_system_path, AnySystemPath};
+
+pub(crate) struct DidChangeTextDocumentHandler;
+
+impl NotificationHandler for DidChangeTextDocumentHandler {
+    type NotificationType = DidChangeTextDocument;
+}
+
+impl SyncNotificationHandler for DidChangeTextDocumentHandler {
+    fn run(
+        session: &mut Session,
+        _notifier: Notifier,
+        _requester: &mut Requester,
+        params: DidChangeTextDocumentParams,
+    ) -> Result<()> {
+        let Ok(path) = url_to_any_system_path(&params.text_document.uri) else {
+            return Ok(());
+        };
+
+        let key = session.key_from_url(params.text_document.uri);
+
+        session
+            .update_text_document(&key, params.content_changes, params.text_document.version)
+            .with_failure_code(ErrorCode::InternalError)?;
+
+        match path {
+            AnySystemPath::System(path) => {
+                let db = match session.workspace_db_for_path_mut(path.as_std_path()) {
+                    Some(db) => db,
+                    None => session.default_workspace_db_mut(),
+                };
+                db.apply_changes(vec![ChangeEvent::file_content_changed(path)], None);
+            }
+            AnySystemPath::SystemVirtual(virtual_path) => {
+                let db = session.default_workspace_db_mut();
+                db.apply_changes(vec![ChangeEvent::ChangedVirtual(virtual_path)], None);
+            }
+        }
+
+        // TODO(dhruvmanila): Publish diagnostics if the client doesnt support pull diagnostics
+
+        Ok(())
+    }
+}

crates/red_knot_server/src/server/api/notifications/did_close.rs

@@ -1,8 +1,7 @@
 use lsp_server::ErrorCode;
 use lsp_types::notification::DidCloseTextDocument;
 use lsp_types::DidCloseTextDocumentParams;
-
-use ruff_db::files::File;
+use red_knot_workspace::watch::ChangeEvent;
 
 use crate::server::api::diagnostics::clear_diagnostics;
 use crate::server::api::traits::{NotificationHandler, SyncNotificationHandler};
@@ -10,7 +9,7 @@ use crate::server::api::LSPResult;
 use crate::server::client::{Notifier, Requester};
 use crate::server::Result;
 use crate::session::Session;
-use crate::system::url_to_system_path;
+use crate::system::{url_to_any_system_path, AnySystemPath};
 
 pub(crate) struct DidCloseTextDocumentHandler;
 
@@ -25,7 +24,7 @@ impl SyncNotificationHandler for DidCloseTextDocumentHandler {
         _requester: &mut Requester,
         params: DidCloseTextDocumentParams,
     ) -> Result<()> {
-        let Ok(path) = url_to_system_path(&params.text_document.uri) else {
+        let Ok(path) = url_to_any_system_path(&params.text_document.uri) else {
             return Ok(());
         };
 
@@ -34,8 +33,9 @@ impl SyncNotificationHandler for DidCloseTextDocumentHandler {
             .close_document(&key)
             .with_failure_code(ErrorCode::InternalError)?;
 
-        if let Some(db) = session.workspace_db_for_path_mut(path.as_std_path()) {
-            File::sync_path(db, &path);
+        if let AnySystemPath::SystemVirtual(virtual_path) = path {
+            let db = session.default_workspace_db_mut();
+            db.apply_changes(vec![ChangeEvent::DeletedVirtual(virtual_path)], None);
         }
 
         clear_diagnostics(key.url(), &notifier)?;

crates/red_knot_server/src/server/api/notifications/did_close_notebook.rs

@@ -1,14 +1,14 @@
 use lsp_types::notification::DidCloseNotebookDocument;
 use lsp_types::DidCloseNotebookDocumentParams;
 
-use ruff_db::files::File;
+use red_knot_workspace::watch::ChangeEvent;
 
 use crate::server::api::traits::{NotificationHandler, SyncNotificationHandler};
 use crate::server::api::LSPResult;
 use crate::server::client::{Notifier, Requester};
 use crate::server::Result;
 use crate::session::Session;
-use crate::system::url_to_system_path;
+use crate::system::{url_to_any_system_path, AnySystemPath};
 
 pub(crate) struct DidCloseNotebookHandler;
 
@@ -23,7 +23,7 @@ impl SyncNotificationHandler for DidCloseNotebookHandler {
         _requester: &mut Requester,
         params: DidCloseNotebookDocumentParams,
     ) -> Result<()> {
-        let Ok(path) = url_to_system_path(&params.notebook_document.uri) else {
+        let Ok(path) = url_to_any_system_path(&params.notebook_document.uri) else {
             return Ok(());
         };
 
@@ -32,8 +32,9 @@ impl SyncNotificationHandler for DidCloseNotebookHandler {
             .close_document(&key)
             .with_failure_code(lsp_server::ErrorCode::InternalError)?;
 
-        if let Some(db) = session.workspace_db_for_path_mut(path.as_std_path()) {
-            File::sync_path(db, &path);
+        if let AnySystemPath::SystemVirtual(virtual_path) = path {
+            let db = session.default_workspace_db_mut();
+            db.apply_changes(vec![ChangeEvent::DeletedVirtual(virtual_path)], None);
         }
 
         Ok(())

crates/red_knot_server/src/server/api/notifications/did_open.rs

@@ -1,13 +1,14 @@
 use lsp_types::notification::DidOpenTextDocument;
 use lsp_types::DidOpenTextDocumentParams;
 
-use ruff_db::files::system_path_to_file;
+use red_knot_workspace::watch::ChangeEvent;
+use ruff_db::Db;
 
 use crate::server::api::traits::{NotificationHandler, SyncNotificationHandler};
 use crate::server::client::{Notifier, Requester};
 use crate::server::Result;
 use crate::session::Session;
-use crate::system::url_to_system_path;
+use crate::system::{url_to_any_system_path, AnySystemPath};
 use crate::TextDocument;
 
 pub(crate) struct DidOpenTextDocumentHandler;
@@ -23,17 +24,25 @@ impl SyncNotificationHandler for DidOpenTextDocumentHandler {
         _requester: &mut Requester,
         params: DidOpenTextDocumentParams,
     ) -> Result<()> {
-        let Ok(path) = url_to_system_path(&params.text_document.uri) else {
+        let Ok(path) = url_to_any_system_path(&params.text_document.uri) else {
             return Ok(());
         };
 
         let document = TextDocument::new(params.text_document.text, params.text_document.version);
         session.open_text_document(params.text_document.uri, document);
 
-        if let Some(db) = session.workspace_db_for_path_mut(path.as_std_path()) {
-            // TODO(dhruvmanila): Store the `file` in `DocumentController`
-            let file = system_path_to_file(db, &path).unwrap();
-            file.sync(db);
+        match path {
+            AnySystemPath::System(path) => {
+                let db = match session.workspace_db_for_path_mut(path.as_std_path()) {
+                    Some(db) => db,
+                    None => session.default_workspace_db_mut(),
+                };
+                db.apply_changes(vec![ChangeEvent::Opened(path)], None);
+            }
+            AnySystemPath::SystemVirtual(virtual_path) => {
+                let db = session.default_workspace_db_mut();
+                db.files().virtual_file(db, &virtual_path);
+            }
         }
 
         // TODO(dhruvmanila): Publish diagnostics if the client doesn't support pull diagnostics

crates/red_knot_server/src/server/api/notifications/did_open_notebook.rs

@@ -2,7 +2,8 @@ use lsp_server::ErrorCode;
 use lsp_types::notification::DidOpenNotebookDocument;
 use lsp_types::DidOpenNotebookDocumentParams;
 
-use ruff_db::files::system_path_to_file;
+use red_knot_workspace::watch::ChangeEvent;
+use ruff_db::Db;
 
 use crate::edit::NotebookDocument;
 use crate::server::api::traits::{NotificationHandler, SyncNotificationHandler};
@@ -10,7 +11,7 @@ use crate::server::api::LSPResult;
 use crate::server::client::{Notifier, Requester};
 use crate::server::Result;
 use crate::session::Session;
-use crate::system::url_to_system_path;
+use crate::system::{url_to_any_system_path, AnySystemPath};
 
 pub(crate) struct DidOpenNotebookHandler;
 
@@ -25,7 +26,7 @@ impl SyncNotificationHandler for DidOpenNotebookHandler {
         _requester: &mut Requester,
         params: DidOpenNotebookDocumentParams,
     ) -> Result<()> {
-        let Ok(path) = url_to_system_path(&params.notebook_document.uri) else {
+        let Ok(path) = url_to_any_system_path(&params.notebook_document.uri) else {
             return Ok(());
         };
 
@@ -38,10 +39,18 @@ impl SyncNotificationHandler for DidOpenNotebookHandler {
         .with_failure_code(ErrorCode::InternalError)?;
         session.open_notebook_document(params.notebook_document.uri.clone(), notebook);
 
-        if let Some(db) = session.workspace_db_for_path_mut(path.as_std_path()) {
-            // TODO(dhruvmanila): Store the `file` in `DocumentController`
-            let file = system_path_to_file(db, &path).unwrap();
-            file.sync(db);
+        match path {
+            AnySystemPath::System(path) => {
+                let db = match session.workspace_db_for_path_mut(path.as_std_path()) {
+                    Some(db) => db,
+                    None => session.default_workspace_db_mut(),
+                };
+                db.apply_changes(vec![ChangeEvent::Opened(path)], None);
+            }
+            AnySystemPath::SystemVirtual(virtual_path) => {
+                let db = session.default_workspace_db_mut();
+                db.files().virtual_file(db, &virtual_path);
+            }
         }
 
         // TODO(dhruvmanila): Publish diagnostics if the client doesn't support pull diagnostics

crates/red_knot_server/src/server/api/requests/diagnostic.rs

@@ -26,13 +26,11 @@ impl BackgroundDocumentRequestHandler for DocumentDiagnosticRequestHandler {
 
     fn run_with_snapshot(
         snapshot: DocumentSnapshot,
-        db: Option<RootDatabase>,
+        db: RootDatabase,
         _notifier: Notifier,
         _params: DocumentDiagnosticParams,
     ) -> Result<DocumentDiagnosticReportResult> {
-        let diagnostics = db
-            .map(|db| compute_diagnostics(&snapshot, &db))
-            .unwrap_or_default();
+        let diagnostics = compute_diagnostics(&snapshot, &db);
 
         Ok(DocumentDiagnosticReportResult::Report(
             DocumentDiagnosticReport::Full(RelatedFullDocumentDiagnosticReport {
@@ -48,10 +46,19 @@ impl BackgroundDocumentRequestHandler for DocumentDiagnosticRequestHandler {
 
 fn compute_diagnostics(snapshot: &DocumentSnapshot, db: &RootDatabase) -> Vec<Diagnostic> {
     let Some(file) = snapshot.file(db) else {
+        tracing::info!(
+            "No file found for snapshot for '{}'",
+            snapshot.query().file_url()
+        );
         return vec![];
     };
-    let Ok(diagnostics) = db.check_file(file) else {
-        return vec![];
+
+    let diagnostics = match db.check_file(file) {
+        Ok(diagnostics) => diagnostics,
+        Err(cancelled) => {
+            tracing::info!("Diagnostics computation {cancelled}");
+            return vec![];
+        }
     };
 
     diagnostics
@@ -65,12 +72,12 @@ fn to_lsp_diagnostic(message: &str) -> Diagnostic {
     let words = message.split(':').collect::<Vec<_>>();
 
     let (range, message) = match words.as_slice() {
-        [_filename, line, column, message] => {
-            let line = line.parse::<u32>().unwrap_or_default();
+        [_, _, line, column, message] | [_, line, column, message] => {
+            let line = line.parse::<u32>().unwrap_or_default().saturating_sub(1);
             let column = column.parse::<u32>().unwrap_or_default();
             (
                 Range::new(
-                    Position::new(line.saturating_sub(1), column.saturating_sub(1)),
+                    Position::new(line, column.saturating_sub(1)),
                     Position::new(line, column),
                 ),
                 message.trim(),

crates/red_knot_server/src/server/api/traits.rs

@@ -34,7 +34,7 @@ pub(super) trait BackgroundDocumentRequestHandler: RequestHandler {
 
     fn run_with_snapshot(
         snapshot: DocumentSnapshot,
-        db: Option<RootDatabase>,
+        db: RootDatabase,
         notifier: Notifier,
         params: <<Self as RequestHandler>::RequestType as Request>::Params,
     ) -> super::Result<<<Self as RequestHandler>::RequestType as Request>::Result>;

crates/red_knot_server/src/session.rs

@@ -6,16 +6,16 @@ use std::path::{Path, PathBuf};
 use std::sync::Arc;
 
 use anyhow::anyhow;
-use lsp_types::{ClientCapabilities, Url};
+use lsp_types::{ClientCapabilities, TextDocumentContentChangeEvent, Url};
 
-use red_knot_python_semantic::{ProgramSettings, PythonVersion, SearchPathSettings};
 use red_knot_workspace::db::RootDatabase;
 use red_knot_workspace::workspace::WorkspaceMetadata;
 use ruff_db::files::{system_path_to_file, File};
 use ruff_db::system::SystemPath;
+use ruff_db::Db;
 
-use crate::edit::{DocumentKey, NotebookDocument};
-use crate::system::{url_to_system_path, LSPSystem};
+use crate::edit::{DocumentKey, DocumentVersion, NotebookDocument};
+use crate::system::{url_to_any_system_path, AnySystemPath, LSPSystem};
 use crate::{PositionEncoding, TextDocument};
 
 pub(crate) use self::capabilities::ResolvedClientCapabilities;
@@ -67,19 +67,10 @@ impl Session {
                 .ok_or_else(|| anyhow!("Workspace path is not a valid UTF-8 path: {:?}", path))?;
             let system = LSPSystem::new(index.clone());
 
-            let metadata = WorkspaceMetadata::from_path(system_path, &system)?;
             // TODO(dhruvmanila): Get the values from the client settings
-            let program_settings = ProgramSettings {
-                target_version: PythonVersion::default(),
-                search_paths: SearchPathSettings {
-                    extra_paths: vec![],
-                    src_root: system_path.to_path_buf(),
-                    site_packages: vec![],
-                    custom_typeshed: None,
-                },
-            };
+            let metadata = WorkspaceMetadata::from_path(system_path, &system, None)?;
             // TODO(micha): Handle the case where the program settings are incorrect more gracefully.
-            workspaces.insert(path, RootDatabase::new(metadata, program_settings, system)?);
+            workspaces.insert(path, RootDatabase::new(metadata, system)?);
         }
 
         Ok(Self {
@@ -92,6 +83,12 @@ impl Session {
         })
     }
 
+    // TODO(dhruvmanila): Ideally, we should have a single method for `workspace_db_for_path_mut`
+    // and `default_workspace_db_mut` but the borrow checker doesn't allow that.
+    // https://github.com/astral-sh/ruff/pull/13041#discussion_r1726725437
+
+    /// Returns a reference to the workspace [`RootDatabase`] corresponding to the given path, if
+    /// any.
     pub(crate) fn workspace_db_for_path(&self, path: impl AsRef<Path>) -> Option<&RootDatabase> {
         self.workspaces
             .range(..=path.as_ref().to_path_buf())
@@ -99,6 +96,8 @@ impl Session {
             .map(|(_, db)| db)
     }
 
+    /// Returns a mutable reference to the workspace [`RootDatabase`] corresponding to the given
+    /// path, if any.
     pub(crate) fn workspace_db_for_path_mut(
         &mut self,
         path: impl AsRef<Path>,
@@ -109,6 +108,19 @@ impl Session {
             .map(|(_, db)| db)
     }
 
+    /// Returns a reference to the default workspace [`RootDatabase`]. The default workspace is the
+    /// minimum root path in the workspace map.
+    pub(crate) fn default_workspace_db(&self) -> &RootDatabase {
+        // SAFETY: Currently, red knot only support a single workspace.
+        self.workspaces.values().next().unwrap()
+    }
+
+    /// Returns a mutable reference to the default workspace [`RootDatabase`].
+    pub(crate) fn default_workspace_db_mut(&mut self) -> &mut RootDatabase {
+        // SAFETY: Currently, red knot only support a single workspace.
+        self.workspaces.values_mut().next().unwrap()
+    }
+
     pub fn key_from_url(&self, url: Url) -> DocumentKey {
         self.index().key_from_url(url)
     }
@@ -135,6 +147,20 @@ impl Session {
         self.index_mut().open_text_document(url, document);
     }
 
+    /// Updates a text document at the associated `key`.
+    ///
+    /// The document key must point to a text document, or this will throw an error.
+    pub(crate) fn update_text_document(
+        &mut self,
+        key: &DocumentKey,
+        content_changes: Vec<TextDocumentContentChangeEvent>,
+        new_version: DocumentVersion,
+    ) -> crate::Result<()> {
+        let position_encoding = self.position_encoding;
+        self.index_mut()
+            .update_text_document(key, content_changes, new_version, position_encoding)
+    }
+
     /// De-registers a document, specified by its key.
     /// Calling this multiple times for the same document is a logic error.
     pub(crate) fn close_document(&mut self, key: &DocumentKey) -> crate::Result<()> {
@@ -221,6 +247,7 @@ impl Drop for MutIndexGuard<'_> {
 
 /// An immutable snapshot of `Session` that references
 /// a specific document.
+#[derive(Debug)]
 pub struct DocumentSnapshot {
     resolved_client_capabilities: Arc<ResolvedClientCapabilities>,
     document_ref: index::DocumentQuery,
@@ -241,7 +268,12 @@ impl DocumentSnapshot {
     }
 
     pub(crate) fn file(&self, db: &RootDatabase) -> Option<File> {
-        let path = url_to_system_path(self.document_ref.file_url()).ok()?;
-        system_path_to_file(db, path).ok()
+        match url_to_any_system_path(self.document_ref.file_url()).ok()? {
+            AnySystemPath::System(path) => system_path_to_file(db, path).ok(),
+            AnySystemPath::SystemVirtual(virtual_path) => db
+                .files()
+                .try_virtual_file(&virtual_path)
+                .map(|virtual_file| virtual_file.file()),
+        }
     }
 }

crates/red_knot_server/src/session/capabilities.rs

@@ -1,5 +1,4 @@
 use lsp_types::ClientCapabilities;
-use ruff_linter::display_settings;
 
 #[derive(Debug, Clone, PartialEq, Eq, Default)]
 #[allow(clippy::struct_excessive_bools)]
@@ -66,20 +65,3 @@ impl ResolvedClientCapabilities {
         }
     }
 }
-
-impl std::fmt::Display for ResolvedClientCapabilities {
-    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
-        display_settings! {
-            formatter = f,
-            namespace = "capabilities",
-            fields = [
-                self.code_action_deferred_edit_resolution,
-                self.apply_edit,
-                self.document_changes,
-                self.workspace_refresh,
-                self.pull_diagnostics,
-            ]
-        };
-        Ok(())
-    }
-}

crates/red_knot_server/src/session/index.rs

@@ -278,18 +278,6 @@ impl DocumentQuery {
         }
     }
 
-    /// Generate a source kind used by the linter.
-    pub(crate) fn make_source_kind(&self) -> ruff_linter::source_kind::SourceKind {
-        match self {
-            Self::Text { document, .. } => {
-                ruff_linter::source_kind::SourceKind::Python(document.contents().to_string())
-            }
-            Self::Notebook { notebook, .. } => {
-                ruff_linter::source_kind::SourceKind::IpyNotebook(notebook.make_ruff_notebook())
-            }
-        }
-    }
-
     /// Attempts to access the underlying notebook document that this query is selecting.
     pub fn as_notebook(&self) -> Option<&NotebookDocument> {
         match self {

crates/red_knot_server/src/system.rs

@@ -8,27 +8,40 @@ use ruff_db::file_revision::FileRevision;
 use ruff_db::system::walk_directory::WalkDirectoryBuilder;
 use ruff_db::system::{
     DirectoryEntry, FileType, Metadata, OsSystem, Result, System, SystemPath, SystemPathBuf,
-    SystemVirtualPath,
+    SystemVirtualPath, SystemVirtualPathBuf,
 };
 use ruff_notebook::{Notebook, NotebookError};
 
 use crate::session::index::Index;
 use crate::DocumentQuery;
 
-/// Converts the given [`Url`] to a [`SystemPathBuf`].
+/// Converts the given [`Url`] to an [`AnySystemPath`].
+///
+/// If the URL scheme is `file`, then the path is converted to a [`SystemPathBuf`]. Otherwise, the
+/// URL is converted to a [`SystemVirtualPathBuf`].
 ///
 /// This fails in the following cases:
-/// * The URL scheme is not `file`.
 /// * The URL cannot be converted to a file path (refer to [`Url::to_file_path`]).
 /// * If the URL is not a valid UTF-8 string.
-pub(crate) fn url_to_system_path(url: &Url) -> std::result::Result<SystemPathBuf, ()> {
+pub(crate) fn url_to_any_system_path(url: &Url) -> std::result::Result<AnySystemPath, ()> {
     if url.scheme() == "file" {
-        Ok(SystemPathBuf::from_path_buf(url.to_file_path()?).map_err(|_| ())?)
+        Ok(AnySystemPath::System(
+            SystemPathBuf::from_path_buf(url.to_file_path()?).map_err(|_| ())?,
+        ))
     } else {
-        Err(())
+        Ok(AnySystemPath::SystemVirtual(
+            SystemVirtualPath::new(url.as_str()).to_path_buf(),
+        ))
     }
 }
 
+/// Represents either a [`SystemPath`] or a [`SystemVirtualPath`].
+#[derive(Debug)]
+pub(crate) enum AnySystemPath {
+    System(SystemPathBuf),
+    SystemVirtual(SystemVirtualPathBuf),
+}
+
 #[derive(Debug)]
 pub(crate) struct LSPSystem {
     /// A read-only copy of the index where the server stores all the open documents and settings.
@@ -144,19 +157,6 @@ impl System for LSPSystem {
         }
     }
 
-    fn virtual_path_metadata(&self, path: &SystemVirtualPath) -> Result<Metadata> {
-        // Virtual paths only exists in the LSP system, so we don't need to check the OS system.
-        let document = self
-            .system_virtual_path_to_document_ref(path)?
-            .ok_or_else(|| virtual_path_not_found(path))?;
-
-        Ok(Metadata::new(
-            document_revision(&document),
-            None,
-            FileType::File,
-        ))
-    }
-
     fn read_virtual_path_to_string(&self, path: &SystemVirtualPath) -> Result<String> {
         let document = self
             .system_virtual_path_to_document_ref(path)?

crates/red_knot_wasm/src/lib.rs

@@ -3,8 +3,8 @@ use std::any::Any;
 use js_sys::Error;
 use wasm_bindgen::prelude::*;
 
-use red_knot_python_semantic::{ProgramSettings, SearchPathSettings};
 use red_knot_workspace::db::RootDatabase;
+use red_knot_workspace::workspace::settings::Configuration;
 use red_knot_workspace::workspace::WorkspaceMetadata;
 use ruff_db::files::{system_path_to_file, File};
 use ruff_db::system::walk_directory::WalkDirectoryBuilder;
@@ -41,16 +41,17 @@ impl Workspace {
     #[wasm_bindgen(constructor)]
     pub fn new(root: &str, settings: &Settings) -> Result<Workspace, Error> {
         let system = WasmSystem::new(SystemPath::new(root));
-        let workspace =
-            WorkspaceMetadata::from_path(SystemPath::new(root), &system).map_err(into_error)?;
+        let workspace = WorkspaceMetadata::from_path(
+            SystemPath::new(root),
+            &system,
+            Some(Configuration {
+                target_version: Some(settings.target_version.into()),
+                ..Configuration::default()
+            }),
+        )
+        .map_err(into_error)?;
 
-        let program_settings = ProgramSettings {
-            target_version: settings.target_version.into(),
-            search_paths: SearchPathSettings::default(),
-        };
-
-        let db =
-            RootDatabase::new(workspace, program_settings, system.clone()).map_err(into_error)?;
+        let db = RootDatabase::new(workspace, system.clone()).map_err(into_error)?;
 
         Ok(Self { db, system })
     }
@@ -109,7 +110,7 @@ impl Workspace {
     pub fn check_file(&self, file_id: &FileHandle) -> Result<Vec<String>, Error> {
         let result = self.db.check_file(file_id.file).map_err(into_error)?;
 
-        Ok(result.to_vec())
+        Ok(result.clone())
     }
 
     /// Checks all open files
@@ -233,13 +234,6 @@ impl System for WasmSystem {
         Notebook::from_source_code(&content)
     }
 
-    fn virtual_path_metadata(
-        &self,
-        _path: &SystemVirtualPath,
-    ) -> ruff_db::system::Result<Metadata> {
-        Err(not_found())
-    }
-
     fn read_virtual_path_to_string(
         &self,
         _path: &SystemVirtualPath,

crates/red_knot_wasm/tests/api.rs

@@ -17,5 +17,8 @@ fn check() {
 
     let result = workspace.check_file(&test).expect("Check to succeed");
 
-    assert_eq!(result, vec!["/test.py:1:8: Unresolved import 'random22'"]);
+    assert_eq!(
+        result,
+        vec!["/test.py:1:8: Import 'random22' could not be resolved.",]
+    );
 }

crates/red_knot_workspace/Cargo.toml

@@ -24,10 +24,10 @@ crossbeam = { workspace = true }
 notify = { workspace = true }
 rustc-hash = { workspace = true }
 salsa = { workspace = true }
-thiserror = { workspace = true }
 tracing = { workspace = true }
 
 [dev-dependencies]
+ruff_db = { workspace = true, features = ["testing"]}
 
 [lints]
 workspace = true

crates/red_knot_workspace/resources/test/corpus/04_assign_named_expr.py

@@ -0,0 +1,2 @@
+x = 0
+(x := x + 1)

crates/red_knot_workspace/resources/test/corpus/10_if_with_named_expr.py

@@ -0,0 +1,3 @@
+x = 0
+if x := x + 1:
+    pass

crates/red_knot_workspace/resources/test/corpus/85_match_guard_with_named_expr.py

@@ -0,0 +1,3 @@
+match x:
+    case [1, 0] if x := x[:0]:
+        y = 1

crates/red_knot_workspace/src/db.rs

@@ -1,17 +1,15 @@
 use std::panic::RefUnwindSafe;
 use std::sync::Arc;
 
-use red_knot_python_semantic::{
-    vendored_typeshed_stubs, Db as SemanticDb, Program, ProgramSettings,
-};
+use salsa::plumbing::ZalsaDatabase;
+use salsa::{Cancelled, Event};
+
+use red_knot_python_semantic::{vendored_typeshed_stubs, Db as SemanticDb, Program};
 use ruff_db::files::{File, Files};
 use ruff_db::system::System;
 use ruff_db::vendored::VendoredFileSystem;
 use ruff_db::{Db as SourceDb, Upcast};
-use salsa::plumbing::ZalsaDatabase;
-use salsa::{Cancelled, Event};
 
-use crate::lint::Diagnostics;
 use crate::workspace::{check_file, Workspace, WorkspaceMetadata};
 
 mod changes;
@@ -28,11 +26,7 @@ pub struct RootDatabase {
 }
 
 impl RootDatabase {
-    pub fn new<S>(
-        workspace: WorkspaceMetadata,
-        settings: ProgramSettings,
-        system: S,
-    ) -> anyhow::Result<Self>
+    pub fn new<S>(workspace: WorkspaceMetadata, system: S) -> anyhow::Result<Self>
     where
         S: System + 'static + Send + Sync + RefUnwindSafe,
     {
@@ -43,11 +37,11 @@ impl RootDatabase {
             system: Arc::new(system),
         };
 
-        let workspace = Workspace::from_metadata(&db, workspace);
         // Initialize the `Program` singleton
-        Program::from_settings(&db, settings)?;
+        Program::from_settings(&db, workspace.settings().program())?;
+
+        db.workspace = Some(Workspace::from_metadata(&db, workspace));
 
-        db.workspace = Some(workspace);
         Ok(db)
     }
 
@@ -61,7 +55,7 @@ impl RootDatabase {
         self.with_db(|db| db.workspace().check(db))
     }
 
-    pub fn check_file(&self, file: File) -> Result<Diagnostics, Cancelled> {
+    pub fn check_file(&self, file: File) -> Result<Vec<String>, Cancelled> {
         self.with_db(|db| check_file(db, file))
     }
 
@@ -115,7 +109,15 @@ impl Upcast<dyn SourceDb> for RootDatabase {
 }
 
 #[salsa::db]
-impl SemanticDb for RootDatabase {}
+impl SemanticDb for RootDatabase {
+    fn is_file_open(&self, file: File) -> bool {
+        let Some(workspace) = &self.workspace else {
+            return false;
+        };
+
+        workspace.is_file_open(self, file)
+    }
+}
 
 #[salsa::db]
 impl SourceDb for RootDatabase {
@@ -153,9 +155,10 @@ impl Db for RootDatabase {}
 
 #[cfg(test)]
 pub(crate) mod tests {
-    use salsa::Event;
     use std::sync::Arc;
 
+    use salsa::Event;
+
     use red_knot_python_semantic::{vendored_typeshed_stubs, Db as SemanticDb};
     use ruff_db::files::Files;
     use ruff_db::system::{DbWithTestSystem, System, TestSystem};
@@ -242,7 +245,12 @@ pub(crate) mod tests {
     }
 
     #[salsa::db]
-    impl red_knot_python_semantic::Db for TestDb {}
+    impl red_knot_python_semantic::Db for TestDb {
+        fn is_file_open(&self, file: ruff_db::files::File) -> bool {
+            !file.path(self).is_vendored_path()
+        }
+    }
+
     #[salsa::db]
     impl Db for TestDb {}
 

crates/red_knot_workspace/src/db/changes.rs

@@ -1,22 +1,33 @@
-use rustc_hash::FxHashSet;
-
+use red_knot_python_semantic::Program;
 use ruff_db::files::{system_path_to_file, File, Files};
 use ruff_db::system::walk_directory::WalkState;
 use ruff_db::system::SystemPath;
 use ruff_db::Db;
+use rustc_hash::FxHashSet;
 
 use crate::db::RootDatabase;
 use crate::watch;
 use crate::watch::{CreatedKind, DeletedKind};
+use crate::workspace::settings::Configuration;
 use crate::workspace::WorkspaceMetadata;
 
 impl RootDatabase {
-    #[tracing::instrument(level = "debug", skip(self, changes))]
-    pub fn apply_changes(&mut self, changes: Vec<watch::ChangeEvent>) {
+    #[tracing::instrument(level = "debug", skip(self, changes, base_configuration))]
+    pub fn apply_changes(
+        &mut self,
+        changes: Vec<watch::ChangeEvent>,
+        base_configuration: Option<&Configuration>,
+    ) {
         let workspace = self.workspace();
         let workspace_path = workspace.root(self).to_path_buf();
+        let program = Program::get(self);
+        let custom_stdlib_versions_path = program
+            .custom_stdlib_search_path(self)
+            .map(|path| path.join("VERSIONS"));
 
         let mut workspace_change = false;
+        // Changes to a custom stdlib path's VERSIONS
+        let mut custom_stdlib_change = false;
         // Packages that need reloading
         let mut changed_packages = FxHashSet::default();
         // Paths that were added
@@ -39,7 +50,7 @@ impl RootDatabase {
         };
 
         for change in changes {
-            if let Some(path) = change.path() {
+            if let Some(path) = change.system_path() {
                 if matches!(
                     path.file_name(),
                     Some(".gitignore" | ".ignore" | "ruff.toml" | ".ruff.toml" | "pyproject.toml")
@@ -54,10 +65,15 @@ impl RootDatabase {
 
                     continue;
                 }
+
+                if Some(path) == custom_stdlib_versions_path.as_deref() {
+                    custom_stdlib_change = true;
+                }
             }
 
             match change {
-                watch::ChangeEvent::Changed { path, kind: _ } => sync_path(self, &path),
+                watch::ChangeEvent::Changed { path, kind: _ }
+                | watch::ChangeEvent::Opened(path) => sync_path(self, &path),
 
                 watch::ChangeEvent::Created { kind, path } => {
                     match kind {
@@ -100,7 +116,13 @@ impl RootDatabase {
                     } else {
                         sync_recursively(self, &path);
 
-                        // TODO: Remove after converting `package.files()` to a salsa query.
+                        if custom_stdlib_versions_path
+                            .as_ref()
+                            .is_some_and(|versions_path| versions_path.starts_with(&path))
+                        {
+                            custom_stdlib_change = true;
+                        }
+
                         if let Some(package) = workspace.package(self, &path) {
                             changed_packages.insert(package);
                         } else {
@@ -109,6 +131,17 @@ impl RootDatabase {
                     }
                 }
 
+                watch::ChangeEvent::CreatedVirtual(path)
+                | watch::ChangeEvent::ChangedVirtual(path) => {
+                    File::sync_virtual_path(self, &path);
+                }
+
+                watch::ChangeEvent::DeletedVirtual(path) => {
+                    if let Some(virtual_file) = self.files().try_virtual_file(&path) {
+                        virtual_file.close(self);
+                    }
+                }
+
                 watch::ChangeEvent::Rescan => {
                     workspace_change = true;
                     Files::sync_all(self);
@@ -118,9 +151,13 @@ impl RootDatabase {
         }
 
         if workspace_change {
-            match WorkspaceMetadata::from_path(&workspace_path, self.system()) {
+            match WorkspaceMetadata::from_path(
+                &workspace_path,
+                self.system(),
+                base_configuration.cloned(),
+            ) {
                 Ok(metadata) => {
-                    tracing::debug!("Reload workspace after structural change.");
+                    tracing::debug!("Reloading workspace after structural change.");
                     // TODO: Handle changes in the program settings.
                     workspace.reload(self, metadata);
                 }
@@ -130,6 +167,11 @@ impl RootDatabase {
             }
 
             return;
+        } else if custom_stdlib_change {
+            let search_paths = workspace.search_path_settings(self).clone();
+            if let Err(error) = program.update_search_paths(self, &search_paths) {
+                tracing::error!("Failed to set the new search paths: {error}");
+            }
         }
 
         let mut added_paths = added_paths.into_iter().filter(|path| {

crates/red_knot_workspace/src/lib.rs

@@ -1,5 +1,4 @@
 pub mod db;
 pub mod lint;
-pub mod site_packages;
 pub mod watch;
 pub mod workspace;

crates/red_knot_workspace/src/lint.rs

@@ -1,5 +1,4 @@
 use std::cell::RefCell;
-use std::ops::Deref;
 use std::time::Duration;
 
 use tracing::debug_span;
@@ -8,7 +7,7 @@ use red_knot_python_semantic::types::Type;
 use red_knot_python_semantic::{HasTy, ModuleName, SemanticModel};
 use ruff_db::files::File;
 use ruff_db::parsed::{parsed_module, ParsedModule};
-use ruff_db::source::{line_index, source_text, SourceText};
+use ruff_db::source::{source_text, SourceText};
 use ruff_python_ast as ast;
 use ruff_python_ast::visitor::{walk_expr, walk_stmt, Visitor};
 use ruff_text_size::{Ranged, TextSize};
@@ -22,7 +21,7 @@ use crate::db::Db;
 pub(crate) fn unwind_if_cancelled(db: &dyn Db) {}
 
 #[salsa::tracked(return_ref)]
-pub(crate) fn lint_syntax(db: &dyn Db, file_id: File) -> Diagnostics {
+pub(crate) fn lint_syntax(db: &dyn Db, file_id: File) -> Vec<String> {
     #[allow(clippy::print_stdout)]
     if std::env::var("RED_KNOT_SLOW_LINT").is_ok() {
         for i in 0..10 {
@@ -49,22 +48,9 @@ pub(crate) fn lint_syntax(db: &dyn Db, file_id: File) -> Diagnostics {
         };
         visitor.visit_body(&ast.body);
         diagnostics = visitor.diagnostics;
-    } else {
-        let path = file_id.path(db);
-        let line_index = line_index(db.upcast(), file_id);
-        diagnostics.extend(parsed.errors().iter().map(|err| {
-            let source_location = line_index.source_location(err.location.start(), source.as_str());
-            format!(
-                "{}:{}:{}: {}",
-                path.as_str(),
-                source_location.row,
-                source_location.column,
-                err,
-            )
-        }));
     }
 
-    Diagnostics::from(diagnostics)
+    diagnostics
 }
 
 fn lint_lines(source: &str, diagnostics: &mut Vec<String>) {
@@ -86,7 +72,7 @@ fn lint_lines(source: &str, diagnostics: &mut Vec<String>) {
 
 #[allow(unreachable_pub)]
 #[salsa::tracked(return_ref)]
-pub fn lint_semantic(db: &dyn Db, file_id: File) -> Diagnostics {
+pub fn lint_semantic(db: &dyn Db, file_id: File) -> Vec<String> {
     let _span = debug_span!("lint_semantic", file=%file_id.path(db)).entered();
 
     let source = source_text(db.upcast(), file_id);
@@ -94,7 +80,7 @@ pub fn lint_semantic(db: &dyn Db, file_id: File) -> Diagnostics {
     let semantic = SemanticModel::new(db.upcast(), file_id);
 
     if !parsed.is_valid() {
-        return Diagnostics::Empty;
+        return vec![];
     }
 
     let context = SemanticLintContext {
@@ -106,7 +92,7 @@ pub fn lint_semantic(db: &dyn Db, file_id: File) -> Diagnostics {
 
     SemanticVisitor { context: &context }.visit_body(parsed.suite());
 
-    Diagnostics::from(context.diagnostics.take())
+    context.diagnostics.take()
 }
 
 fn format_diagnostic(context: &SemanticLintContext, message: &str, start: TextSize) -> String {
@@ -116,48 +102,13 @@ fn format_diagnostic(context: &SemanticLintContext, message: &str, start: TextSi
         .source_location(start, context.source_text());
     format!(
         "{}:{}:{}: {}",
-        context.semantic.file_path().as_str(),
+        context.semantic.file_path(),
         source_location.row,
         source_location.column,
         message,
     )
 }
 
-fn lint_unresolved_imports(context: &SemanticLintContext, import: AnyImportRef) {
-    // TODO: this treats any symbol with `Type::Unknown` as an unresolved import,
-    // which isn't really correct: if it exists but has `Type::Unknown` in the
-    // module we're importing it from, we shouldn't really emit a diagnostic here,
-    // but currently do.
-    match import {
-        AnyImportRef::Import(import) => {
-            for alias in &import.names {
-                let ty = alias.ty(&context.semantic);
-
-                if ty.is_unknown() {
-                    context.push_diagnostic(format_diagnostic(
-                        context,
-                        &format!("Unresolved import '{}'", &alias.name),
-                        alias.start(),
-                    ));
-                }
-            }
-        }
-        AnyImportRef::ImportFrom(import) => {
-            for alias in &import.names {
-                let ty = alias.ty(&context.semantic);
-
-                if ty.is_unknown() {
-                    context.push_diagnostic(format_diagnostic(
-                        context,
-                        &format!("Unresolved import '{}'", &alias.name),
-                        alias.start(),
-                    ));
-                }
-            }
-        }
-    }
-}
-
 fn lint_maybe_undefined(context: &SemanticLintContext, name: &ast::ExprName) {
     if !matches!(name.ctx, ast::ExprContext::Load) {
         return;
@@ -280,17 +231,8 @@ struct SemanticVisitor<'a> {
 
 impl Visitor<'_> for SemanticVisitor<'_> {
     fn visit_stmt(&mut self, stmt: &ast::Stmt) {
-        match stmt {
-            ast::Stmt::ClassDef(class) => {
-                lint_bad_override(self.context, class);
-            }
-            ast::Stmt::Import(import) => {
-                lint_unresolved_imports(self.context, AnyImportRef::Import(import));
-            }
-            ast::Stmt::ImportFrom(import) => {
-                lint_unresolved_imports(self.context, AnyImportRef::ImportFrom(import));
-            }
-            _ => {}
+        if let ast::Stmt::ClassDef(class) = stmt {
+            lint_bad_override(self.context, class);
         }
 
         walk_stmt(self, stmt);
@@ -308,53 +250,6 @@ impl Visitor<'_> for SemanticVisitor<'_> {
     }
 }
 
-#[derive(Debug, Clone, PartialEq, Eq)]
-pub enum Diagnostics {
-    Empty,
-    List(Vec<String>),
-}
-
-impl Diagnostics {
-    pub fn as_slice(&self) -> &[String] {
-        match self {
-            Diagnostics::Empty => &[],
-            Diagnostics::List(list) => list.as_slice(),
-        }
-    }
-}
-
-impl Deref for Diagnostics {
-    type Target = [String];
-    fn deref(&self) -> &Self::Target {
-        self.as_slice()
-    }
-}
-
-impl From<Vec<String>> for Diagnostics {
-    fn from(value: Vec<String>) -> Self {
-        if value.is_empty() {
-            Diagnostics::Empty
-        } else {
-            Diagnostics::List(value)
-        }
-    }
-}
-
-#[derive(Copy, Clone, Debug)]
-enum AnyImportRef<'a> {
-    Import(&'a ast::StmtImport),
-    ImportFrom(&'a ast::StmtImportFrom),
-}
-
-impl Ranged for AnyImportRef<'_> {
-    fn range(&self) -> ruff_text_size::TextRange {
-        match self {
-            AnyImportRef::Import(import) => import.range(),
-            AnyImportRef::ImportFrom(import) => import.range(),
-        }
-    }
-}
-
 #[cfg(test)]
 mod tests {
     use red_knot_python_semantic::{Program, ProgramSettings, PythonVersion, SearchPathSettings};
@@ -363,7 +258,7 @@ mod tests {
 
     use crate::db::tests::TestDb;
 
-    use super::{lint_semantic, Diagnostics};
+    use super::lint_semantic;
 
     fn setup_db() -> TestDb {
         setup_db_with_root(SystemPathBuf::from("/src"))
@@ -378,14 +273,9 @@ mod tests {
 
         Program::from_settings(
             &db,
-            ProgramSettings {
+            &ProgramSettings {
                 target_version: PythonVersion::default(),
-                search_paths: SearchPathSettings {
-                    extra_paths: Vec::new(),
-                    src_root,
-                    site_packages: vec![],
-                    custom_typeshed: None,
-                },
+                search_paths: SearchPathSettings::new(src_root),
             },
         )
         .expect("Valid program settings");
@@ -409,9 +299,9 @@ mod tests {
         .unwrap();
 
         let file = system_path_to_file(&db, "/src/a.py").expect("file to exist");
-        let Diagnostics::List(messages) = lint_semantic(&db, file) else {
-            panic!("expected some diagnostics");
-        };
+        let messages = lint_semantic(&db, file);
+
+        assert_ne!(messages, &[] as &[String], "expected some diagnostics");
 
         assert_eq!(
             *messages,

crates/red_knot_workspace/src/watch.rs

@@ -1,4 +1,4 @@
-use ruff_db::system::{SystemPath, SystemPathBuf};
+use ruff_db::system::{SystemPath, SystemPathBuf, SystemVirtualPathBuf};
 pub use watcher::{directory_watcher, EventHandler, Watcher};
 pub use workspace_watcher::WorkspaceWatcher;
 
@@ -20,6 +20,9 @@ mod workspace_watcher;
 /// event instead of emitting an event for each file or subdirectory in that path.
 #[derive(Debug, PartialEq, Eq)]
 pub enum ChangeEvent {
+    /// The file corresponding to the given path was opened in an editor.
+    Opened(SystemPathBuf),
+
     /// A new path was created
     Created {
         path: SystemPathBuf,
@@ -38,6 +41,15 @@ pub enum ChangeEvent {
         kind: DeletedKind,
     },
 
+    /// A new virtual path was created.
+    CreatedVirtual(SystemVirtualPathBuf),
+
+    /// The content of a virtual path was changed.
+    ChangedVirtual(SystemVirtualPathBuf),
+
+    /// A virtual path was deleted.
+    DeletedVirtual(SystemVirtualPathBuf),
+
     /// The file watcher failed to observe some changes and now is out of sync with the file system.
     ///
     /// This can happen if many files are changed at once. The consumer should rescan all files to catch up
@@ -46,16 +58,27 @@ pub enum ChangeEvent {
 }
 
 impl ChangeEvent {
-    pub fn file_name(&self) -> Option<&str> {
-        self.path().and_then(|path| path.file_name())
+    /// Creates a new [`Changed`] event for the file content at the given path.
+    ///
+    /// [`Changed`]: ChangeEvent::Changed
+    pub fn file_content_changed(path: SystemPathBuf) -> ChangeEvent {
+        ChangeEvent::Changed {
+            path,
+            kind: ChangedKind::FileContent,
+        }
     }
 
-    pub fn path(&self) -> Option<&SystemPath> {
+    pub fn file_name(&self) -> Option<&str> {
+        self.system_path().and_then(|path| path.file_name())
+    }
+
+    pub fn system_path(&self) -> Option<&SystemPath> {
         match self {
-            ChangeEvent::Created { path, .. }
+            ChangeEvent::Opened(path)
+            | ChangeEvent::Created { path, .. }
             | ChangeEvent::Changed { path, .. }
             | ChangeEvent::Deleted { path, .. } => Some(path),
-            ChangeEvent::Rescan => None,
+            _ => None,
         }
     }
 }

crates/red_knot_workspace/src/watch/watcher.rs

@@ -109,7 +109,7 @@ struct WatcherInner {
 impl Watcher {
     /// Sets up file watching for `path`.
     pub fn watch(&mut self, path: &SystemPath) -> notify::Result<()> {
-        tracing::debug!("Watching path: {path}.");
+        tracing::debug!("Watching path: '{path}'.");
 
         self.inner_mut()
             .watcher
@@ -118,7 +118,7 @@ impl Watcher {
 
     /// Stops file watching for `path`.
     pub fn unwatch(&mut self, path: &SystemPath) -> notify::Result<()> {
-        tracing::debug!("Unwatching path: {path}.");
+        tracing::debug!("Unwatching path: '{path}'.");
 
         self.inner_mut().watcher.unwatch(path.as_std_path())
     }
@@ -351,7 +351,7 @@ impl Debouncer {
             }
 
             EventKind::Any => {
-                tracing::debug!("Skip any FS event for {path}.");
+                tracing::debug!("Skipping any FS event for '{path}'.");
                 return;
             }
         };

crates/red_knot_workspace/src/workspace.rs

@@ -4,21 +4,26 @@ use rustc_hash::{FxBuildHasher, FxHashSet};
 use salsa::{Durability, Setter as _};
 
 pub use metadata::{PackageMetadata, WorkspaceMetadata};
-use ruff_db::source::{source_text, SourceDiagnostic};
+use red_knot_python_semantic::types::check_types;
+use red_knot_python_semantic::SearchPathSettings;
+use ruff_db::parsed::parsed_module;
+use ruff_db::source::{line_index, source_text, SourceDiagnostic};
 use ruff_db::{
     files::{system_path_to_file, File},
     system::{walk_directory::WalkState, SystemPath, SystemPathBuf},
 };
 use ruff_python_ast::{name::Name, PySourceType};
+use ruff_text_size::Ranged;
 
-use crate::workspace::files::{Index, IndexedFiles, PackageFiles};
+use crate::workspace::files::{Index, Indexed, PackageFiles};
 use crate::{
     db::Db,
-    lint::{lint_semantic, lint_syntax, Diagnostics},
+    lint::{lint_semantic, lint_syntax},
 };
 
 mod files;
 mod metadata;
+pub mod settings;
 
 /// The project workspace as a Salsa ingredient.
 ///
@@ -79,6 +84,10 @@ pub struct Workspace {
     /// The (first-party) packages in this workspace.
     #[return_ref]
     package_tree: BTreeMap<SystemPathBuf, Package>,
+
+    /// The unresolved search path configuration.
+    #[return_ref]
+    pub search_path_settings: SearchPathSettings,
 }
 
 /// A first-party package in a workspace.
@@ -92,8 +101,8 @@ pub struct Package {
     root_buf: SystemPathBuf,
 
     /// The files that are part of this package.
-    #[return_ref]
     #[default]
+    #[return_ref]
     file_set: PackageFiles,
     // TODO: Add the loaded settings.
 }
@@ -107,10 +116,14 @@ impl Workspace {
             packages.insert(package.root.clone(), Package::from_metadata(db, package));
         }
 
-        Workspace::builder(metadata.root, packages)
-            .durability(Durability::MEDIUM)
-            .open_fileset_durability(Durability::LOW)
-            .new(db)
+        Workspace::builder(
+            metadata.root,
+            packages,
+            metadata.settings.program.search_paths,
+        )
+        .durability(Durability::MEDIUM)
+        .open_fileset_durability(Durability::LOW)
+        .new(db)
     }
 
     pub fn root(self, db: &dyn Db) -> &SystemPath {
@@ -141,9 +154,12 @@ impl Workspace {
             new_packages.insert(path, package);
         }
 
-        self.set_package_tree(db)
-            .with_durability(Durability::MEDIUM)
-            .to(new_packages);
+        if &metadata.settings.program.search_paths != self.search_path_settings(db) {
+            self.set_search_path_settings(db)
+                .to(metadata.settings.program.search_paths);
+        }
+
+        self.set_package_tree(db).to(new_packages);
     }
 
     pub fn update_package(self, db: &mut dyn Db, metadata: PackageMetadata) -> anyhow::Result<()> {
@@ -197,7 +213,7 @@ impl Workspace {
     ///
     /// This changes the behavior of `check` to only check the open files rather than all files in the workspace.
     pub fn open_file(self, db: &mut dyn Db, file: File) {
-        tracing::debug!("Opening file {}", file.path(db));
+        tracing::debug!("Opening file '{}'", file.path(db));
 
         let mut open_files = self.take_open_files(db);
         open_files.insert(file);
@@ -206,7 +222,7 @@ impl Workspace {
 
     /// Closes a file in the workspace.
     pub fn close_file(self, db: &mut dyn Db, file: File) -> bool {
-        tracing::debug!("Closing file {}", file.path(db));
+        tracing::debug!("Closing file '{}'", file.path(db));
 
         let mut open_files = self.take_open_files(db);
         let removed = open_files.remove(&file);
@@ -249,6 +265,23 @@ impl Workspace {
             FxHashSet::default()
         }
     }
+
+    /// Returns `true` if the file is open in the workspace.
+    ///
+    /// A file is considered open when:
+    /// * explicitly set as an open file using [`open_file`](Self::open_file)
+    /// * It has a [`SystemPath`] and belongs to a package's `src` files
+    /// * It has a [`SystemVirtualPath`](ruff_db::system::SystemVirtualPath)
+    pub fn is_file_open(self, db: &dyn Db, file: File) -> bool {
+        if let Some(open_files) = self.open_files(db) {
+            open_files.contains(&file)
+        } else if let Some(system_path) = file.path(db).as_system_path() {
+            self.package(db, system_path)
+                .map_or(false, |package| package.contains_file(db, file))
+        } else {
+            file.path(db).is_system_virtual_path()
+        }
+    }
 }
 
 #[salsa::tracked]
@@ -259,13 +292,13 @@ impl Package {
 
     /// Returns `true` if `file` is a first-party file part of this package.
     pub fn contains_file(self, db: &dyn Db, file: File) -> bool {
-        self.files(db).read().contains(&file)
+        self.files(db).contains(&file)
     }
 
     #[tracing::instrument(level = "debug", skip(db))]
     pub fn remove_file(self, db: &mut dyn Db, file: File) {
         tracing::debug!(
-            "Remove file {} from package {}",
+            "Removing file '{}' from package '{}'",
             file.path(db),
             self.name(db)
         );
@@ -278,7 +311,11 @@ impl Package {
     }
 
     pub fn add_file(self, db: &mut dyn Db, file: File) {
-        tracing::debug!("Add file {} to package {}", file.path(db), self.name(db));
+        tracing::debug!(
+            "Adding file '{}' to package '{}'",
+            file.path(db),
+            self.name(db)
+        );
 
         let Some(mut index) = PackageFiles::indexed_mut(db, self) else {
             return;
@@ -289,10 +326,10 @@ impl Package {
 
     #[tracing::instrument(level = "debug", skip(db))]
     pub(crate) fn check(self, db: &dyn Db) -> Vec<String> {
-        tracing::debug!("Checking package {}", self.root(db));
+        tracing::debug!("Checking package '{}'", self.root(db));
 
         let mut result = Vec::new();
-        for file in &self.files(db).read() {
+        for file in &self.files(db) {
             let diagnostics = check_file(db, file);
             result.extend_from_slice(&diagnostics);
         }
@@ -301,15 +338,16 @@ impl Package {
     }
 
     /// Returns the files belonging to this package.
-    #[salsa::tracked]
-    pub fn files(self, db: &dyn Db) -> IndexedFiles {
-        let _entered = tracing::debug_span!("files").entered();
+    pub fn files(self, db: &dyn Db) -> Indexed<'_> {
         let files = self.file_set(db);
 
         let indexed = match files.get() {
             Index::Lazy(vacant) => {
-                tracing::debug!("Indexing files for package {}", self.name(db));
+                let _entered =
+                    tracing::debug_span!("index_package_files", package = %self.name(db)).entered();
+
                 let files = discover_package_files(db, self.root(db));
+                tracing::info!("Found {} files in package '{}'", files.len(), self.name(db));
                 vacant.set(files)
             }
             Index::Indexed(indexed) => indexed,
@@ -330,14 +368,12 @@ impl Package {
         assert_eq!(root, metadata.root());
 
         if self.name(db) != metadata.name() {
-            self.set_name(db)
-                .with_durability(Durability::MEDIUM)
-                .to(metadata.name);
+            self.set_name(db).to(metadata.name);
         }
     }
 
     pub fn reload_files(self, db: &mut dyn Db) {
-        tracing::debug!("Reload files for package {}", self.name(db));
+        tracing::debug!("Reloading files for package '{}'", self.name(db));
 
         if !self.file_set(db).is_lazy() {
             // Force a re-index of the files in the next revision.
@@ -347,10 +383,10 @@ impl Package {
 }
 
 #[salsa::tracked]
-pub(super) fn check_file(db: &dyn Db, file: File) -> Diagnostics {
+pub(super) fn check_file(db: &dyn Db, file: File) -> Vec<String> {
     let path = file.path(db);
     let _span = tracing::debug_span!("check_file", file=%path).entered();
-    tracing::debug!("Checking file {path}");
+    tracing::debug!("Checking file '{path}'");
 
     let mut diagnostics = Vec::new();
 
@@ -363,13 +399,36 @@ pub(super) fn check_file(db: &dyn Db, file: File) -> Diagnostics {
     );
 
     // Abort checking if there are IO errors.
-    if source_text(db.upcast(), file).has_read_error() {
-        return Diagnostics::from(diagnostics);
+    let source = source_text(db.upcast(), file);
+
+    if source.has_read_error() {
+        return diagnostics;
+    }
+
+    let parsed = parsed_module(db.upcast(), file);
+
+    if !parsed.errors().is_empty() {
+        let path = file.path(db);
+        let line_index = line_index(db.upcast(), file);
+        diagnostics.extend(parsed.errors().iter().map(|err| {
+            let source_location = line_index.source_location(err.location.start(), source.as_str());
+            format!("{path}:{source_location}: {message}", message = err.error)
+        }));
+    }
+
+    for diagnostic in check_types(db.upcast(), file) {
+        let index = line_index(db.upcast(), diagnostic.file());
+        let location = index.source_location(diagnostic.start(), source.as_str());
+        diagnostics.push(format!(
+            "{path}:{location}: {message}",
+            path = file.path(db),
+            message = diagnostic.message()
+        ));
     }
 
     diagnostics.extend_from_slice(lint_syntax(db, file));
     diagnostics.extend_from_slice(lint_semantic(db, file));
-    Diagnostics::from(diagnostics)
+    diagnostics
 }
 
 fn discover_package_files(db: &dyn Db, path: &SystemPath) -> FxHashSet<File> {
@@ -423,7 +482,7 @@ mod tests {
     use ruff_db::testing::assert_function_query_was_not_run;
 
     use crate::db::tests::TestDb;
-    use crate::lint::{lint_syntax, Diagnostics};
+    use crate::lint::lint_syntax;
     use crate::workspace::check_file;
 
     #[test]
@@ -441,9 +500,7 @@ mod tests {
         assert_eq!(source_text(&db, file).as_str(), "");
         assert_eq!(
             check_file(&db, file),
-            Diagnostics::List(vec![
-                "Failed to read file: No such file or directory".to_string()
-            ])
+            vec!["Failed to read file: No such file or directory".to_string()]
         );
 
         let events = db.take_salsa_events();
@@ -454,7 +511,7 @@ mod tests {
         db.write_file(path, "").unwrap();
 
         assert_eq!(source_text(&db, file).as_str(), "");
-        assert_eq!(check_file(&db, file), Diagnostics::Empty);
+        assert_eq!(check_file(&db, file), vec![] as Vec<String>);
 
         Ok(())
     }

crates/red_knot_workspace/src/workspace/files.rs

@@ -1,4 +1,4 @@
-use std::iter::FusedIterator;
+use std::marker::PhantomData;
 use std::ops::Deref;
 use std::sync::Arc;
 
@@ -10,6 +10,9 @@ use ruff_db::files::File;
 use crate::db::Db;
 use crate::workspace::Package;
 
+/// Cheap cloneable hash set of files.
+type FileSet = Arc<FxHashSet<File>>;
+
 /// The indexed files of a package.
 ///
 /// The indexing happens lazily, but the files are then cached for subsequent reads.
@@ -18,7 +21,7 @@ use crate::workspace::Package;
 /// The implementation uses internal mutability to transition between the lazy and indexed state
 /// without triggering a new salsa revision. This is safe because the initial indexing happens on first access,
 /// so no query can be depending on the contents of the indexed files before that. All subsequent mutations to
-/// the indexed files must go through `IndexedFilesMut`, which uses the Salsa setter `package.set_file_set` to
+/// the indexed files must go through `IndexedMut`, which uses the Salsa setter `package.set_file_set` to
 /// ensure that Salsa always knows when the set of indexed files have changed.
 #[derive(Debug)]
 pub struct PackageFiles {
@@ -32,46 +35,67 @@ impl PackageFiles {
         }
     }
 
-    fn indexed(indexed_files: IndexedFiles) -> Self {
+    fn indexed(files: FileSet) -> Self {
         Self {
-            state: std::sync::Mutex::new(State::Indexed(indexed_files)),
+            state: std::sync::Mutex::new(State::Indexed(files)),
         }
     }
 
-    pub fn get(&self) -> Index {
+    pub(super) fn get(&self) -> Index {
         let state = self.state.lock().unwrap();
 
         match &*state {
             State::Lazy => Index::Lazy(LazyFiles { files: state }),
-            State::Indexed(files) => Index::Indexed(files.clone()),
+            State::Indexed(files) => Index::Indexed(Indexed {
+                files: Arc::clone(files),
+                _lifetime: PhantomData,
+            }),
         }
     }
 
-    pub fn is_lazy(&self) -> bool {
+    pub(super) fn is_lazy(&self) -> bool {
         matches!(*self.state.lock().unwrap(), State::Lazy)
     }
 
     /// Returns a mutable view on the index that allows cheap in-place mutations.
     ///
     /// The changes are automatically written back to the database once the view is dropped.
-    pub fn indexed_mut(db: &mut dyn Db, package: Package) -> Option<IndexedFilesMut> {
+    pub(super) fn indexed_mut(db: &mut dyn Db, package: Package) -> Option<IndexedMut> {
         // Calling `zalsa_mut` cancels all pending salsa queries. This ensures that there are no pending
         // reads to the file set.
         // TODO: Use a non-internal API instead https://salsa.zulipchat.com/#narrow/stream/333573-salsa-3.2E0/topic/Expose.20an.20API.20to.20cancel.20other.20queries
         let _ = db.as_dyn_database_mut().zalsa_mut();
 
-        let files = package.file_set(db);
-
-        let indexed = match &*files.state.lock().unwrap() {
-            State::Lazy => return None,
-            State::Indexed(indexed) => indexed.clone(),
+        // Replace the state with lazy. The `IndexedMut` guard restores the state
+        // to `State::Indexed`  or sets a new `PackageFiles` when it gets dropped to ensure the state
+        // is restored to how it has been before replacing the value.
+        //
+        // It isn't necessary to hold on to the lock after this point:
+        // * The above call to `zalsa_mut` guarantees that there's exactly **one** DB reference.
+        // * `Indexed` has a `'db` lifetime, and this method requires a `&mut db`.
+        //   This means that there can't be any pending reference to `Indexed` because Rust
+        //   doesn't allow borrowing `db` as mutable (to call this method) and immutable (`Indexed<'db>`) at the same time.
+        //   There can't be any other `Indexed<'db>` references created by clones of this DB because
+        //   all clones must have been dropped at this point and the `Indexed`
+        //   can't outlive the database (constrained by the `db` lifetime).
+        let state = {
+            let files = package.file_set(db);
+            let mut locked = files.state.lock().unwrap();
+            std::mem::replace(&mut *locked, State::Lazy)
         };
 
-        Some(IndexedFilesMut {
+        let indexed = match state {
+            // If it's already lazy, just return. We also don't need to restore anything because the
+            // replace above was a no-op.
+            State::Lazy => return None,
+            State::Indexed(indexed) => indexed,
+        };
+
+        Some(IndexedMut {
             db: Some(db),
             package,
-            new_revision: indexed.revision,
-            indexed,
+            files: indexed,
+            did_change: false,
         })
     }
 }
@@ -88,152 +112,93 @@ enum State {
     Lazy,
 
     /// The files are indexed. Stores the known files of a package.
-    Indexed(IndexedFiles),
+    Indexed(FileSet),
 }
 
-pub enum Index<'a> {
+pub(super) enum Index<'db> {
     /// The index has not yet been computed. Allows inserting the files.
-    Lazy(LazyFiles<'a>),
+    Lazy(LazyFiles<'db>),
 
-    Indexed(IndexedFiles),
+    Indexed(Indexed<'db>),
 }
 
 /// Package files that have not been indexed yet.
-pub struct LazyFiles<'a> {
-    files: std::sync::MutexGuard<'a, State>,
+pub(super) struct LazyFiles<'db> {
+    files: std::sync::MutexGuard<'db, State>,
 }
 
-impl<'a> LazyFiles<'a> {
+impl<'db> LazyFiles<'db> {
     /// Sets the indexed files of a package to `files`.
-    pub fn set(mut self, files: FxHashSet<File>) -> IndexedFiles {
-        let files = IndexedFiles::new(files);
-        *self.files = State::Indexed(files.clone());
+    pub(super) fn set(mut self, files: FxHashSet<File>) -> Indexed<'db> {
+        let files = Indexed {
+            files: Arc::new(files),
+            _lifetime: PhantomData,
+        };
+        *self.files = State::Indexed(Arc::clone(&files.files));
         files
     }
 }
 
 /// The indexed files of a package.
 ///
-/// # Salsa integration
-/// The type is cheap clonable and allows for in-place mutation of the files. The in-place mutation requires
-/// extra care because the type is used as the result of Salsa queries and Salsa relies on a type's equality
-/// to determine if the output has changed. This is accomplished by using a `revision` that gets incremented
-/// whenever the files are changed. The revision ensures that salsa's comparison of the
-/// previous [`IndexedFiles`] with the next [`IndexedFiles`] returns false even though they both
-/// point to the same underlying hash set.
-///
-/// # Equality
-/// Two [`IndexedFiles`] are only equal if they have the same revision and point to the **same** (identity) hash set.
-#[derive(Debug, Clone)]
-pub struct IndexedFiles {
-    revision: u64,
-    files: Arc<std::sync::Mutex<FxHashSet<File>>>,
+/// Note: This type is intentionally non-cloneable. Making it cloneable requires
+/// revisiting the locking behavior in [`PackageFiles::indexed_mut`].
+#[derive(Debug, PartialEq, Eq)]
+pub struct Indexed<'db> {
+    files: FileSet,
+    // Preserve the lifetime of `PackageFiles`.
+    _lifetime: PhantomData<&'db ()>,
 }
 
-impl IndexedFiles {
-    fn new(files: FxHashSet<File>) -> Self {
-        Self {
-            files: Arc::new(std::sync::Mutex::new(files)),
-            revision: 0,
-        }
-    }
-
-    /// Locks the file index for reading.
-    pub fn read(&self) -> IndexedFilesGuard {
-        IndexedFilesGuard {
-            guard: self.files.lock().unwrap(),
-        }
-    }
-}
-
-impl PartialEq for IndexedFiles {
-    fn eq(&self, other: &Self) -> bool {
-        self.revision == other.revision && Arc::ptr_eq(&self.files, &other.files)
-    }
-}
-
-impl Eq for IndexedFiles {}
-
-pub struct IndexedFilesGuard<'a> {
-    guard: std::sync::MutexGuard<'a, FxHashSet<File>>,
-}
-
-impl Deref for IndexedFilesGuard<'_> {
+impl Deref for Indexed<'_> {
     type Target = FxHashSet<File>;
 
     fn deref(&self) -> &Self::Target {
-        &self.guard
+        &self.files
     }
 }
 
-impl<'a> IntoIterator for &'a IndexedFilesGuard<'a> {
+impl<'a> IntoIterator for &'a Indexed<'_> {
     type Item = File;
-    type IntoIter = IndexedFilesIter<'a>;
+    type IntoIter = std::iter::Copied<std::collections::hash_set::Iter<'a, File>>;
 
     fn into_iter(self) -> Self::IntoIter {
-        IndexedFilesIter {
-            inner: self.guard.iter(),
-        }
+        self.files.iter().copied()
     }
 }
 
-/// Iterator over the indexed files.
-///
-/// # Locks
-/// Holding on to the iterator locks the file index for reading.
-pub struct IndexedFilesIter<'a> {
-    inner: std::collections::hash_set::Iter<'a, File>,
-}
-
-impl<'a> Iterator for IndexedFilesIter<'a> {
-    type Item = File;
-
-    fn next(&mut self) -> Option<Self::Item> {
-        self.inner.next().copied()
-    }
-
-    fn size_hint(&self) -> (usize, Option<usize>) {
-        self.inner.size_hint()
-    }
-}
-
-impl FusedIterator for IndexedFilesIter<'_> {}
-
-impl ExactSizeIterator for IndexedFilesIter<'_> {}
-
 /// A Mutable view of a package's indexed files.
 ///
 /// Allows in-place mutation of the files without deep cloning the hash set.
 /// The changes are written back when the mutable view is dropped or by calling [`Self::set`] manually.
-pub struct IndexedFilesMut<'db> {
+pub(super) struct IndexedMut<'db> {
     db: Option<&'db mut dyn Db>,
     package: Package,
-    indexed: IndexedFiles,
-    new_revision: u64,
+    files: FileSet,
+    did_change: bool,
 }
 
-impl IndexedFilesMut<'_> {
-    pub fn insert(&mut self, file: File) -> bool {
-        if self.indexed.files.lock().unwrap().insert(file) {
-            self.new_revision += 1;
+impl IndexedMut<'_> {
+    pub(super) fn insert(&mut self, file: File) -> bool {
+        if self.files_mut().insert(file) {
+            self.did_change = true;
             true
         } else {
             false
         }
     }
 
-    pub fn remove(&mut self, file: File) -> bool {
-        if self.indexed.files.lock().unwrap().remove(&file) {
-            self.new_revision += 1;
+    pub(super) fn remove(&mut self, file: File) -> bool {
+        if self.files_mut().remove(&file) {
+            self.did_change = true;
             true
         } else {
             false
         }
     }
 
-    /// Writes the changes back to the database.
-    pub fn set(mut self) {
-        self.set_impl();
+    fn files_mut(&mut self) -> &mut FxHashSet<File> {
+        Arc::get_mut(&mut self.files).expect("All references to `FilesSet` to have been dropped")
     }
 
     fn set_impl(&mut self) {
@@ -241,19 +206,70 @@ impl IndexedFilesMut<'_> {
             return;
         };
 
-        if self.indexed.revision != self.new_revision {
+        let files = Arc::clone(&self.files);
+
+        if self.did_change {
+            // If there are changes, set the new file_set to trigger a salsa revision change.
             self.package
                 .set_file_set(db)
-                .to(PackageFiles::indexed(IndexedFiles {
-                    revision: self.new_revision,
-                    files: self.indexed.files.clone(),
-                }));
+                .to(PackageFiles::indexed(files));
+        } else {
+            // The `indexed_mut` replaced the `state` with Lazy. Restore it back to the indexed state.
+            *self.package.file_set(db).state.lock().unwrap() = State::Indexed(files);
         }
     }
 }
 
-impl Drop for IndexedFilesMut<'_> {
+impl Drop for IndexedMut<'_> {
     fn drop(&mut self) {
         self.set_impl();
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use rustc_hash::FxHashSet;
+
+    use ruff_db::files::system_path_to_file;
+    use ruff_db::system::{DbWithTestSystem, SystemPathBuf};
+    use ruff_python_ast::name::Name;
+
+    use crate::db::tests::TestDb;
+    use crate::workspace::files::Index;
+    use crate::workspace::Package;
+
+    #[test]
+    fn re_entrance() -> anyhow::Result<()> {
+        let mut db = TestDb::new();
+
+        db.write_file("test.py", "")?;
+
+        let package = Package::new(&db, Name::new("test"), SystemPathBuf::from("/test"));
+
+        let file = system_path_to_file(&db, "test.py").unwrap();
+
+        let files = match package.file_set(&db).get() {
+            Index::Lazy(lazy) => lazy.set(FxHashSet::from_iter([file])),
+            Index::Indexed(files) => files,
+        };
+
+        // Calling files a second time should not dead-lock.
+        // This can e.g. happen when `check_file` iterates over all files and
+        // `is_file_open` queries the open files.
+        let files_2 = package.file_set(&db).get();
+
+        match files_2 {
+            Index::Lazy(_) => {
+                panic!("Expected indexed files, got lazy files");
+            }
+            Index::Indexed(files_2) => {
+                assert_eq!(
+                    files_2.iter().collect::<Vec<_>>(),
+                    files.iter().collect::<Vec<_>>()
+                );
+            }
+        }
+
+        Ok(())
+    }
+}

crates/red_knot_workspace/src/workspace/metadata.rs

@@ -1,3 +1,4 @@
+use crate::workspace::settings::{Configuration, WorkspaceSettings};
 use ruff_db::system::{System, SystemPath, SystemPathBuf};
 use ruff_python_ast::name::Name;
 
@@ -7,6 +8,8 @@ pub struct WorkspaceMetadata {
 
     /// The (first-party) packages in this workspace.
     pub(super) packages: Vec<PackageMetadata>,
+
+    pub(super) settings: WorkspaceSettings,
 }
 
 /// A first-party package in a workspace.
@@ -21,7 +24,11 @@ pub struct PackageMetadata {
 
 impl WorkspaceMetadata {
     /// Discovers the closest workspace at `path` and returns its metadata.
-    pub fn from_path(path: &SystemPath, system: &dyn System) -> anyhow::Result<WorkspaceMetadata> {
+    pub fn from_path(
+        path: &SystemPath,
+        system: &dyn System,
+        base_configuration: Option<Configuration>,
+    ) -> anyhow::Result<WorkspaceMetadata> {
         assert!(
             system.is_directory(path),
             "Workspace root path must be a directory"
@@ -38,9 +45,20 @@ impl WorkspaceMetadata {
             root: root.clone(),
         };
 
+        // TODO: Load the configuration from disk.
+        let mut configuration = Configuration::default();
+
+        if let Some(base_configuration) = base_configuration {
+            configuration.extend(base_configuration);
+        }
+
+        // TODO: Respect the package configurations when resolving settings (e.g. for the target version).
+        let settings = configuration.into_workspace_settings(&root);
+
         let workspace = WorkspaceMetadata {
             root,
             packages: vec![package],
+            settings,
         };
 
         Ok(workspace)
@@ -53,6 +71,10 @@ impl WorkspaceMetadata {
     pub fn packages(&self) -> &[PackageMetadata] {
         &self.packages
     }
+
+    pub fn settings(&self) -> &WorkspaceSettings {
+        &self.settings
+    }
 }
 
 impl PackageMetadata {

crates/red_knot_workspace/src/workspace/settings.rs

@@ -0,0 +1,89 @@
+use red_knot_python_semantic::{ProgramSettings, PythonVersion, SearchPathSettings, SitePackages};
+use ruff_db::system::{SystemPath, SystemPathBuf};
+
+/// The resolved configurations.
+///
+/// The main difference to [`Configuration`] is that default values are filled in.
+#[derive(Debug, Clone)]
+pub struct WorkspaceSettings {
+    pub(super) program: ProgramSettings,
+}
+
+impl WorkspaceSettings {
+    pub fn program(&self) -> &ProgramSettings {
+        &self.program
+    }
+}
+
+/// The configuration for the workspace or a package.
+#[derive(Debug, Default, Clone)]
+pub struct Configuration {
+    pub target_version: Option<PythonVersion>,
+    pub search_paths: SearchPathConfiguration,
+}
+
+impl Configuration {
+    /// Extends this configuration by using the values from `with` for all values that are absent in `self`.
+    pub fn extend(&mut self, with: Configuration) {
+        self.target_version = self.target_version.or(with.target_version);
+        self.search_paths.extend(with.search_paths);
+    }
+
+    pub fn into_workspace_settings(self, workspace_root: &SystemPath) -> WorkspaceSettings {
+        WorkspaceSettings {
+            program: ProgramSettings {
+                target_version: self.target_version.unwrap_or_default(),
+                search_paths: self.search_paths.into_settings(workspace_root),
+            },
+        }
+    }
+}
+
+#[derive(Debug, Default, Clone, Eq, PartialEq)]
+pub struct SearchPathConfiguration {
+    /// List of user-provided paths that should take first priority in the module resolution.
+    /// Examples in other type checkers are mypy's MYPYPATH environment variable,
+    /// or pyright's stubPath configuration setting.
+    pub extra_paths: Option<Vec<SystemPathBuf>>,
+
+    /// The root of the workspace, used for finding first-party modules.
+    pub src_root: Option<SystemPathBuf>,
+
+    /// Optional path to a "custom typeshed" directory on disk for us to use for standard-library types.
+    /// If this is not provided, we will fallback to our vendored typeshed stubs for the stdlib,
+    /// bundled as a zip file in the binary
+    pub custom_typeshed: Option<SystemPathBuf>,
+
+    /// The path to the user's `site-packages` directory, where third-party packages from ``PyPI`` are installed.
+    pub site_packages: Option<SitePackages>,
+}
+
+impl SearchPathConfiguration {
+    pub fn into_settings(self, workspace_root: &SystemPath) -> SearchPathSettings {
+        let site_packages = self.site_packages.unwrap_or(SitePackages::Known(vec![]));
+
+        SearchPathSettings {
+            extra_paths: self.extra_paths.unwrap_or_default(),
+            src_root: self
+                .src_root
+                .unwrap_or_else(|| workspace_root.to_path_buf()),
+            custom_typeshed: self.custom_typeshed,
+            site_packages,
+        }
+    }
+
+    pub fn extend(&mut self, with: SearchPathConfiguration) {
+        if let Some(extra_paths) = with.extra_paths {
+            self.extra_paths.get_or_insert(extra_paths);
+        }
+        if let Some(src_root) = with.src_root {
+            self.src_root.get_or_insert(src_root);
+        }
+        if let Some(custom_typeshed) = with.custom_typeshed {
+            self.custom_typeshed.get_or_insert(custom_typeshed);
+        }
+        if let Some(site_packages) = with.site_packages {
+            self.site_packages.get_or_insert(site_packages);
+        }
+    }
+}

crates/red_knot_workspace/tests/check.rs

@@ -1,6 +1,7 @@
-use red_knot_python_semantic::{
-    HasTy, ProgramSettings, PythonVersion, SearchPathSettings, SemanticModel,
-};
+use std::fs;
+use std::path::PathBuf;
+
+use red_knot_python_semantic::{HasTy, SemanticModel};
 use red_knot_workspace::db::RootDatabase;
 use red_knot_workspace::workspace::WorkspaceMetadata;
 use ruff_db::files::{system_path_to_file, File};
@@ -9,23 +10,11 @@ use ruff_db::system::{OsSystem, SystemPath, SystemPathBuf};
 use ruff_python_ast::visitor::source_order;
 use ruff_python_ast::visitor::source_order::SourceOrderVisitor;
 use ruff_python_ast::{Alias, Expr, Parameter, ParameterWithDefault, Stmt};
-use std::fs;
-use std::path::PathBuf;
 
-fn setup_db(workspace_root: SystemPathBuf) -> anyhow::Result<RootDatabase> {
-    let system = OsSystem::new(&workspace_root);
-    let workspace = WorkspaceMetadata::from_path(&workspace_root, &system)?;
-    let search_paths = SearchPathSettings {
-        extra_paths: vec![],
-        src_root: workspace_root,
-        custom_typeshed: None,
-        site_packages: vec![],
-    };
-    let settings = ProgramSettings {
-        target_version: PythonVersion::default(),
-        search_paths,
-    };
-    RootDatabase::new(workspace, settings, system)
+fn setup_db(workspace_root: &SystemPath) -> anyhow::Result<RootDatabase> {
+    let system = OsSystem::new(workspace_root);
+    let workspace = WorkspaceMetadata::from_path(workspace_root, &system, None)?;
+    RootDatabase::new(workspace, system)
 }
 
 /// Test that all snippets in testcorpus can be checked without panic
@@ -33,8 +22,9 @@ fn setup_db(workspace_root: SystemPathBuf) -> anyhow::Result<RootDatabase> {
 #[allow(clippy::print_stdout)]
 fn corpus_no_panic() -> anyhow::Result<()> {
     let corpus = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("resources/test/corpus");
-    let system_corpus = SystemPath::from_std_path(&corpus).expect("corpus path to be UTF8");
-    let db = setup_db(system_corpus.to_path_buf())?;
+    let system_corpus =
+        SystemPathBuf::from_path_buf(corpus.clone()).expect("corpus path to be UTF8");
+    let db = setup_db(&system_corpus)?;
 
     for path in fs::read_dir(&corpus).expect("corpus to be a directory") {
         let path = path.expect("path to not be an error").path();

crates/ruff/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "ruff"
-version = "0.6.1"
+version = "0.6.2"
 publish = true
 authors = { workspace = true }
 edition = { workspace = true }

crates/ruff_benchmark/benches/red_knot.rs

@@ -1,7 +1,9 @@
 #![allow(clippy::disallowed_names)]
 
-use red_knot_python_semantic::{ProgramSettings, PythonVersion, SearchPathSettings};
+use red_knot_python_semantic::PythonVersion;
 use red_knot_workspace::db::RootDatabase;
+use red_knot_workspace::watch::{ChangeEvent, ChangedKind};
+use red_knot_workspace::workspace::settings::Configuration;
 use red_knot_workspace::workspace::WorkspaceMetadata;
 use ruff_benchmark::criterion::{criterion_group, criterion_main, BatchSize, Criterion};
 use ruff_benchmark::TestFile;
@@ -12,13 +14,53 @@ use ruff_db::system::{MemoryFileSystem, SystemPath, TestSystem};
 struct Case {
     db: RootDatabase,
     fs: MemoryFileSystem,
-    parser: File,
     re: File,
     re_path: &'static SystemPath,
 }
 
 const TOMLLIB_312_URL: &str = "https://raw.githubusercontent.com/python/cpython/8e8a4baf652f6e1cee7acde9d78c4b6154539748/Lib/tomllib";
 
+// This first "unresolved import" is because we don't understand `*` imports yet.
+// The following "unresolved import" violations are because we can't distinguish currently from
+// "Symbol exists in the module but its type is unknown" and
+// "Symbol does not exist in the module"
+static EXPECTED_DIAGNOSTICS: &[&str] = &[
+    "/src/tomllib/_parser.py:7:29: Could not resolve import of 'Iterable' from 'collections.abc'",
+    "/src/tomllib/_parser.py:10:20: Could not resolve import of 'Any' from 'typing'",
+    "/src/tomllib/_parser.py:13:5: Could not resolve import of 'RE_DATETIME' from '._re'",
+    "/src/tomllib/_parser.py:14:5: Could not resolve import of 'RE_LOCALTIME' from '._re'",
+    "/src/tomllib/_parser.py:15:5: Could not resolve import of 'RE_NUMBER' from '._re'",
+    "/src/tomllib/_parser.py:20:21: Could not resolve import of 'Key' from '._types'",
+    "/src/tomllib/_parser.py:20:26: Could not resolve import of 'ParseFloat' from '._types'",
+    "Line 69 is too long (89 characters)",
+    "Use double quotes for strings",
+    "Use double quotes for strings",
+    "Use double quotes for strings",
+    "Use double quotes for strings",
+    "Use double quotes for strings",
+    "Use double quotes for strings",
+    "Use double quotes for strings",
+    "/src/tomllib/_parser.py:153:22: Name 'key' used when not defined.",
+    "/src/tomllib/_parser.py:153:27: Name 'flag' used when not defined.",
+    "/src/tomllib/_parser.py:159:16: Name 'k' used when not defined.",
+    "/src/tomllib/_parser.py:161:25: Name 'k' used when not defined.",
+    "/src/tomllib/_parser.py:168:16: Name 'k' used when not defined.",
+    "/src/tomllib/_parser.py:169:22: Name 'k' used when not defined.",
+    "/src/tomllib/_parser.py:170:25: Name 'k' used when not defined.",
+    "/src/tomllib/_parser.py:180:16: Name 'k' used when not defined.",
+    "/src/tomllib/_parser.py:182:31: Name 'k' used when not defined.",
+    "/src/tomllib/_parser.py:206:16: Name 'k' used when not defined.",
+    "/src/tomllib/_parser.py:207:22: Name 'k' used when not defined.",
+    "/src/tomllib/_parser.py:208:25: Name 'k' used when not defined.",
+    "/src/tomllib/_parser.py:330:32: Name 'header' used when not defined.",
+    "/src/tomllib/_parser.py:330:41: Name 'key' used when not defined.",
+    "/src/tomllib/_parser.py:333:26: Name 'cont_key' used when not defined.",
+    "/src/tomllib/_parser.py:334:71: Name 'cont_key' used when not defined.",
+    "/src/tomllib/_parser.py:337:31: Name 'cont_key' used when not defined.",
+    "/src/tomllib/_parser.py:628:75: Name 'e' used when not defined.",
+    "/src/tomllib/_parser.py:686:23: Name 'parse_float' used when not defined.",
+];
+
 fn get_test_file(name: &str) -> TestFile {
     let path = format!("tomllib/{name}");
     let url = format!("{TOMLLIB_312_URL}/{name}");
@@ -28,31 +70,34 @@ fn get_test_file(name: &str) -> TestFile {
 fn setup_case() -> Case {
     let system = TestSystem::default();
     let fs = system.memory_file_system().clone();
-    let init_path = SystemPath::new("/src/tomllib/__init__.py");
     let parser_path = SystemPath::new("/src/tomllib/_parser.py");
     let re_path = SystemPath::new("/src/tomllib/_re.py");
-    let types_path = SystemPath::new("/src/tomllib/_types.py");
     fs.write_files([
-        (init_path, get_test_file("__init__.py").code()),
+        (
+            SystemPath::new("/src/tomllib/__init__.py"),
+            get_test_file("__init__.py").code(),
+        ),
         (parser_path, get_test_file("_parser.py").code()),
         (re_path, get_test_file("_re.py").code()),
-        (types_path, get_test_file("_types.py").code()),
+        (
+            SystemPath::new("/src/tomllib/_types.py"),
+            get_test_file("_types.py").code(),
+        ),
     ])
     .unwrap();
 
     let src_root = SystemPath::new("/src");
-    let metadata = WorkspaceMetadata::from_path(src_root, &system).unwrap();
-    let settings = ProgramSettings {
-        target_version: PythonVersion::PY312,
-        search_paths: SearchPathSettings {
-            extra_paths: vec![],
-            src_root: src_root.to_path_buf(),
-            site_packages: vec![],
-            custom_typeshed: None,
-        },
-    };
+    let metadata = WorkspaceMetadata::from_path(
+        src_root,
+        &system,
+        Some(Configuration {
+            target_version: Some(PythonVersion::PY312),
+            ..Configuration::default()
+        }),
+    )
+    .unwrap();
 
-    let mut db = RootDatabase::new(metadata, settings, system).unwrap();
+    let mut db = RootDatabase::new(metadata, system).unwrap();
     let parser = system_path_to_file(&db, parser_path).unwrap();
 
     db.workspace().open_file(&mut db, parser);
@@ -62,7 +107,6 @@ fn setup_case() -> Case {
     Case {
         db,
         fs,
-        parser,
         re,
         re_path,
     }
@@ -72,8 +116,8 @@ fn benchmark_incremental(criterion: &mut Criterion) {
     criterion.bench_function("red_knot_check_file[incremental]", |b| {
         b.iter_batched_ref(
             || {
-                let mut case = setup_case();
-                case.db.check_file(case.parser).unwrap();
+                let case = setup_case();
+                case.db.check().unwrap();
 
                 case.fs
                     .write_file(
@@ -82,14 +126,22 @@ fn benchmark_incremental(criterion: &mut Criterion) {
                     )
                     .unwrap();
 
-                case.re.sync(&mut case.db);
                 case
             },
             |case| {
-                let Case { db, parser, .. } = case;
-                let result = db.check_file(*parser).unwrap();
+                let Case { db, .. } = case;
 
-                assert_eq!(result.len(), 34);
+                db.apply_changes(
+                    vec![ChangeEvent::Changed {
+                        path: case.re_path.to_path_buf(),
+                        kind: ChangedKind::FileContent,
+                    }],
+                    None,
+                );
+
+                let result = db.check().unwrap();
+
+                assert_eq!(result, EXPECTED_DIAGNOSTICS);
             },
             BatchSize::SmallInput,
         );
@@ -101,10 +153,10 @@ fn benchmark_cold(criterion: &mut Criterion) {
         b.iter_batched_ref(
             setup_case,
             |case| {
-                let Case { db, parser, .. } = case;
-                let result = db.check_file(*parser).unwrap();
+                let Case { db, .. } = case;
+                let result = db.check().unwrap();
 
-                assert_eq!(result.len(), 34);
+                assert_eq!(result, EXPECTED_DIAGNOSTICS);
             },
             BatchSize::SmallInput,
         );

crates/ruff_db/src/files.rs

@@ -12,7 +12,7 @@ use ruff_notebook::{Notebook, NotebookError};
 use crate::file_revision::FileRevision;
 use crate::files::file_root::FileRoots;
 use crate::files::private::FileStatus;
-use crate::system::{Metadata, SystemPath, SystemPathBuf, SystemVirtualPath, SystemVirtualPathBuf};
+use crate::system::{SystemPath, SystemPathBuf, SystemVirtualPath, SystemVirtualPathBuf};
 use crate::vendored::{VendoredPath, VendoredPathBuf};
 use crate::{vendored, Db, FxDashMap};
 
@@ -60,8 +60,8 @@ struct FilesInner {
     /// so that queries that depend on the existence of a file are re-executed when the file is created.
     system_by_path: FxDashMap<SystemPathBuf, File>,
 
-    /// Lookup table that maps [`SystemVirtualPathBuf`]s to salsa interned [`File`] instances.
-    system_virtual_by_path: FxDashMap<SystemVirtualPathBuf, File>,
+    /// Lookup table that maps [`SystemVirtualPathBuf`]s to [`VirtualFile`] instances.
+    system_virtual_by_path: FxDashMap<SystemVirtualPathBuf, VirtualFile>,
 
     /// Lookup table that maps vendored files to the salsa [`File`] ingredients.
     vendored_by_path: FxDashMap<VendoredPathBuf, File>,
@@ -85,7 +85,7 @@ impl Files {
             .system_by_path
             .entry(absolute.clone())
             .or_insert_with(|| {
-                tracing::trace!("Adding file {path}");
+                tracing::trace!("Adding file '{path}'");
 
                 let metadata = db.system().path_metadata(path);
                 let durability = self
@@ -131,7 +131,7 @@ impl Files {
                     Err(_) => return Err(FileError::NotFound),
                 };
 
-                tracing::trace!("Adding vendored file {}", path);
+                tracing::trace!("Adding vendored file '{}'", path);
                 let file = File::builder(FilePath::Vendored(path.to_path_buf()))
                     .permissions(Some(0o444))
                     .revision(metadata.revision())
@@ -147,31 +147,31 @@ impl Files {
         Ok(file)
     }
 
-    /// Looks up a virtual file by its `path`.
+    /// Create a new virtual file at the given path and store it for future lookups.
     ///
-    /// For a non-existing file, creates a new salsa [`File`] ingredient and stores it for future lookups.
-    ///
-    /// The operations fails if the system failed to provide a metadata for the path.
-    pub fn add_virtual_file(&self, db: &dyn Db, path: &SystemVirtualPath) -> Option<File> {
-        let file = match self.inner.system_virtual_by_path.entry(path.to_path_buf()) {
-            Entry::Occupied(entry) => *entry.get(),
-            Entry::Vacant(entry) => {
-                let metadata = db.system().virtual_path_metadata(path).ok()?;
+    /// This will always create a new file, overwriting any existing file at `path` in the internal
+    /// storage.
+    pub fn virtual_file(&self, db: &dyn Db, path: &SystemVirtualPath) -> VirtualFile {
+        tracing::trace!("Adding virtual file {}", path);
+        let virtual_file = VirtualFile(
+            File::builder(FilePath::SystemVirtual(path.to_path_buf()))
+                .status(FileStatus::Exists)
+                .revision(FileRevision::zero())
+                .permissions(None)
+                .new(db),
+        );
+        self.inner
+            .system_virtual_by_path
+            .insert(path.to_path_buf(), virtual_file);
+        virtual_file
+    }
 
-                tracing::trace!("Adding virtual file {}", path);
-
-                let file = File::builder(FilePath::SystemVirtual(path.to_path_buf()))
-                    .revision(metadata.revision())
-                    .permissions(metadata.permissions())
-                    .new(db);
-
-                entry.insert(file);
-
-                file
-            }
-        };
-
-        Some(file)
+    /// Tries to look up a virtual file by its path. Returns `None` if no such file exists yet.
+    pub fn try_virtual_file(&self, path: &SystemVirtualPath) -> Option<VirtualFile> {
+        self.inner
+            .system_virtual_by_path
+            .get(&path.to_path_buf())
+            .map(|entry| *entry.value())
     }
 
     /// Looks up the closest  root for `path`. Returns `None` if `path` isn't enclosed by any source root.
@@ -211,7 +211,7 @@ impl Files {
     /// That's why [`File::sync_path`] and [`File::sync_path`] is preferred if it is known that the path is a file.
     pub fn sync_recursively(db: &mut dyn Db, path: &SystemPath) {
         let path = SystemPath::absolute(path, db.system().current_directory());
-        tracing::debug!("Syncing all files in {path}");
+        tracing::debug!("Syncing all files in '{path}'");
 
         let inner = Arc::clone(&db.files().inner);
         for entry in inner.system_by_path.iter_mut() {
@@ -224,9 +224,7 @@ impl Files {
 
         for root in roots.all() {
             if root.path(db).starts_with(&path) {
-                root.set_revision(db)
-                    .with_durability(Durability::HIGH)
-                    .to(FileRevision::now());
+                root.set_revision(db).to(FileRevision::now());
             }
         }
     }
@@ -249,9 +247,7 @@ impl Files {
         let roots = inner.roots.read().unwrap();
 
         for root in roots.all() {
-            root.set_revision(db)
-                .with_durability(Durability::HIGH)
-                .to(FileRevision::now());
+            root.set_revision(db).to(FileRevision::now());
         }
     }
 
@@ -322,6 +318,9 @@ impl File {
             }
             FilePath::Vendored(vendored) => db.vendored().read_to_string(vendored),
             FilePath::SystemVirtual(system_virtual) => {
+                // Add a dependency on the revision to ensure the operation gets re-executed when the file changes.
+                let _ = self.revision(db);
+
                 db.system().read_virtual_path_to_string(system_virtual)
             }
         }
@@ -346,6 +345,9 @@ impl File {
                 "Reading a notebook from the vendored file system is not supported.",
             ))),
             FilePath::SystemVirtual(system_virtual) => {
+                // Add a dependency on the revision to ensure the operation gets re-executed when the file changes.
+                let _ = self.revision(db);
+
                 db.system().read_virtual_path_to_notebook(system_virtual)
             }
         }
@@ -358,6 +360,13 @@ impl File {
         Self::sync_system_path(db, &absolute, None);
     }
 
+    /// Increments the revision for the virtual file at `path`.
+    pub fn sync_virtual_path(db: &mut dyn Db, path: &SystemVirtualPath) {
+        if let Some(virtual_file) = db.files().try_virtual_file(path) {
+            virtual_file.sync(db);
+        }
+    }
+
     /// Syncs the [`File`]'s state with the state of the file on the system.
     pub fn sync(self, db: &mut dyn Db) {
         let path = self.path(db).clone();
@@ -370,35 +379,20 @@ impl File {
             FilePath::Vendored(_) => {
                 // Readonly, can never be out of date.
             }
-            FilePath::SystemVirtual(system_virtual) => {
-                Self::sync_system_virtual_path(db, &system_virtual, self);
+            FilePath::SystemVirtual(_) => {
+                VirtualFile(self).sync(db);
             }
         }
     }
 
+    /// Private method providing the implementation for [`Self::sync_path`] and [`Self::sync`] for
+    /// system paths.
     fn sync_system_path(db: &mut dyn Db, path: &SystemPath, file: Option<File>) {
         let Some(file) = file.or_else(|| db.files().try_system(db, path)) else {
             return;
         };
-        let metadata = db.system().path_metadata(path);
-        let durability = db.files().root(db, path).map(|root| root.durability(db));
-        Self::sync_impl(db, metadata, file, durability);
-    }
 
-    fn sync_system_virtual_path(db: &mut dyn Db, path: &SystemVirtualPath, file: File) {
-        let metadata = db.system().virtual_path_metadata(path);
-        Self::sync_impl(db, metadata, file, None);
-    }
-
-    /// Private method providing the implementation for [`Self::sync_system_path`] and
-    /// [`Self::sync_system_virtual_path`].
-    fn sync_impl(
-        db: &mut dyn Db,
-        metadata: crate::system::Result<Metadata>,
-        file: File,
-        durability: Option<Durability>,
-    ) {
-        let (status, revision, permission) = match metadata {
+        let (status, revision, permission) = match db.system().path_metadata(path) {
             Ok(metadata) if metadata.file_type().is_file() => (
                 FileStatus::Exists,
                 metadata.revision(),
@@ -410,25 +404,19 @@ impl File {
             _ => (FileStatus::NotFound, FileRevision::zero(), None),
         };
 
-        let durability = durability.unwrap_or_default();
-
         if file.status(db) != status {
-            tracing::debug!("Updating the status of {}", file.path(db),);
-            file.set_status(db).with_durability(durability).to(status);
+            tracing::debug!("Updating the status of '{}'", file.path(db),);
+            file.set_status(db).to(status);
         }
 
         if file.revision(db) != revision {
-            tracing::debug!("Updating the revision of {}", file.path(db));
-            file.set_revision(db)
-                .with_durability(durability)
-                .to(revision);
+            tracing::debug!("Updating the revision of '{}'", file.path(db));
+            file.set_revision(db).to(revision);
         }
 
         if file.permissions(db) != permission {
-            tracing::debug!("Updating the permissions of {}", file.path(db),);
-            file.set_permissions(db)
-                .with_durability(durability)
-                .to(permission);
+            tracing::debug!("Updating the permissions of '{}'", file.path(db),);
+            file.set_permissions(db).to(permission);
         }
     }
 
@@ -438,6 +426,35 @@ impl File {
     }
 }
 
+/// A virtual file that doesn't exist on the file system.
+///
+/// This is a wrapper around a [`File`] that provides additional methods to interact with a virtual
+/// file.
+#[derive(Copy, Clone)]
+pub struct VirtualFile(File);
+
+impl VirtualFile {
+    /// Returns the underlying [`File`].
+    pub fn file(&self) -> File {
+        self.0
+    }
+
+    /// Increments the revision of the underlying [`File`].
+    fn sync(&self, db: &mut dyn Db) {
+        let file = self.0;
+        tracing::debug!("Updating the revision of '{}'", file.path(db));
+        let current_revision = file.revision(db);
+        file.set_revision(db)
+            .to(FileRevision::new(current_revision.as_u128() + 1));
+    }
+
+    /// Closes the virtual file.
+    pub fn close(&self, db: &mut dyn Db) {
+        tracing::debug!("Closing virtual file '{}'", self.0.path(db));
+        self.0.set_status(db).to(FileStatus::NotFound);
+    }
+}
+
 // The types in here need to be public because they're salsa ingredients but we
 // don't want them to be publicly accessible. That's why we put them into a private module.
 mod private {

crates/ruff_db/src/parsed.rs

@@ -121,9 +121,9 @@ mod tests {
 
         db.write_virtual_file(path, "x = 10");
 
-        let file = db.files().add_virtual_file(&db, path).unwrap();
+        let virtual_file = db.files().virtual_file(&db, path);
 
-        let parsed = parsed_module(&db, file);
+        let parsed = parsed_module(&db, virtual_file.file());
 
         assert!(parsed.is_valid());
 
@@ -137,9 +137,9 @@ mod tests {
 
         db.write_virtual_file(path, "%timeit a = b");
 
-        let file = db.files().add_virtual_file(&db, path).unwrap();
+        let virtual_file = db.files().virtual_file(&db, path);
 
-        let parsed = parsed_module(&db, file);
+        let parsed = parsed_module(&db, virtual_file.file());
 
         assert!(parsed.is_valid());
 

crates/ruff_db/src/source.rs

@@ -22,7 +22,7 @@ pub fn source_text(db: &dyn Db, file: File) -> SourceText {
     let kind = if is_notebook(file.path(db)) {
         file.read_to_notebook(db)
             .unwrap_or_else(|error| {
-                tracing::debug!("Failed to read notebook {path}: {error}");
+                tracing::debug!("Failed to read notebook '{path}': {error}");
 
                 has_read_error = true;
                 SourceDiagnostic(Arc::new(SourceTextError::FailedToReadNotebook(error)))
@@ -33,7 +33,7 @@ pub fn source_text(db: &dyn Db, file: File) -> SourceText {
     } else {
         file.read_to_string(db)
             .unwrap_or_else(|error| {
-                tracing::debug!("Failed to read file {path}: {error}");
+                tracing::debug!("Failed to read file '{path}': {error}");
 
                 has_read_error = true;
                 SourceDiagnostic(Arc::new(SourceTextError::FailedToReadFile(error))).accumulate(db);

crates/ruff_db/src/system.rs

@@ -63,9 +63,6 @@ pub trait System: Debug {
     /// representation fall-back to deserializing the notebook from a string.
     fn read_to_notebook(&self, path: &SystemPath) -> std::result::Result<Notebook, NotebookError>;
 
-    /// Reads the metadata of the virtual file at `path`.
-    fn virtual_path_metadata(&self, path: &SystemVirtualPath) -> Result<Metadata>;
-
     /// Reads the content of the virtual file at `path` into a [`String`].
     fn read_virtual_path_to_string(&self, path: &SystemVirtualPath) -> Result<String>;
 

crates/ruff_db/src/system/memory_fs.rs

@@ -136,22 +136,6 @@ impl MemoryFileSystem {
         ruff_notebook::Notebook::from_source_code(&content)
     }
 
-    pub(crate) fn virtual_path_metadata(
-        &self,
-        path: impl AsRef<SystemVirtualPath>,
-    ) -> Result<Metadata> {
-        let virtual_files = self.inner.virtual_files.read().unwrap();
-        let file = virtual_files
-            .get(&path.as_ref().to_path_buf())
-            .ok_or_else(not_found)?;
-
-        Ok(Metadata {
-            revision: file.last_modified.into(),
-            permissions: Some(MemoryFileSystem::PERMISSION),
-            file_type: FileType::File,
-        })
-    }
-
     pub(crate) fn read_virtual_path_to_string(
         &self,
         path: impl AsRef<SystemVirtualPath>,

crates/ruff_db/src/system/os.rs

@@ -77,10 +77,6 @@ impl System for OsSystem {
         Notebook::from_path(path.as_std_path())
     }
 
-    fn virtual_path_metadata(&self, _path: &SystemVirtualPath) -> Result<Metadata> {
-        Err(not_found())
-    }
-
     fn read_virtual_path_to_string(&self, _path: &SystemVirtualPath) -> Result<String> {
         Err(not_found())
     }

crates/ruff_db/src/system/test.rs

@@ -69,13 +69,6 @@ impl System for TestSystem {
         }
     }
 
-    fn virtual_path_metadata(&self, path: &SystemVirtualPath) -> Result<Metadata> {
-        match &self.inner {
-            TestSystemInner::Stub(fs) => fs.virtual_path_metadata(path),
-            TestSystemInner::System(system) => system.virtual_path_metadata(path),
-        }
-    }
-
     fn read_virtual_path_to_string(&self, path: &SystemVirtualPath) -> Result<String> {
         match &self.inner {
             TestSystemInner::Stub(fs) => fs.read_virtual_path_to_string(path),

crates/ruff_db/src/testing.rs

@@ -31,10 +31,20 @@ pub fn assert_const_function_query_was_not_run<Db, Q, QDb, R>(
     Db: salsa::Database,
     Q: Fn(QDb) -> R,
 {
-    let (query_name, will_execute_event) = find_will_execute_event(db, query, (), events);
+    // Salsa now interns singleton ingredients. But we know that it is a singleton, so we can just search for
+    // any event of that ingredient.
+    let query_name = query_name(&query);
+
+    let event = events.iter().find(|event| {
+        if let salsa::EventKind::WillExecute { database_key } = event.kind {
+            db.ingredient_debug_name(database_key.ingredient_index()) == query_name
+        } else {
+            false
+        }
+    });
 
     db.attach(|_| {
-        if let Some(will_execute_event) = will_execute_event {
+        if let Some(will_execute_event) = event {
             panic!(
                 "Expected query {query_name}() not to have run but it did: {will_execute_event:?}"
             );

crates/ruff_db/src/vendored.rs

@@ -97,7 +97,16 @@ impl VendoredFileSystem {
         fn read_to_string(fs: &VendoredFileSystem, path: &VendoredPath) -> Result<String> {
             let mut archive = fs.lock_archive();
             let mut zip_file = archive.lookup_path(&NormalizedVendoredPath::from(path))?;
-            let mut buffer = String::new();
+
+            // Pre-allocate the buffer with the size specified in the ZIP file metadata
+            // because `read_to_string` passes `None` as the size hint.
+            // But let's not trust the zip file metadata (even though it's vendored)
+            // and limit it to a reasonable size.
+            let mut buffer = String::with_capacity(
+                usize::try_from(zip_file.size())
+                    .unwrap_or(usize::MAX)
+                    .min(10_000_000),
+            );
             zip_file.read_to_string(&mut buffer)?;
             Ok(buffer)
         }

crates/ruff_linter/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "ruff_linter"
-version = "0.6.1"
+version = "0.6.2"
 publish = false
 authors = { workspace = true }
 edition = { workspace = true }

crates/ruff_linter/resources/test/fixtures/flake8_pyi/PYI053.pyi

@@ -66,3 +66,6 @@ def not_warnings_dot_deprecated(
 def not_a_deprecated_function() -> None: ...
 
 fbaz: str = f"51 character {foo} stringgggggggggggggggggggggggggg"  # Error: PYI053
+
+# see https://github.com/astral-sh/ruff/issues/12995
+def foo(bar: typing.Literal["a", "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"]):...

crates/ruff_linter/resources/test/fixtures/flake8_simplify/SIM115.py

@@ -47,7 +47,6 @@ with contextlib.ExitStack() as exit_stack:
 open("filename", "w").close()
 pathlib.Path("filename").open("w").close()
 
-
 # OK (custom context manager)
 class MyFile:
     def __init__(self, filename: str):
@@ -58,3 +57,189 @@ class MyFile:
 
     def __exit__(self, exc_type, exc_val, exc_tb):
         self.file.close()
+
+
+import tempfile
+import tarfile
+from tarfile import TarFile
+import zipfile
+import io
+import codecs
+import bz2
+import gzip
+import dbm
+import dbm.gnu
+import dbm.ndbm
+import dbm.dumb
+import lzma
+import shelve
+import tokenize
+import wave
+import fileinput
+
+f = tempfile.NamedTemporaryFile()
+f = tempfile.TemporaryFile()
+f = tempfile.SpooledTemporaryFile()
+f = tarfile.open("foo.tar")
+f = TarFile("foo.tar").open()
+f = tarfile.TarFile("foo.tar").open()
+f = tarfile.TarFile().open()
+f = zipfile.ZipFile("foo.zip").open("foo.txt")
+f = io.open("foo.txt")
+f = io.open_code("foo.txt")
+f = codecs.open("foo.txt")
+f = bz2.open("foo.txt")
+f = gzip.open("foo.txt")
+f = dbm.open("foo.db")
+f = dbm.gnu.open("foo.db")
+f = dbm.ndbm.open("foo.db")
+f = dbm.dumb.open("foo.db")
+f = lzma.open("foo.xz")
+f = lzma.LZMAFile("foo.xz")
+f = shelve.open("foo.db")
+f = tokenize.open("foo.py")
+f = wave.open("foo.wav")
+f = tarfile.TarFile.taropen("foo.tar")
+f = fileinput.input("foo.txt")
+f = fileinput.FileInput("foo.txt")
+
+with contextlib.suppress(Exception):
+    # The following line is for example's sake.
+    # For some f's above, this would raise an error (since it'd be f.readline() etc.)
+    data = f.read()
+
+f.close()
+
+# OK
+with tempfile.TemporaryFile() as f:
+    data = f.read()
+
+# OK
+with tarfile.open("foo.tar") as f:
+    data = f.add("foo.txt")
+
+# OK
+with tarfile.TarFile("foo.tar") as f:
+    data = f.add("foo.txt")
+
+# OK
+with tarfile.TarFile("foo.tar").open() as f:
+    data = f.add("foo.txt")
+
+# OK
+with zipfile.ZipFile("foo.zip") as f:
+    data = f.read("foo.txt")
+
+# OK
+with zipfile.ZipFile("foo.zip").open("foo.txt") as f:
+    data = f.read()
+
+# OK
+with zipfile.ZipFile("foo.zip") as zf:
+    with zf.open("foo.txt") as f:
+        data = f.read()
+
+# OK
+with io.open("foo.txt") as f:
+    data = f.read()
+
+# OK
+with io.open_code("foo.txt") as f:
+    data = f.read()
+
+# OK
+with codecs.open("foo.txt") as f:
+    data = f.read()
+
+# OK
+with bz2.open("foo.txt") as f:
+    data = f.read()
+
+# OK
+with gzip.open("foo.txt") as f:
+    data = f.read()
+
+# OK
+with dbm.open("foo.db") as f:
+    data = f.get("foo")
+
+# OK
+with dbm.gnu.open("foo.db") as f:
+    data = f.get("foo")
+
+# OK
+with dbm.ndbm.open("foo.db") as f:
+    data = f.get("foo")
+
+# OK
+with dbm.dumb.open("foo.db") as f:
+    data = f.get("foo")
+
+# OK
+with lzma.open("foo.xz") as f:
+    data = f.read()
+
+# OK
+with lzma.LZMAFile("foo.xz") as f:
+    data = f.read()
+
+# OK
+with shelve.open("foo.db") as f:
+    data = f["foo"]
+
+# OK
+with tokenize.open("foo.py") as f:
+    data = f.read()
+
+# OK
+with wave.open("foo.wav") as f:
+    data = f.readframes(1024)
+
+# OK
+with tarfile.TarFile.taropen("foo.tar") as f:
+    data = f.add("foo.txt")
+
+# OK
+with fileinput.input("foo.txt") as f:
+    data = f.readline()
+
+# OK
+with fileinput.FileInput("foo.txt") as f:
+    data = f.readline()
+
+# OK (quick one-liner to clear file contents)
+tempfile.NamedTemporaryFile().close()
+tempfile.TemporaryFile().close()
+tempfile.SpooledTemporaryFile().close()
+tarfile.open("foo.tar").close()
+tarfile.TarFile("foo.tar").close()
+tarfile.TarFile("foo.tar").open().close()
+tarfile.TarFile.open("foo.tar").close()
+zipfile.ZipFile("foo.zip").close()
+zipfile.ZipFile("foo.zip").open("foo.txt").close()
+io.open("foo.txt").close()
+io.open_code("foo.txt").close()
+codecs.open("foo.txt").close()
+bz2.open("foo.txt").close()
+gzip.open("foo.txt").close()
+dbm.open("foo.db").close()
+dbm.gnu.open("foo.db").close()
+dbm.ndbm.open("foo.db").close()
+dbm.dumb.open("foo.db").close()
+lzma.open("foo.xz").close()
+lzma.LZMAFile("foo.xz").close()
+shelve.open("foo.db").close()
+tokenize.open("foo.py").close()
+wave.open("foo.wav").close()
+tarfile.TarFile.taropen("foo.tar").close()
+fileinput.input("foo.txt").close()
+fileinput.FileInput("foo.txt").close()
+
+def aliased():
+    from shelve import open as open_shelf
+    x = open_shelf("foo.dbm")
+    x.close()
+
+    from tarfile import TarFile as TF
+    f = TF("foo").open()
+    f.close()

crates/ruff_linter/resources/test/fixtures/ruff/RUF027_0.py

@@ -38,7 +38,7 @@ def tripled_quoted():
     c = a
     single_line = """ {a} """  # RUF027
     # RUF027
-    multi_line = a = """b { # comment
+    multi_line = """b { # comment
     c}  d
     """
 
@@ -72,3 +72,16 @@ def method_calls():
 def format_specifiers():
     a = 4
     b = "{a:b} {a:^5}"
+
+
+def implicitly_concatenated_fstring(x: int):
+    y = f"{x}" "{x}"  # RUF027
+    z = "{x}" f"{x}"  # RUF027
+
+
+def method_call_passed_to_logging(x: int):
+    # We special-case bindings that are passed to logging calls later on,
+    # but not bindings that involve arbitrary call expressions!
+    y = foo("{x}")  # RUF027
+    import logging
+    logging.log(0, y)

crates/ruff_linter/resources/test/fixtures/ruff/RUF027_1.py

@@ -68,3 +68,33 @@ def negative_cases():
     @app.get("/items/{item_id}")
     async def read_item(item_id):
         return {"item_id": item_id}
+
+
+# we shouldn't flag either of these, because the bindings are used elsewhere
+# in contexts that make it clear that they're not meant to be f-strings
+GLOBAL_STRING = "foo {bar}"
+LOGGING_TEMPLATE = "{foo} bar"
+
+def uses_global_strings():
+    print(GLOBAL_STRING.format(bar="whatever"))
+
+    import logging
+    logging.error(LOGGING_TEMPLATE, 42)
+
+
+def binding_defined_after_string():
+    if bool():
+        x = "{foo}"
+    else:
+        x = "{foo}"
+    foo = 42
+    print(x.format(foo=foo))
+
+
+def variable_immediately_used_after_more_complex_binding(arg: str, as_pypath: bool):
+    msg = (
+        "module or package not found: {arg} (missing __init__.py?)"
+        if as_pypath
+        else "file or directory not found: {arg}"
+    )
+    raise TypeError(msg.format(arg=arg))

crates/ruff_linter/resources/test/fixtures/ruff/RUF032.py

@@ -0,0 +1,120 @@
+import decimal
+
+# Tests with fully qualified import
+decimal.Decimal(0)
+
+decimal.Decimal(0.0)  # Should error
+
+decimal.Decimal("0.0")
+
+decimal.Decimal(10)
+
+decimal.Decimal(10.0)  # Should error
+
+decimal.Decimal("10.0")
+
+decimal.Decimal(-10)
+
+decimal.Decimal(-10.0)  # Should error
+
+decimal.Decimal("-10.0")
+
+a = 10.0
+
+decimal.Decimal(a)
+
+
+# Tests with relative import
+from decimal import Decimal
+
+
+val = Decimal(0)
+
+val = Decimal(0.0)  # Should error
+
+val = Decimal("0.0")
+
+val = Decimal(10)
+
+val = Decimal(10.0)  # Should error
+
+val = Decimal("10.0")
+
+val = Decimal(-10)
+
+val = Decimal(-10.0)  # Should error
+
+val = Decimal("-10.0")
+
+a = 10.0
+
+val = Decimal(a)
+
+
+# Tests with shadowed name
+class Decimal():
+    value: float | int | str
+
+    def __init__(self, value: float | int | str) -> None:
+        self.value = value
+
+
+val = Decimal(0.0)
+
+val = Decimal("0.0")
+
+val = Decimal(10.0)
+
+val = Decimal("10.0")
+
+val = Decimal(-10.0)
+
+val = Decimal("-10.0")
+
+a = 10.0
+
+val = Decimal(a)
+
+
+# Retest with fully qualified import
+
+val = decimal.Decimal(0.0)  # Should error
+
+val = decimal.Decimal("0.0")
+
+val = decimal.Decimal(10.0)  # Should error
+
+val = decimal.Decimal("10.0")
+
+val = decimal.Decimal(-10.0)  # Should error
+
+val = decimal.Decimal("-10.0")
+
+a = 10.0
+
+val = decimal.Decimal(a)
+
+
+class decimal():
+    class Decimal():
+        value: float | int | str
+
+        def __init__(self, value: float | int | str) -> None:
+            self.value = value
+
+
+val = decimal.Decimal(0.0)
+
+val = decimal.Decimal("0.0")
+
+val = decimal.Decimal(10.0)
+
+val = decimal.Decimal("10.0")
+
+val = decimal.Decimal(-10.0)
+
+val = decimal.Decimal("-10.0")
+
+a = 10.0
+
+val = decimal.Decimal(a)

crates/ruff_linter/src/checkers/ast/analyze/bindings.rs

@@ -15,6 +15,7 @@ pub(crate) fn bindings(checker: &mut Checker) {
         Rule::UnconventionalImportAlias,
         Rule::UnsortedDunderSlots,
         Rule::UnusedVariable,
+        Rule::MissingFStringSyntax,
     ]) {
         return;
     }
@@ -77,5 +78,11 @@ pub(crate) fn bindings(checker: &mut Checker) {
                 checker.diagnostics.push(diagnostic);
             }
         }
+        if checker.enabled(Rule::MissingFStringSyntax) {
+            if let Some(diagnostic) = ruff::rules::missing_fstring_syntax_binding(checker, binding)
+            {
+                checker.diagnostics.push(diagnostic);
+            }
+        }
     }
 }

crates/ruff_linter/src/checkers/ast/analyze/expression.rs

@@ -883,7 +883,7 @@ pub(crate) fn expression(expr: &Expr, checker: &mut Checker) {
                 flake8_simplify::rules::use_capital_environment_variables(checker, expr);
             }
             if checker.enabled(Rule::OpenFileWithContextHandler) {
-                flake8_simplify::rules::open_file_with_context_handler(checker, func);
+                flake8_simplify::rules::open_file_with_context_handler(checker, call);
             }
             if checker.enabled(Rule::DictGetWithNoneDefault) {
                 flake8_simplify::rules::dict_get_with_none_default(checker, expr);
@@ -1011,6 +1011,9 @@ pub(crate) fn expression(expr: &Expr, checker: &mut Checker) {
             if checker.enabled(Rule::UnnecessaryIterableAllocationForFirstElement) {
                 ruff::rules::unnecessary_iterable_allocation_for_first_element(checker, expr);
             }
+            if checker.enabled(Rule::DecimalFromFloatLiteral) {
+                ruff::rules::decimal_from_float_literal_syntax(checker, call);
+            }
             if checker.enabled(Rule::IntOnSlicedStr) {
                 refurb::rules::int_on_sliced_str(checker, call);
             }
@@ -1077,7 +1080,7 @@ pub(crate) fn expression(expr: &Expr, checker: &mut Checker) {
             }
             if checker.enabled(Rule::MissingFStringSyntax) {
                 for string_literal in value.literals() {
-                    ruff::rules::missing_fstring_syntax(checker, string_literal);
+                    ruff::rules::missing_fstring_syntax_expr(checker, string_literal);
                 }
             }
         }
@@ -1373,7 +1376,7 @@ pub(crate) fn expression(expr: &Expr, checker: &mut Checker) {
             }
             if checker.enabled(Rule::MissingFStringSyntax) {
                 for string_literal in value.as_slice() {
-                    ruff::rules::missing_fstring_syntax(checker, string_literal);
+                    ruff::rules::missing_fstring_syntax_expr(checker, string_literal);
                 }
             }
         }

crates/ruff_linter/src/codes.rs

@@ -959,6 +959,7 @@ pub fn code_to_rule(linter: Linter, code: &str) -> Option<(RuleGroup, Rule)> {
         (Ruff, "029") => (RuleGroup::Preview, rules::ruff::rules::UnusedAsync),
         (Ruff, "030") => (RuleGroup::Preview, rules::ruff::rules::AssertWithPrintMessage),
         (Ruff, "031") => (RuleGroup::Preview, rules::ruff::rules::IncorrectlyParenthesizedTupleInSubscript),
+        (Ruff, "032") => (RuleGroup::Preview, rules::ruff::rules::DecimalFromFloatLiteral),
         (Ruff, "100") => (RuleGroup::Stable, rules::ruff::rules::UnusedNOQA),
         (Ruff, "101") => (RuleGroup::Stable, rules::ruff::rules::RedirectedNOQA),
 

crates/ruff_linter/src/rules/flake8_async/mod.rs

@@ -11,6 +11,7 @@ mod tests {
 
     use crate::assert_messages;
     use crate::registry::Rule;
+    use crate::settings::types::PythonVersion;
     use crate::settings::LinterSettings;
     use crate::test::test_path;
 
@@ -36,4 +37,18 @@ mod tests {
         assert_messages!(snapshot, diagnostics);
         Ok(())
     }
+
+    #[test_case(Path::new("ASYNC109_0.py"); "asyncio")]
+    #[test_case(Path::new("ASYNC109_1.py"); "trio")]
+    fn async109_python_310_or_older(path: &Path) -> Result<()> {
+        let diagnostics = test_path(
+            Path::new("flake8_async").join(path),
+            &LinterSettings {
+                target_version: PythonVersion::Py310,
+                ..LinterSettings::for_rule(Rule::AsyncFunctionWithTimeout)
+            },
+        )?;
+        assert_messages!(path.file_name().unwrap().to_str().unwrap(), diagnostics);
+        Ok(())
+    }
 }

crates/ruff_linter/src/rules/flake8_async/rules/async_function_with_timeout.rs

@@ -6,6 +6,7 @@ use ruff_text_size::Ranged;
 
 use crate::checkers::ast::Checker;
 use crate::rules::flake8_async::helpers::AsyncModule;
+use crate::settings::types::PythonVersion;
 
 /// ## What it does
 /// Checks for `async` functions with a `timeout` argument.
@@ -86,6 +87,11 @@ pub(crate) fn async_function_with_timeout(
         AsyncModule::AsyncIo
     };
 
+    // asyncio.timeout feature was first introduced in Python 3.11
+    if module == AsyncModule::AsyncIo && checker.settings.target_version < PythonVersion::Py311 {
+        return;
+    }
+
     checker.diagnostics.push(Diagnostic::new(
         AsyncFunctionWithTimeout { module },
         timeout.range(),

crates/ruff_linter/src/rules/flake8_async/snapshots/ruff_linter__rules__flake8_async__tests__ASYNC109_0.py.snap

@@ -0,0 +1,18 @@
+---
+source: crates/ruff_linter/src/rules/flake8_async/mod.rs
+---
+ASYNC109_0.py:8:16: ASYNC109 Async function definition with a `timeout` parameter
+  |
+8 | async def func(timeout):
+  |                ^^^^^^^ ASYNC109
+9 |     ...
+  |
+  = help: Use `trio.fail_after` instead
+
+ASYNC109_0.py:12:16: ASYNC109 Async function definition with a `timeout` parameter
+   |
+12 | async def func(timeout=10):
+   |                ^^^^^^^^^^ ASYNC109
+13 |     ...
+   |
+   = help: Use `trio.fail_after` instead

crates/ruff_linter/src/rules/flake8_async/snapshots/ruff_linter__rules__flake8_async__tests__ASYNC109_1.py.snap

@@ -0,0 +1,4 @@
+---
+source: crates/ruff_linter/src/rules/flake8_async/mod.rs
+---
+

crates/ruff_linter/src/rules/flake8_pyi/rules/string_or_bytes_too_long.rs

@@ -59,6 +59,10 @@ pub(crate) fn string_or_bytes_too_long(checker: &mut Checker, string: StringLike
         return;
     }
 
+    if semantic.in_annotation() {
+        return;
+    }
+
     let length = match string {
         StringLike::String(ast::ExprStringLiteral { value, .. }) => value.chars().count(),
         StringLike::Bytes(ast::ExprBytesLiteral { value, .. }) => value.len(),

crates/ruff_linter/src/rules/flake8_pyi/snapshots/ruff_linter__rules__flake8_pyi__tests__PYI053_PYI053.pyi.snap

@@ -152,6 +152,8 @@ PYI053.pyi:68:13: PYI053 [*] String and bytes literals longer than 50 characters
 67 | 
 68 | fbaz: str = f"51 character {foo} stringgggggggggggggggggggggggggg"  # Error: PYI053
    |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ PYI053
+69 | 
+70 | # see https://github.com/astral-sh/ruff/issues/12995
    |
    = help: Replace with `...`
 
@@ -161,3 +163,6 @@ PYI053.pyi:68:13: PYI053 [*] String and bytes literals longer than 50 characters
 67 67 | 
 68    |-fbaz: str = f"51 character {foo} stringgggggggggggggggggggggggggg"  # Error: PYI053
    68 |+fbaz: str = ...  # Error: PYI053
+69 69 | 
+70 70 | # see https://github.com/astral-sh/ruff/issues/12995
+71 71 | def foo(bar: typing.Literal["a", "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"]):...

crates/ruff_linter/src/rules/flake8_pytest_style/rules/fixture.rs

@@ -42,7 +42,7 @@ use super::helpers::{
 /// import pytest
 ///
 ///
-/// @pytest.fixture
+/// @pytest.fixture()
 /// def my_fixture(): ...
 /// ```
 ///
@@ -52,7 +52,7 @@ use super::helpers::{
 /// import pytest
 ///
 ///
-/// @pytest.fixture()
+/// @pytest.fixture
 /// def my_fixture(): ...
 /// ```
 ///

crates/ruff_linter/src/rules/flake8_simplify/mod.rs

@@ -58,6 +58,7 @@ mod tests {
     }
 
     #[test_case(Rule::IfElseBlockInsteadOfIfExp, Path::new("SIM108.py"))]
+    #[test_case(Rule::OpenFileWithContextHandler, Path::new("SIM115.py"))]
     fn preview_rules(rule_code: Rule, path: &Path) -> Result<()> {
         let snapshot = format!(
             "preview__{}_{}",

crates/ruff_linter/src/rules/flake8_simplify/rules/open_file_with_context_handler.rs

@@ -8,14 +8,20 @@ use ruff_text_size::Ranged;
 use crate::checkers::ast::Checker;
 
 /// ## What it does
-/// Checks for uses of the builtin `open()` function without an associated context
-/// manager.
+/// Checks for cases where files are opened (e.g., using the builtin `open()` function)
+/// without using a context manager.
 ///
 /// ## Why is this bad?
 /// If a file is opened without a context manager, it is not guaranteed that
 /// the file will be closed (e.g., if an exception is raised), which can cause
 /// resource leaks.
 ///
+/// ## Preview-mode behavior
+/// If [preview] mode is enabled, this rule will detect a wide array of IO calls where
+/// context managers could be used, such as `tempfile.TemporaryFile()` or
+/// `tarfile.TarFile(...).gzopen()`. If preview mode is not enabled, only `open()`,
+/// `builtins.open()` and `pathlib.Path(...).open()` are detected.
+///
 /// ## Example
 /// ```python
 /// file = open("foo.txt")
@@ -37,7 +43,7 @@ pub struct OpenFileWithContextHandler;
 impl Violation for OpenFileWithContextHandler {
     #[derive_message_formats]
     fn message(&self) -> String {
-        format!("Use context handler for opening files")
+        format!("Use a context manager for opening files")
     }
 }
 
@@ -113,14 +119,14 @@ fn match_exit_stack(semantic: &SemanticModel) -> bool {
 }
 
 /// Return `true` if `func` is the builtin `open` or `pathlib.Path(...).open`.
-fn is_open(semantic: &SemanticModel, func: &Expr) -> bool {
+fn is_open(semantic: &SemanticModel, call: &ast::ExprCall) -> bool {
     // Ex) `open(...)`
-    if semantic.match_builtin_expr(func, "open") {
+    if semantic.match_builtin_expr(&call.func, "open") {
         return true;
     }
 
     // Ex) `pathlib.Path(...).open()`
-    let Expr::Attribute(ast::ExprAttribute { attr, value, .. }) = func else {
+    let Expr::Attribute(ast::ExprAttribute { attr, value, .. }) = &*call.func else {
         return false;
     };
 
@@ -140,6 +146,63 @@ fn is_open(semantic: &SemanticModel, func: &Expr) -> bool {
         .is_some_and(|qualified_name| matches!(qualified_name.segments(), ["pathlib", "Path"]))
 }
 
+/// Return `true` if the expression is an `open` call or temporary file constructor.
+fn is_open_preview(semantic: &SemanticModel, call: &ast::ExprCall) -> bool {
+    let func = &*call.func;
+
+    // Ex) `open(...)`
+    if let Some(qualified_name) = semantic.resolve_qualified_name(func) {
+        return matches!(
+            qualified_name.segments(),
+            [
+                "" | "builtins"
+                    | "bz2"
+                    | "codecs"
+                    | "dbm"
+                    | "gzip"
+                    | "tarfile"
+                    | "shelve"
+                    | "tokenize"
+                    | "wave",
+                "open"
+            ] | ["dbm", "gnu" | "ndbm" | "dumb", "open"]
+                | ["fileinput", "FileInput" | "input"]
+                | ["io", "open" | "open_code"]
+                | ["lzma", "LZMAFile" | "open"]
+                | ["tarfile", "TarFile", "taropen"]
+                | [
+                    "tempfile",
+                    "TemporaryFile" | "NamedTemporaryFile" | "SpooledTemporaryFile"
+                ]
+        );
+    }
+
+    // Ex) `pathlib.Path(...).open()`
+    let Expr::Attribute(ast::ExprAttribute { attr, value, .. }) = func else {
+        return false;
+    };
+
+    let Expr::Call(ast::ExprCall { func, .. }) = &**value else {
+        return false;
+    };
+
+    // E.g. for `pathlib.Path(...).open()`, `qualified_name_of_instance.segments() == ["pathlib", "Path"]`
+    let Some(qualified_name_of_instance) = semantic.resolve_qualified_name(func) else {
+        return false;
+    };
+
+    matches!(
+        (qualified_name_of_instance.segments(), &**attr),
+        (
+            ["pathlib", "Path"] | ["zipfile", "ZipFile"] | ["lzma", "LZMAFile"],
+            "open"
+        ) | (
+            ["tarfile", "TarFile"],
+            "open" | "taropen" | "gzopen" | "bz2open" | "xzopen"
+        )
+    )
+}
+
 /// Return `true` if the current expression is followed by a `close` call.
 fn is_closed(semantic: &SemanticModel) -> bool {
     let Some(expr) = semantic.current_expression_grandparent() else {
@@ -165,11 +228,17 @@ fn is_closed(semantic: &SemanticModel) -> bool {
 }
 
 /// SIM115
-pub(crate) fn open_file_with_context_handler(checker: &mut Checker, func: &Expr) {
+pub(crate) fn open_file_with_context_handler(checker: &mut Checker, call: &ast::ExprCall) {
     let semantic = checker.semantic();
 
-    if !is_open(semantic, func) {
-        return;
+    if checker.settings.preview.is_disabled() {
+        if !is_open(semantic, call) {
+            return;
+        }
+    } else {
+        if !is_open_preview(semantic, call) {
+            return;
+        }
     }
 
     // Ex) `open("foo.txt").close()`
@@ -201,7 +270,8 @@ pub(crate) fn open_file_with_context_handler(checker: &mut Checker, func: &Expr)
         }
     }
 
-    checker
-        .diagnostics
-        .push(Diagnostic::new(OpenFileWithContextHandler, func.range()));
+    checker.diagnostics.push(Diagnostic::new(
+        OpenFileWithContextHandler,
+        call.func.range(),
+    ));
 }

crates/ruff_linter/src/rules/flake8_simplify/snapshots/ruff_linter__rules__flake8_simplify__tests__SIM115_SIM115.py.snap

@@ -1,7 +1,7 @@
 ---
 source: crates/ruff_linter/src/rules/flake8_simplify/mod.rs
 ---
-SIM115.py:8:5: SIM115 Use context handler for opening files
+SIM115.py:8:5: SIM115 Use a context manager for opening files
    |
  7 | # SIM115
  8 | f = open("foo.txt")
@@ -10,7 +10,7 @@ SIM115.py:8:5: SIM115 Use context handler for opening files
 10 | f = pathlib.Path("foo.txt").open()
    |
 
-SIM115.py:9:5: SIM115 Use context handler for opening files
+SIM115.py:9:5: SIM115 Use a context manager for opening files
    |
  7 | # SIM115
  8 | f = open("foo.txt")
@@ -20,7 +20,7 @@ SIM115.py:9:5: SIM115 Use context handler for opening files
 11 | f = pl.Path("foo.txt").open()
    |
 
-SIM115.py:10:5: SIM115 Use context handler for opening files
+SIM115.py:10:5: SIM115 Use a context manager for opening files
    |
  8 | f = open("foo.txt")
  9 | f = Path("foo.txt").open()
@@ -30,7 +30,7 @@ SIM115.py:10:5: SIM115 Use context handler for opening files
 12 | f = P("foo.txt").open()
    |
 
-SIM115.py:11:5: SIM115 Use context handler for opening files
+SIM115.py:11:5: SIM115 Use a context manager for opening files
    |
  9 | f = Path("foo.txt").open()
 10 | f = pathlib.Path("foo.txt").open()
@@ -40,7 +40,7 @@ SIM115.py:11:5: SIM115 Use context handler for opening files
 13 | data = f.read()
    |
 
-SIM115.py:12:5: SIM115 Use context handler for opening files
+SIM115.py:12:5: SIM115 Use a context manager for opening files
    |
 10 | f = pathlib.Path("foo.txt").open()
 11 | f = pl.Path("foo.txt").open()
@@ -50,7 +50,7 @@ SIM115.py:12:5: SIM115 Use context handler for opening files
 14 | f.close()
    |
 
-SIM115.py:39:9: SIM115 Use context handler for opening files
+SIM115.py:39:9: SIM115 Use a context manager for opening files
    |
 37 | # SIM115
 38 | with contextlib.ExitStack():
@@ -59,5 +59,3 @@ SIM115.py:39:9: SIM115 Use context handler for opening files
 40 | 
 41 | # OK
    |
-
-

crates/ruff_linter/src/rules/flake8_simplify/snapshots/ruff_linter__rules__flake8_simplify__tests__preview__SIM115_SIM115.py.snap

@@ -0,0 +1,326 @@
+---
+source: crates/ruff_linter/src/rules/flake8_simplify/mod.rs
+---
+SIM115.py:8:5: SIM115 Use a context manager for opening files
+   |
+ 7 | # SIM115
+ 8 | f = open("foo.txt")
+   |     ^^^^ SIM115
+ 9 | f = Path("foo.txt").open()
+10 | f = pathlib.Path("foo.txt").open()
+   |
+
+SIM115.py:9:5: SIM115 Use a context manager for opening files
+   |
+ 7 | # SIM115
+ 8 | f = open("foo.txt")
+ 9 | f = Path("foo.txt").open()
+   |     ^^^^^^^^^^^^^^^^^^^^ SIM115
+10 | f = pathlib.Path("foo.txt").open()
+11 | f = pl.Path("foo.txt").open()
+   |
+
+SIM115.py:10:5: SIM115 Use a context manager for opening files
+   |
+ 8 | f = open("foo.txt")
+ 9 | f = Path("foo.txt").open()
+10 | f = pathlib.Path("foo.txt").open()
+   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ SIM115
+11 | f = pl.Path("foo.txt").open()
+12 | f = P("foo.txt").open()
+   |
+
+SIM115.py:11:5: SIM115 Use a context manager for opening files
+   |
+ 9 | f = Path("foo.txt").open()
+10 | f = pathlib.Path("foo.txt").open()
+11 | f = pl.Path("foo.txt").open()
+   |     ^^^^^^^^^^^^^^^^^^^^^^^ SIM115
+12 | f = P("foo.txt").open()
+13 | data = f.read()
+   |
+
+SIM115.py:12:5: SIM115 Use a context manager for opening files
+   |
+10 | f = pathlib.Path("foo.txt").open()
+11 | f = pl.Path("foo.txt").open()
+12 | f = P("foo.txt").open()
+   |     ^^^^^^^^^^^^^^^^^ SIM115
+13 | data = f.read()
+14 | f.close()
+   |
+
+SIM115.py:39:9: SIM115 Use a context manager for opening files
+   |
+37 | # SIM115
+38 | with contextlib.ExitStack():
+39 |     f = open("filename")
+   |         ^^^^ SIM115
+40 | 
+41 | # OK
+   |
+
+SIM115.py:80:5: SIM115 Use a context manager for opening files
+   |
+78 | import fileinput
+79 | 
+80 | f = tempfile.NamedTemporaryFile()
+   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^ SIM115
+81 | f = tempfile.TemporaryFile()
+82 | f = tempfile.SpooledTemporaryFile()
+   |
+
+SIM115.py:81:5: SIM115 Use a context manager for opening files
+   |
+80 | f = tempfile.NamedTemporaryFile()
+81 | f = tempfile.TemporaryFile()
+   |     ^^^^^^^^^^^^^^^^^^^^^^ SIM115
+82 | f = tempfile.SpooledTemporaryFile()
+83 | f = tarfile.open("foo.tar")
+   |
+
+SIM115.py:82:5: SIM115 Use a context manager for opening files
+   |
+80 | f = tempfile.NamedTemporaryFile()
+81 | f = tempfile.TemporaryFile()
+82 | f = tempfile.SpooledTemporaryFile()
+   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ SIM115
+83 | f = tarfile.open("foo.tar")
+84 | f = TarFile("foo.tar").open()
+   |
+
+SIM115.py:83:5: SIM115 Use a context manager for opening files
+   |
+81 | f = tempfile.TemporaryFile()
+82 | f = tempfile.SpooledTemporaryFile()
+83 | f = tarfile.open("foo.tar")
+   |     ^^^^^^^^^^^^ SIM115
+84 | f = TarFile("foo.tar").open()
+85 | f = tarfile.TarFile("foo.tar").open()
+   |
+
+SIM115.py:84:5: SIM115 Use a context manager for opening files
+   |
+82 | f = tempfile.SpooledTemporaryFile()
+83 | f = tarfile.open("foo.tar")
+84 | f = TarFile("foo.tar").open()
+   |     ^^^^^^^^^^^^^^^^^^^^^^^ SIM115
+85 | f = tarfile.TarFile("foo.tar").open()
+86 | f = tarfile.TarFile().open()
+   |
+
+SIM115.py:85:5: SIM115 Use a context manager for opening files
+   |
+83 | f = tarfile.open("foo.tar")
+84 | f = TarFile("foo.tar").open()
+85 | f = tarfile.TarFile("foo.tar").open()
+   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ SIM115
+86 | f = tarfile.TarFile().open()
+87 | f = zipfile.ZipFile("foo.zip").open("foo.txt")
+   |
+
+SIM115.py:86:5: SIM115 Use a context manager for opening files
+   |
+84 | f = TarFile("foo.tar").open()
+85 | f = tarfile.TarFile("foo.tar").open()
+86 | f = tarfile.TarFile().open()
+   |     ^^^^^^^^^^^^^^^^^^^^^^ SIM115
+87 | f = zipfile.ZipFile("foo.zip").open("foo.txt")
+88 | f = io.open("foo.txt")
+   |
+
+SIM115.py:87:5: SIM115 Use a context manager for opening files
+   |
+85 | f = tarfile.TarFile("foo.tar").open()
+86 | f = tarfile.TarFile().open()
+87 | f = zipfile.ZipFile("foo.zip").open("foo.txt")
+   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ SIM115
+88 | f = io.open("foo.txt")
+89 | f = io.open_code("foo.txt")
+   |
+
+SIM115.py:88:5: SIM115 Use a context manager for opening files
+   |
+86 | f = tarfile.TarFile().open()
+87 | f = zipfile.ZipFile("foo.zip").open("foo.txt")
+88 | f = io.open("foo.txt")
+   |     ^^^^^^^ SIM115
+89 | f = io.open_code("foo.txt")
+90 | f = codecs.open("foo.txt")
+   |
+
+SIM115.py:89:5: SIM115 Use a context manager for opening files
+   |
+87 | f = zipfile.ZipFile("foo.zip").open("foo.txt")
+88 | f = io.open("foo.txt")
+89 | f = io.open_code("foo.txt")
+   |     ^^^^^^^^^^^^ SIM115
+90 | f = codecs.open("foo.txt")
+91 | f = bz2.open("foo.txt")
+   |
+
+SIM115.py:90:5: SIM115 Use a context manager for opening files
+   |
+88 | f = io.open("foo.txt")
+89 | f = io.open_code("foo.txt")
+90 | f = codecs.open("foo.txt")
+   |     ^^^^^^^^^^^ SIM115
+91 | f = bz2.open("foo.txt")
+92 | f = gzip.open("foo.txt")
+   |
+
+SIM115.py:91:5: SIM115 Use a context manager for opening files
+   |
+89 | f = io.open_code("foo.txt")
+90 | f = codecs.open("foo.txt")
+91 | f = bz2.open("foo.txt")
+   |     ^^^^^^^^ SIM115
+92 | f = gzip.open("foo.txt")
+93 | f = dbm.open("foo.db")
+   |
+
+SIM115.py:92:5: SIM115 Use a context manager for opening files
+   |
+90 | f = codecs.open("foo.txt")
+91 | f = bz2.open("foo.txt")
+92 | f = gzip.open("foo.txt")
+   |     ^^^^^^^^^ SIM115
+93 | f = dbm.open("foo.db")
+94 | f = dbm.gnu.open("foo.db")
+   |
+
+SIM115.py:93:5: SIM115 Use a context manager for opening files
+   |
+91 | f = bz2.open("foo.txt")
+92 | f = gzip.open("foo.txt")
+93 | f = dbm.open("foo.db")
+   |     ^^^^^^^^ SIM115
+94 | f = dbm.gnu.open("foo.db")
+95 | f = dbm.ndbm.open("foo.db")
+   |
+
+SIM115.py:94:5: SIM115 Use a context manager for opening files
+   |
+92 | f = gzip.open("foo.txt")
+93 | f = dbm.open("foo.db")
+94 | f = dbm.gnu.open("foo.db")
+   |     ^^^^^^^^^^^^ SIM115
+95 | f = dbm.ndbm.open("foo.db")
+96 | f = dbm.dumb.open("foo.db")
+   |
+
+SIM115.py:95:5: SIM115 Use a context manager for opening files
+   |
+93 | f = dbm.open("foo.db")
+94 | f = dbm.gnu.open("foo.db")
+95 | f = dbm.ndbm.open("foo.db")
+   |     ^^^^^^^^^^^^^ SIM115
+96 | f = dbm.dumb.open("foo.db")
+97 | f = lzma.open("foo.xz")
+   |
+
+SIM115.py:96:5: SIM115 Use a context manager for opening files
+   |
+94 | f = dbm.gnu.open("foo.db")
+95 | f = dbm.ndbm.open("foo.db")
+96 | f = dbm.dumb.open("foo.db")
+   |     ^^^^^^^^^^^^^ SIM115
+97 | f = lzma.open("foo.xz")
+98 | f = lzma.LZMAFile("foo.xz")
+   |
+
+SIM115.py:97:5: SIM115 Use a context manager for opening files
+   |
+95 | f = dbm.ndbm.open("foo.db")
+96 | f = dbm.dumb.open("foo.db")
+97 | f = lzma.open("foo.xz")
+   |     ^^^^^^^^^ SIM115
+98 | f = lzma.LZMAFile("foo.xz")
+99 | f = shelve.open("foo.db")
+   |
+
+SIM115.py:98:5: SIM115 Use a context manager for opening files
+    |
+ 96 | f = dbm.dumb.open("foo.db")
+ 97 | f = lzma.open("foo.xz")
+ 98 | f = lzma.LZMAFile("foo.xz")
+    |     ^^^^^^^^^^^^^ SIM115
+ 99 | f = shelve.open("foo.db")
+100 | f = tokenize.open("foo.py")
+    |
+
+SIM115.py:99:5: SIM115 Use a context manager for opening files
+    |
+ 97 | f = lzma.open("foo.xz")
+ 98 | f = lzma.LZMAFile("foo.xz")
+ 99 | f = shelve.open("foo.db")
+    |     ^^^^^^^^^^^ SIM115
+100 | f = tokenize.open("foo.py")
+101 | f = wave.open("foo.wav")
+    |
+
+SIM115.py:100:5: SIM115 Use a context manager for opening files
+    |
+ 98 | f = lzma.LZMAFile("foo.xz")
+ 99 | f = shelve.open("foo.db")
+100 | f = tokenize.open("foo.py")
+    |     ^^^^^^^^^^^^^ SIM115
+101 | f = wave.open("foo.wav")
+102 | f = tarfile.TarFile.taropen("foo.tar")
+    |
+
+SIM115.py:101:5: SIM115 Use a context manager for opening files
+    |
+ 99 | f = shelve.open("foo.db")
+100 | f = tokenize.open("foo.py")
+101 | f = wave.open("foo.wav")
+    |     ^^^^^^^^^ SIM115
+102 | f = tarfile.TarFile.taropen("foo.tar")
+103 | f = fileinput.input("foo.txt")
+    |
+
+SIM115.py:102:5: SIM115 Use a context manager for opening files
+    |
+100 | f = tokenize.open("foo.py")
+101 | f = wave.open("foo.wav")
+102 | f = tarfile.TarFile.taropen("foo.tar")
+    |     ^^^^^^^^^^^^^^^^^^^^^^^ SIM115
+103 | f = fileinput.input("foo.txt")
+104 | f = fileinput.FileInput("foo.txt")
+    |
+
+SIM115.py:103:5: SIM115 Use a context manager for opening files
+    |
+101 | f = wave.open("foo.wav")
+102 | f = tarfile.TarFile.taropen("foo.tar")
+103 | f = fileinput.input("foo.txt")
+    |     ^^^^^^^^^^^^^^^ SIM115
+104 | f = fileinput.FileInput("foo.txt")
+    |
+
+SIM115.py:104:5: SIM115 Use a context manager for opening files
+    |
+102 | f = tarfile.TarFile.taropen("foo.tar")
+103 | f = fileinput.input("foo.txt")
+104 | f = fileinput.FileInput("foo.txt")
+    |     ^^^^^^^^^^^^^^^^^^^ SIM115
+105 | 
+106 | with contextlib.suppress(Exception):
+    |
+
+SIM115.py:240:9: SIM115 Use a context manager for opening files
+    |
+238 | def aliased():
+239 |     from shelve import open as open_shelf
+240 |     x = open_shelf("foo.dbm")
+    |         ^^^^^^^^^^ SIM115
+241 |     x.close()
+    |
+
+SIM115.py:244:9: SIM115 Use a context manager for opening files
+    |
+243 |     from tarfile import TarFile as TF
+244 |     f = TF("foo").open()
+    |         ^^^^^^^^^^^^^^ SIM115
+245 |     f.close()
+    |

crates/ruff_linter/src/rules/flake8_type_checking/rules/typing_only_runtime_import.rs

@@ -300,7 +300,7 @@ pub(crate) fn typing_only_runtime_import(
             // Categorize the import, using coarse-grained categorization.
             let import_type = match categorize(
                 &qualified_name.to_string(),
-                0,
+                qualified_name.is_unresolved_import(),
                 &checker.settings.src,
                 checker.package(),
                 checker.settings.isort.detect_same_package,

crates/ruff_linter/src/rules/isort/categorize.rs

@@ -91,7 +91,7 @@ enum Reason<'a> {
 #[allow(clippy::too_many_arguments)]
 pub(crate) fn categorize<'a>(
     module_name: &str,
-    level: u32,
+    is_relative: bool,
     src: &[PathBuf],
     package: Option<&Path>,
     detect_same_package: bool,
@@ -103,14 +103,14 @@ pub(crate) fn categorize<'a>(
 ) -> &'a ImportSection {
     let module_base = module_name.split('.').next().unwrap();
     let (mut import_type, mut reason) = {
-        if level == 0 && module_base == "__future__" {
+        if !is_relative && module_base == "__future__" {
             (&ImportSection::Known(ImportType::Future), Reason::Future)
         } else if no_sections {
             (
                 &ImportSection::Known(ImportType::FirstParty),
                 Reason::NoSections,
             )
-        } else if level > 0 {
+        } else if is_relative {
             (
                 &ImportSection::Known(ImportType::LocalFolder),
                 Reason::NonZeroLevel,
@@ -132,7 +132,7 @@ pub(crate) fn categorize<'a>(
                 &ImportSection::Known(ImportType::FirstParty),
                 Reason::SourceMatch(src),
             )
-        } else if level == 0 && module_name == "__main__" {
+        } else if !is_relative && module_name == "__main__" {
             (
                 &ImportSection::Known(ImportType::FirstParty),
                 Reason::KnownFirstParty,
@@ -190,7 +190,7 @@ pub(crate) fn categorize_imports<'a>(
     for (alias, comments) in block.import {
         let import_type = categorize(
             &alias.module_name(),
-            0,
+            false,
             src,
             package,
             detect_same_package,
@@ -210,7 +210,7 @@ pub(crate) fn categorize_imports<'a>(
     for (import_from, aliases) in block.import_from {
         let classification = categorize(
             &import_from.module_name(),
-            import_from.level,
+            import_from.level > 0,
             src,
             package,
             detect_same_package,
@@ -230,7 +230,7 @@ pub(crate) fn categorize_imports<'a>(
     for ((import_from, alias), aliases) in block.import_from_as {
         let classification = categorize(
             &import_from.module_name(),
-            import_from.level,
+            import_from.level > 0,
             src,
             package,
             detect_same_package,
@@ -250,7 +250,7 @@ pub(crate) fn categorize_imports<'a>(
     for (import_from, comments) in block.import_from_star {
         let classification = categorize(
             &import_from.module_name(),
-            import_from.level,
+            import_from.level > 0,
             src,
             package,
             detect_same_package,

crates/ruff_linter/src/rules/pydocstyle/rules/not_missing.rs

@@ -218,6 +218,9 @@ impl Violation for UndocumentedPublicClass {
 ///             raise ValueError("Tried to greet an unhappy cat.")
 /// ```
 ///
+/// ## Options
+/// - `lint.pydocstyle.ignore-decorators`
+///
 /// ## References
 /// - [PEP 257 – Docstring Conventions](https://peps.python.org/pep-0257/)
 /// - [PEP 287 – reStructuredText Docstring Format](https://peps.python.org/pep-0287/)
@@ -305,6 +308,9 @@ impl Violation for UndocumentedPublicMethod {
 ///         raise FasterThanLightError from exc
 /// ```
 ///
+/// ## Options
+/// - `lint.pydocstyle.ignore-decorators`
+///
 /// ## References
 /// - [PEP 257 – Docstring Conventions](https://peps.python.org/pep-0257/)
 /// - [PEP 287 – reStructuredText Docstring Format](https://peps.python.org/pep-0287/)
@@ -402,6 +408,9 @@ impl Violation for UndocumentedPublicPackage {
 /// print(cat)  # "Cat: Dusty"
 /// ```
 ///
+/// ## Options
+/// - `lint.pydocstyle.ignore-decorators`
+///
 /// ## References
 /// - [PEP 257 – Docstring Conventions](https://peps.python.org/pep-0257/)
 /// - [PEP 287 – reStructuredText Docstring Format](https://peps.python.org/pep-0287/)
@@ -502,6 +511,9 @@ impl Violation for UndocumentedPublicNestedClass {
 ///         self.population: int = population
 /// ```
 ///
+/// ## Options
+/// - `lint.pydocstyle.ignore-decorators`
+///
 /// ## References
 /// - [PEP 257 – Docstring Conventions](https://peps.python.org/pep-0257/)
 /// - [PEP 287 – reStructuredText Docstring Format](https://peps.python.org/pep-0287/)

crates/ruff_linter/src/rules/pyflakes/rules/unused_import.rs

@@ -6,8 +6,7 @@ use std::collections::BTreeMap;
 
 use ruff_diagnostics::{Applicability, Diagnostic, Fix, FixAvailability, Violation};
 use ruff_macros::{derive_message_formats, violation};
-use ruff_python_ast as ast;
-use ruff_python_ast::{Stmt, StmtImportFrom};
+use ruff_python_ast::{self as ast, Stmt};
 use ruff_python_semantic::{
     AnyImport, BindingKind, Exceptions, Imported, NodeId, Scope, SemanticModel, SubmoduleImport,
 };
@@ -218,10 +217,11 @@ enum UnusedImportContext {
     Other,
 }
 
-fn is_first_party(qualified_name: &str, level: u32, checker: &Checker) -> bool {
+fn is_first_party(import: &AnyImport, checker: &Checker) -> bool {
+    let qualified_name = import.qualified_name();
     let category = isort::categorize(
-        qualified_name,
-        level,
+        &qualified_name.to_string(),
+        qualified_name.is_unresolved_import(),
         &checker.settings.src,
         checker.package(),
         checker.settings.isort.detect_same_package,
@@ -343,13 +343,6 @@ pub(crate) fn unused_import(checker: &Checker, scope: &Scope, diagnostics: &mut
         let in_except_handler =
             exceptions.intersects(Exceptions::MODULE_NOT_FOUND_ERROR | Exceptions::IMPORT_ERROR);
         let multiple = bindings.len() > 1;
-        let level = match checker.semantic().statement(import_statement) {
-            Stmt::Import(_) => 0,
-            Stmt::ImportFrom(StmtImportFrom { level, .. }) => *level,
-            _ => {
-                continue;
-            }
-        };
 
         // pair each binding with context; divide them by how we want to fix them
         let (to_reexport, to_remove): (Vec<_>, Vec<_>) = bindings
@@ -357,9 +350,7 @@ pub(crate) fn unused_import(checker: &Checker, scope: &Scope, diagnostics: &mut
             .map(|binding| {
                 let context = if in_except_handler {
                     UnusedImportContext::ExceptHandler
-                } else if in_init
-                    && is_first_party(&binding.import.qualified_name().to_string(), level, checker)
-                {
+                } else if in_init && is_first_party(&binding.import, checker) {
                     UnusedImportContext::DunderInitFirstParty {
                         dunder_all_count: DunderAllCount::from(dunder_all_exprs.len()),
                         submodule_import: binding.import.is_submodule_import(),

crates/ruff_linter/src/rules/ruff/mod.rs

@@ -57,6 +57,7 @@ mod tests {
     #[test_case(Rule::UnusedAsync, Path::new("RUF029.py"))]
     #[test_case(Rule::AssertWithPrintMessage, Path::new("RUF030.py"))]
     #[test_case(Rule::IncorrectlyParenthesizedTupleInSubscript, Path::new("RUF031.py"))]
+    #[test_case(Rule::DecimalFromFloatLiteral, Path::new("RUF032.py"))]
     #[test_case(Rule::RedirectedNOQA, Path::new("RUF101.py"))]
     fn rules(rule_code: Rule, path: &Path) -> Result<()> {
         let snapshot = format!("{}_{}", rule_code.noqa_code(), path.to_string_lossy());

crates/ruff_linter/src/rules/ruff/rules/decimal_from_float_literal.rs

@@ -0,0 +1,85 @@
+use ruff_diagnostics::{AlwaysFixableViolation, Diagnostic, Edit, Fix};
+use ruff_macros::{derive_message_formats, violation};
+use ruff_python_ast::{self as ast};
+use ruff_python_codegen::Stylist;
+use ruff_text_size::{Ranged, TextRange};
+
+use crate::checkers::ast::Checker;
+
+/// ## What it does
+/// Checks for `Decimal` calls passing a float literal.
+///
+/// ## Why is this bad?
+/// Float literals have limited precision that can lead to unexpected results.
+/// The `Decimal` class is designed to handle numbers with fixed-point precision,
+/// so a string literal should be used instead.
+///
+/// ## Example
+///
+/// ```python
+/// num = Decimal(1.2345)
+/// ```
+///
+/// Use instead:
+/// ```python
+/// num = Decimal("1.2345")
+/// ```
+///
+/// ## Fix Safety
+/// This rule's fix is marked as unsafe because it changes the underlying value
+/// of the `Decimal` instance that is constructed. This can lead to unexpected
+/// behavior if your program relies on the previous value (whether deliberately or not).
+#[violation]
+pub struct DecimalFromFloatLiteral;
+
+impl AlwaysFixableViolation for DecimalFromFloatLiteral {
+    #[derive_message_formats]
+    fn message(&self) -> String {
+        format!("`Decimal()` called with float literal argument")
+    }
+
+    fn fix_title(&self) -> String {
+        "Use a string literal instead".to_string()
+    }
+}
+
+/// RUF032: `Decimal()` called with float literal argument
+pub(crate) fn decimal_from_float_literal_syntax(checker: &mut Checker, call: &ast::ExprCall) {
+    let Some(arg) = call.arguments.args.first() else {
+        return;
+    };
+
+    if !is_arg_float_literal(arg) {
+        return;
+    }
+
+    if checker
+        .semantic()
+        .resolve_qualified_name(call.func.as_ref())
+        .is_some_and(|qualified_name| matches!(qualified_name.segments(), ["decimal", "Decimal"]))
+    {
+        let diagnostic =
+            Diagnostic::new(DecimalFromFloatLiteral, arg.range()).with_fix(fix_float_literal(
+                arg.range(),
+                &checker.generator().expr(arg),
+                checker.stylist(),
+            ));
+        checker.diagnostics.push(diagnostic);
+    }
+}
+
+fn is_arg_float_literal(arg: &ast::Expr) -> bool {
+    match arg {
+        ast::Expr::NumberLiteral(ast::ExprNumberLiteral {
+            value: ast::Number::Float(_),
+            ..
+        }) => true,
+        ast::Expr::UnaryOp(ast::ExprUnaryOp { operand, .. }) => is_arg_float_literal(operand),
+        _ => false,
+    }
+}
+
+fn fix_float_literal(range: TextRange, float_literal: &str, stylist: &Stylist) -> Fix {
+    let content = format!("{quote}{float_literal}{quote}", quote = stylist.quote());
+    Fix::unsafe_edit(Edit::range_replacement(content, range))
+}

crates/ruff_linter/src/rules/ruff/rules/missing_fstring_syntax.rs

@@ -4,7 +4,7 @@ use ruff_python_ast as ast;
 use ruff_python_literal::format::FormatSpec;
 use ruff_python_parser::parse_expression;
 use ruff_python_semantic::analyze::logging::is_logger_candidate;
-use ruff_python_semantic::{Modules, SemanticModel};
+use ruff_python_semantic::{Binding, Modules, ScopeId, SemanticModel};
 use ruff_source_file::Locator;
 use ruff_text_size::{Ranged, TextRange};
 
@@ -36,6 +36,8 @@ use crate::rules::fastapi::rules::is_fastapi_route_call;
 /// 6. Any format specifiers in the potential f-string are invalid.
 /// 7. The string is part of a function call that is known to expect a template string rather than an
 ///    evaluated f-string: for example, a [`logging`] call, a [`gettext`] call, or a [`fastAPI` path].
+/// 8. The string is assigned to a symbol where any of the *references* to that symbol are part of a
+///    function call that is known to expect a template string rather than an evaluated f-string.
 ///
 /// ## Example
 ///
@@ -61,16 +63,27 @@ pub struct MissingFStringSyntax;
 impl AlwaysFixableViolation for MissingFStringSyntax {
     #[derive_message_formats]
     fn message(&self) -> String {
-        format!(r#"Possible f-string without an `f` prefix"#)
+        format!("Possible f-string without an `f` prefix")
     }
 
     fn fix_title(&self) -> String {
-        "Add `f` prefix".into()
+        "Add `f` prefix".to_string()
     }
 }
 
 /// RUF027
-pub(crate) fn missing_fstring_syntax(checker: &mut Checker, literal: &ast::StringLiteral) {
+///
+/// Analyze a [`ast::StringLiteral`] node to see if it looks like it could be a string that was
+/// meant to be an f-string, but is missing an `f` prefix. If so, emit a diagnostic.
+///
+/// This routine skips any string literals that are part of "simple assignments", e.g. `x = "foo"`
+/// or `x: str = "foo"`. These are checked by the [`missing_fstring_syntax_binding`] function
+/// below, which is part of the same check.
+pub(crate) fn missing_fstring_syntax_expr(checker: &mut Checker, literal: &ast::StringLiteral) {
+    if !has_brackets(&literal.value) {
+        return;
+    }
+
     let semantic = checker.semantic();
 
     // we want to avoid statement expressions that are just a string literal.
@@ -82,35 +95,168 @@ pub(crate) fn missing_fstring_syntax(checker: &mut Checker, literal: &ast::Strin
         }
     }
 
+    // Simple assignments will be dealt with by `missing_fstring_syntax_binding`
+    if is_simple_assignment_to_literal(literal, semantic) {
+        return;
+    }
+
     let logger_objects = &checker.settings.logger_objects;
     let fastapi_seen = semantic.seen_module(Modules::FASTAPI);
 
+    let mut arg_names = FxHashSet::default();
+
     // We also want to avoid:
     // - Expressions inside `gettext()` calls
     // - Expressions passed to logging calls (since the `logging` module evaluates them lazily:
     //   https://docs.python.org/3/howto/logging-cookbook.html#using-particular-formatting-styles-throughout-your-application)
     // - `fastAPI` paths: https://fastapi.tiangolo.com/tutorial/path-params/
     // - Expressions where a method is immediately called on the string literal
-    if semantic
+    for call_expr in semantic
         .current_expressions()
         .filter_map(ast::Expr::as_call_expr)
-        .any(|call_expr| {
-            is_method_call_on_literal(call_expr, literal)
-                || is_gettext(call_expr, semantic)
-                || is_logger_candidate(&call_expr.func, semantic, logger_objects)
-                || (fastapi_seen && is_fastapi_route_call(call_expr, semantic))
-        })
     {
-        return;
+        if is_method_call_on_literal(call_expr, literal) {
+            return;
+        }
+        if is_gettext(call_expr, semantic) {
+            return;
+        }
+        if is_logger_candidate(&call_expr.func, semantic, logger_objects) {
+            return;
+        }
+        if fastapi_seen && is_fastapi_route_call(call_expr, semantic) {
+            return;
+        }
+        let ast::Arguments { keywords, args, .. } = &call_expr.arguments;
+        for keyword in &**keywords {
+            if let Some(ident) = keyword.arg.as_ref() {
+                arg_names.insert(&ident.id);
+            }
+        }
+        for arg in &**args {
+            if let ast::Expr::Name(ast::ExprName { id, .. }) = arg {
+                arg_names.insert(id);
+            }
+        }
     }
 
-    if should_be_fstring(literal, checker.locator(), semantic) {
+    if should_be_fstring(
+        literal,
+        checker.locator(),
+        semantic,
+        semantic.scope_id,
+        &arg_names,
+    ) {
         let diagnostic = Diagnostic::new(MissingFStringSyntax, literal.range())
             .with_fix(fix_fstring_syntax(literal.range()));
         checker.diagnostics.push(diagnostic);
     }
 }
 
+/// RUF027
+///
+/// Analyze a [`Binding`] to see if the bound variable is a string literal
+/// that's part of a "simple assignment", e.g. `x = "foo"` or `x: str = "foo"`.
+/// If it is, check to see if the string looks like it's meant to be an f-string,
+/// but is missing an `f` prefix. False positives are minimized by analyzing the references
+/// to the binding, to see if the binding is ever used in a way that indicates that the
+/// string is clearly meant to be a string template rather than an f-string.
+pub(crate) fn missing_fstring_syntax_binding(
+    checker: &Checker,
+    binding: &Binding,
+) -> Option<Diagnostic> {
+    let semantic = checker.semantic();
+    let locator = checker.locator();
+
+    let stmt = binding.statement(semantic)?;
+    let string_literal = match stmt {
+        ast::Stmt::Assign(ast::StmtAssign { targets, value, .. }) => match targets.as_slice() {
+            [_] => value.as_string_literal_expr()?,
+            _ => return None,
+        },
+        ast::Stmt::AnnAssign(ast::StmtAnnAssign {
+            value: Some(value), ..
+        }) => value.as_string_literal_expr()?,
+        _ => return None,
+    };
+    let [string_literal] = string_literal.value.as_slice() else {
+        return None;
+    };
+    if !has_brackets(&string_literal.value) {
+        return None;
+    }
+
+    let logger_objects = &checker.settings.logger_objects;
+    let fastapi_seen = semantic.seen_module(Modules::FASTAPI);
+    let mut arg_names = FxHashSet::default();
+
+    for reference in binding.references().map(|id| semantic.reference(id)) {
+        let Some(expr_id) = reference.expression_id() else {
+            continue;
+        };
+        for call_expr in semantic
+            .expressions(expr_id)
+            .filter_map(ast::Expr::as_call_expr)
+        {
+            if is_method_call_on_name(call_expr, binding.name(locator)) {
+                return None;
+            }
+            if is_gettext(call_expr, semantic) {
+                return None;
+            }
+            if is_logger_candidate(&call_expr.func, semantic, logger_objects) {
+                return None;
+            }
+            if fastapi_seen && is_fastapi_route_call(call_expr, semantic) {
+                return None;
+            }
+            let ast::Arguments { keywords, args, .. } = &call_expr.arguments;
+            for keyword in &**keywords {
+                if let Some(ident) = keyword.arg.as_ref() {
+                    arg_names.insert(&ident.id);
+                }
+            }
+            for arg in &**args {
+                if let ast::Expr::Name(ast::ExprName { id, .. }) = arg {
+                    arg_names.insert(id);
+                }
+            }
+        }
+    }
+
+    should_be_fstring(string_literal, locator, semantic, binding.scope, &arg_names).then(|| {
+        Diagnostic::new(MissingFStringSyntax, string_literal.range())
+            .with_fix(fix_fstring_syntax(string_literal.range()))
+    })
+}
+
+/// Determine whether `stmt` represents a "simple assignment" to the string literal `literal`.
+///
+/// For our purposes here, a "simple assignment" is any assignment
+/// where `literal` appears on the right-hand side
+/// and `literal` is not part of an implicitly concatenated string.
+fn is_simple_assignment_to_literal(literal: &ast::StringLiteral, semantic: &SemanticModel) -> bool {
+    if semantic.current_expressions().any(|expr| match expr {
+        ast::Expr::Call(_) => true,
+        ast::Expr::StringLiteral(ast::ExprStringLiteral { value, .. }) => {
+            value.is_implicit_concatenated()
+        }
+        ast::Expr::FString(ast::ExprFString { value, .. }) => value.is_implicit_concatenated(),
+        _ => false,
+    }) {
+        return false;
+    }
+    match semantic.current_statement() {
+        ast::Stmt::Assign(ast::StmtAssign { value, .. }) => {
+            value.range().contains_range(literal.range())
+        }
+        ast::Stmt::AnnAssign(ast::StmtAnnAssign {
+            value: Some(value), ..
+        }) => value.range().contains_range(literal.range()),
+        _ => false,
+    }
+}
+
 /// Returns `true` if an expression appears to be a `gettext` call.
 ///
 /// We want to avoid statement expressions and assignments related to aliases
@@ -162,17 +308,28 @@ fn is_method_call_on_literal(call_expr: &ast::ExprCall, literal: &ast::StringLit
     value.as_slice().contains(literal)
 }
 
+/// Determine whether `call_expr` represents a method call on a symbol bound to the name `name`.
+///
+/// E.g., `foo.format(bar=3)`, where `name == "foo"`.
+fn is_method_call_on_name(call_expr: &ast::ExprCall, name: &str) -> bool {
+    let ast::Expr::Attribute(ast::ExprAttribute { value, .. }) = &*call_expr.func else {
+        return false;
+    };
+    let ast::Expr::Name(ast::ExprName { id, .. }) = &**value else {
+        return false;
+    };
+    id == name
+}
+
 /// Returns `true` if `literal` is likely an f-string with a missing `f` prefix.
 /// See [`MissingFStringSyntax`] for the validation criteria.
 fn should_be_fstring(
     literal: &ast::StringLiteral,
     locator: &Locator,
     semantic: &SemanticModel,
+    scope: ScopeId,
+    relevant_argument_names: &FxHashSet<&ast::name::Name>,
 ) -> bool {
-    if !has_brackets(&literal.value) {
-        return false;
-    }
-
     let fstring_expr = format!("f{}", locator.slice(literal));
     let Ok(parsed) = parse_expression(&fstring_expr) else {
         return false;
@@ -183,35 +340,21 @@ fn should_be_fstring(
         return false;
     };
 
-    let mut arg_names = FxHashSet::default();
-    for expr in semantic
-        .current_expressions()
-        .filter_map(ast::Expr::as_call_expr)
-    {
-        let ast::Arguments { keywords, args, .. } = &expr.arguments;
-        for keyword in &**keywords {
-            if let Some(ident) = keyword.arg.as_ref() {
-                arg_names.insert(&ident.id);
-            }
-        }
-        for arg in &**args {
-            if let ast::Expr::Name(ast::ExprName { id, .. }) = arg {
-                arg_names.insert(id);
-            }
-        }
-    }
-
     for f_string in value.f_strings() {
         let mut has_name = false;
         for element in f_string.elements.expressions() {
             if let ast::Expr::Name(ast::ExprName { id, .. }) = element.expression.as_ref() {
-                if arg_names.contains(id) {
+                if relevant_argument_names.contains(id) {
                     return false;
                 }
-                if semantic
-                    .lookup_symbol(id)
-                    .map_or(true, |id| semantic.binding(id).kind.is_builtin())
-                {
+                let Some(binding_id) = semantic.lookup_symbol_in_scope(id, scope, false) else {
+                    return false;
+                };
+                let binding = semantic.binding(binding_id);
+                if binding.kind.is_builtin() {
+                    return false;
+                }
+                if binding.start() > literal.end() {
                     return false;
                 }
                 has_name = true;

crates/ruff_linter/src/rules/ruff/rules/mod.rs

@@ -3,6 +3,7 @@ pub(crate) use assert_with_print_message::*;
 pub(crate) use assignment_in_assert::*;
 pub(crate) use asyncio_dangling_task::*;
 pub(crate) use collection_literal_concatenation::*;
+pub(crate) use decimal_from_float_literal::*;
 pub(crate) use default_factory_kwarg::*;
 pub(crate) use explicit_f_string_type_conversion::*;
 pub(crate) use function_call_in_dataclass_default::*;
@@ -36,6 +37,7 @@ mod assignment_in_assert;
 mod asyncio_dangling_task;
 mod collection_literal_concatenation;
 mod confusables;
+mod decimal_from_float_literal;
 mod default_factory_kwarg;
 mod explicit_f_string_type_conversion;
 mod function_call_in_dataclass_default;