Cyclic0007/ruff
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[red-knot] Use arena-allocated association lists for narrowing constraints (#16306)

Browse Source 
This PR adds an implementation of [association
lists](https://en.wikipedia.org/wiki/Association_list), and uses them to
replace the previous `BitSet`/`SmallVec` representation for narrowing
constraints.

An association list is a linked list of key/value pairs. We additionally
guarantee that the elements of an association list are sorted (by their
keys), and that they do not contain any entries with duplicate keys.

Association lists have fallen out of favor in recent decades, since you
often need operations that are inefficient on them. In particular,
looking up a random element by index is O(n), just like a linked list;
and looking up an element by key is also O(n), since you must do a
linear scan of the list to find the matching element. Luckily we don't
need either of those operations for narrowing constraints!

The typical implementation also suffers from poor cache locality and
high memory allocation overhead, since individual list cells are
typically allocated separately from the heap. We solve that last problem
by storing the cells of an association list in an `IndexVec` arena.

---------

Co-authored-by: Carl Meyer <carl@astral.sh>
This commit is contained in:
Douglas Creager
2025-02-25 10:58:56 -05:00
committed by GitHub
parent 5c007db7e2
commit fa76f6cbb2
13 changed files with 1073 additions and 322 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
crates/red_knot_python_semantic/src/lib.rs
View File

@@ -27,7 +27,6 @@ pub(crate) mod symbol;
pub mod types;
mod unpack;
mod util;
mod visibility_constraints;
type FxOrderSet<V> = ordermap::set::OrderSet<V, BuildHasherDefault<FxHasher>>;

2
crates/red_knot_python_semantic/src/semantic_index.rs
View File

@@ -28,8 +28,10 @@ mod builder;
pub(crate) mod constraint;
pub mod definition;
pub mod expression;
mod narrowing_constraints;
pub mod symbol;
mod use_def;
mod visibility_constraints;
pub(crate) use self::use_def::{
BindingWithConstraints, BindingWithConstraintsIterator, DeclarationWithConstraint,

20
crates/red_knot_python_semantic/src/semantic_index/builder.rs
View File

@@ -15,10 +15,14 @@ use crate::module_name::ModuleName;
use crate::semantic_index::ast_ids::node_key::ExpressionNodeKey;
use crate::semantic_index::ast_ids::AstIdsBuilder;
use crate::semantic_index::attribute_assignment::{AttributeAssignment, AttributeAssignments};
use crate::semantic_index::constraint::{PatternConstraintKind, ScopedConstraintId};
use crate::semantic_index::constraint::{
Constraint, ConstraintNode, PatternConstraint, PatternConstraintKind, ScopedConstraintId,
};
use crate::semantic_index::definition::{
AssignmentDefinitionNodeRef, ComprehensionDefinitionNodeRef, Definition, DefinitionNodeKey,
DefinitionNodeRef, ForStmtDefinitionNodeRef, ImportFromDefinitionNodeRef,
AssignmentDefinitionNodeRef, ComprehensionDefinitionNodeRef, Definition, DefinitionCategory,
DefinitionNodeKey, DefinitionNodeRef, ExceptHandlerDefinitionNodeRef, ForStmtDefinitionNodeRef,
ImportDefinitionNodeRef, ImportFromDefinitionNodeRef, MatchPatternDefinitionNodeRef,
WithItemDefinitionNodeRef,
};
use crate::semantic_index::expression::{Expression, ExpressionKind};
use crate::semantic_index::symbol::{
@@ -28,17 +32,13 @@ use crate::semantic_index::symbol::{
use crate::semantic_index::use_def::{
EagerBindingsKey, FlowSnapshot, ScopedEagerBindingsId, UseDefMapBuilder,
};
use crate::semantic_index::visibility_constraints::{
ScopedVisibilityConstraintId, VisibilityConstraintsBuilder,
};
use crate::semantic_index::SemanticIndex;
use crate::unpack::{Unpack, UnpackValue};
use crate::visibility_constraints::{ScopedVisibilityConstraintId, VisibilityConstraintsBuilder};
use crate::Db;
use super::constraint::{Constraint, ConstraintNode, PatternConstraint};
use super::definition::{
DefinitionCategory, ExceptHandlerDefinitionNodeRef, ImportDefinitionNodeRef,
MatchPatternDefinitionNodeRef, WithItemDefinitionNodeRef,
};
mod except_handlers;
/// Are we in a state where a `break` statement is allowed?

151
crates/red_knot_python_semantic/src/semantic_index/narrowing_constraints.rs Normal file
View File

@@ -0,0 +1,151 @@
//! # Narrowing constraints
//!
//! When building a semantic index for a file, we associate each binding with _narrowing
//! constraints_. The narrowing constraint is used to constrain the type of the binding's symbol.
//! Note that a binding can be associated with a different narrowing constraint at different points
//! in a file. See the [`use_def`][crate::semantic_index::use_def] module for more details.
//!
//! This module defines how narrowing constraints are stored internally.
//!
//! A _narrowing constraint_ consists of a list of _clauses_, each of which corresponds with an
//! expression in the source file (represented by a [`Constraint`]). We need to support the
//! following operations on narrowing constraints:
//!
//! - Adding a new clause to an existing constraint
//! - Merging two constraints together, which produces the _intersection_ of their clauses
//! - Iterating through the clauses in a constraint
//!
//! In particular, note that we do not need random access to the clauses in a constraint. That
//! means that we can use a simple [_sorted association list_][ruff_index::list] as our data
//! structure. That lets us use a single 32-bit integer to store each narrowing constraint, no
//! matter how many clauses it contains. It also makes merging two narrowing constraints fast,
//! since alists support fast intersection.
//!
//! Because we visit the contents of each scope in source-file order, and assign scoped IDs in
//! source-file order, that means that we will tend to visit narrowing constraints in order by
//! their IDs. This is exactly how to get the best performance from our alist implementation.
//!
//! [`Constraint`]: crate::semantic_index::constraint::Constraint
use ruff_index::list::{ListBuilder, ListSetReverseIterator, ListStorage};
use ruff_index::newtype_index;
use crate::semantic_index::constraint::ScopedConstraintId;
/// A narrowing constraint associated with a live binding.
///
/// A constraint is a list of clauses, each of which is a [`Constraint`] that constrains the type
/// of the binding's symbol.
///
/// An instance of this type represents a _non-empty_ narrowing constraint. You will often wrap
/// this in `Option` and use `None` to represent an empty narrowing constraint.
///
/// [`Constraint`]: crate::semantic_index::constraint::Constraint
#[newtype_index]
pub(crate) struct ScopedNarrowingConstraintId;
/// One of the clauses in a narrowing constraint, which is a [`Constraint`] that constrains the
/// type of the binding's symbol.
///
/// Note that those [`Constraint`]s are stored in [their own per-scope
/// arena][crate::semantic_index::constraint::Constraints], so internally we use a
/// [`ScopedConstraintId`] to refer to the underlying constraint.
///
/// [`Constraint`]: crate::semantic_index::constraint::Constraint
#[derive(Clone, Copy, Debug, Eq, Ord, PartialEq, PartialOrd)]
pub(crate) struct ScopedNarrowingConstraintClause(ScopedConstraintId);
impl ScopedNarrowingConstraintClause {
/// Returns (the ID of) the `Constraint` for this clause
pub(crate) fn constraint(self) -> ScopedConstraintId {
self.0
}
}
impl From<ScopedConstraintId> for ScopedNarrowingConstraintClause {
fn from(constraint: ScopedConstraintId) -> ScopedNarrowingConstraintClause {
ScopedNarrowingConstraintClause(constraint)
}
}
/// A collection of narrowing constraints for a given scope.
#[derive(Debug, Eq, PartialEq)]
pub(crate) struct NarrowingConstraints {
lists: ListStorage<ScopedNarrowingConstraintId, ScopedNarrowingConstraintClause>,
}
// Building constraints
// --------------------
/// A builder for creating narrowing constraints.
#[derive(Debug, Default, Eq, PartialEq)]
pub(crate) struct NarrowingConstraintsBuilder {
lists: ListBuilder<ScopedNarrowingConstraintId, ScopedNarrowingConstraintClause>,
}
impl NarrowingConstraintsBuilder {
pub(crate) fn build(self) -> NarrowingConstraints {
NarrowingConstraints {
lists: self.lists.build(),
}
}
/// Adds a clause to an existing narrowing constraint.
pub(crate) fn add(
&mut self,
constraint: Option<ScopedNarrowingConstraintId>,
clause: ScopedNarrowingConstraintClause,
) -> Option<ScopedNarrowingConstraintId> {
self.lists.insert(constraint, clause)
}
/// Returns the intersection of two narrowing constraints. The result contains the clauses that
/// appear in both inputs.
pub(crate) fn intersect(
&mut self,
a: Option<ScopedNarrowingConstraintId>,
b: Option<ScopedNarrowingConstraintId>,
) -> Option<ScopedNarrowingConstraintId> {
self.lists.intersect(a, b)
}
}
// Iteration
// ---------
pub(crate) type NarrowingConstraintsIterator<'a> = std::iter::Copied<
ListSetReverseIterator<'a, ScopedNarrowingConstraintId, ScopedNarrowingConstraintClause>,
>;
impl NarrowingConstraints {
/// Iterates over the clauses in a narrowing constraint.
pub(crate) fn iter_clauses(
&self,
set: Option<ScopedNarrowingConstraintId>,
) -> NarrowingConstraintsIterator<'_> {
self.lists.iter_set_reverse(set).copied()
}
}
// Test support
// ------------
#[cfg(test)]
mod tests {
use super::*;
impl ScopedNarrowingConstraintClause {
pub(crate) fn as_u32(self) -> u32 {
self.0.as_u32()
}
}
impl NarrowingConstraintsBuilder {
pub(crate) fn iter_constraints(
&self,
set: Option<ScopedNarrowingConstraintId>,
) -> NarrowingConstraintsIterator<'_> {
self.lists.iter_set_reverse(set).copied()
}
}
}

44
crates/red_knot_python_semantic/src/semantic_index/use_def.rs
View File

@@ -260,20 +260,22 @@ use ruff_index::{newtype_index, IndexVec};
use rustc_hash::FxHashMap;
use self::symbol_state::{
ConstraintIndexIterator, LiveBindingsIterator, LiveDeclaration, LiveDeclarationsIterator,
ScopedDefinitionId, SymbolBindings, SymbolDeclarations, SymbolState,
LiveBindingsIterator, LiveDeclaration, LiveDeclarationsIterator, ScopedDefinitionId,
SymbolBindings, SymbolDeclarations, SymbolState,
};
use crate::semantic_index::ast_ids::ScopedUseId;
use crate::semantic_index::constraint::{
Constraint, Constraints, ConstraintsBuilder, ScopedConstraintId,
};
use crate::semantic_index::definition::Definition;
use crate::semantic_index::narrowing_constraints::{
NarrowingConstraints, NarrowingConstraintsBuilder, NarrowingConstraintsIterator,
};
use crate::semantic_index::symbol::{FileScopeId, ScopedSymbolId};
use crate::visibility_constraints::{
use crate::semantic_index::visibility_constraints::{
ScopedVisibilityConstraintId, VisibilityConstraints, VisibilityConstraintsBuilder,
};
mod bitset;
mod symbol_state;
/// Applicable definitions and constraints for every use of a name.
@@ -286,6 +288,9 @@ pub(crate) struct UseDefMap<'db> {
/// Array of [`Constraint`] in this scope.
constraints: Constraints<'db>,
/// Array of narrowing constraints in this scope.
narrowing_constraints: NarrowingConstraints,
/// Array of visibility constraints in this scope.
visibility_constraints: VisibilityConstraints,
@@ -370,6 +375,7 @@ impl<'db> UseDefMap<'db> {
BindingWithConstraintsIterator {
all_definitions: &self.all_definitions,
constraints: &self.constraints,
narrowing_constraints: &self.narrowing_constraints,
visibility_constraints: &self.visibility_constraints,
inner: bindings.iter(),
}
@@ -416,6 +422,7 @@ type EagerBindings = IndexVec<ScopedEagerBindingsId, SymbolBindings>;
pub(crate) struct BindingWithConstraintsIterator<'map, 'db> {
all_definitions: &'map IndexVec<ScopedDefinitionId, Option<Definition<'db>>>,
pub(crate) constraints: &'map Constraints<'db>,
pub(crate) narrowing_constraints: &'map NarrowingConstraints,
pub(crate) visibility_constraints: &'map VisibilityConstraints,
inner: LiveBindingsIterator<'map>,
}
@@ -425,14 +432,16 @@ impl<'map, 'db> Iterator for BindingWithConstraintsIterator<'map, 'db> {
fn next(&mut self) -> Option<Self::Item> {
let constraints = self.constraints;
let narrowing_constraints = self.narrowing_constraints;
self.inner
.next()
.map(|live_binding| BindingWithConstraints {
binding: self.all_definitions[live_binding.binding],
narrowing_constraints: ConstraintsIterator {
narrowing_constraint: ConstraintsIterator {
constraints,
constraint_ids: live_binding.narrowing_constraints.iter(),
constraint_ids: narrowing_constraints
.iter_clauses(live_binding.narrowing_constraint),
},
visibility_constraint: live_binding.visibility_constraint,
})
@@ -443,13 +452,13 @@ impl std::iter::FusedIterator for BindingWithConstraintsIterator<'_, '_> {}
pub(crate) struct BindingWithConstraints<'map, 'db> {
pub(crate) binding: Option<Definition<'db>>,
pub(crate) narrowing_constraints: ConstraintsIterator<'map, 'db>,
pub(crate) narrowing_constraint: ConstraintsIterator<'map, 'db>,
pub(crate) visibility_constraint: ScopedVisibilityConstraintId,
}
pub(crate) struct ConstraintsIterator<'map, 'db> {
constraints: &'map Constraints<'db>,
constraint_ids: ConstraintIndexIterator<'map>,
constraint_ids: NarrowingConstraintsIterator<'map>,
}
impl<'db> Iterator for ConstraintsIterator<'_, 'db> {
@@ -458,7 +467,7 @@ impl<'db> Iterator for ConstraintsIterator<'_, 'db> {
fn next(&mut self) -> Option<Self::Item> {
self.constraint_ids
.next()
.map(|constraint_id| self.constraints[ScopedConstraintId::from_u32(constraint_id)])
.map(|narrowing_constraint| self.constraints[narrowing_constraint.constraint()])
}
}
@@ -509,7 +518,10 @@ pub(super) struct UseDefMapBuilder<'db> {
all_definitions: IndexVec<ScopedDefinitionId, Option<Definition<'db>>>,
/// Builder of constraints.
constraints: ConstraintsBuilder<'db>,
pub(super) constraints: ConstraintsBuilder<'db>,
/// Builder of narrowing constraints.
pub(super) narrowing_constraints: NarrowingConstraintsBuilder,
/// Builder of visibility constraints.
pub(super) visibility_constraints: VisibilityConstraintsBuilder,
@@ -542,6 +554,7 @@ impl Default for UseDefMapBuilder<'_> {
Self {
all_definitions: IndexVec::from_iter([None]),
constraints: ConstraintsBuilder::default(),
narrowing_constraints: NarrowingConstraintsBuilder::default(),
visibility_constraints: VisibilityConstraintsBuilder::default(),
scope_start_visibility: ScopedVisibilityConstraintId::ALWAYS_TRUE,
bindings_by_use: IndexVec::new(),
@@ -578,8 +591,9 @@ impl<'db> UseDefMapBuilder<'db> {
}
pub(super) fn record_constraint_id(&mut self, constraint: ScopedConstraintId) {
let narrowing_constraint = constraint.into();
for state in &mut self.symbol_states {
state.record_constraint(constraint);
state.record_constraint(&mut self.narrowing_constraints, narrowing_constraint);
}
}
@@ -737,10 +751,15 @@ impl<'db> UseDefMapBuilder<'db> {
let mut snapshot_definitions_iter = snapshot.symbol_states.into_iter();
for current in &mut self.symbol_states {
if let Some(snapshot) = snapshot_definitions_iter.next() {
current.merge(snapshot, &mut self.visibility_constraints);
current.merge(
snapshot,
&mut self.narrowing_constraints,
&mut self.visibility_constraints,
);
} else {
current.merge(
SymbolState::undefined(snapshot.scope_start_visibility),
&mut self.narrowing_constraints,
&mut self.visibility_constraints,
);
// Symbol not present in snapshot, so it's unbound/undeclared from that path.
@@ -763,6 +782,7 @@ impl<'db> UseDefMapBuilder<'db> {
UseDefMap {
all_definitions: self.all_definitions,
constraints: self.constraints.build(),
narrowing_constraints: self.narrowing_constraints.build(),
visibility_constraints: self.visibility_constraints.build(),
bindings_by_use: self.bindings_by_use,
public_symbols: self.symbol_states,

234
crates/red_knot_python_semantic/src/semantic_index/use_def/bitset.rs
View File

@@ -1,234 +0,0 @@
/// Ordered set of `u32`.
///
/// Uses an inline bit-set for small values (up to 64 * B), falls back to heap allocated vector of
/// blocks for larger values.
#[derive(Debug, Clone, PartialEq, Eq)]
pub(super) enum BitSet<const B: usize> {
/// Bit-set (in 64-bit blocks) for the first 64 * B entries.
Inline([u64; B]),
/// Overflow beyond 64 * B.
Heap(Vec<u64>),
}
impl<const B: usize> Default for BitSet<B> {
fn default() -> Self {
// B * 64 must fit in a u32, or else we have unusable bits; this assertion makes the
// truncating casts to u32 below safe. This would be better as a const assertion, but
// that's not possible on stable with const generic params. (B should never really be
// anywhere close to this large.)
assert!(B * 64 < (u32::MAX as usize));
// This implementation requires usize >= 32 bits.
static_assertions::const_assert!(usize::BITS >= 32);
Self::Inline([0; B])
}
}
impl<const B: usize> BitSet<B> {
/// Convert from Inline to Heap, if needed, and resize the Heap vector, if needed.
fn resize(&mut self, value: u32) {
let num_blocks_needed = (value / 64) + 1;
self.resize_blocks(num_blocks_needed as usize);
}
fn resize_blocks(&mut self, num_blocks_needed: usize) {
match self {
Self::Inline(blocks) => {
let mut vec = blocks.to_vec();
vec.resize(num_blocks_needed, 0);
*self = Self::Heap(vec);
}
Self::Heap(vec) => {
vec.resize(num_blocks_needed, 0);
}
}
}
fn blocks_mut(&mut self) -> &mut [u64] {
match self {
Self::Inline(blocks) => blocks.as_mut_slice(),
Self::Heap(blocks) => blocks.as_mut_slice(),
}
}
fn blocks(&self) -> &[u64] {
match self {
Self::Inline(blocks) => blocks.as_slice(),
Self::Heap(blocks) => blocks.as_slice(),
}
}
/// Insert a value into the [`BitSet`].
///
/// Return true if the value was newly inserted, false if already present.
pub(super) fn insert(&mut self, value: u32) -> bool {
let value_usize = value as usize;
let (block, index) = (value_usize / 64, value_usize % 64);
if block >= self.blocks().len() {
self.resize(value);
}
let blocks = self.blocks_mut();
let missing = blocks[block] & (1 << index) == 0;
blocks[block] |= 1 << index;
missing
}
/// Intersect in-place with another [`BitSet`].
pub(super) fn intersect(&mut self, other: &BitSet<B>) {
let my_blocks = self.blocks_mut();
let other_blocks = other.blocks();
let min_len = my_blocks.len().min(other_blocks.len());
for i in 0..min_len {
my_blocks[i] &= other_blocks[i];
}
for block in my_blocks.iter_mut().skip(min_len) {
*block = 0;
}
}
/// Return an iterator over the values (in ascending order) in this [`BitSet`].
pub(super) fn iter(&self) -> BitSetIterator<'_, B> {
let blocks = self.blocks();
BitSetIterator {
blocks,
current_block_index: 0,
current_block: blocks[0],
}
}
}
/// Iterator over values in a [`BitSet`].
#[derive(Debug)]
pub(super) struct BitSetIterator<'a, const B: usize> {
/// The blocks we are iterating over.
blocks: &'a [u64],
/// The index of the block we are currently iterating through.
current_block_index: usize,
/// The block we are currently iterating through (and zeroing as we go.)
current_block: u64,
}
impl<const B: usize> Iterator for BitSetIterator<'_, B> {
type Item = u32;
fn next(&mut self) -> Option<Self::Item> {
while self.current_block == 0 {
if self.current_block_index + 1 >= self.blocks.len() {
return None;
}
self.current_block_index += 1;
self.current_block = self.blocks[self.current_block_index];
}
let lowest_bit_set = self.current_block.trailing_zeros();
// reset the lowest set bit, without a data dependency on `lowest_bit_set`
self.current_block &= self.current_block.wrapping_sub(1);
// SAFETY: `lowest_bit_set` cannot be more than 64, `current_block_index` cannot be more
// than `B - 1`, and we check above that `B * 64 < u32::MAX`. So both `64 *
// current_block_index` and the final value here must fit in u32.
#[allow(clippy::cast_possible_truncation)]
Some(lowest_bit_set + (64 * self.current_block_index) as u32)
}
}
impl<const B: usize> std::iter::FusedIterator for BitSetIterator<'_, B> {}
#[cfg(test)]
mod tests {
use super::BitSet;
impl<const B: usize> BitSet<B> {
/// Create and return a new [`BitSet`] with a single `value` inserted.
pub(super) fn with(value: u32) -> Self {
let mut bitset = Self::default();
bitset.insert(value);
bitset
}
}
fn assert_bitset<const B: usize>(bitset: &BitSet<B>, contents: &[u32]) {
assert_eq!(bitset.iter().collect::<Vec<_>>(), contents);
}
#[test]
fn iter() {
let mut b = BitSet::<1>::with(3);
b.insert(27);
b.insert(6);
assert!(matches!(b, BitSet::Inline(_)));
assert_bitset(&b, &[3, 6, 27]);
}
#[test]
fn iter_overflow() {
let mut b = BitSet::<1>::with(140);
b.insert(100);
b.insert(129);
assert!(matches!(b, BitSet::Heap(_)));
assert_bitset(&b, &[100, 129, 140]);
}
#[test]
fn intersect() {
let mut b1 = BitSet::<1>::with(4);
let mut b2 = BitSet::<1>::with(4);
b1.insert(23);
b2.insert(5);
b1.intersect(&b2);
assert_bitset(&b1, &[4]);
}
#[test]
fn intersect_mixed_1() {
let mut b1 = BitSet::<1>::with(4);
let mut b2 = BitSet::<1>::with(4);
b1.insert(89);
b2.insert(5);
b1.intersect(&b2);
assert_bitset(&b1, &[4]);
}
#[test]
fn intersect_mixed_2() {
let mut b1 = BitSet::<1>::with(4);
let mut b2 = BitSet::<1>::with(4);
b1.insert(23);
b2.insert(89);
b1.intersect(&b2);
assert_bitset(&b1, &[4]);
}
#[test]
fn intersect_heap() {
let mut b1 = BitSet::<1>::with(4);
let mut b2 = BitSet::<1>::with(4);
b1.insert(89);
b2.insert(90);
b1.intersect(&b2);
assert_bitset(&b1, &[4]);
}
#[test]
fn intersect_heap_2() {
let mut b1 = BitSet::<1>::with(89);
let mut b2 = BitSet::<1>::with(89);
b1.insert(91);
b2.insert(90);
b1.intersect(&b2);
assert_bitset(&b1, &[89]);
}
#[test]
fn multiple_blocks() {
let mut b = BitSet::<2>::with(120);
b.insert(45);
assert!(matches!(b, BitSet::Inline(_)));
assert_bitset(&b, &[45, 120]);
}
}

154
crates/red_knot_python_semantic/src/semantic_index/use_def/symbol_state.rs
View File

@@ -36,10 +36,8 @@
//! dominates, but it does dominate the `x = 1 if flag2 else None` binding, so we have to keep
//! track of that.
//!
//! The data structures used here ([`BitSet`] and [`smallvec::SmallVec`]) optimize for keeping all
//! data inline (avoiding lots of scattered allocations) in small-to-medium cases, and falling back
//! to heap allocation to be able to scale to arbitrary numbers of live bindings and constraints
//! when needed.
//! The data structures use `IndexVec` arenas to store all data compactly and contiguously, while
//! supporting very cheap clones.
//!
//! Tracking live declarations is simpler, since constraints are not involved, but otherwise very
//! similar to tracking live bindings.
@@ -48,10 +46,12 @@ use itertools::{EitherOrBoth, Itertools};
use ruff_index::newtype_index;
use smallvec::{smallvec, SmallVec};
use crate::semantic_index::constraint::ScopedConstraintId;
use crate::semantic_index::use_def::bitset::{BitSet, BitSetIterator};
use crate::semantic_index::use_def::VisibilityConstraintsBuilder;
use crate::visibility_constraints::ScopedVisibilityConstraintId;
use crate::semantic_index::narrowing_constraints::{
NarrowingConstraintsBuilder, ScopedNarrowingConstraintClause, ScopedNarrowingConstraintId,
};
use crate::semantic_index::visibility_constraints::{
ScopedVisibilityConstraintId, VisibilityConstraintsBuilder,
};
/// A newtype-index for a definition in a particular scope.
#[newtype_index]
@@ -67,18 +67,10 @@ impl ScopedDefinitionId {
pub(super) const UNBOUND: ScopedDefinitionId = ScopedDefinitionId::from_u32(0);
}
/// Can reference this * 64 total constraints inline; more will fall back to the heap.
const INLINE_CONSTRAINT_BLOCKS: usize = 2;
/// Can keep inline this many live bindings or declarations per symbol at a given time; more will
/// go to heap.
const INLINE_DEFINITIONS_PER_SYMBOL: usize = 4;
/// Which constraints apply to a given binding?
type Constraints = BitSet<INLINE_CONSTRAINT_BLOCKS>;
pub(super) type ConstraintIndexIterator<'a> = BitSetIterator<'a, INLINE_CONSTRAINT_BLOCKS>;
/// Live declarations for a single symbol at some point in control flow, with their
/// corresponding visibility constraints.
#[derive(Clone, Debug, Default, PartialEq, Eq, salsa::Update)]
@@ -197,7 +189,7 @@ pub(super) struct SymbolBindings {
#[derive(Clone, Debug, PartialEq, Eq)]
pub(super) struct LiveBinding {
pub(super) binding: ScopedDefinitionId,
pub(super) narrowing_constraints: Constraints,
pub(super) narrowing_constraint: Option<ScopedNarrowingConstraintId>,
pub(super) visibility_constraint: ScopedVisibilityConstraintId,
}
@@ -207,7 +199,7 @@ impl SymbolBindings {
fn unbound(scope_start_visibility: ScopedVisibilityConstraintId) -> Self {
let initial_binding = LiveBinding {
binding: ScopedDefinitionId::UNBOUND,
narrowing_constraints: Constraints::default(),
narrowing_constraint: None,
visibility_constraint: scope_start_visibility,
};
Self {
@@ -226,15 +218,20 @@ impl SymbolBindings {
self.live_bindings.clear();
self.live_bindings.push(LiveBinding {
binding,
narrowing_constraints: Constraints::default(),
narrowing_constraint: None,
visibility_constraint,
});
}
/// Add given constraint to all live bindings.
pub(super) fn record_constraint(&mut self, constraint_id: ScopedConstraintId) {
pub(super) fn record_constraint(
&mut self,
narrowing_constraints: &mut NarrowingConstraintsBuilder,
constraint: ScopedNarrowingConstraintClause,
) {
for binding in &mut self.live_bindings {
binding.narrowing_constraints.insert(constraint_id.into());
binding.narrowing_constraint =
narrowing_constraints.add(binding.narrowing_constraint, constraint);
}
}
@@ -273,7 +270,12 @@ impl SymbolBindings {
}
}
fn merge(&mut self, b: Self, visibility_constraints: &mut VisibilityConstraintsBuilder) {
fn merge(
&mut self,
b: Self,
narrowing_constraints: &mut NarrowingConstraintsBuilder,
visibility_constraints: &mut VisibilityConstraintsBuilder,
) {
let a = std::mem::take(self);
// Invariant: merge_join_by consumes the two iterators in sorted order, which ensures that
@@ -289,8 +291,8 @@ impl SymbolBindings {
// If the same definition is visible through both paths, any constraint
// that applies on only one path is irrelevant to the resulting type from
// unioning the two paths, so we intersect the constraints.
let mut narrowing_constraints = a.narrowing_constraints;
narrowing_constraints.intersect(&b.narrowing_constraints);
let narrowing_constraint = narrowing_constraints
.intersect(a.narrowing_constraint, b.narrowing_constraint);
// For visibility constraints, we merge them using a ternary OR operation:
let visibility_constraint = visibility_constraints
@@ -298,7 +300,7 @@ impl SymbolBindings {
self.live_bindings.push(LiveBinding {
binding: a.binding,
narrowing_constraints,
narrowing_constraint,
visibility_constraint,
});
}
@@ -338,8 +340,13 @@ impl SymbolState {
}
/// Add given constraint to all live bindings.
pub(super) fn record_constraint(&mut self, constraint_id: ScopedConstraintId) {
self.bindings.record_constraint(constraint_id);
pub(super) fn record_constraint(
&mut self,
narrowing_constraints: &mut NarrowingConstraintsBuilder,
constraint: ScopedNarrowingConstraintClause,
) {
self.bindings
.record_constraint(narrowing_constraints, constraint);
}
/// Add given visibility constraint to all live bindings.
@@ -373,9 +380,11 @@ impl SymbolState {
pub(super) fn merge(
&mut self,
b: SymbolState,
narrowing_constraints: &mut NarrowingConstraintsBuilder,
visibility_constraints: &mut VisibilityConstraintsBuilder,
) {
self.bindings.merge(b.bindings, visibility_constraints);
self.bindings
.merge(b.bindings, narrowing_constraints, visibility_constraints);
self.declarations
.merge(b.declarations, visibility_constraints);
}
@@ -393,8 +402,14 @@ impl SymbolState {
mod tests {
use super::*;
use crate::semantic_index::constraint::ScopedConstraintId;
#[track_caller]
fn assert_bindings(symbol: &SymbolState, expected: &[&str]) {
fn assert_bindings(
narrowing_constraints: &NarrowingConstraintsBuilder,
symbol: &SymbolState,
expected: &[&str],
) {
let actual = symbol
.bindings()
.iter()
@@ -405,10 +420,9 @@ mod tests {
} else {
def_id.as_u32().to_string()
};
let constraints = live_binding
.narrowing_constraints
.iter()
.map(|idx| idx.to_string())
let constraints = narrowing_constraints
.iter_constraints(live_binding.narrowing_constraint)
.map(|idx| idx.as_u32().to_string())
.collect::<Vec<_>>()
.join(", ");
format!("{def}<{constraints}>")
@@ -440,36 +454,41 @@ mod tests {
#[test]
fn unbound() {
let narrowing_constraints = NarrowingConstraintsBuilder::default();
let sym = SymbolState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
assert_bindings(&sym, &["unbound<>"]);
assert_bindings(&narrowing_constraints, &sym, &["unbound<>"]);
}
#[test]
fn with() {
let narrowing_constraints = NarrowingConstraintsBuilder::default();
let mut sym = SymbolState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
sym.record_binding(
ScopedDefinitionId::from_u32(1),
ScopedVisibilityConstraintId::ALWAYS_TRUE,
);
assert_bindings(&sym, &["1<>"]);
assert_bindings(&narrowing_constraints, &sym, &["1<>"]);
}
#[test]
fn record_constraint() {
let mut narrowing_constraints = NarrowingConstraintsBuilder::default();
let mut sym = SymbolState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
sym.record_binding(
ScopedDefinitionId::from_u32(1),
ScopedVisibilityConstraintId::ALWAYS_TRUE,
);
sym.record_constraint(ScopedConstraintId::from_u32(0));
let constraint = ScopedConstraintId::from_u32(0).into();
sym.record_constraint(&mut narrowing_constraints, constraint);
assert_bindings(&sym, &["1<0>"]);
assert_bindings(&narrowing_constraints, &sym, &["1<0>"]);
}
#[test]
fn merge() {
let mut narrowing_constraints = NarrowingConstraintsBuilder::default();
let mut visibility_constraints = VisibilityConstraintsBuilder::default();
// merging the same definition with the same constraint keeps the constraint
@@ -478,18 +497,24 @@ mod tests {
ScopedDefinitionId::from_u32(1),
ScopedVisibilityConstraintId::ALWAYS_TRUE,
);
sym1a.record_constraint(ScopedConstraintId::from_u32(0));
let constraint = ScopedConstraintId::from_u32(0).into();
sym1a.record_constraint(&mut narrowing_constraints, constraint);
let mut sym1b = SymbolState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
sym1b.record_binding(
ScopedDefinitionId::from_u32(1),
ScopedVisibilityConstraintId::ALWAYS_TRUE,
);
sym1b.record_constraint(ScopedConstraintId::from_u32(0));
let constraint = ScopedConstraintId::from_u32(0).into();
sym1b.record_constraint(&mut narrowing_constraints, constraint);
sym1a.merge(sym1b, &mut visibility_constraints);
sym1a.merge(
sym1b,
&mut narrowing_constraints,
&mut visibility_constraints,
);
let mut sym1 = sym1a;
assert_bindings(&sym1, &["1<0>"]);
assert_bindings(&narrowing_constraints, &sym1, &["1<0>"]);
// merging the same definition with differing constraints drops all constraints
let mut sym2a = SymbolState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
@@ -497,18 +522,24 @@ mod tests {
ScopedDefinitionId::from_u32(2),
ScopedVisibilityConstraintId::ALWAYS_TRUE,
);
sym2a.record_constraint(ScopedConstraintId::from_u32(1));
let constraint = ScopedConstraintId::from_u32(1).into();
sym2a.record_constraint(&mut narrowing_constraints, constraint);
let mut sym1b = SymbolState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
sym1b.record_binding(
ScopedDefinitionId::from_u32(2),
ScopedVisibilityConstraintId::ALWAYS_TRUE,
);
sym1b.record_constraint(ScopedConstraintId::from_u32(2));
let constraint = ScopedConstraintId::from_u32(2).into();
sym1b.record_constraint(&mut narrowing_constraints, constraint);
sym2a.merge(sym1b, &mut visibility_constraints);
sym2a.merge(
sym1b,
&mut narrowing_constraints,
&mut visibility_constraints,
);
let sym2 = sym2a;
assert_bindings(&sym2, &["2<>"]);
assert_bindings(&narrowing_constraints, &sym2, &["2<>"]);
// merging a constrained definition with unbound keeps both
let mut sym3a = SymbolState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
@@ -516,18 +547,27 @@ mod tests {
ScopedDefinitionId::from_u32(3),
ScopedVisibilityConstraintId::ALWAYS_TRUE,
);
sym3a.record_constraint(ScopedConstraintId::from_u32(3));
let constraint = ScopedConstraintId::from_u32(3).into();
sym3a.record_constraint(&mut narrowing_constraints, constraint);
let sym2b = SymbolState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
sym3a.merge(sym2b, &mut visibility_constraints);
sym3a.merge(
sym2b,
&mut narrowing_constraints,
&mut visibility_constraints,
);
let sym3 = sym3a;
assert_bindings(&sym3, &["unbound<>", "3<3>"]);
assert_bindings(&narrowing_constraints, &sym3, &["unbound<>", "3<3>"]);
// merging different definitions keeps them each with their existing constraints
sym1.merge(sym3, &mut visibility_constraints);
sym1.merge(
sym3,
&mut narrowing_constraints,
&mut visibility_constraints,
);
let sym = sym1;
assert_bindings(&sym, &["unbound<>", "1<0>", "3<3>"]);
assert_bindings(&narrowing_constraints, &sym, &["unbound<>", "1<0>", "3<3>"]);
}
#[test]
@@ -556,6 +596,7 @@ mod tests {
#[test]
fn record_declaration_merge() {
let mut narrowing_constraints = NarrowingConstraintsBuilder::default();
let mut visibility_constraints = VisibilityConstraintsBuilder::default();
let mut sym = SymbolState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
sym.record_declaration(ScopedDefinitionId::from_u32(1));
@@ -563,20 +604,29 @@ mod tests {
let mut sym2 = SymbolState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
sym2.record_declaration(ScopedDefinitionId::from_u32(2));
sym.merge(sym2, &mut visibility_constraints);
sym.merge(
sym2,
&mut narrowing_constraints,
&mut visibility_constraints,
);
assert_declarations(&sym, &["1", "2"]);
}
#[test]
fn record_declaration_merge_partial_undeclared() {
let mut narrowing_constraints = NarrowingConstraintsBuilder::default();
let mut visibility_constraints = VisibilityConstraintsBuilder::default();
let mut sym = SymbolState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
sym.record_declaration(ScopedDefinitionId::from_u32(1));
let sym2 = SymbolState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
sym.merge(sym2, &mut visibility_constraints);
sym.merge(
sym2,
&mut narrowing_constraints,
&mut visibility_constraints,
);
assert_declarations(&sym, &["undeclared", "1"]);
}

3
crates/red_knot_python_semantic/src/visibility_constraints.rs → crates/red_knot_python_semantic/src/semantic_index/visibility_constraints.rs
View File

@@ -281,8 +281,7 @@ const AMBIGUOUS: ScopedVisibilityConstraintId = ScopedVisibilityConstraintId::AM
const ALWAYS_FALSE: ScopedVisibilityConstraintId = ScopedVisibilityConstraintId::ALWAYS_FALSE;
const SMALLEST_TERMINAL: ScopedVisibilityConstraintId = ALWAYS_FALSE;
/// A collection of visibility constraints. This is currently stored in `UseDefMap`, which means we
/// maintain a separate set of visibility constraints for each scope in file.
/// A collection of visibility constraints for a given scope.
#[derive(Debug, PartialEq, Eq, salsa::Update)]
pub(crate) struct VisibilityConstraints {
interiors: IndexVec<ScopedVisibilityConstraintId, InteriorNode>,

20
crates/red_knot_python_semantic/src/symbol.rs
View File

@@ -8,7 +8,7 @@ use crate::semantic_index::{
symbol_table, BindingWithConstraints, BindingWithConstraintsIterator, DeclarationsIterator,
};
use crate::types::{
binding_type, declaration_type, narrowing_constraint, todo_type, IntersectionBuilder,
binding_type, declaration_type, infer_narrowing_constraint, todo_type, IntersectionBuilder,
KnownClass, Truthiness, Type, TypeAndQualifiers, TypeQualifiers, UnionBuilder, UnionType,
};
use crate::{resolve_module, Db, KnownModule, Module, Program};
@@ -550,7 +550,7 @@ fn symbol_from_bindings_impl<'db>(
Some(BindingWithConstraints {
binding,
visibility_constraint,
narrowing_constraints: _,
narrowing_constraint: _,
}) if binding.map_or(true, is_non_exported) => {
visibility_constraints.evaluate(db, constraints, *visibility_constraint)
}
@@ -560,7 +560,7 @@ fn symbol_from_bindings_impl<'db>(
let mut types = bindings_with_constraints.filter_map(
|BindingWithConstraints {
binding,
narrowing_constraints,
narrowing_constraint,
visibility_constraint,
}| {
let binding = binding?;
@@ -576,21 +576,23 @@ fn symbol_from_bindings_impl<'db>(
return None;
}
let mut constraint_tys = narrowing_constraints
.filter_map(|constraint| narrowing_constraint(db, constraint, binding))
.peekable();
let constraint_tys: Vec<_> = narrowing_constraint
.filter_map(|constraint| infer_narrowing_constraint(db, constraint, binding))
.collect();
let binding_ty = binding_type(db, binding);
if constraint_tys.peek().is_some() {
if constraint_tys.is_empty() {
Some(binding_ty)
} else {
let intersection_ty = constraint_tys
.into_iter()
.rev()
.fold(
IntersectionBuilder::new(db).add_positive(binding_ty),
IntersectionBuilder::add_positive,
)
.build();
Some(intersection_ty)
} else {
Some(binding_ty)
}
},
);

2
crates/red_knot_python_semantic/src/types.rs
View File

@@ -34,7 +34,7 @@ use crate::types::class_base::ClassBase;
use crate::types::diagnostic::{INVALID_TYPE_FORM, UNSUPPORTED_BOOL_CONVERSION};
use crate::types::infer::infer_unpack_types;
use crate::types::mro::{Mro, MroError, MroIterator};
pub(crate) use crate::types::narrow::narrowing_constraint;
pub(crate) use crate::types::narrow::infer_narrowing_constraint;
use crate::types::signatures::{Parameter, ParameterKind, Parameters};
use crate::{Db, FxOrderSet, Module, Program};
pub(crate) use class::{Class, ClassLiteralType, InstanceType, KnownClass, KnownInstanceType};

2
crates/red_knot_python_semantic/src/types/narrow.rs
View File

@@ -35,7 +35,7 @@ use std::sync::Arc;
///
/// But if we called this with the same `test` expression, but the `definition` of `y`, no
/// constraint is applied to that definition, so we'd just return `None`.
pub(crate) fn narrowing_constraint<'db>(
pub(crate) fn infer_narrowing_constraint<'db>(
db: &'db dyn Db,
constraint: Constraint<'db>,
definition: Definition<'db>,