diff --git a/crates/red_knot_python_semantic/build.rs b/crates/red_knot_python_semantic/build.rs
new file mode 100644
index 0000000000..10815f6a89
--- /dev/null
+++ b/crates/red_knot_python_semantic/build.rs
@@ -0,0 +1,4 @@
+/// Rebuild the crate if a test file is added or removed from
+pub fn main() {
+    println!("cargo:rerun-if-changed=resources/mdtest");
+}
diff --git a/crates/red_knot_test/README.md b/crates/red_knot_test/README.md
index 9a5964db5c..3c35862b4d 100644
--- a/crates/red_knot_test/README.md
+++ b/crates/red_knot_test/README.md
@@ -34,6 +34,18 @@ syntax, it's just how this README embeds an example mdtest Markdown document.)
 See actual example mdtest suites in
 [`crates/red_knot_python_semantic/resources/mdtest`](https://github.com/astral-sh/ruff/tree/main/crates/red_knot_python_semantic/resources/mdtest).
 
+> ℹ️ Note: If you use `rstest` to generate a separate test for all Markdown files in a certain directory,
+> as with the example in `crates/red_knot_python_semantic/tests/mdtest.rs`,
+> you will likely want to also make sure that the crate the tests are in is rebuilt every time a
+> Markdown file is added or removed from the directory. See
+> [`crates/red_knot_python_semantic/build.rs`](https://github.com/astral-sh/ruff/tree/main/crates/red_knot_python_semantic/build.rs)
+> for an example of how to do this.
+>
+> This is because `rstest` generates its tests at build time rather than at runtime.
+> Without the `build.rs` file to force a rebuild when a Markdown file is added or removed,
+> a new Markdown test suite might not be run unless some other change in the crate caused a rebuild
+> following the addition of the new test file.
+
 ## Assertions
 
 Two kinds of assertions are supported: `# revealed:` (shown above) and `# error:`.