Disabling Joins in DataFusion with Custom Optimizer Rules

October 2026

I wanted to start making changes to Apache DataFusion, and I needed a first project small enough to finish but big enough to make me read real code. I picked a slightly silly one: turn off joins. Any query that needs a join should fail with a clear error.


Nobody wants a query engine without joins. The point was that joins show up in more places than you would guess, so chasing them down forces you through most of the planner. That turned out to be true, and I learned more from the places where my plan was wrong than from the parts that worked.


The plan I started with

My first sketch had three steps:

  1. Fail at the SQL parser. Low hanging fruit.
  2. The planners sometimes rewrite queries into joins (a cross product, for example), so tell them to stop doing that.
  3. Brace for a noisy test suite: wrong results, missing plan nodes, that kind of thing.

All three changed.


The parser idea died first. DataFusion doesn’t own its SQL parser. Parsing is done by the external sqlparser-rs crate, so failing there means forking a different project. The next layer down, SqlToRel in datafusion/sql, turns the parsed SQL into a logical plan, and I thought about putting the check there instead. Then I remembered that SQL is only one way into DataFusion. The DataFrame API builds logical plans directly, and so does datafusion-substrait when it reads a Substrait plan. A check in SqlToRel would only ever see SQL. It could give SQL users a nicer early error, but it couldn’t enforce anything.


The second idea was wrong in a more interesting way. A cross product doesn’t get rewritten into a join. It already is one. SELECT * FROM a, b becomes a Join node with no condition (DataFusion folded the old CrossJoin plan node into Join a while ago). And “tell the planner not to make joins” falls apart when you ask what it should make instead. For a lot of queries a join is the only way DataFusion knows how to compute the answer. Blocking the rewrite doesn’t make those queries run without a join; they just fail later, with a worse error message.


So the contract got simpler: any query that needs a join fails, with an error that says so.


The third worry mostly went away once I decided not to fork anything. DataFusion is built to be extended. You can register your own optimizer rules on a session, and the rest of the engine, tests included, never notices. I decided to write two rules: a logical OptimizerRule first, then a PhysicalOptimizerRule, and compare how each one behaves.


Version one: a logical optimizer rule

The rule looks at each node in the logical plan and refuses joins:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

#[derive(Default, Debug)]
struct OptimizerRuleRejectJoins;
 
impl OptimizerRule for OptimizerRuleRejectJoins {
    fn name(&self) -> &str {
        "optimizer_rule_reject_joins"
    }
 
    fn apply_order(&self) -> Option<ApplyOrder> {
        Some(ApplyOrder::TopDown)
    }
 
    fn rewrite(
        &self,
        plan: LogicalPlan,
        _config: &dyn OptimizerConfig,
    ) -> Result<Transformed<LogicalPlan>> {
        match &plan {
            LogicalPlan::Join(_) | LogicalPlan::AsOfJoin(_) => {
                not_impl_err!("JOIN operations are disabled")
            }
            _ => Ok(Transformed::no(plan)),
        }
    }
}

Returning Some(ApplyOrder::TopDown) tells the optimizer to walk the tree for me and call rewrite on every node, root included, so the rule only has to look at the node it’s given. AsOfJoin is its own LogicalPlan variant on main, so it needs matching alongside Join.

Registered with ctx.add_optimizer_rule(...), it did what I wanted for a simple query. Before the rule, the optimized plan looked like this:

1
2
3
4
5
6

Projection: c.id, c.name, o.amount_paid
  Inner Join: c.name = o.name
    SubqueryAlias: c
      TableScan: customers projection=[id, name]
    SubqueryAlias: o
      TableScan: orders projection=[amount_paid, name]

After it, I got:

1

Error: Context("Optimizer rule 'optimizer_rule_reject_joins' failed", NotImplemented("JOIN operations are disabled"))

I also had to decide whether to throw an error or rewrite the plan. An optimizer rewrite has to preserve the query’s results, and there’s no join-free plan that computes a join. Any “rewrite” would be a join under another name or a wrong answer, so the rule throws.


Where joins come from when you don’t write JOIN

This was the part I enjoyed most. Several queries get rejected even though they never say JOIN:

  • SELECT * FROM customers, orders is a join from the moment the plan is built.
  • INTERSECT and EXCEPT are also built as joins right away. In datafusion/expr/src/logical_plan/builder.rs, LogicalPlanBuilder::intersect builds a LeftSemi join and except builds a LeftAnti join. I didn’t expect that one at all.
  • IN (SELECT ...), EXISTS (...) and correlated scalar subqueries start out as subquery expressions. The optimizer then rewrites them into semi joins or left joins (the rules live in decorrelate_predicate_subquery.rs and scalar_subquery_to_join.rs in datafusion/optimizer). DataFusion has no way to run those without a join, so the rule rejects them too. Uncorrelated scalar subqueries are the exception: by default (the enable_physical_uncorrelated_scalar_subquery option) DataFusion runs them with a dedicated ScalarSubqueryExec operator, no join involved.

That ordering detail matters. Subquery joins don’t exist until partway through optimization, so a rule that runs earlier can’t see them. Custom optimizer rules get appended after the built-in ones, which puts mine after decorrelation.


The join that got away

Running after the built-in rules has a downside, and I found it with this query:

1

SELECT DISTINCT c.name FROM customers c LEFT JOIN orders o ON c.name = o.name

It has JOIN right there in the text, and my rule let it through. Here’s the plan my rule saw:

1
2
3

Aggregate: groupBy=[[c.name]], aggr=[[]]
  SubqueryAlias: c
    TableScan: customers projection=[name]

The orders table is gone completely, along with the join. That’s the built-in EliminateJoin rule at work (datafusion/optimizer/src/eliminate_join.rs). It noticed that no column from orders is used above the join and that DISTINCT makes duplicate rows irrelevant. A left join keeps every customer anyway, so orders can’t affect the result, and the rule replaced the join with its left input. Separately, DISTINCT became a group-by with no aggregates, which is how DataFusion runs it.


The rewrite is correct, and my rule did its job: by the time it ran, the plan had no join in it.


Picking a contract

This query forced me to decide what “disable joins” means. I had two options.


Option A: no join is ever executed. Keep the rule where it is. The DISTINCT query passing is correct, because the engine never runs a join for it.

Option B: no query that uses a join is accepted. Then the check has to run before the optimizer, as an AnalyzerRule, while the join is still in the plan. Analyzer errors also always propagate, which helps with a problem I’ll get to below. The cost is that subqueries haven’t been decorrelated yet at that point, so an analyzer rule would also need to reject IN, EXISTS and scalar subquery expressions by hand.


I went with option A. It describes what the engine does instead of what the user typed, and it was a smaller rule.


The hole: skip_failed_rules

DataFusion has a setting, datafusion.optimizer.skip_failed_rules. It’s false by default, which is why I saw my error. When it’s true, the optimizer treats a rule error as “this optimization didn’t work out”: it logs the error, restores the previous plan and keeps going. The join runs.


Anyone can turn that on, including with a SQL SET. A logical optimizer rule is a fine place to optimize and a weak place to enforce, because the optimizer is allowed to give up on it. That gap is the main reason for a second rule.


Version two: a physical optimizer rule

The physical optimizer doesn’t have that escape hatch. In physical_planner.rs, optimize_physical_plan wraps any rule error with the rule’s name and returns it, every time. A physical rule also looks at the plan that will run.


The rule has the same shape as the logical one, with two differences. ExecutionPlan is a trait object, not an enum, so you can’t match on it; you check each node’s concrete type at runtime. And optimize is called once with the whole plan, so the rule walks the tree itself.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

use datafusion::physical_plan::joins::{
    AsOfJoinExec, CrossJoinExec, HashJoinExec, NestedLoopJoinExec,
    PiecewiseMergeJoinExec, SortMergeJoinExec, SymmetricHashJoinExec,
};
 
#[derive(Default, Debug)]
pub struct PhysicalOptimizerRuleRejectJoins;
 
fn is_join(node: &dyn ExecutionPlan) -> bool {
    node.is::<HashJoinExec>()
        || node.is::<SortMergeJoinExec>()
        || node.is::<NestedLoopJoinExec>()
        || node.is::<CrossJoinExec>()
        || node.is::<SymmetricHashJoinExec>()
        || node.is::<PiecewiseMergeJoinExec>()
        || node.is::<AsOfJoinExec>()
}
 
impl PhysicalOptimizerRule for PhysicalOptimizerRuleRejectJoins {
    fn name(&self) -> &str {
        "physical_optimizer_rule_reject_joins"
    }
 
    fn optimize(
        &self,
        plan: Arc<dyn ExecutionPlan>,
        _config: &ConfigOptions,
    ) -> Result<Arc<dyn ExecutionPlan>> {
        plan.apply(|node| {
            if is_join(node.as_ref()) {
                return not_impl_err!("JOIN operations are disabled (found {})", node.name());
            }
            Ok(TreeNodeRecursion::Continue)
        })?;
        Ok(plan)
    }
 
    fn schema_check(&self) -> bool {
        true
    }
}

Wiring it up

ctx.add_optimizer_rule only takes logical rules, and SessionContext has no method for physical ones. A physical rule goes in when the session state is built. Use with_physical_optimizer_rule, which appends to the built-in rules; the plural with_physical_optimizer_rules replaces the whole list and drops every built-in optimization.

1
2
3
4
5
6
7
8

let state = SessionStateBuilder::new_with_default_features()
    .with_physical_optimizer_rule(Arc::new(PhysicalOptimizerRuleRejectJoins))
    .build();
let ctx = SessionContext::new_with_state(state);
 
// into_optimized_plan() stops after the logical optimizer, so it never reaches this rule.
// create_physical_plan() runs physical planning and the physical optimizer.
let physical_plan = ctx.sql(sql).await?.create_physical_plan().await?;

collect() and show() go through the same steps before executing, so the rule fires there too. With the logical rule switched off, this one caught the joins on its own.


What I took away

I expected a small project about one rule. Instead I got a decent map of how a query moves through DataFusion: analyzer, logical optimizer passes, physical planning, physical optimizer. Along the way I learned that rule order decides what each rule can see, that apply_order decides who walks the tree, and that the same check behaves differently depending on which layer it runs in. The logical rule is simple and future-proof but can be skipped. The physical rule can’t be skipped but has to list every join operator by name.