Don't strip shebang in expr-ctxt include!(…)

[fmease]

No longer strip shebang interpreter directives in files that were included in expression (statement) contexts.

[fmease]

Re. I-lang-nominated Nominated for discussion during a lang team meeting. :

Dear T-lang,

I honestly don't know if we should make this change or not. Therefore I hope you might have an answer.
The overall context is frontmatter and its proposed stabilization in #148051 (which is still lang-nominated).

One goal which its RFC establishes is the following: "This [frontmatter stripping] applies anywhere shebang stripping is performed. For example, if include! strips shebangs, then it will also frontmatter.". This goal couldn't be fully accomplished:

1. We can't strip frontmatter in proc_macro::TokenStream::from_str contrary to shebang. See issue Introduction of frontmatter was a theoretical breaking change (via TokenStream::from_str) #145520 for details (PR reduce the amount of panics in {TokenStream, Literal}::from_str calls #147859 might actually solve this, I'm not entirely sure, cc reduce the amount of panics in {TokenStream, Literal}::from_str calls #147859 (comment)).
2. More relevantly here: We can't strip frontmatter in expression-context includes since that would theoretically regress “manifold negation” (see Theoretical breaking change from a combination of frontmatter and include! #145945 (comment) for details).

We still strip frontmatter in item-context includes to be consistent with shebang where possible and since it truly can't regress any code. This leads us to this PR in which I propose to stop stripping shebang in expression-context includes, too, to make the behavior of include more consistent across these two special lexical forms.

I don't know if this is a good idea. It feels slightly surprising that include is “context-dependent”¹ on a lexical level (it being context-dependent on a syntactical level is a given), that's a first I believe. Should we keep that behavior (which I've intentionally introduced in #146340) or should we accept the inconsistency (i.e., close this PR)? Or more drastically, should we go back to lexing/stripping frontmatter in expression-context includes (ultimately meaning: in all contexts) and accept the theoretical breaking change related to the “manifold negation” (or make this edition-sensitive) [STRIP_ALL_EVERYWHERE]?

---

rustc@main	strip shebang?	strip frontmatter?
item-ctxt include	✅	✅
expr-ctxt include	✅	❌

PRO: No breaking changes, fully behavior-preserving.
CON: Inconsistency between shebang & frontmatter.
CON: We have “lexical context-dependence”¹ (“not all Rust source files lex the same”).

rustc@no-incl-shebang-expr	strip shebang?	strip frontmatter?
item-ctxt include	✅	✅
expr-ctxt include	❌ (theoretical breakage)	❌

PRO: Consistency between shebang & frontmatter
PRO(?): Gets rid of an “odd” “feature” (shebang in a file consisting of a Rust expression).
CON: Theoretical breaking change.
CON(?): Shebang in a “Rust expression file” might be useful?
CON: We have “lexical context-dependence”¹ (“not all Rust source files lex the same”).

[STRIP_ALL_EVERYWHERE]	strip shebang?	strip frontmatter?
item-ctxt include	✅	✅
expr-ctxt include	✅	✅ (breaks manifold negation)

PRO: Full consistency.
PRO: No “lexical context-dependence”¹ (“all Rust source files lex the same”).
CON: Theoretical breaking change.

Footnotes

1. I couldn't come up with a better name. Just note that this is completely unrelated to the concept of context-sensitive grammar. ↩ ↩² ↩³ ↩⁴

[traviscross]

@bors2 try

[joshtriplett]

We talked about this in today's @rust-lang/lang meeting. We'd like to get some data to back this up, in the form of crater runs.

We'd love to see a crater run that validates no Rust code relies on include! stripping of shebangs.

[joshtriplett]

Speaking for myself, I'd also love to see a crater run that removes all stripping of shebang and frontmatter from include!, including in item context, and possibly errors if seen so that we can clearly identify if anything is relying on that. That would simplify include! further and make it not context-dependent.

[jackh726]

We talked about this in today's @rust-lang/lang meeting. We'd like to get some data to back this up, in the form of crater runs.

We'd love to see a crater run that validates no Rust code relies on include! stripping of shebangs.

To clarify: I think the crater runs should be "error if a shebang is stripped in an expr-ctxt include!()", not just a crater run of this PR as it stands - this ensures that we can evaluate all cases where stripping is relied upon, not just those that influence behavior in a visible way (though, as Josh pointed out in the meeting, it's pretty unlikely that code will compile without stripping if it relied on it).

This is more important for the item-ctxt shebang stripping, but still better to be prudent here imo and detect all cases, rather than a (potential) subset of them.

Speaking for myself, I'd also love to see a crater run that removes all stripping of shebang and frontmatter from include!, including in item context, and possibly errors if seen so that we can clearly identify if anything is relying on that. That would simplify include! further and make it not context-dependent.

I'm a second on this - not blocking this PR on that, because it isn't a strong "logical extension", but I am filing a concern on #148051, since that "closes some doors" as far as decisions if we stabilize frontmatter with or without stripping and later wish to change it. Though - I'm hoping that minimal extra work would need to be done alongside the crater run requested for this PR.

[rust-bors]

☀️ Try build successful (CI)
Build commit: 3a4cb0e (3a4cb0edb4040379c037e06efeb5409e44be7b77, parent: c797096598b80ba4a40397ad7e91584fcd3df5f1)

[traviscross]

@craterbot check

[craterbot]

👌 Experiment pr-146377 created and queued.
🤖 Automatically detected try build 3a4cb0e
🔍 You can check out the queue and this experiment's details.

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more