PHP 8.5 | Tokenizer/PHP: properly fix "Using null as an array offset" deprecation

If a slash/hash comment is on its own line, the new line token is merged into the comment token for PHP 8.0+, and the new line is skipped by setting it to `null`.

The PHP Tokenizer did not take the possibility of a token being set to `null` into account when retokening a `?` to either `T_NULLABLE` or `T_INLINE_THEN`, leading to the PHP 8.5 deprecation notice.
In that case, the sniff first looks forward to see if we can draw a conclusion based on the non-empty tokens following the `?` and if not, walks backward to look at the tokens before the `?`.
The problem occurs when a `null` token exists in the sequence before the `?`.

This `null` token will only have been injected with a specific code sequence: when there is a slash/hash comment followed by a new line in the token sequence before the `?` and there is no indentation/new line whitespace on the next line.

Also see the detailed analysis by andrewnicols in PHPCSStandards/PHP_CodeSniffer 1216#issuecomment-3274198443

There are only two situations in which this causes the tokenizer to hit the PHP 8.5 "Using null as an array offset" deprecation notice:
1. If the `?` token is the last token in the file (live coding - the issue was discovered via a test related to live coding).
2. If there are tokens after the `?`, but not tokens which allows us to draw a conclusion (yet) and there is a slash/hash comment between the `?` and the previous non-empty token.

This commit:
1. Adds dedicated tests for both situations.
2. Adds a new fix for situation [1] as if there are no tokens after a `?`, we cannot determine definitively whether the `?` is a nullable operator or an inline then. For BC, this should be tokenized as `T_INLINE_THEN`.
3. Makes a small tweak to the previously added fix which addressed situation [2].
    Alternatively, we could consider switching to using the "$finalTokens" token stack to do the token walking instead, but as that is a bigger change and this part of the code has nowhere near sufficient tests, that change is left for later.

Fixes 1216

Author: jrfnl

src/Tokenizers/PHP.php

@@ -2244,6 +2244,24 @@ protected function tokenize($string)
                     break;
                 }//end for
 
+                // Handle live coding/parse errors elegantly.
+                // If the "?" is the last non-empty token in the file, we cannot draw a definitive conclusion,
+                // so tokenize as T_INLINE_THEN.
+                if ($i === $numTokens) {
+                    if (PHP_CODESNIFFER_VERBOSITY > 1) {
+                        echo "\t\t* token $stackPtr at end of file changed from ? to T_INLINE_THEN".PHP_EOL;
+                    }
+
+                    $newToken['code'] = T_INLINE_THEN;
+                    $newToken['type'] = 'T_INLINE_THEN';
+
+                    $insideInlineIf[] = $stackPtr;
+
+                    $finalTokens[$newStackPtr] = $newToken;
+                    $newStackPtr++;
+                    continue;
+                }
+
                 /*
                  * This can still be a nullable type or a ternary.
                  * Do additional checking.
@@ -2253,11 +2271,13 @@ protected function tokenize($string)
                 $lastSeenNonEmpty = null;
 
                 for ($i = ($stackPtr - 1); $i >= 0; $i--) {
+                    if (isset($tokens[$i]) === false) {
+                        // Ignore skipped tokens (related to PHP 8+ slash/hash comment vs new line retokenization).
+                        continue;
+                    }
+
                     if (is_array($tokens[$i]) === true) {
                         $tokenType = $tokens[$i][0];
-                    } else if ($tokens[$i] === null) {
-                        // Ignore skipped tokens.
-                        continue;
                     } else {
                         $tokenType = $tokens[$i];
                     }

tests/Core/Tokenizers/PHP/NullableVsInlineThenParseErrorTest.inc

@@ -0,0 +1,9 @@
+<?php
+
+// This should be the only test in the file.
+// Ref: https://github.com/PHPCSStandards/PHP_CodeSniffer/issues/1216
+
+/* testLiveCoding */
+$ternary = true
+    # This must be a slash or hash comment and the next line must **NOT** have any indentation for the PHP 8.5 deprecation notice to occur.
+? //comment.

tests/Core/Tokenizers/PHP/NullableVsInlineThenParseErrorTest.php

@@ -0,0 +1,40 @@
+<?php
+/**
+ * Tests the retokenization of ? to T_NULLABLE or T_INLINE_THEN.
+ *
+ * @copyright 2025 PHPCSStandards and contributors
+ * @license   https://github.com/PHPCSStandards/PHP_CodeSniffer/blob/master/licence.txt BSD Licence
+ */
+
+namespace PHP_CodeSniffer\Tests\Core\Tokenizers\PHP;
+
+use PHP_CodeSniffer\Tests\Core\Tokenizers\AbstractTokenizerTestCase;
+
+/**
+ * Tests the retokenization of ? to T_NULLABLE or T_INLINE_THEN.
+ *
+ * @covers PHP_CodeSniffer\Tokenizers\PHP::tokenize
+ */
+final class NullableVsInlineThenParseErrorTest extends AbstractTokenizerTestCase
+{
+
+
+    /**
+     * Verify that a "?" as the last functional token in a file (live coding) is tokenized as `T_INLINE_THEN`
+     * as it cannot yet be determined what the token would be once the code is finalized.
+     *
+     * @return void
+     */
+    public function testInlineThenAtEndOfFile()
+    {
+        $tokens     = $this->phpcsFile->getTokens();
+        $target     = $this->getTargetToken('/* testLiveCoding */', [T_NULLABLE, T_INLINE_THEN]);
+        $tokenArray = $tokens[$target];
+
+        $this->assertSame(T_INLINE_THEN, $tokenArray['code'], 'Token tokenized as '.$tokenArray['type'].', not T_INLINE_THEN (code)');
+        $this->assertSame('T_INLINE_THEN', $tokenArray['type'], 'Token tokenized as '.$tokenArray['type'].', not T_INLINE_THEN (type)');
+
+    }//end testInlineThenAtEndOfFile()
+
+
+}//end class

tests/Core/Tokenizers/PHP/NullableVsInlineThenTest.inc

@@ -22,7 +22,11 @@ $closure = function (
     /* testClosureParamTypeNullableInt */
     ?Int $a,
     /* testClosureParamTypeNullableCallable */
-    ?    Callable $b
+    ?    Callable $b,
+    /* testClosureParamTypeNullableStringWithAttributeAndSlashComment */
+    #[AttributeForParam]
+    // This must be a slash or hash comment and the next line must **NOT** have any indentation for the PHP 8.5 deprecation notice (issue PHPCSStandards/PHP_CodeSniffer#1216) to occur.
+?string $c
 /* testClosureReturnTypeNullableInt */
 ) :?INT{};
 

tests/Core/Tokenizers/PHP/NullableVsInlineThenTest.php

@@ -50,16 +50,17 @@ public function testNullable($testMarker)
     public static function dataNullable()
     {
         return [
-            'property declaration, readonly, no visibility'  => ['/* testNullableReadonlyOnly */'],
-            'property declaration, private set'              => ['/* testNullablePrivateSet */'],
-            'property declaration, public and protected set' => ['/* testNullablePublicProtectedSet */'],
-            'property declaration, final, no visibility'     => ['/* testNullableFinalOnly */'],
-            'property declaration, abstract, no visibility'  => ['/* testNullableAbstractOnly */'],
-
-            'closure param type, nullable int'               => ['/* testClosureParamTypeNullableInt */'],
-            'closure param type, nullable callable'          => ['/* testClosureParamTypeNullableCallable */'],
-            'closure return type, nullable int'              => ['/* testClosureReturnTypeNullableInt */'],
-            'function return type, nullable callable'        => ['/* testFunctionReturnTypeNullableCallable */'],
+            'property declaration, readonly, no visibility'                 => ['/* testNullableReadonlyOnly */'],
+            'property declaration, private set'                             => ['/* testNullablePrivateSet */'],
+            'property declaration, public and protected set'                => ['/* testNullablePublicProtectedSet */'],
+            'property declaration, final, no visibility'                    => ['/* testNullableFinalOnly */'],
+            'property declaration, abstract, no visibility'                 => ['/* testNullableAbstractOnly */'],
+
+            'closure param type, nullable int'                              => ['/* testClosureParamTypeNullableInt */'],
+            'closure param type, nullable callable'                         => ['/* testClosureParamTypeNullableCallable */'],
+            'closure param type, nullable string with comment, issue #1216' => ['/* testClosureParamTypeNullableStringWithAttributeAndSlashComment */'],
+            'closure return type, nullable int'                             => ['/* testClosureReturnTypeNullableInt */'],
+            'function return type, nullable callable'                       => ['/* testFunctionReturnTypeNullableCallable */'],
         ];
 
     }//end dataNullable()