Experimental markdown output #1303

jrturton · 2025-09-26T13:35:50Z

Summary

This PR addresses the changes proposed in #1301. It involves a new translator / semantic visitor / markup walker to create the new outputs from DocumentationNode. This is done for each node, after the render JSON is generated.

Dependencies

None

Testing

There are extensive unit tests in MarkdownOutputTests.swift. Testers can also run the convert command against any catalog with the --enable-experimental-markdown-output and --enable-experimental-markdown-output-manifest flags set.

Checklist

Make sure you check off the following items. If they cannot be completed, provide a reason.

Added tests
Ran the ./bin/test script and it succeeded
Updated documentation if necessary

…ert feature flags

…wn output

d-ronnqvist

There's a lot to unpack here and I've only had the time to skim thought the code but these are a few things that stand out to me:

If this is intended as experimental feature, then it can't add actual public API because consumers of SwiftDocC would have no way of knowing what API is stable and what is experimental and could still have breaking changes or be removed.

If a goal of this API is that clients can use it while it's still experimental then it needs to be marked as SPI (using @_spi(SomeName) so that those clients make a deliberate decision to use it and acknowledge that it may have breaking changes.
The added tests follow a pattern not seen elsewhere in the repo. I left some more specific comments about that.

Sources/SwiftDocC/SwiftDocCMarkdownOutput/MarkdownOutputManifest.swift

Sources/SwiftDocC/Model/MarkdownOutput/Translation/MarkdownOutputSemanticVisitor.swift

Tests/SwiftDocCTests/Rendering/Markdown/MarkdownOutputTests.swift

Tests/SwiftDocCTests/Test Bundles/MarkdownOutput.docc/MarkdownOutput.symbols.json

jrturton · 2025-10-02T15:08:01Z

I have removed or @_spi-decorated all new public API, with the exception of the feature flags themselves, I assumed that would be OK

…are rendering links-with-abstracts

d-ronnqvist

There's quite a lot going on here. To name a few, there's:

logic bugs or bugs where the values are based of incorrect data
"public" properties that are very loosely typed (either String or Data) without accompanying documentation that describe how the caller can decode and use that information.
several symbol sections that's silently excluded from the output
a fair amount of supported markdown content that's never tested against (and AFAICT isn't defined what the expected output would be)
some supported tutorial content that's silently excluded from the output
some terminology that's not seen elsewhere in the project
a few design choices that could be worth revisiting
quite a lot of formatting that's different from existing code in this projected
some unclear "public" SPI where it's not clear to me what the use-case is and there's also no documentation to give the caller/client context to how and when they should use this "public" SPI.

d-ronnqvist · 2025-12-02T09:46:47Z

Sources/SwiftDocCUtilities/Action/Actions/Convert/JSONEncodingRenderNodeWriter.swift

+        let fileSafePath = NodeURLGenerator.fileSafeReferencePath(
+            markdownNode.identifier,
+            lowercased: true
+        )
+
+        // The path on disk to write the markdown file at.
+        let markdownNodeTargetFileURL = renderNodeURLGenerator
+            .urlForReference(
+                markdownNode.identifier,
+                fileSafePath: fileSafePath
+            )
+            .appendingPathExtension("md")
+
+        let markdownNodeTargetFolderURL = markdownNodeTargetFileURL.deletingLastPathComponent()
+
+        // On Linux sometimes it takes a moment for the directory to be created and that leads to
+        // errors when trying to write files concurrently in the same target location.
+        // We keep an index in `directoryIndex` and create new sub-directories as needed.
+        // When the symbol's directory already exists no code is executed during the lock below
+        // besides the set lookup.
+        try directoryIndex.sync { directoryIndex in
+            let (insertedMarkdownNodeTargetFolderURL, _) = directoryIndex.insert(markdownNodeTargetFolderURL)
+            if insertedMarkdownNodeTargetFolderURL {
+                try fileManager.createDirectory(
+                    at: markdownNodeTargetFolderURL,
+                    withIntermediateDirectories: true,
+                    attributes: nil
+                )
+            }
+        }
+
+        try fileManager.createFile(at: markdownNodeTargetFileURL, contents: markdownNode.nodeData, options: nil)


minor: This is duplicating some non-trivial code with only a few minor differences. Can we extract the common code into a private helper function that both write(_:) methods can call?

d-ronnqvist · 2025-12-02T09:54:34Z

Sources/SwiftDocC/SwiftDocCMarkdownOutput/MarkdownOutputNode.swift

+extension MarkdownOutputNode {
+    public struct Metadata: Codable, Sendable {
+
+        static let version = "0.1.0"


Please use SemanticVersion rather than strings for version information. See for example DiagnosticFile, SerializableLinkResolutionInformation, and RenderIndex.

d-ronnqvist · 2025-12-02T09:58:18Z

Sources/SwiftDocC/SwiftDocCMarkdownOutput/MarkdownOutputNode.swift

+                // If no introduced, we are unavailable
+                self.unavailable = unavailable || introduced == nil


I don't think this logic is correct. A framework can have availability information configured in its Info.plist that doesn't include an introduced version. That configuration specifies that the framework is available for that platform.

d-ronnqvist · 2025-12-02T10:01:03Z

Sources/SwiftDocC/SwiftDocCMarkdownOutput/MarkdownOutputNode.swift

+                // Can't have deprecated without an introduced
+                self.introduced = introduced ?? deprecated


I don't know if this is correct. It's possible to specify @available(macOS, deprecated: 26, message: "This symbol is deprecated") as an availability attribute in Swift.

Generally, no introduced version means that a symbol is available as far back as the framework itself supports.

d-ronnqvist · 2025-12-02T10:05:57Z

Sources/SwiftDocC/SwiftDocCMarkdownOutput/MarkdownOutputNode.swift

+        }
+
+        public struct Symbol: Codable, Sendable {
+            public let kind: String


Question: Is this representing the tooling facing kind identifier (for example "struct" or "enum.case") or is this representing a display name (for example "Structure" or "Enumeration Case")?

If it's the former; is there a more precise type—for example SymbolKit/SymbolGraph/Symbol/KindIdentifier —that we could use here?

If it's the latter; please rename this to kindDisplayName or something similar that clarifies what this value represents.

d-ronnqvist · 2025-12-02T12:54:36Z

Sources/SwiftDocC/Model/MarkdownOutput/Translation/MarkdownOutputMarkdownWalker.swift

+    }
+
+    var markdown = ""
+    var outgoingReferences: Set<MarkdownOutputManifest.Relationship> = []


Question: what does this represent? Based on the implementation it looks like it's "topic curation" (documentation organization) but this property name sounds broader than that. It would be good to clarify this name and maybe add a documentation comment if the name alone isn't clear.

d-ronnqvist · 2025-12-02T12:55:30Z

Package.resolved

@@ -1,4 +1,5 @@
 {
+  "originHash" : "6a2fba12d9d8cf682094fa377600d56c196ce947a70bf82d12a7faff1b81c530",


Question: what does this hash represent?

d-ronnqvist · 2025-12-02T12:59:04Z

Sources/SwiftDocC/Infrastructure/ConvertOutputConsumer.swift

+
+}
+
+// Merge into ConvertOutputMarkdownConsumer when no longer SPI


What is this comment referring to? This is the ConvertOutputMarkdownConsumer protocol.

d-ronnqvist · 2025-12-02T13:00:41Z

Sources/SwiftDocC/Infrastructure/ConvertOutputConsumer.swift

+
+// Merge into ConvertOutputMarkdownConsumer when no longer SPI
+@_spi(MarkdownOutput)
+public protocol ConvertOutputMarkdownConsumer {


Question: Is there a use case for callers/clients to consume the in-memory representations of this information as opposed to decoding the output files? If not, can we make this internal only?

d-ronnqvist · 2025-12-02T13:08:50Z

Sources/SwiftDocC/SwiftDocCMarkdownOutput/MarkdownOutputManifest.swift

+
+        public let sourceURI: String
+        public let relationshipType: RelationshipType
+        public let subtype: String?


Question: what does this represent? Is there a more precise type—either existing or new—that this could use rather than a string value?

Another way to look at it; what is the caller expected to do with this information? It's public but just a string so it could represent anything. With an enumeration there would be a known list of cases the the caller could check against and make decisions based on.

Richard Turton added 23 commits September 2, 2025 16:25

Add experimental markdown output flag and pass it through to the conv…

cf7ebc6

…ert feature flags

Initial export of Markdown from Article

4b1b94a

Initial processing of a type-level symbol

55e7836

Adds symbol declarations and article reference links

53a6196

Output tutorials to markdown

3513a73

Be smarter about removing indentation from within block directives

81d2d5a

Baseline for adding new tests for markdown output

f3fa5ab

Basic test infrastructure for markdown output

5013a09

Adds symbol link tests to markdown output

6798162

Tutoorial code rendering markdown tests

132cd6c

Adding metadata to markdown output

1753cca

Include package source for markdown output test catalog

301d7da

Output metadata updates

9607e3b

Adds default availability for modules to markdown export

5244f0f

Move availability out of symbol and in to general metadata for markdo…

a6a740e

…wn output

Refactor markdown output so the final node type is standalone

d0dbf44

Add generated markdown flag to render node metadata

595831a

Only include unavailable in markdown header if it is true

104f9eb

Initial setup of manifest output, no references

a2aa8f1

Output of manifest / relationships

b5ed559

Manifest output format updates

74b6023

Remove member symbol relationship

198d8e8

More compact availability, deal with metadata availability for symbols

53ba222

jrturton requested review from d-ronnqvist, franklinsch and heckj September 26, 2025 13:35

Merge branch 'main' into 159600318/experimental-markdown-ouput

af890be

jrturton linked an issue Sep 26, 2025 that may be closed by this pull request

LLM-accessible output #1301

Open

d-ronnqvist requested changes Sep 26, 2025

View reviewed changes

Update tests so all inputs are locally defined

4784380

Richard Turton added 5 commits October 2, 2025 09:38

Remove print statements from unused visitors

abc9985

Merge branch 'main' into 159600318/experimental-markdown-ouput

edc74da

Added snippet handling

eb77d39

Remove or _spi-hide new public API

0e867d7

Remove or _spi-hide new public API (part 2)

f07f683

d-ronnqvist added the needs forum discussion Needs to be discussed in the Swift Forums label Oct 10, 2025

Richard Turton added 6 commits November 6, 2025 10:02

Prevent recursion when the abstract of a link contains a link and we …

6ccfe8f

…are rendering links-with-abstracts

Test and temporary fix for colspan issue swiftlang/swift-markdown#238

4fcd9fe

Remove separate SwiftDocCMarkdownOutput target

ba716ed

Bump swift-markdown, remove temporary bug fix

5c835b0

Merge branch 'main' into 159600318/experimental-markdown-ouput

85ea19b

Updates to handle removed bundle parameter

43efc75

d-ronnqvist requested changes Dec 2, 2025

View reviewed changes

		// If no introduced, we are unavailable
		self.unavailable = unavailable \|\| introduced == nil

		// Can't have deprecated without an introduced
		self.introduced = introduced ?? deprecated

		@@ -1,4 +1,5 @@
		{
		"originHash" : "6a2fba12d9d8cf682094fa377600d56c196ce947a70bf82d12a7faff1b81c530",


		}

		// Merge into ConvertOutputMarkdownConsumer when no longer SPI

Experimental markdown output #1303

Are you sure you want to change the base?

Experimental markdown output #1303

Uh oh!

Conversation

jrturton commented Sep 26, 2025

Summary

Dependencies

Testing

Checklist

Uh oh!

d-ronnqvist left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jrturton commented Oct 2, 2025

Uh oh!

d-ronnqvist left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

d-ronnqvist left a comment •

edited

Loading