Support including per-page minimal semantic HTML content in the index.html files

[d-ronnqvist]

Bug/issue #, if applicable: rdar://163326857

Summary

This is an early version of an enhancement to DocC's static hosting files that embeds a very minimal representation of the per-page documentation content as semantic HTML inside of the <noscript> tag of each index.html file.

You can see a live example of this if you open https://swiftlang.github.io/swift-docc/documentation/swiftdocc/documentationcontext/ and disable JavaScript in your web browser and refresh the page.
For the combined archive of SwiftDocC and SwiftDocCUtilities (the command line actions) this increases the archive size by just 0.2% (from 85.1 MB on disk to 85.3 MB on disk). The reason for such a small size increase is that only 46 of the 5730 index.html files become larger than 4 kB (the file system block size on macOS and many Linux distributions).

---

Up until this point, I didn't feel that this enhancement was sufficiently complete in order for others to get a sense of the bigger picture or to for me to have a clear sense of how (if at all) I think its API would change.

This feature/enhancement started out with the goal of producing semantic HTML pages meant to be read by people and later shifted focus to producing very minimal HTML pages that's primarily meant to be processed by tools like SEO indexers. However, I've found that there's a lot of API surface and a lot of logic that's the same for both and still believe that static semantic HTML for people to read is an output format that DocC should support. So rather than removing the code that produces richer HTML content for people, I designed the core rendering code to be able to conditionally—based on an enum value—produce either richer HTML content for people or concise HTML content for tools.

In order to make this easy to land in the toolchain, I've completely avoided external dependencies and instead based core HTML rendering code around Foundation's XMLNode. I've also made sure that this PR doesn't add any public API so that we can completely reimplement the internals in the future if we want to. For example, that's one of the reasons why I added a new (package access) HTMLContentConsumer protocol rather than adding methods to ConvertOutputConsumer which is public API.

---

I don't know what the easiest way to review these changes would be, but I'm open to suggestions.

All in all it's about 2k lines of code and 2k lines of tests. This is admittedly a bit large for a good PR. However, one good thing it as going for it is that it shows the full history of how the code evolved.

I don't think it would be meaningful to slice up the history into smaller chunks and review this iteratively, because I learnt things and made changes along the way that would mean that some details in the early commits have already been invalidated in later commits. Instead, the only way I think it makes sense to slice this work would be into chunks of files/functionality in the state it's currently in as a single "squashed" commit per PR. Those slices would probably be:

• DocCHTML/MarkdownRenderer.swift, DocCHTML/LinkProvider.swift, DocCHTML/WordBreak.swift, DocCHTML/XMLNode+element.swift and their accompanying test in DocCHTMLTests/MarkdownRendererTests.swift and DocCHTMLTests/WordBreakTests.swift in the first PR. This would only add code that's not exercised outside of those tests.

• DocCHTML/MarkupRenderer+Availability.swift, DocCHTML/MarkupRenderer+Breadcrumbs.swift, DocCHTML/MarkupRenderer+Declaration.swift, DocCHTML/MarkupRenderer+Parameters.swift, DocCHTML/MarkupRenderer+Returns.swift, DocCHTML/MarkupRenderer+Topics.swiftand their accompanying tests inDocCHTMLTests/MarkdownRenderer+PageElementsTests.swift`. This would also only add code that's not exercised outside of those tests.

• All the remaining changes that integrate this enhancement into DocC, adds a CLI feature flags, etc. It's only at this point, that there would be any code that a user could exercise by invoking docc convert.

That said, one could go about reviewing the same changes in that file order in this PR as well, with the added benefit that there's history to see how each file evolved to its current state.

Dependencies

None

Testing

• Build documentation for some project and pass the --experimental-transform-for-static-hosting-with-content flag
• Start a basic web server from within the documentation archive, for example ruby -run -ehttpd . -p8000.
  Note: the later steps won't work with docc prevew
• Browse the local documentation.
  • Everything should look and work the same as today
• Disable JavaScript in your web browser
• Refresh the currently open documentation page
  • You should see a very plain version the content, without most of the styling and layout
• Click though links in the content, curation, and breadcrumbs
  • You should be taken to the page that you clicked on

• Navigate to one of the index.html files in the file system and open (it by double clicking).
  Alternatively, on macOS you can run open /path/to/SomeModule.doccarchive/documentation/somemodule/index.html
  • You should see a completely plain version the content, without any of the styling and layout
• Click though links in the content, curation, and breadcrumbs
  • You should be taken to the page that you clicked on

Checklist

Make sure you check off the following items. If they cannot be completed, provide a reason.

• Added tests
• Ran the ./bin/test script and it succeeded
• Updated documentation if necessary

[d-ronnqvist]

@swift-ci please test

[d-ronnqvist]

@swift-ci please test

[d-ronnqvist]

@swift-ci please test

[d-ronnqvist]

@swift-ci please test

[d-ronnqvist]

I've started a cross-repo test in swiftlang/swift#84510 (comment) to verify that the new DocCHTML target it set up correctly and doesn't cause any trouble.

[d-ronnqvist]

@swift-ci please test

[d-ronnqvist]

Because this is too large for a PR I'll mark this as a draft and start opening smaller PRs to land these changes incrementally (but squashed). Starting with the first PR here #1369