Feature request: Better support for non-CLI usage - or: nextest-as-a-lib

[TobiasvdVen]

Motivation

I have a pet project that is basically a GUI for running tests.

I'd like to leverage nextest for the actual mechanics of discovering, running and parsing tests. If I can do this in code, rather than needing to parse CLI output, that would be vastly preferable.

There's a few minor things blocking my ability to do so, however, mainly related to visibility.

I've outlined what seems to me a "minimal" way to be able to do what I need in the "Proposal" section.

What I'm wondering is, is this something that nextest wants to support? Is it strictly intended to be used as a CLI tool, or is it also meant to serve as a lib?

Proposal

Having poked around a bit, it seems nextest is composed and run with CargoNextestApp::parse() as its primary starting point. Through this API, it seems there is currently no way to leverage the TestEvent stream that TestRunner::execute(...) provides.

To gain access to that, it seems to me that the CargoOptions type must be made pub (instead of pub(crate)). This then allows access to CargoOptions::compute_binary_list(), which produces the binaries that are needed to run the tests. The function acquire_graph_data() must also be made pub, as the metadata it gathers is required for the previous step.

Likewise, the fields of OutputContext must be made pub (instead of pub(crate)), as the OutputContext is required for the functions above. There may be a way to avoid this, as an OutputContext is presumably not a true dependency when one is running without wanting nextest to actually write to the process output.

There also needs to be a way to construct and initialize a CargoOptions instance. I have a Fork over here where I've made these minimal changes. I use bon to facilitate the CargoOptions construction though, and I'm hesitant to create a PR for these changes. Adding a whole crate just for an edge-case like this may be too much. I'm curious about others' thoughts.

Alternatives

As an alternative to bon, it would be possible to manually craft a builder pattern, or just add a massive new function and rely on consumers of nextest to deal with the construction of CargoOptions (at least it would be possible that way, as long as the new is pub).

There may also be alternatives that modify the existing API landscape. This could reduce the number of additional types or functions that need to be made pub, by changing at what point they are injected.

Additional context

As additional context, this is a rough example of how the nextest API could be used to run some tests, given the changes outlined above:

let working_dir = ...;
let target_dir = ...;
let manifest_path = ...;

let cargo_options = cargo_options()
    .target_dir(target_dir.clone())
    .call();

let build_platforms = BuildPlatforms::new_with_no_target());

let output_context = OutputContext {
    verbose: true,
    color: cargo_nextest::Color::Auto
};

let graph_data = acquire_graph_data(Some(&manifest_path), Some(&target_dir), &options, &build_platforms, output_context);
let graph = PackageGraph::from_json(graph_data);
let parse_context = ParseContext::new(&graph);
let config_file = None;
let tool_config_files = Vec::new();
let experimental = BTreeSet::new();
let nextest_config = NextestConfig::from_sources(working_dir.clone(), &parse_context, config_file, tool_config_files, &experimental);

let binary_list = options.compute_binary_list(&graph, Some(&manifest_path), output_context, build_platforms.clone());

let path_mapper = PathMapper::noop();
let rust_build_meta = binary_list.rust_build_meta.map_paths(&path_mapper);
let platform_filter = None;
let artifacts = RustTestArtifact::from_binary_list(&graph, Arc::new(binary_list), &rust_build_meta, &path_mapper, platform_filter);

let double_spawn = DoubleSpawnInfo::disabled();
let target_runner = TargetRunner::empty();

let profile = nextest_config.profile(NextestConfig::DEFAULT_PROFILE);

let profile = profile.apply_build_platforms(&build_platforms);

let context = TestExecuteContext {
    profile_name: NextestConfig::DEFAULT_PROFILE,
    double_spawn: &double_spawn,
    target_runner: &target_runner
};

let test_filter_builder = TestFilterBuilder::default_set(RunIgnored::Default);

let cli_configs: Vec<String> = Vec::new();
let cargo_configs = CargoConfigs::new(cli_configs.into_iter());

let env = EnvironmentMap::new(&cargo_configs);

let test_list = TestList::new(
    &context,
    artifacts.into_iter(),
    rust_build_meta,
    &test_filter_builder,
    working_dir,
    env,
    &profile,
    FilterBound::DefaultSet,
    get_num_cpus()
);

let runner = nextest_runner::runner::TestRunnerBuilder::default().build(
    &test_list,
    &profile,
    vec![], // we aren't testing CLI args at the moment
    SignalHandlerKind::Noop,
    InputHandlerKind::Noop,
    DoubleSpawnInfo::disabled(),
    TargetRunner::empty()
);

runner.execute(|test_event| {
// Do custom stuff here
});

[sunshowers]

Thanks for the report. In general I'm okay accepting minor changes like making methods pub, but nextest's internals don't provide any stability guarantees, and will be changed as needed. Are you OK with that?

I appreciate what the bon folks are doing, but (if required) would prefer a manually-implemented builder pattern for the avoidance of proc-macro magic.

[TobiasvdVen]

Sounds good to me :)

I've gotten rid of bon, instead opting for a new function. Also tried to add some brief documentation comments to the things that I made public, to avoid introducing new warnings.

See the PR here: #2622