Help debugging why unnecessary tasks are running

[davidalbers]

Hi 👋

I've setup AMD in my project and created a custom task to run unit tests for a specific flavor:

    customTasks = [
            new AffectedModuleConfiguration.CustomTask(
                    "runAffectedStagingDebugUnitTests",
                    "testStagingDebugUnitTest",
                    "Run Affected Staging Debug Unit Tests"
            )
    ]

When I run

./gradle runAffectedStagingDebugUnitTests -Paffected_module_detector.enable

As expected, the unitTest task only runs on affected modules, example log output:

...
> Task  :some:affected:module:testStagingDebugUnitTest
...
... test output
...
> Task :some:unaffected:module:testStagingDebugUnitTest SKIPPED

However, I still see some dependent tasks from the unaffected modules still running and contributing to the build time:

...
> Task :some:unaffected:module:preStagingDebugUnitTestBuild
> Task :some:unaffected:module:checkStagingDebugUnitTestAarMetadata
...

Since these are dependencies for :some:unaffected:module:testStagingDebugUnitTest I'd expect them not to run.
Is that right? Do you have any ideas how I can debug what's going on?

[davidalbers]

Update: I think I've worked out a solution. For all subtasks, I create a list of all the projects that use that particular subtask. Then I check if any of the projects in the list are affected.

I'm not 100% sure if this is just a quirk of my project or an actual issue with the library so I'm hesitant to open a PR. If anyone can confirm that this will be an improvement, I'd be happy to open a PR though.

Here's roughly my fix:

// in a custom plugin that I based on AffectedModuleDetectorPlugin.kt

internal fun registerCustomTasks(
    rootProject: Project,
    customTasks: Set<AffectedModuleTaskType>
) {
    customTasks.forEach { taskType ->
        rootProject.tasks.register(taskType.commandByImpact) { task ->
            //...
            val dependencyMap = hashMapOf<Task, ArrayList<Project>>()

            rootProject.subprojects { project ->
                pluginIds.forEach { pluginId ->
                    withPlugin(pluginId, task, taskType, project, dependencyMap)
                }
            }

            markTasksAsAffected(dependencyMap)
        }
    }
}

private fun withPlugin(
        pluginId: String,
        task: Task,
        testType: AffectedModuleTaskType,
        project: Project,
        dependencyMap: HashMap<Task, ArrayList<Project>>,
) {
    //...
    project.tasks.findByPath(path)?.let {
        createDependencyMapForTask(task, project, )
    }
}

private fun createDependencyMapForTask(
    task: Task,
    rootProject: Project,
    dependencyMap: HashMap<Task, ArrayList<Project>>,
) {
    val projects = dependencyMap.getOrDefault(task, arrayListOf())
    if (!projects.contains(rootProject)) {
        projects.add(rootProject)
        dependencyMap[task] = projects
        addSubTasksToDependencyMap(task, rootProject, dependencyMap)
        addFinalizedByTasksToDependencyMap(task, rootProject, dependencyMap)
    }
}

private fun addSubTasksToDependencyMap(
    task: Task,
    rootProject: Project,
    dependencyMap: HashMap<Task, ArrayList<Project>>,
) {
    task.taskDependencies.getDependencies(task).forEach { subTask ->
        createDependencyMapForTask(subTask, rootProject, dependencyMap)
    }
}

private fun addFinalizedByTasksToDependencyMap(
    task: Task,
    rootProject: Project,
    dependencyMap: HashMap<Task, ArrayList<Project>>,
) {
    task.finalizedBy.getDependencies(task).forEach { subTask ->
        createDependencyMapForTask(subTask, rootProject, dependencyMap)
    }
}

private fun markTasksAsAffected(dependencyMap: HashMap<Task, ArrayList<Project>>) {
    dependencyMap.entries.forEach { (dependentTask, projects) ->
        dependentTask.onlyIf {
            projects.any { dependentProject ->
                AffectedModuleDetector.isProjectAffected(dependentProject)
            }
        }
    }
}

[joshafeinberg]

100% open a PR for this. We have a similar report in #223 and I couldn't figure out a clean way. Would love to test this out and put out a release with this fix cause it could be huge gains for some projects!

[ss0930]

Update: I think I've worked out a solution. For all subtasks, I create a list of all the projects that use that particular subtask. Then I check if any of the projects in the list are affected.

I'm not 100% sure if this is just a quirk of my project or an actual issue with the library so I'm hesitant to open a PR. If anyone can confirm that this will be an improvement, I'd be happy to open a PR though.

Here's roughly my fix:

// in a custom plugin that I based on AffectedModuleDetectorPlugin.kt

internal fun registerCustomTasks(
    rootProject: Project,
    customTasks: Set<AffectedModuleTaskType>
) {
    customTasks.forEach { taskType ->
        rootProject.tasks.register(taskType.commandByImpact) { task ->
            //...
            val dependencyMap = hashMapOf<Task, ArrayList<Project>>()

            rootProject.subprojects { project ->
                pluginIds.forEach { pluginId ->
                    withPlugin(pluginId, task, taskType, project, dependencyMap)
                }
            }

            markTasksAsAffected(dependencyMap)
        }
    }
}

private fun withPlugin(
        pluginId: String,
        task: Task,
        testType: AffectedModuleTaskType,
        project: Project,
        dependencyMap: HashMap<Task, ArrayList<Project>>,
) {
    //...
    project.tasks.findByPath(path)?.let {
        createDependencyMapForTask(task, project, )
    }
}

private fun createDependencyMapForTask(
    task: Task,
    rootProject: Project,
    dependencyMap: HashMap<Task, ArrayList<Project>>,
) {
    val projects = dependencyMap.getOrDefault(task, arrayListOf())
    if (!projects.contains(rootProject)) {
        projects.add(rootProject)
        dependencyMap[task] = projects
        addSubTasksToDependencyMap(task, rootProject, dependencyMap)
        addFinalizedByTasksToDependencyMap(task, rootProject, dependencyMap)
    }
}

private fun addSubTasksToDependencyMap(
    task: Task,
    rootProject: Project,
    dependencyMap: HashMap<Task, ArrayList<Project>>,
) {
    task.taskDependencies.getDependencies(task).forEach { subTask ->
        createDependencyMapForTask(subTask, rootProject, dependencyMap)
    }
}

private fun addFinalizedByTasksToDependencyMap(
    task: Task,
    rootProject: Project,
    dependencyMap: HashMap<Task, ArrayList<Project>>,
) {
    task.finalizedBy.getDependencies(task).forEach { subTask ->
        createDependencyMapForTask(subTask, rootProject, dependencyMap)
    }
}

private fun markTasksAsAffected(dependencyMap: HashMap<Task, ArrayList<Project>>) {
    dependencyMap.entries.forEach { (dependentTask, projects) ->
        dependentTask.onlyIf {
            projects.any { dependentProject ->
                AffectedModuleDetector.isProjectAffected(dependentProject)
            }
        }
    }
}

I am testing this out with a fork of this repo. I am running into incredibly long configuration time and seeing

Configuration 'releaseCompileClasspath' was resolved during configuration time.
This is a build performance and scalability issue.
See https://github.com/gradle/gradle/issues/2298
Run with --info for a stacktrace.
Configuration 'releaseRuntimeClasspath' was resolved during configuration time.
This is a build performance and scalability issue.
See https://github.com/gradle/gradle/issues/2298
Run with --info for a stacktrace.
...
many more

gradle/gradle#2298

I see the unaffected module tasks being skipped but their dependent tasks are still executed.

[joshafeinberg]

Ah yeah, that is a problem. In order to know at the time which routes to run we'd need to potentially trigger configuring all those dependencies.

I haven't had a chance to look at this but it might be a hard problem to handle