[MNT] diagnose and address long test runtimes

[fkiraly]

Current test runtimes are in the order of 1 hour.

We should diagnose the biggest time sinks and think about strategies to mitigate them, while optimally not reducing coverage.

[geetu040]

Some info from this run: https://github.com/openml/openml-python/actions/runs/22063995363/:

"production" and "sklearn" included tests are responsible for the major time sink

some tests are taking too long, though they involve a single simple API call
it could be something with the server, which is not able to hold the load of multiple jobs from multiple PRs
though I believe it should be sporadic, but some specific tests are taking too long, even when run locally, these single tests can take up to 30-60 mins to exexute, here are some noted from a few recent jobs

tests/test_study/test_study_functions.py::TestStudyFunctions::test_get_study_old
tests/test_evaluations/test_evaluation_functions.py::TestEvaluationFunctions::test_list_evaluations_setups_filter_flow
tests/test_runs/test_run_functions.py::TestRun::test_get_runs_list_by_tag
tests/test_tasks/test_task_functions.py::TestTask::test_list_clustering_task
tests/test_evaluations/test_evaluation_functions.py::TestEvaluationFunctions::test_list_evaluations_setups_filter_task
tests/test_flows/test_flow_functions.py::TestFlowFunctions::test_list_flows_paginate
tests/test_flows/test_flow_functions.py::TestFlowFunctions::test_list_flows_output_format
tests/test_evaluations/test_evaluations_example.py::TestEvaluationsExample::test_example_python_paper
tests/test_tasks/test_task_functions.py::TestTask::test_get_task_different_types
tests/test_runs/test_run_functions.py::TestRun::test_get_runs_list_by_task
tests/test_runs/test_run_functions.py::TestRun::test_get_runs_pagination
tests/test_study/test_study_functions.py::TestStudyFunctions::test_get_suite

[fkiraly]

yes, looks like it is coming from "production" - here is a list with runtimes from the durations flag:

3740.37s call     tests/test_study/test_study_functions.py::TestStudyFunctions::test_get_study_old
2094.26s call     tests/test_tasks/test_task_functions.py::TestTask::test_list_clustering_task
1980.66s call     tests/test_runs/test_run_functions.py::TestRun::test_get_runs_list_by_tag
175.46s call     tests/test_runs/test_run_functions.py::TestRun::test_get_runs_list_by_task
64.49s call     tests/test_flows/test_flow_functions.py::TestFlowFunctions::test_list_flows
42.39s call     tests/test_flows/test_flow_functions.py::TestFlowFunctions::test_list_flows_output_format
25.99s call     tests/test_evaluations/test_evaluation_functions.py::TestEvaluationFunctions::test_evaluation_list_sort
24.68s call     tests/test_flows/test_flow_functions.py::TestFlowFunctions::test_list_flows_paginate
22.22s call     tests/test_runs/test_run_functions.py::TestRun::test_online_run_metric_score
21.01s call     tests/test_runs/test_run_functions.py::TestRun::test_get_runs_pagination
17.99s call     tests/test_setups/test_setup_functions.py::TestSetupFunctions::test_get_setup
13.51s call     tests/test_evaluations/test_evaluation_functions.py::TestEvaluationFunctions::test_list_evaluations_setups_filter_flow
13.30s call     tests/test_tasks/test_task_functions.py::TestTask::test_get_task_different_types
12.72s call     tests/test_evaluations/test_evaluation_functions.py::TestEvaluationFunctions::test_list_evaluations_setups_filter_task
11.71s call     tests/test_runs/test_run_functions.py::TestRun::test_format_prediction_non_supervised
11.11s call     tests/test_flows/test_flow_functions.py::TestFlowFunctions::test_list_flows_empty
9.60s call     tests/test_study/test_study_functions.py::TestStudyFunctions::test_get_study_error
9.54s call     tests/test_datasets/test_dataset.py::OpenMLDatasetTestSparse::test_get_sparse_categorical_data_id_395
7.70s call     tests/test_tasks/test_clustering_task.py::OpenMLClusteringTaskTest::test_get_dataset
7.18s call     tests/test_setups/test_setup_functions.py::TestSetupFunctions::test_list_setups_output_format

[PGijsbers]

The slowest tests should take no longer than 1-2 minutes. These are tests that:

• run some local experiment that's time consuming (we should probably find lightweight equivalents)
• require a cronjob to run on the server (e.g., uploading a dataset and checking it's processed correctly by the server)

There should be no more than a few of those in the entire test suite. Any test that takes longer, especially such as reported above, is caused by connectivity issues or server-sided issues.

I think it would be useful to look at longer running tests (such as the ones that require the server cronjob) and make sure they can run without a server (e.g., through mocking). This is beneficial twice: our tests are more consistent and faster, but also those tests typically poll the server and for that reason may be more likely to cause server-sided issues in the first place.

[geetu040]

also since the connection_n_retries is set to 5 with a longer delay with each retry, a failing test will take even longer to fail

would it be fine to set retries to possibly 3?

[areebahmeddd]

Hi, could i pick this issue up? thanks!

[geetu040]

Hi, could i pick this issue up? thanks!

please mention down first how you are going to fix it, which test cases you are going to cover and what will you do to reduce their time

[kronzter]

Hi, could i pick this issue up? thanks!

please mention down first how you are going to fix it, which test cases you are going to cover and what will you do to reduce their time

Hi, Id like to try tackling this issue.
I have gone through the code base and have some questions and approaches.

Issues as ive tried to understand

1. Some tests call list_flows(), list_runs()etc. with no size limit so it is triggering _list_all to fetch the entire production database in batches of 10,000 records. For example, test_list_flows fetches all 15,000+ flows just to assert len >= 1500.
2. The as_robot fixture in conftest.py applies n_retries=20 with a sigmoid delay (approaching 60s/retry) to the entire test session. A single network error on a production test can stall and cause large delays
3. Several @pytest.mark.production_server tests hit the live server only because the test server lacks that data (study 34/99, setups, clustering tasks)not because they need production behavior. Can't these be replaced with mock fixtures using the create_request_response pattern already in the codebase?

Approach i'm planning:

• Cap uncapped listing calls with size=N where the assertion doesn't actually need the full dataset
• Reduce n_retries in the as_robot fixture
• Replace single-GET production tests with XML fixture mocks following the existing pattern in test_dataset_functions.py

Questions:

1. For tests like test_list_flows where the assertion is len >= 1500 is it okay to reduce this to something like len >= 100 with size=200, or would you prefer keeping the original assertion and instead just mocking the full response?
2. For the as_robot retry count is n_retries=3 an acceptable floor, or is this intentional?
   Because as mentioned above in thread I also think such a long delay is not required. From what I understand, such a long delay should only be for prod environment, isnt it fine if for just test suite, it fails faster?
3. Should the mocked tests keep the @pytest.mark.production_server marker (just using a mock instead of the real server), or should that marker be removed entirely if run without production server?
   Happy to discuss further before diving in!

[geetu040]

@kronzter yes you are looking in the right direction

• For tests like test_list_flows where the assertion is len >= 1500 is it okay to reduce this to something like len >= 100 with size=200, or would you prefer keeping the original assertion and instead just mocking the full response?

Ultimately for any given test-case it's better to mock than rely on live server call (unless it's an explicit integration test)
For mocking this is the relevant issue: #1649
CC: @PGijsbers

• For the as_robot retry count is n_retries=3 an acceptable floor, or is this intentional?
  Because as mentioned above in thread I also think such a long delay is not required. From what I understand, such a long delay should only be for prod environment, isnt it fine if for just test suite, it fails faster?

I would prefer a lower number and would be fine with n_retries=3, but would leave this question for @PGijsbers since they set it up to this value in the first place.

• Should the mocked tests keep the @pytest.mark.production_server marker (just using a mock instead of the real server), or should that marker be removed entirely if run without production server?
  Happy to discuss further before diving in!

yes you can remove the marker if it's not sending an api call to the production server

[kronzter]

@kronzter yes you are looking in the right direction

• For tests like test_list_flows where the assertion is len >= 1500 is it okay to reduce this to something like len >= 100 with size=200, or would you prefer keeping the original assertion and instead just mocking the full response?

Ultimately for any given test-case it's better to mock than rely on live server call (unless it's an explicit integration test) For mocking this is the relevant issue: #1649 CC: @PGijsbers

• For the as_robot retry count is n_retries=3 an acceptable floor, or is this intentional?
  Because as mentioned above in thread I also think such a long delay is not required. From what I understand, such a long delay should only be for prod environment, isnt it fine if for just test suite, it fails faster?

I would prefer a lower number and would be fine with n_retries=3, but would leave this question for @PGijsbers since they set it up to this value in the first place.

• Should the mocked tests keep the @pytest.mark.production_server marker (just using a mock instead of the real server), or should that marker be removed entirely if run without production server?
  Happy to discuss further before diving in!

yes you can remove the marker if it's not sending an api call to the production server

Will await confirmation regarding retry count and then push a fix.

[geetu040]

@kronzter please see this comment #1633 (comment). Can you collect the relevant tests mentioned there? You can run these tests locally using the markers "production_server" and "test_server" to run specific tests.

[kronzter]

Hi @geetu040 ,
I've grouped them basis of test_server and production_server

With 40 longest durations :
Check snippet
test_server

Production_server

Except the one that failed looks like only one has run durations even reaching 90 secs

Rest are under a minute.

Please let me know if ive missed something.