diff --git a/docs/android-startup-scenarios.md b/docs/android-startup-scenarios.md new file mode 100644 index 00000000000..075ea14a10e --- /dev/null +++ b/docs/android-startup-scenarios.md @@ -0,0 +1,64 @@ + +# Android Startup Test + +## Prereqs + +- Ensure `python` is installed and available. Any currently supported `python` 3.* version should work. Downloads are available at https://www.python.org/downloads/. +- Ensure `dotnet` is installed and available with the `dotnet` command for easy xharness installation. Any supported .NET Core version should work. [Dotnet Download](https://dotnet.microsoft.com/en-us/download) or [Daily Dotnet Download](https://github.com/dotnet/sdk/blob/main/documentation/package-table.md). +- Ensure `xharness` is installed and available with the `xharness` command. The current version in use can be found in the `eng/performance/maui_scenarios_android.proj` file at line 7 (under the tag `MicrosoftDotNetXHarnessCLIVersion`), although any recent version should work. [XHarness Install Instructions](https://github.com/dotnet/xharness?tab=readme-ov-file#installation-and-usage). +- Have an Android app APK available for testing. +- Have an Android Device (with developer mode enabled) or emulator connected to computer, and viewable with `xharness android device` or `xharness android adb -- devices -l`. + +## Steps + +1. Initialize the environment (note the . for bash): + + ```sh + cd src/scenarios + . ./init.sh # or `.\init.ps1` on Windows. Can specify custom dotnet install with -dotnetdir , but dotnet install should not impact Android Startup testing itself. + ``` + +2. Navigate to the `genericandroidstartup` scenario directory: + + ```sh + cd genericandroidstartup + ``` + +3. Copy the APK into the `genericandroidstartup` directory. +4. Run the test: + + ```sh + python test.py devicestartup --device-type android --package-path --package-name [--disable-animations] [--use-fully-drawn-time --fully-drawn-extra-delay ] + ``` + +* Refer to the [Notes](./android-startup-scenarios.md#notes) below about specifying --use-fully-drawn-time --fully-drawn-extra-delay parameters. + +5. Read the output: + + During the running of the test you will see the loop of the activity being started to get the startup times. + Once the testing is completed, you will see output similar to the following: + + ```txt + [2025/01/29 11:15:44][INFO] Found Value (ms): 713 + [2025/01/29 11:15:44][INFO] Found Value (ms): 715 + [2025/01/29 11:15:44][INFO] Found Value (ms): 728 + [2025/01/29 11:15:44][INFO] Found Value (ms): 716 + [2025/01/29 11:15:44][INFO] Found Value (ms): 715 + [2025/01/29 11:15:44][INFO] Found Value (ms): 734 + [2025/01/29 11:15:44][INFO] Found Value (ms): 716 + [2025/01/29 11:15:44][INFO] Found Value (ms): 718 + [2025/01/29 11:15:44][INFO] Found Value (ms): 713 + [2025/01/29 11:15:44][INFO] Found Value (ms): 706 + [2025/01/29 11:15:44][INFO] Device Startup - Maui Android Default NoAnimation + [2025/01/29 11:15:44][INFO] Metric |Average |Min |Max + [2025/01/29 11:15:44][INFO] ----------------|---------------|---------------|--------------- + [2025/01/29 11:15:44][INFO] Generic Startup |717.400 ms |706.000 ms |734.000 ms + ``` + + The Found Value's are the individual test run startup times with the overall stats at the bottom. The stats provided include the following startup stats: average, minimum, and maximum times. + +## Notes + +- Specific example command such as when using the runtime android example app: `python test.py devicestartup --device-type android --package-path HelloAndroid.apk --package-name net.dot.HelloAndroid`. +- Other example commands and additional logic can be found in the `maui_scenarios_android.proj` and `runner.py` files in the `performance` repository. +- If using `[--use-fully-drawn-time --fully-drawn-extra-delay ]` arguments, the Android app must have reportFullyDrawn() called on a ComponentActivity. Reference: https://developer.android.com/topic/performance/vitals/launch-time#retrieve-TTFD. diff --git a/docs/scenarios-workflow.md b/docs/scenarios-workflow.md index b57191933a7..cbbdeb441b1 100644 --- a/docs/scenarios-workflow.md +++ b/docs/scenarios-workflow.md @@ -3,7 +3,7 @@ ## Overview -Our existing scenario tests are under `src\scenarios` in this repo, where each subdirectory contains a test asset that can be combined with a specific set of commands to do measurements. Currently we have scenario tests for [SDK](./sdk-scenarios.md), [Crossgen](./crossgen-scenarios.md), [Blazor](./blazor-scenarios.md) and [other scenarios](./basic-scenarios.md). +Our existing scenario tests are under `src\scenarios` in this repo, where each subdirectory contains a test asset that can be combined with a specific set of commands to do measurements. Currently we have scenario tests for [SDK](./sdk-scenarios.md), [Crossgen](./crossgen-scenarios.md), [Blazor](./blazor-scenarios.md), [Android Startup](./android-startup-scenarios.md), and [other scenarios](./basic-scenarios.md). ## Running scenario tests @@ -12,6 +12,7 @@ This is a general guideline on how the scenario tests are arranged in this repo. - [How to run SDK scenario tests](./sdk-scenarios.md) - [How to run Crossgen scenario tests](./crossgen-scenarios.md) - [How to run Blazor tests](./blazor-scenarios.md) +- [How to run Android Startup tests](./android-startup-scenarios.md) - [How to run other Scenario tests](./basic-scenarios.md) ### Prerequisites @@ -138,4 +139,5 @@ Some command options are only applicable for certain test assets. Refer to the c - [SDK Command Matrix](./sdk-scenarios.md#command-matrix) - [Crossgen Command Matrix](./crossgen-scenarios.md#command-matrix) - [Blazor Command Matrix](./blazor-scenarios.md#command-matrix) +- [How to run Android Startup tests](./android-startup-scenarios.md) - [Other Scenarios Command Matrix](./basic-scenarios.md#command-matrix) diff --git a/src/scenarios/genericandroidstartup/post.py b/src/scenarios/genericandroidstartup/post.py new file mode 100644 index 00000000000..c87a49b4b7d --- /dev/null +++ b/src/scenarios/genericandroidstartup/post.py @@ -0,0 +1,7 @@ +''' +post cleanup script +''' + +from shared.postcommands import clean_directories + +clean_directories() diff --git a/src/scenarios/genericandroidstartup/pre.py b/src/scenarios/genericandroidstartup/pre.py new file mode 100644 index 00000000000..ebb277b8a83 --- /dev/null +++ b/src/scenarios/genericandroidstartup/pre.py @@ -0,0 +1,34 @@ +''' +pre-command +''' +import sys +import os +from zipfile import ZipFile +from performance.logger import setup_loggers, getLogger +from shutil import copyfile +from shared.precommands import PreCommands +from shared.const import PUBDIR +from argparse import ArgumentParser + +setup_loggers(True) + +parser = ArgumentParser() +parser.add_argument( + '--apk-name', + dest='apk', + required=True, + type=str, + help='Name of the APK to setup (with .apk)') +args = parser.parse_args() + +if not os.path.exists(PUBDIR): + os.mkdir(PUBDIR) +apkname = args.apk +if not os.path.exists(apkname): + getLogger().error('Cannot find %s' % (apkname)) + exit(-1) +else: + copyfile(apkname, os.path.join(PUBDIR, apkname)) + + + diff --git a/src/scenarios/genericandroidstartup/test.py b/src/scenarios/genericandroidstartup/test.py new file mode 100644 index 00000000000..049ceab2bdd --- /dev/null +++ b/src/scenarios/genericandroidstartup/test.py @@ -0,0 +1,12 @@ +''' +C# Console app +''' +from shared.runner import TestTraits, Runner + +EXENAME = 'GenericAndroidStartup' + +if __name__ == "__main__": + traits = TestTraits(exename=EXENAME, + guiapp='false', + ) + Runner(traits).run()