final draft

This commit is contained in:
Sarah Jamie Lewis 2023-02-02 12:06:12 -08:00
parent 7b8d8eaf26
commit 6fd364aea6
3 changed files with 17 additions and 16 deletions

View File

@ -12,9 +12,9 @@ authors:
image_url: /img/sarah.jpg
---
We first [introduced UI tests in last January](https://openprivacy.ca/discreet-log/23-cucumber-testing/), at the time we had developed a suite of UI tests that could be run manually in a development environment. However, we faced a number of issues consistently running these tests in our automated pipelines.
We first [introduced UI tests last January](https://openprivacy.ca/discreet-log/23-cucumber-testing/). At the time we had developed a suite of UI tests that could be run manually in a development environment. However, we faced a number of issues consistently running these tests in our automated pipelines.
One of the main threads of work that needs to be complete early in the Cwtch Stable roadmap is integrating UI tests into our CI pipelines, in addition to expanding their scope. Now that Flutter 3 has stabilized desktop support, and we have invested effort in improving Cwtch performance, it is time to ensure these tests are running on every build.
One of the main threads of work that needs to be complete early in the [Cwtch Stable roadmap](https://docs.cwtch.im/blog/path-to-cwtch-stable) is integrating UI tests into our CI pipelines, in addition to expanding their scope. Now that Flutter 3 has stabilized desktop support, and we have invested effort in improving Cwtch performance, it is time to ensure these tests are running on every build.
![](/img/devlog5.png)
@ -22,24 +22,24 @@ One of the main threads of work that needs to be complete early in the Cwtch Sta
## Current Limitations of Flutter Gherkin
The original [flutter_gherkin](https://pub.dev/packages/flutter_gherkin) is under semi-active development, however the latest published versions don't support using it with `flutter test`.
The original [flutter_gherkin](https://pub.dev/packages/flutter_gherkin) is under semi-active development; however, the latest published versions don't support using it with `flutter test`.
- **Flutter Test** - was originally intended to run single widget/unit tests for a flutter project.
- **Flutter Drive** - was originally intended to run integration tests *on a device or an emulator*.
- **Flutter Test** was originally intended to run single widget/unit tests for a Flutter project.
- **Flutter Drive** was originally intended to run integration tests *on a device or an emulator*.
However, in recent releases these lines have become blurred. The new [integration_test](https://docs.flutter.dev/testing/integration-tests) package that comes built into newer flutter releases has support for both `flutter drive` and `flutter test`. This was a great change because decreases the required overhead to run larger integration tests (`flutter drive` setups up a host-controller model that requires a dedicated control channel to be setup, `flutter test` can take advantage of the knowledge that it is being run in the same process, and is noticeably faster - very important when the goal is to run tests as often as possible).
However, in recent releases these lines have become blurred. The new [integration_test](https://docs.flutter.dev/testing/integration-tests) package that comes built into newer Flutter releases has support for both `flutter drive` and `flutter test`. This was a great change because it decreases the required overhead to run larger integration tests (`flutter drive` sets up a host-controller model that requires a dedicated control channel to be setup, whereas `flutter test` can take advantage of the knowledge that it is being run in the same process, and is noticeably faster - very important when the goal is to run tests as often as possible).
There is thankfully code in the `flutter_gherkin` repository that supports running tests with `flutter test`, however this code currently has a few issues:
- the test code generation produces code that doesn't compile with minor changes.
- certain functionality like "take a screenshot" does not work on desktop.
- The test code generation produces code that doesn't compile without minor changes.
- Certain functionality like "take a screenshot" does not work on desktop.
Additionally, there are a few limitations in built-in flutter_gherkin steps that we noticed our tests running into:
- certain tests that fail with async timeouts will cause flutter exceptions instead of a failed test.
- certain flutter widgets like `DropdownButton` are not compatible with built-in steps like `tap` because they internally contain multiple copies of the same widget
- Certain tests that fail with async timeouts will cause Flutter exceptions instead of a failed test.
- Certain Flutter widgets like `DropdownButton` are not compatible with built-in steps like `tap` because they internally contain multiple copies of the same widget.
Because of the above issues we have chosen to [fork flutter_gherkin](https://git.openprivacy.ca/openprivacy/flutter_gherkin) to fix some of these issues, with the intent of contributing significant fixes upstream, while allowing us to iterate faster on flutter UI testing.
Because of the above issues we have chosen to [fork flutter_gherkin](https://git.openprivacy.ca/openprivacy/flutter_gherkin) to fix some of these issues, with the intent of contributing significant fixes upstream, while allowing us to iterate faster on Flutter UI testing.
## Integrating Tests into the Pipeline
@ -50,7 +50,7 @@ Thankfully it is possible to use [Xfvb](https://en.wikipedia.org/wiki/Xvfb) to c
export DISPLAY=:99
Xvfb -ac :99 -screen 0 1280x1024x24 > /dev/null 2>&1 &
This allows us to neutralize the main issue with `flutter test`, and efficiently run tests in our pipeline.
This allows us to neutralize our main issue with `flutter test`, and efficiently run tests in our pipeline.
## Catching Bugs!
@ -59,15 +59,16 @@ This small amount of integration work has already caught its first bug.
Once we had fixed most of the issues outlined above, we were still seeing failures on what should have been a very basic scenario. [02_save_load.feature](https://git.openprivacy.ca/cwtch.im/cwtch-ui/src/branch/trunk/integration_test/features/01_general/02_save_load.feature) simply turns a set of experiments on and checks that the state is saved. This test runs perfectly fine on
development environments, but when uploaded to our build pipeline it always failed in the same place - turning on the file sharing experiment.
The cause of this was an actual bug Cwtch UI. The file sharing experiment failed to turn on if the directory `$USER_HOME/Downloads` didn't exist. This is rarely the case on most real world systems, but is the case in our build pipelines. We have since fixed this behaviour to allow file sharing to be turned on even if the usual Download directories are not available.
The cause of this was an actual bug in Cwtch UI. The file sharing experiment failed to turn on if the directory `$USER_HOME/Downloads` didn't exist. This is rarely the case on most real world systems, but is the case in our build pipelines. We have since fixed this behaviour to allow file sharing to be turned on even if the usual Download directories are not available.
As we enable more of our UI tests in our pipeline, and across more platforms, we expect to catch more subtle issues like the above - a big win for people who use Cwtch!
## Next Steps
- **More automated tests** - we have a nice collection of pre-written tests that we can begin to automatacally run within pipelines. We have already begun this work, and anticipate finishing it before Cwtch 1.11.
- **More platforms** - right now UI tests only run on Linux. In order to fully take advantage of these tests we need to be able to run them across our target platforms. We expect to start this work soon, expect more news in a future Cwtch Testing update!
- **More steps** - one of our longer-term goals with UI testing was to produce a language around Cwtch testing that went beyond wdigets. We had begun to explore last year with the `expect to see the message` step. As we grow our test library we will be looking for opportunities to build out addition higher-level and cwtch-specific constructs e.g. `send a file` or `set profile picture`.
- **More automated tests:** We have a nice collection of pre-written tests that we can begin to automatically run within pipelines. We have already begun this work, and anticipate finishing it before Cwtch 1.11.
- **More platforms:** Right now UI tests only run on Linux. In order to fully take advantage of these tests we need to be able to run them across [our target platforms](https://docs.cwtch.im/docs/getting-started/supported_platforms). We expect to start this work soon; expect more news in a future Cwtch Testing update!
- **More steps:** One of our longer-term goals with UI testing was to produce a language around Cwtch testing that went beyond widgets. We had begun to explore this last year with the `expect to see the message` step. As we grow our test library we will be looking for opportunities to build out additional higher-level and Cwtch-specific constructs, e.g. `send a file` or `set profile picture`.
## Help us go further!

Binary file not shown.

Before

Width:  |  Height:  |  Size: 639 KiB

After

Width:  |  Height:  |  Size: 334 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 380 KiB