initial draft of testing devlog 5
This commit is contained in:
parent
9a27bf6c4e
commit
7b8d8eaf26
|
|
@ -0,0 +1,83 @@
|
|||
---
|
||||
title: Notes on Cwtch UI Testing
|
||||
description: "In this development log we provide an update on automated UI integration testing!"
|
||||
slug: cwtch-testing-i
|
||||
tags: [cwtch, cwtch-stable,support, testing]
|
||||
image: /img/devlog5_small.png
|
||||
hide_table_of_contents: false
|
||||
toc_max_heading_level: 4
|
||||
authors:
|
||||
- name: Sarah Jamie Lewis
|
||||
title: Executive Director, Open Privacy Research Society
|
||||
image_url: /img/sarah.jpg
|
||||
---
|
||||
|
||||
We first [introduced UI tests in last January](https://openprivacy.ca/discreet-log/23-cucumber-testing/), at the time we had developed a suite of UI tests that could be run manually in a development environment. However, we faced a number of issues consistently running these tests in our automated pipelines.
|
||||
|
||||
One of the main threads of work that needs to be complete early in the Cwtch Stable roadmap is integrating UI tests into our CI pipelines, in addition to expanding their scope. Now that Flutter 3 has stabilized desktop support, and we have invested effort in improving Cwtch performance, it is time to ensure these tests are running on every build.
|
||||
|
||||
|
||||
|
||||
<!--truncate-->
|
||||
|
||||
## Current Limitations of Flutter Gherkin
|
||||
|
||||
The original [flutter_gherkin](https://pub.dev/packages/flutter_gherkin) is under semi-active development, however the latest published versions don't support using it with `flutter test`.
|
||||
|
||||
- **Flutter Test** - was originally intended to run single widget/unit tests for a flutter project.
|
||||
- **Flutter Drive** - was originally intended to run integration tests *on a device or an emulator*.
|
||||
|
||||
However, in recent releases these lines have become blurred. The new [integration_test](https://docs.flutter.dev/testing/integration-tests) package that comes built into newer flutter releases has support for both `flutter drive` and `flutter test`. This was a great change because decreases the required overhead to run larger integration tests (`flutter drive` setups up a host-controller model that requires a dedicated control channel to be setup, `flutter test` can take advantage of the knowledge that it is being run in the same process, and is noticeably faster - very important when the goal is to run tests as often as possible).
|
||||
|
||||
There is thankfully code in the `flutter_gherkin` repository that supports running tests with `flutter test`, however this code currently has a few issues:
|
||||
|
||||
- the test code generation produces code that doesn't compile with minor changes.
|
||||
- certain functionality like "take a screenshot" does not work on desktop.
|
||||
|
||||
Additionally, there are a few limitations in built-in flutter_gherkin steps that we noticed our tests running into:
|
||||
|
||||
- certain tests that fail with async timeouts will cause flutter exceptions instead of a failed test.
|
||||
- certain flutter widgets like `DropdownButton` are not compatible with built-in steps like `tap` because they internally contain multiple copies of the same widget
|
||||
|
||||
Because of the above issues we have chosen to [fork flutter_gherkin](https://git.openprivacy.ca/openprivacy/flutter_gherkin) to fix some of these issues, with the intent of contributing significant fixes upstream, while allowing us to iterate faster on flutter UI testing.
|
||||
|
||||
## Integrating Tests into the Pipeline
|
||||
|
||||
One of the major limitations of `flutter test` is the lack of a headless mode. In order to successfully run tests in our pipeline we need a headless mode, as most of the containers we use do not have any kind of active display.
|
||||
|
||||
Thankfully it is possible to use [Xfvb](https://en.wikipedia.org/wiki/Xvfb) to create a virtual framebuffer, and set `DISPLAY` to render to that buffer:
|
||||
|
||||
export DISPLAY=:99
|
||||
Xvfb -ac :99 -screen 0 1280x1024x24 > /dev/null 2>&1 &
|
||||
|
||||
This allows us to neutralize the main issue with `flutter test`, and efficiently run tests in our pipeline.
|
||||
|
||||
## Catching Bugs!
|
||||
|
||||
This small amount of integration work has already caught its first bug.
|
||||
|
||||
Once we had fixed most of the issues outlined above, we were still seeing failures on what should have been a very basic scenario. [02_save_load.feature](https://git.openprivacy.ca/cwtch.im/cwtch-ui/src/branch/trunk/integration_test/features/01_general/02_save_load.feature) simply turns a set of experiments on and checks that the state is saved. This test runs perfectly fine on
|
||||
development environments, but when uploaded to our build pipeline it always failed in the same place - turning on the file sharing experiment.
|
||||
|
||||
The cause of this was an actual bug Cwtch UI. The file sharing experiment failed to turn on if the directory `$USER_HOME/Downloads` didn't exist. This is rarely the case on most real world systems, but is the case in our build pipelines. We have since fixed this behaviour to allow file sharing to be turned on even if the usual Download directories are not available.
|
||||
|
||||
As we enable more of our UI tests in our pipeline, and across more platforms, we expect to catch more subtle issues like the above - a big win for people who use Cwtch!
|
||||
|
||||
## Next Steps
|
||||
|
||||
- **More automated tests** - we have a nice collection of pre-written tests that we can begin to automatacally run within pipelines. We have already begun this work, and anticipate finishing it before Cwtch 1.11.
|
||||
- **More platforms** - right now UI tests only run on Linux. In order to fully take advantage of these tests we need to be able to run them across our target platforms. We expect to start this work soon, expect more news in a future Cwtch Testing update!
|
||||
- **More steps** - one of our longer-term goals with UI testing was to produce a language around Cwtch testing that went beyond wdigets. We had begun to explore last year with the `expect to see the message` step. As we grow our test library we will be looking for opportunities to build out addition higher-level and cwtch-specific constructs e.g. `send a file` or `set profile picture`.
|
||||
|
||||
## Help us go further!
|
||||
|
||||
We couldn't do what we do without all the wonderful community support we get, from [one-off donations](https://openprivacy.ca/donate) to [recurring support via Patreon](https://www.patreon.com/openprivacy).
|
||||
|
||||
If you want to see us move faster on some of these goals and are in a position to, please [donate](https://openprivacy.ca/donate). If you happen to be at a company that wants to do more for the community and this aligns, please consider donating or sponsoring a developer.
|
||||
|
||||
Donations of **$5 or more** can opt to receive stickers as a thank-you gift!
|
||||
|
||||
For more information about donating to Open Privacy and claiming a thank you gift [please visit the Open Privacy Donate page](https://openprivacy.ca/donate/).
|
||||
|
||||
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 639 KiB |
Loading…
Reference in New Issue