Failed tests don't fail the step

[johnhess]

When I run this step with an intentionally failing espresso test, the logs show that 0/1 tests pass. But the step itself succeeds, so we don't get notified in GitHub.

I would expect failing tests to fail the step. Otherwise, why bother waiting for results?

+------------------------------------------------------------------------------+

| (9) BrowserStack App Automate - Espresso                                     |
+------------------------------------------------------------------------------+
| id: browserstack-bitrise-espresso-step                                       |
| version: 1.0.0                                                               |
| collection: https://github.com./bitrise-io/bitrise-steplib.git                |
| toolkit: go                                                                  |
| time: 2023-02-08T23:16:47Z                                                   |
+------------------------------------------------------------------------------+
|                                                                              |
Start installing (git) with apt-get
 * [OK] Step dependency (git) installed, available.
Start installing (wget) with apt-get
 * [OK] Step dependency (wget) installed, available.
2023/02/08 23:16:48 Starting the build on BrowserStack App Automate
2023/02/08 23:16:48 Uploading app on BrowserStack App Automate
2023/02/08 23:16:53 Successfully uploaded the app
2023/02/08 23:16:53 Uploading test suite on BrowserStack App Automate
2023/02/08 23:16:55 Successfully uploaded the test suite
2023/02/08 23:16:56 Successfully started the build
2023/02/08 23:16:56 Waiting for results
2023/02/08 23:17:26 Build is running (BrowserStack build id 26fe8078974f544840991df6090cf121bea6397d)
2023/02/08 23:17:56 Build is running (BrowserStack build id 26fe8078974f544840991df6090cf121bea6397d)
2023/02/08 23:17:56 Build finished
2023/02/08 23:17:56 Test results summary:
+------------------------------------------+--------------------------+---------------------+
|                 BUILD ID                 |         DEVICES          |       STATUS        |
+------------------------------------------+--------------------------+---------------------+
| 26fe8078974f544840991df6090cf121bea6397d | Samsung Galaxy S22 Ultra | FAILED (0/1 passed) |
+------------------------------------------+--------------------------+---------------------+
|                                                                              |
+---+---------------------------------------------------------------+----------+
| ✓ | BrowserStack App Automate - Espresso                          | 1.2 min |
+---+---------------------------------------------------------------+----------+

[Gosunet]

I have the same problem.

You can make a script step after this to check the output BROWSERSTACK_BUILD_STATUS and fail the step.
But yes it will be better if this step failed (if we want to, we can maybe add an input to the step ?)