Skip to main content

What is wdio-ocr-service

Background#

Sometimes it can be hard to find an element in a native app with the default WebdriverIO Mobile selectors. In that case it would be nice if you would be able to use something like OCR (Optical Character Recognition) to interact with elements on your screen.

This service provides you that option to interact with elements on your mobile screen based on visible text. It will provide multiple commands to:

  • wait
  • search
  • and interact

with an element, all based on text.

How does it work#

This service will

  1. create a screenshot of your screen
  2. optimize the result for OCR by turning the screenshot into a black/white with a high contrast screenshot (the high contrast is needed to prevent a lot of image background noise).
  3. uses Optical Character Recognition from Tesseract to get all text from the screen and highlight all found text on an image. Can support several languages, all supported languages can be found here.
  4. uses Fuzzy Logic from Fuse.js to find strings that are approximately equal to a given pattern (rather than exactly). This means for example that the search value Username can also find the text Usename or visa versa.

An example of step 1, 2 and 3 can be found in this image

Process steps

It works with ZERO system dependencies (besides from what WebdriverIO uses), but if needed it can also work with a local installation from Tesseract which will reduce the execution time drastically! (See also the Test Execution Optimization on how to speed up your tests.)

Enthusiastic? Start using it today by following the installation instructions below.

Important

There are a variety of reasons you might not get good quality output from Tesseract. One of the biggest reasons that could be related to your app and this module could be the fact that there is no proper color distinguish between the text that needs to be found, and the background. For example, a white text on a dark background can easily be found, but a light text on a white background or a dark text on a dark background can hardly be found.

See also this page for more information from Tesseract.

Also don't forget to read the FAQ.

Edit this page