All posts by Tomáš Andrle

Mobile app testing checklist

Some things to check before releasing a mobile application. Slightly iOS specific.

Startup

  • Start in different orientations
  • Start app with no internet connection
  • Delete all user data. Is the first start behaving correctly?
  • Install and use previous version of app. User data migrated correctly?

OS

  • Dark mode support working? Switching on any screen?
  • OS-wide font size applied?
  • Locale/language changes applied correctly?
  • Accesibility labels for everything?

Network

  • Initial start without internet
  • Try using each page of the app without internet
  • Turn off internet, go back to previous page
  • Is network outage during video playback handled well?
  • Wake up after a long time (week-month). Detected & refreshed?
  • Is the app usable on a slow or unreliable network connection?

Layout & navigation

  • Rotate back and forth each page
  • Rotate and go back to previous page
  • Play fullscreen video, rotate during playback, close. Layout updated correctly?
  • Blurry text anywhere? Enable highlighting of misaligned layers in Simulator
  • During a call, does the double height status bar break any layouts?
  • Tap a button 2-3 times really quickly. Only one screen instance created?
  • Look for navigation loops. Detail -> push info -> push detail -> push info etc.

Buttons

  • All buttons/elements trigger on touch up and NOT touch down?
  • All disabled/highlighted/hover/pushed states displayed correctly?
  • Are buttons large enough to hit on small screen?

Lists & views

  • Do all lists scroll to top after changing the sorting order?
  • Do all views show an “empty results” text when no data is available?
  • Showing a loading indicator when loading data takes >1s?

Tracking

  • Is analytics tracking working? Correct tracking IDs?
  • Crash reports actually sent and saved? Symbolication working?

Prolonged usage

  • Keep app running, use whole day, overnight. Still performing well?
  • Open & close a screen (open detail, go back). Repeat ~15 times. If it leaks memory it can eventually lead to a crash.

Input fields

  • Is the keyboard mode correct for all input fields?
  • Email keyboard layout for email input, numeric for phone numbers etc.
  • Placeholder text in fields that need it?
  • Is there a clear “X” button in fields that need it?
  • In email input fields, is the email format verified?
  • If maximum length is specified for fields, is the limit really applied? Even when a longer text is pasted? From input fields to local storage, to server and back?
  • If no maximum was specified, does the app actually work with very long inputs, long words, special characters? Does it break any layout, internal DB, server backend API?
  • Is it easy to hide the onscreen keyboard when needed?
  • Does auto-completion work as it should? Does it limit the number of results after typing a single letter? Does input still work when auto-completion API is offline?

Search input

  • Make a search, open one of the results, go back. Is the search text still there?
  • Does search for empty string work correctly? Single letter? Spaces? Special @#?:/ unicode emoji characters? Extra long strings? Extra long words?

Interruptions

  • Put phone to sleep and on again
  • Switch to another app and back
  • Simulate memory warning, go back to previous page
  • Simulate incoming call. App resumes correctly when call ends?

Date & time

  • Date formatting correct for different locale settings? No broken layouts?
  • Does it handle daylight saving changes correctly? Really?
  • Does it handle timezone changes correctly? Start app, suspend, change timezone, resume app. Works?

Sharing

  • Does social sharing actually work? Posts appear on FB/Twitter site? Correct title, body, images, links?
  • Correct Facebook app ID?
  • Does sharing work with and without the official Facebook/Twitter app installed on the device? Automatic login through the app?
  • Does the app show correct feedback message after sending/cancelling/failing?
  • Is the sharing email correctly formatted? In GMail, Outlook and Mail? On a mobile device?

Storage & output

  • If the app has some web caching mechanism, does it correctly restrict itself from using too much storage?
  • Does the app behave correctly when the device storage is full? (Load music or videos to your device to fill it up OR place simulator folder in a APFS volume/disk image with limited size)
  • Does the app correctly handle having its cache folders deleted by iOS? (Run in simulator and delete using Finder)
  • Does the app limit its console output in Release mode? ie. no excess debug information printed – only errors.

Legal

  • Acknowledgements page up to date with all 3rd party code/asset licensing info?
  • Is it possible to turn off usage tracking? Is the user warned about usage tracking on first start?
  • If the app has a built-in browser, is it OK to receive a 17+ rating in the app store?

OS versions

  • Fully functional on all supported OS versions?
  • Is it possible to run and use the last supported version on older devices/systems? Legacy API?

Devices

  • Does the app support all screen sizes in both orientations?

Development tools 2019

Here is a list of development tools I’m using at the moment.

Hardware

  • iMac retina 27″ 2015, 32GB RAM
  • iPhone SE
  • iPad Mini 4
  • XBox wireless controller

Software

Libraries

Notes

  • Recent versions of Photoshop have been not so great. Pixelmator development has almost ceased. Considering switching to Affinity Photo.
  • Starting to use Blender for 2D vector graphics as well (icons etc).

Previous installments

2006201220142017

Git activity graph

Number of commits is not a good metric for productivity, project health or anything else really. But let’s pretend! It may be OK as a simple activity indicator…

Here’s a Python script I made that graphs daily commit counts on a git repository by year.

Download the script here. Requires Python 3 and PIL/Pillow.

Usage:

pip3 install pillow
python3 activity.py /path/to/imgui imgui

Output:

imgui-2014.png
imgui-2015.png
imgui-2016.png
imgui-2017.png
imgui-2018.png
imgui-2019.png

Example output for imgui:

Syncthing

The latest annoying update from Dropbox made me look around for an alternative.

iCloud, Google Drive and OneDrive might seem like the obvious candidates but I’d rather avoid them. Why? They are too much like Dropbox. Sooner or later they are going to introduce a similarly annoying UI blunder, crippling limitation or an unfriendly plan.

So I decided to try Syncthing instead. Simple, free, self-hosted, with Linux, Mac and Windows clients. Sounds good…

You deleted 9094 files from Dropbox
Yep. Do I want them back? Nah, I’m good.

From a business standpoint it probably makes sense that Dropbox doesn’t offer a pay-for-what-you-use plan but that is exactly what I want. With Syncthing, it’s no problem.

I set up a simple DigitalOcean VPS as a “master” node that’s always online and adding extra storage space is super easy and cheap.

I’ve been using Syncthing for about two weeks now and it seems to be working really well. It is solid software, complete and functional with no extra fluff. Filesystem changes are picked up and synced quickly.

★★★★★ / ★★★★★

Syncthing web UI
The device management & overview of sync status.

Superforce temporarily removed from App Store

I just got a notice from Apple that Superforce has been removed from the App Store. Oh no!

Luckily, it was not due to violation of any of the intricate compatibility or privacy rules… it just had not been updated for a while. Not fresh enough!

I’m going to prepare a new build, test it on as many recent devices as I can get my hands on and submit it for review. Superforce is coming back to the App Store soon.

The Witness Review

Following the development of the Witness was fun. Jonathan Blow, a very talented game designer, took over 7 years to make it. When the game finally came out, I thought I had seen enough screenshots and read enough blog posts about the process that actually playing it wasn’t necessary.

But one day it went on sale on Steam and I decided to give it a try, just to see if it would run on my iMac at all.

It did. And I got sucked in immediately. From the first minute, it was clear that this was a game for me.

The world of The Witness is vibrant and beautiful. The visual style is perfect. Just the right amount of geometry and texture to make everything clear. Exquisite lighting. I also found it delightful to hear proper footstep sounds for different kinds of surfaces.

The game is designed so that there’s no need for a tutorial, a guide or other hints that would normally be put in a game. In The Witness, those would look out of place. Everything is seamless. No words, no hand holding. You just start playing.

There are several areas on the island, each with a different type of puzzles. The rules are not written anywhere or shown – for each new type of puzzle you need to experiment a little to see what works. Sometimes you get it wrong. The rules you devised worked for the first puzzle or two but not the next one. Go back and think again – same moves, different reasoning.

Generally, the difficulty goes up for each puzzle type as you progress through the area but it is fine to leave and go elsewhere if you get stuck. I was able to finish some areas without much effort, some others I found devilishly difficult. I guess other players could easily have it the other way around because they see and think differently.

Later in the game there are places where you need to combine several puzzle types. This adds more challenge and also acts as a key/lock mechanism to prevent you from going places where you shouldn’t be able to get yet. Find the place to learn what you need and come back later.

Sometimes, after going through a few increasingly difficult puzzles, I’d stumble on one that just made me laugh. A deliberate joke one can only understand knowing the rules and having seen the previous puzzles. This is a nice reminder that the puzzles have been carefuly crafted – not randomly generated.

Running around the island reminded me of orienteering. A person has been there before me, setting up challenges for me to discover and solve.

I was also reminded of OK Go videos. Sometimes it all seems simple and effortless but when you think about it a lot of work has gone into every detail to make it all work. A true sign of a well crafted piece of art.

Excellent game.

I’d love to see a special edition of some sort with developer commentaries in the style of Half-Life 2: walk up to an area, hear a quick clip of Jonathan Blow talking about his intentions and challenges related to that place etc.

★★★★★ / ★★★★★

Trailer