Xamarin sics army of 1,000 test devices on hapless mobile apps
Cloud-based testing service puts code through virtual wringer
Mobile app-dev specialists Xamarin have launched the Xamarin Test Cloud, an automated, internet-based service that lets developers test their code on more than 1,000 devices.
"With Xamarin Test Cloud, you can create your own test scripts using our powerful testing framework, run them locally against emulators or devices, and then press a button to run the same tests on as many devices as you would like in the cloud," Xamarin CEO Nat Friedman said in a blog post.
The service works with an automation framework called Calabash, which was originally developed by LessPainful, a startup Xamarin acquired in 2013.
Developers write their test scripts in either C# or Ruby, but the service can be used to test any hybrid or native Android or iOS app written in any language.
Although they're automated, the tests can involve practically anything you might do when holding a device in your hand, short of dropping it on the ground. They can simulate tapping and swiping the screen, rotating the device, changing the GPS location, pressing physical buttons, activating the camera, and more.
They can also simulate varying mobile network conditions, from EDGE to 4G and everything in between, including lost connections.
Developers can select devices to test against from a library of over 1,000 models, filtering by such factors as manufacturer, operating system, or form factor.
The service provides detailed reports on the outcomes of each test, including full-resolution screenshots of each step. Developers can access CPU, memory, test duration, and other performance data, in addition to such debugging information as stack traces and device logs.
Because Xamarin wants to see mobile developers test their apps more frequently and thoroughly, the service also works with continuous integration tools such as Jenkins, Team Foundation Server, and TeamCity.
"We believe that testing is not something you should do at the end of the development cycle, just before submitting to the app store," Friedman said. "Instead, your app should be continuously tested during development in an automated, scalable way."
As powerful as Xamarin's cloud-based service is, however, it will cost you. Xamarin Test Cloud starts at $12,000 per year for the Basic version, which lets you test two apps for a total of 200 device-hours per month, with email support only.
On the high end, the Enterprise version of the service costs $144,000 per year, which gets you license to test 20 apps for 2,400 device-hours per month, in addition to a number of gold-plated service and support benefits. The not-quite-as-plush Professional and Business service levels offer varying levels of service and go for $60,000 and $96,000 per year, respectively. ®