Shutting Down a Service with 500MM Requests per Month

May 29, 2015

This week I finished shutting down my app-detection service, iHasApp. I disabled the signup forms, grandfathered the current users on their plans, sent out emails, and updated the landing page with a message to any new visitors. After all of this, I thought it might be interesting to chronicle what it was, how it did, and how it feels to shut down a project after multiple years of work.

How I Ended Up Here

Four years ago, I wrote a blog post about detecting installed iOS apps. In truth, I wrote it simply because I needed something to write about, and I thought it was an interesting workaround. The way iHasApp worked was by leveraging an API that would allow developers to query for the existence of a custom URL handler on the user's device. Therefore, with a large enough list of URL schemes and the apps that they represent, iHasApp could iterate through the list to determine which apps are installed on a user's device.

I eventually open sourced the code and dataset for others to use, and saw the repository's traffic rise. Because the accuracy of iHasApp was dependent on having as big of a list of URL schemes as possible, and over one million apps on the App Store, mining the required data is no simple task--in addition to the data becoming stale as time goes on. To solve this issue, in addition to the impracticality of storing all of the mappings in the application data bundle, I ended up making a paid offering of iHasApp that would match against significantly more URL schemes than I had open sourced, as well as matching executable names from the running processes list--all of this with less network bandwidth and CPU time necessary than the open source library.

The majority of my work up until then was client-side Objective-C, so I spent a lot of time building out and optimizing the service's internals. This was the first project where I would also be handling billing users, so I didn't want to end up in a situation where I under-bill, or even worse, wildly over-bill.

API Usage

The iHasApp service was originally in beta for a very, very long time. As developers inquired about a hosted and curated solution, I would add them to the beta and help them get started. For about a year, usage grew very slowly from one of my own apps and a few dozen indie apps that mostly powered recommendation engines. Then all of a sudden in mid-2014, it seemed like the mobile ad industry hit a sort of critical mass (possibly in response to Facebook's massive mobile advertising success). I began receiving inquiries from multiple mobile ad networks who had already been using the open sourced codebase.

As time went on, the usage climbed to a daily average of approximately 11,500 requests per minute, which came out to just shy of 500 million requests per month in the last few months. Up until now, all of my personal server-side work consisted of simple CRUD services to support persistence and email functionality in a number of my iPhone apps--with this being my first project where all of the logic was server-side with multiple moving pieces--billing (Stripe), metering (self-built with Mongo/Redis), caching (Redis/in-memory), dataset updates (separate service).

All of this usage might make it look like I was rolling in money, but in reality my margins were very small. On one side, from a competitive perspective, I had to compete with my own open source code on speed and accuracy, so I gave sharp discounts to heavy users. On the other side, even with a number of optimizations, the average response size for the detection dataset (one of the two primary API calls) was over 100kB, which at the end of the month meant that I was sending out dozens of terabytes of bandwidth. At a few cents per gigabyte, the bills came in at four digits per month.

The Apple Ban Hammer

Three weeks ago I received one of the first rejections for "using a public API in a manner not prescribed by Apple," and later had a phone call with a member of the App Store Review Team. Essentially any usage of the core API powering iHasApp would be rejected for anything other than providing improved inter-app UI/UX. Since then, many other apps and developers have received similar rejections, with more recent rejections specifically mentioning iHasApp in the rejection notes.

This isn't the first time that I've built something for the App Store that eventually got banned, let alone rejected. The cliché lesson is that you should not be dependent on the whims of someone else's platform. The obvious lesson is that you shouldn't build a service that skirts right on the edge of "Apple definitely won't like this." While Android has an API for fetching a user's installed applications, Apple did not include one, and even removed the ability to access a device's UDID--steps clearly made with the intention of preventing the profiling of users and the leaking of personal information.

In reality, I am more surprised that iHasApp lasted as long as it did--even after Twitter implemented their own version as App Graph. For years I've gotten Google Alerts of people tweeting links to iHasApp in discussion of the dirty tricks and security vulnerabilities that developers abuse to scrape user information.

Going Forward

I am very appreciative to all of the developers who helped me during iHasApp's beta, and to all of the developers who continued to use the iHasApp service and lifted it to such high usage levels. I have emailed all current accounts to notify them that they have been grandfathered in their current plans until 2016 and will no longer be billed. Going forward, the only remaining iHasApp will be the codebase that I open sourced. You can find it on GitHub if you are interested.

If you're a developer who is or was working on something related, feel free to reach out to me at any time to talk about app detection on iOS: [email protected]