TechiesPad

We built three separate crawlers in Python and Scrapy, one per store, so each can be installed, scheduled and maintained on its own when a store changes its layout. They crawl politely, with randomised delays between requests and configurable proxies, so the stores treat the traffic as normal and the client's server is not blocked. Images are downloaded at the right sizes, and each crawler was tested across the stores' categories, because listings that look identical often differ in the details.

We kept the data useful rather than exhaustive. Popular apps carry hundreds of thousands of reviews, so we recommended collecting the most recent ten to fifteen instead, which keeps the database lean without losing anything a visitor would read. Ratings that the stores show only as star graphics were converted into numbers the site can use. The Microsoft and Apple crawlers run on scheduled jobs, and Google Play uses a separate on-demand approach that suited that store better.

On the WordPress side we built a plugin with custom post types and an admin panel to search the collected data and import it, so the setup can be installed on any of the client's sites, plus a custom theme for the home page, app listings and app pages. Each import fills in the client's standard install guide with the app's name, and every page stays fully editable afterwards.

We also advised on how downloads should work. The site does not host the apps, which the client is not licensed to distribute, so each page sends visitors to the app's official store or developer download.