Web Scraping

Swipr - Server

Thursday, December 13, 2018

Recall our arch diagram from the two chapters ago: The Database Service is a collection of methods opening and closing SQLite connections and, while important, is really exceedingly boring and furthermore, unsurprising. As such, we’ll leave it out of the discussion. We’ll also leave out building the Razor Pages and the views, dealing with identity, migrating our initial user tables with EF Core and other such minutiae of running an ASP.NET Core server. …

Swipr - Datastore

Thursday, December 13, 2018

We need a datastore. Primarily so that we can stow a given user’s Tinder information – Facebooked ID, Facebooked password, Facebooked access token, and Tinder access token, – but also because we now have a complicated application it’s state must be preserved somehow. We’re going to skip talking about tables relating to how ASP.NET Core projects do identity because there are a lot of tables for that. The only thing worth pointing out with respect to that is many of our tables will keep a foreign key to the autogenerated AspNetUsers.Id field. …

Swipr - LibSwipr

Thursday, December 13, 2018

As a harness for our model, we’ll create a server that serves as a centralized access point. This server provides the necessary facilities to register an account with it, interact with Tinder, and receive responses from the model. The general architecture for the server will go something like this: SwiprServer is the server, in our case, an ASP.NET Core 2 project, and LibSwipr is our custom tooling around all the other interactions. …

Swipr - Swipr Script Service

Wednesday, December 12, 2018

Now that we have a trained model, we need to consider how to interact with it. The full tech of the script can be found here on GitHub but we’ll walk through it here too. The overview of this step is that we have a python script acting as a local server where the client side of the script receives input and returns output to to external consumers, and the server side loads the PyTorch model and runs the computation. …

Swipr - Fast.ai and CNNs

Tuesday, December 11, 2018

Finally we can consider creating the image classification model. The jupyter notebook that was used for this process can be found here on GitHub. DataLoader The first thing the creation of a PyTorch model requires is the creation of a DataLoader, which is a neat structure for handling the data we want to go over and what their labels should be. It loads x’s (inputs) and maps them to y’s (outputs). …

Swipr - Data Cleaning

Tuesday, December 11, 2018

Back in part 3, we said: Of course, pictures of beaches, food, and dogs are all common Instagram subjects, even on profiles of stereotypically self centered selfie-obsessed young twenties girls. We will come back to this point in detail later on, but for now, let's assume that each profile is pure and ideal for its category. Now it is time to come back to that. A bunch of Instagram photos, no matter from whose profile, are of course not pure and ideal for their category, and pictures of beaches, foods, and boyfriends abound. …

Swipr - Data Collection (Part 2 of 2)

Tuesday, December 11, 2018

Downloading from Instagram There are a number of solutions for scraping Instagram, and it’s mostly a pick-your-poison kind of affair. we picked the first well-supported google result for “Instagram scaper” and came up with Instalooter. It seemed to come with good documentation and sufficient automation facilities. Rate Limits We are told that prior to April 2018, downloading from Instagram was a much more lackadaisical affair. There was allegedly a generous rate limit for a given Instagram login token, upwards of 5,000 page requests per hour. This mythical time must have been the golden era for Instagram scraping. …

Swipr - Data Collection (Part 1 of 2)

Tuesday, December 11, 2018

Data Collection In previous posts we laid out what we expect our end result to look like, but underlying all of that is of course the ML model making our auto-tinder more than just a simple always-swipe-right bot. Like all deep learning projects, before we can consider training our model or even what our model architecture will be, we need to consider the problem of data. What, exactly do we need to collect? …

Swipr - Scope

Tuesday, December 11, 2018

From the outset we wanted to have, at the most basic level, the ability to use a CPU-only machine to judge whether or not we should swipe right on a given photograph. Additionally, it does us no good to just have a model that does this, it needs to be useful in some form beyond merely existing, meaning it has to exist as some kind of accessible application. Finally, it would be preferable to have this system be accessible from anywhere. Binding it to our local machine is useful insofar as this being an academic exercise is concerned, but it’d be a lot nicer to turn it on and off as we see fit, when we see fit. …

Swipr - Overview

Tuesday, November 27, 2018

The Goal Continuing the quest to learn modern Machine Learning, I thought it would be fun to create a Tinder auto-swiper that could potentially reject matches I don’t want and accept matches I do want. One of my friends had the brilliant idea of naming this tinderbot “Swiper”, like the fox from Dora the Explorer, famous for being repelled by the intrepid protagonists with the incantation of “Swiper, no swiping!” …