Mashup

Odds are you’re already familiar, but head to Google Maps anyway at https://www.google.com/maps. Input 42.374490, -71.117185 into the search box up top, and you should find yourself at Harvard. Input 41.3163284, -72.9245318, and you should find yourself at Yale.

Interesting! It seems Google Maps understands GPS coordinates (i.e., latitude and longitude). In fact, search for 28.410, -81.584. Perhaps you’d rather be there? (You might need to zoom out.)

It turns out that Google Maps offers an API that allows you to embed Google’s maps into your own web apps. Hey, that’s one of the ingredients we need! Go ahead and familiarize yourself with Google Maps Javascript API by perusing the three sections below of its Developer’s Guide. Read through any sample code carefully, clicking View example below it, if present, to see the code in action.

Okay, now we need us some news. If you happen to have a Google Account (e.g., Gmail), head to https://news.google.com/ and click the Personalize button at top-right. Below the icon should appear Personalize Google News, down below which is Add Any News Topic. Input, say, Cambridge, Massachusetts or New Haven, Connecticut into that box, then click +. When you refresh your page you should see the new section added to your side bar. Additionally, you can search for location based news by modifying parameters in the URL, like the ones below. For a full list of parameters you can check out this guide.

Not to worry if you don’t have a Google Account. Just head straight to either URL.

Interesting, it looks like our input is now the value of an HTTP parameter, geo, though there’s a bunch of other parameters too. (Know that + is one way a browser can encode a space in a URL. Another way is with %20.) One at a time, delete each of those other key-value pairs plus an ampersand (e.g., first cf=all&, then pz=1&, then &ned=us, then &redirect=true, hitting Enter after each deletion so as to reload the page via a shorter and shorter URL. You should find that Google still returns news for Cambridge or New Haven even when the URL is either of the below?

Nice. Never fear a bit of trial and error! Now try changing the value of geo to, say, 02138 or 06511 and then hit Enter again. You should find yourself at the URL below? The articles might change (since Cambridge and New Haven have more than one postal code each), but the news should still be about Cambridge or New Haven?

Very nice. Though the page you’re looking at, of course, is written in HTML. And all we want, if the staff’s solution is any indication, is a bulleted list of articles' titles and links. How to get those without "scraping" this page’s (surely complicated) HTML? Scroll down to the page’s bottom and look for RSS. Click that link, and you should find yourself at one of the URLs below?

As before, delete any parameters that don’t feel core to the mission at hand, leaving only, say, geo and, now, output. You should find yourself at one of the (still fully functional) URLs below.

Deleting those parameters probably isn’t necessary (and, who knows, their absence might break things eventually), but whittling a URL down to its essence does feel like better design, so let’s stick with simple.

Now, what’s all this markup that’s now on your screen? It looks a bit like HTML, but you’re actually looking at an "RSS feed," a flavor of XML (a tag-based markup language). For quite some time, RSS was all the rage insofar as it enabled websites to "syndicate" articles in a standard format that "RSS readers" could read. RSS isn’t quite as hip anymore these days, but it’s still a terrific find for us because it’s "machine-readable". Because it adheres to a standard format, we can parse it (pretty easily!) with software. Here’s what an RSS feed generally looks like (sans actual data):

In other words, an RSS feed contains a root element called rss, the child of which is an element called channel. Inside of channel are elements called title, description, and link, followed by one or more elements called item, each of which represents an article (or blog post or the like). Each item, meanwhile, contains elements called guid, title, link, description, category, and pubDate. Of course, between most of these start tags and end tags should be actual data (e.g., an article’s actual title). For more details, see https://cyber.law.harvard.edu/rss/rss.html.

Ultimately, we’ll parse RSS feeds from Google News using Python and then return articles' titles and links to our web app via Ajax as JSON. But more on that in a bit.

Recall that jQuery is a popular JavaScript library that "makes things like HTML document traversal and manipulation, event handling, animation, and Ajax much simpler with an easy-to-use API that works across a multitude of browsers." To be fair, though, it’s not without a learning curve. Read through a few sections of jQuery’s documentation if you’d like.

jQuery’s documentation isn’t the most user-friendly, though, so odds are you’ll ultimately find Google and Stack Overflow handier resources.

Recall that $ is usually (though not always) an alias for a global object that’s otherwise called jQuery.

Now take a look at some examples of Twitter’s typeahead.js library, a jQuery "plugin" that adds support for autocompletion to HTML text fields. Play with The Basics, Custom Templates, and Scrollable Dropdown Menu in particular.

http://twitter.github.io/typeahead.js/examples/

And now skim the documentation for a "fork" (i.e., someone else’s version) of that same library:

https://github.com/corejavascript/typeahead.js/blob/master/doc/jquery_typeahead.md

Note that Twitter hasn’t updated their own version of the library for quite some time, so take care to rely on github.com/corejavascript/typeahead.js, not github.com/twitter/typeahead.js.

If you need to find that value later (to copy and paste it again), visit console.developers.google.com and click Credentials at top-left.

Per readme.txt, US.txt is quite like a CSV file except that its fields are delimited with \t (a tab character) instead of a comma. Conveniently, SQLite allows you to import CSV files and, as it turns out, TSV (tab-separated values) files as well. But you first need a table into which to import such a file.

Using phpLiteAdmin or sqlite3, create a table in mashup.db called places that has these twelve fields, in this order:

See readme.txt (or US.txt itself) for clues as to appropriate types for these fields. Don’t include an id field (else you can’t do what we’re about to do!).

Rather than INSERT the rows from US.txt into your newly created table, let’s now import them in bulk as follows:

If you see any errors, odds are your schema for places isn’t quite right, in which case you’ll want to ALTER (or DROP and re-CREATE) it accordingly. To confirm that an import’s successful, execute

to count how many rows are in US.txt. (That command-line argument is a hyphen followed by a lowercase L.) Then execute a query like

in sqlite3 or phpLiteAdmin. The counts should match!

First, toward the top of scripts.js, you’ll see an anonymous function, inside of which is a definition of options, an object, one of whose keys is center, the value of which is an object with two keys of its own, lat, and lng. Per the comment alongside that object, your mashup’s map is currently centered on Stanford, California. (D’oh.) Change the coordinates of your map’s center to Cambridge (42.3770, -71.1256) or New Haven (41.3184, -72.9318) or anywhere else! (Though be sure to choose coordinates in the US if you downloaded US.txt!) Once you save your changes and reload your map, you should find yourself there! Zoom out as needed to confirm visually.

As before, feel free to tinker with the staff’s solution at http://mashup.cs50.net/, inspecting its HTTP requests via Chrome’s Network tab as needed, if unsure how your own code should work!

If you see this message upon running .import in sqlite3, odds are you haven’t run sqlite3 in the same directory as mashup.db. If so, exit sqlite3 with .exit, cd to your mashup directory, and then re-run sqlite3 mashup.db.