(Originally posted August 9, 2013)

I was working on a little messaging app for a Django site and I had a little error when performing tests. I had named the app “messages” considering that was the function performed. Well Django also has a built in app named “messages as well. This app allows for messages to be sent to users notifying them of site updates and things like. Not the same functionality as what I was building. So when I wanted to run the tests on my messaging app,

$ python manage.py test messages

Django picked their built in app to run the tests agains, all of which took way longer than my app’s tests should have.

The fix for this is really easy, just make sure that Django’s messaging app is listed below the custom app since Django’s manage.py module stops when it finds a name match. This actually brings up a better point that you should probably just avoid duplicate names in the first place.

(Originally posted August 5, 2013)

I ran across an odd error when I was working on a messaging app. As you can imagine, I had a an app named messages. As usual with south, to create the migrations, we run

$ python manage.py schemamigration --initial messages
$ python manage.py migrate messages

With this, I saw the correct output from south and everything seemed fine… except I kept getting error messages that said that there weren’t tables created for other apps. After redoing the database a couple times, and getting different, but similar errors, I stepped back and looked at the output from syncdb. What I saw was that django.contrib.messages had been integrated into south. South apparently takes the name and matches it to the first match it finds in installed apps. Since this was listed above my messages app, I wasn’t seeing the migrations from my one. And I saw that there was no migration folder in the app like there should be.

After removing django.contrib.messages from installed apps (since I wasn’t using it anyway), the migrations worked and the errors stopped.

(Originally posted July 30, 2013)

I recently ran into a little hitch when working with Django Rest Framework. Since the model that I was working with had to be attached to a user, I needed to have some way to set the member before the serializer was saved, but after validation. The solution turns out to be very similar to what you do in a form.

def post(self, request, format=None):
  serializer = GoalSerializer(data=request.DATA)
  if serializer.is_valid():
    serializer.object.member = self.request.user.get_profile()
    serializer.save()
    return Response(serializer.data, status=status.HTTP_201_CREATED)
  return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)

Very simple and working solution. There should be another way to do this using the pre_save method that comes with subclassing the view with the DRF Create mixin, but I wasn’t able to get that working right now. At the moment, I just wanted something that worked so I can keep developing and moving forward. The other solution is less hack-ish, so when I have time to go back, I’ll update this with the “correct” answer.

(Originally posted July 28, 2013)

Web scraping is fun. I think it has something to do with the fact that you’re seemingly stealing information that’s shouldn’t be stolen. In reality, it’s all public anyway so there isn’t really any issue other than possibly private policy issues and making sure you aren’t accidentally ddos-ing the site. Actually, the private policy thing might be a real issues since people are overly protective of their data when they shouldn’t be but that’s another issue. Either way, it’s fun, and you can make some really cool things with it.

Currently, I’m working on a fantasy golf app for some friends. For this, there are two parts in this that need scraping. The first is getting all the information about the players, including their world ranking, into the database. The second is getting up to date information on the tournament results when there is a tournament happening. Theoretically, the first could be done by hand since it’s just inputting names and only takes place once a quarter according to the rules of the fantasy game, but doing things by hand when you can do them programmatically is a giant waste of time. Because of this, I’m going to show how to use web scraping to update the Django database at the same time. Since later we’re going to be scraping another website with auto run using celery, we’re going to write this in tasks.py.

We’re going to need some new installs for the scraping. The first is requests, which makes getting web pages really simple compared to urllib2 or something like that. The other is beautifulsoup, which is what we’re going to use on the dom. Assuming that your virtualenv is running,

(env)$ pip install requests
(env)$ pip install beautifulsoup4

The initial setup of the scraping function is the following.

import requests
from bs4 import BeautifulSoup

def wr_scrape():
  r = requests.get('http://www.officialworldgolfranking.com/rankings/default.sps')
  soup = BeautifulSoup(r.
1ff8
text)

wr_scrape()

After you have the dom loaded into Beautiful Soup, it sort of turns into an art to get access to the information you’re looking for. I’m by no means an expert, but I’ve done enough to know a few tricks. First, you need to go searching through the dom using something like chrome’s Developer Tools to find a tag that includes everything you want. Hopefully there is a class or id on that which you can use to get access to quickly. Then loop through, or splice out the information from there. In this case, we are looking for the table that has the info for each player, and it looks like it has the title “Click on player names to be taken to their individual tournaments page”. So we’ll use that as the identifier.

tables = soup.find_all('table', title="Click on player names to be taken to their individual tournaments page")

This oddly returns two such tables, only one of which we want. From here we want to loop the rows in the table with the player’s info. I’m going to stop here in describing all the steps I took since I can go on forever with the little details. Like I mentioned earlier, this is an art. There is a bunch of trial and error that goes into getting the info. Practice and you’ll get better. After some finagling, here is the code to get the relevant information. We want the player’s name, their rank, and the unique id that the World Ranking website gives to the player.

import requests
from bs4 import BeautifulSoup
import cgi

def wr_scrape():
 import pdb
 r = requests.get('http://www.officialworldgolfranking.com/rankings/default.sps')
 soup = BeautifulSoup(r.text)
 tables = soup.find_all('table', title="Click on player names to be taken to their individual tournaments page")
 table = tables[1]
 for info in table.contents:
   try:
     data = info.contents[7]
     player_name = data.a.string
     player_url = data.a['href']
     qs = cgi.parse_qs(data.a['href'])
     player_rank = qs['Rank'][0]
     player_ID = qs['/players/bio.sps?ID'][0]
     print '%s, %s, %s' % (player_name, player_rank, player_ID)
   except AttributeError: # incase it's just a string
     pass
   except IndexError: #this is if there is no [7]
     pass

wr_scrape()

We put everything in a try block and use exception catching to filter out the false rows in the table. After we execute the info.contents[7] and data.a[‘href’] lines, anything that doesn’t raise the exception passes through. The print line at the end prints out the info as a check to make sure that we have all the information that we want and that it’s correct.

Now we want to interface with the models. This is the current version of the model.

class Player(models.Model):
  name = models.CharField(max_length=40)
  current_wr = models.IntegerField()
  id_wr = models.IntegerField()
  id_pga = models.IntegerField()

We have three of the four attributes that we want. The other, id_pga, is for scraping the leaderboards, which we’ll deal with later. In order to deal with models and other Django elements in the script, we need to set up the Django settings in the script. To do this, we want to do a few things. The first is to create a directory in the same level as manage.py called bin. Inside there, we want to create a file that runs the python command to run the script as a module, so we can use relative imports. This file is called “update_wr”.

#!/bin/bash
python -m fg.apps.players.tasks

Then we

$ chmod u+x bin/update_wr

so we can run the command from the command line. Then, in the tasks.py file, we need to do a few things. The first is make sure that the settings are configured to allow for Django model use. Since we aren’t calling this from manage.py, we need to do the configuration ourselves at the top of tasks.py.

from django.conf import settings
if not settings.configured:
  from ... import settings
  from django.core.management import setup_environ
  setup_environ(settings)

We check to see if the settings are configured, and if they aren’t, we use relative imports to import them and then setup the environment. I should mention that there is another way to make this run with the Django settings by making it callable from the command line in the same way as we do syncdb or runserver. We also want to move the call of wr_update() at the bottom for the file to

if __name__ == '__main__':
  wr_scrape()

This is because we only want to script to run if we call it specifically. The last step here is to add the part of the script that adds to the database!

try:
  player = Player.objects.get(id_wr=player_ID)
except Player.DoesNotExist:
  player = None
  if player is None:
    player = Player(name=player_name, current_wr=player_rank, id_wr=player_ID, id_pga=-1)
    player.save()
  else:
    player.current_wr = player_rank
    player.save()

This snippet should put right before the print statement (I kept that in so I can see that everything is working fine). If you’ve imported the Player model at the top of the file, but under the settings configuration code, then you should now have populated the first 50 players in the world rankings! The final thing to do is to get more than just the top 50, something like the top 250 would be reasonable for now. Looking at the urls when we ask for 50-100, we can see that it just follows an incrementing paging system. If we just change the page number in the url, we get different values.

base_url = 'http://www.officialworldgolfranking.com/rankings/default.sps?region=world&PageCount='
for i in range(1,6):
  url = base_url + str(i)
  r = requests.get(url)

With put this snippet at the start of the function, and if you make sure that all the lines are indented correctly, then we should have just added players in the world ranking from 1 – 250!

You can see the code in it’s entirety on github here. We still have a little more scraping to do – getting the tournament leaderboards into the database and matching with the players we have. But that’s for next time.

(Originally posted July 22, 2013)

With all the basic functionality finished in the last post, now is the time to add the little features that make the experience better for the users.

The first thing we’re going to do is randomize the id’s for the room. The way we had it before, the primary key in the model was autoincrementing, which means that it would be pretty easy to guess what rooms had already been created. Since we’re not using password authentication, we need some little measure of security to help prevent people from guessing open rooms. The fix here is pretty simple. All we have to do is change the __init__ function for the Room model.

The loop here just obtains a random integer from the python module uuid (which also needs to be imported at the top of the file), and performs a check to see if we’ve already generated that key. Considering there are 10^6 different possibilities, for the id, the odds aren’t very good, but we should perform the check anyway. Fire up the server again and create a new room and check out the new random ids. One thing to note is that we don’t want it to be too large of an int since we have to share it with others, but six seems like a decent size.

Currently, if there are no songs in the unplayed queue, meaning that the player isn’t playing a video, we have no way of getting it to play unless we reload the page. This is because we only start a video on the signaled event that a video has ended. This is a really quick fix in the update function.

The first check if to see if the player has been initialized. The we just check to see if anything is playing and if we have videos cued up, then play it if we do. This check should go at the bottom of the success function in update after we updated the queues.

The next thing we want to do is change the markup for the video title if we’ve played it or not. If you recall, in the update function, when we created the new element, we added an extra class — vid-played — if the song had already been played. Now we get to use that. In a css file, probably the one you’re using for the entire site (or you can just include it in the html file) you can add the following rule.

.vid-played {
color: #AAAAAA;
}

If you put this in a separate file, you also need to make sure that you include the link somewhere, generally in the head where the other css files are. This rule simply turns the text, which in this case is the title of the video gray. You need to reload the page to see the effects but you can skip to the end of the first video to see the effect.

Another issue we want to fix is to let the user know that we’re processing the request for url upload. We want a few things to happen, all of which will take effect before the ajax call, and will be resolved on success. The first is that we want to disable the upload button so we don’t have someone spamming the site until we finished with one. The next is we want to show them some information that we are working on their request. Something like a spinning wheel you sometimes see.

To get an animation of a spinner, do a google search along the lines of processing spinner to find a gif. We then want to add that that as an image next to the submit button in room.html like this.

Since we don’t want the spinner going constantly, we set the visibility to hidden. We then move on to the javascript portion, which will handle the turning on and off of the elements.

This piece of code should be put at before the ajax call in the submit override function. This disables the submit button and shows the spinner. We want to save $(this) in a variable so we can use is in the success function since the form goes out of scope at that point. Then, in the success function below, add

This is simply the reverse of the actions above and set the form back to its normal state. Refresh the page to get the updated js and you should see the animation!

The last thing we want to do is to error check the video upload, and return errors if there are any. First step is adding a span next to the image we just created. This span will hold the errors that we put in there from javascript.

Back in the views, we need to add checking for the two error we can likely expect. A url not from youtube, and a video not found. I’ll include the entire view for reference.

The errors here are passed back to the client with the keyword errors for easy use. The last step is to change the javascript to display the errors. The flow we want is to have an error be presented if there are errors, and have the message be removed on next upload attempt.

This snippet lives in the success function, just below where we changed the effects of the spinner and the upload button. It simply puts the text in that span we created.

This line lives below the effects we perform before the ajax call. It removes the text if there is any. And that’s it. A refresh and you can test out giving bad input to the server.

With all those little changes fixed, all that’s left for the mvp is making it look pretty and creating a welcome page that entices users.

(Originally posted July 22, 2013)

With the basic functionality finished in the last post, we need to address the obvious point in the room… the fact that this has all been all on one url. How are we supposed to share with people we want when everyone is on the same stream? To do this, we want to allow the person to create a “room” where they can share the code and let everyone input together.

We’re going to need to change a lot of things in to get this to work, but most of those changes are just refactoring of the code. But we have to start somewhere, so it might as well be with the models. Since we’re adding support for different rooms, we might as well make a model for a room. Besides linking to the url class, we also need to make a slight change to the url class itself so it knows that there is a link to a room. The resulting models.py will look like the following.

This example was taken right from the SqlAlchemy example pages so you can refer to them if there’s any confusion. Now we want to take on changing of the routes. For the urls, we want to have each room at url.com/room/roomid where roomid is the id assigned to the room. What we want to do is shift the logic that we had at the index route to that url. We also have to update the gathering and processing of the video ids to reflect the new models. The resulting view, which is in views.py if you forgot, should look like the following.

The only things different are the database queries for the objects and the routing at the top. Note how we grab the info from the url in the second and third lines. I included the entire function so everyone can know what’s going on. The view for the update also needs to change to reflect the new models.

This is very similar to the changes above. I should mention that we also need to change the urls in the javascript files, since they need to point to the correct /room/ urls. To do this, just put the room id somewhere in the html as an attribute and use jquery to extract it and concatinate that with the normal url. Here is an example of the new titles on the room pages.

The last thing we have to worry about is the creation of the rooms. We are going to do this at the base url, so users know what to do at the beginning. The html for this view just needs to include a basic form where the user just inputs what they want to name their room. The view for this is also simple.

In the html, all you have to do is make sure that the input for the name has the attribute name=”room-name” so that you can get the value from request.form. Note that we redirect the user to the room when they create it. This first argument in url_for is the name of the function in views.py that we want to redirect to, and then we use keyword arguments after for the room’s id.

With this, we should now have the ability to create separate rooms for people to queue their music! With all the basic functionality done, we now need to clean up the presentation before we can call it a minimum viable product. Next post will be on that subject.

(Originally posted July 22, 2013)

Last time we actually got the video stream going by adding more and better information to the models, getting queues for the videos going, and used the youtube api to load and play all the videos. This time, we’re going to add a search functionality so we don’t have to rely on going to a different tab to find the video, and moving to the site to upload the link. When it’s done, we should have a way to type in a query, see the resulting videos and, when clicked, asynchronously post the url to the queue.

The first step is to look at the youtube search api. As we’ve seen before, youtube has a fantastic api support for not only getting info on a video, but also searching. Using a string like https://gdata.youtube.com/feeds/api/videos?q=avett&max-results=5&v=2&alt=json, we should be able to get the results by changing the attribute q. If we load that url in a browser, we can look at the resulting json and figure out what information we want to display.

With that looking reasonable to work with, we’ll start by adding a search bar, and intercepting the input with jquery so we can have it later to query against. Since we’re going to replace the upload bar with the search, we might as well just hijack that for our purposes. Go ahead and add an id attribute to the input bar, I’m going to use “yt-search”. Jquery has a fantastic autocomplete functionality which we can use to make it dead simple. The final code that we want to add to our javascript file is as follows.

What this does is populate the dropdown menu with the titles of the most relevant videos according to what was typed. We need to add a theme so the ui looks ok, and jquery has a bunch to choose from if you google jquery themes and adding a link to a css file of one of them will make it look better.

As mentioned in the post before, we want to override the submit functionality on the form, and submit the url using ajax so we don’t start over the queue when we add another. This snippet intercepts the form submit, and posts via ajax, and updates it.

If you have the same view function as before, this should be all you need. Now there are some ui things that would make this much better. Something that lets the user know that the request is being processed. Error messages would be nice too, but in terms of functionality, this is what we wanted.

(Originally posted July 22, 2013)

Last time we added long polling functionality so the queue is updated continuously across users. In this post, we’re going to add more information to the models so they look better to the user, and allow us to keep the information better.

The videos we play are identified on youtube by a key, and all of the urls have them, so we want to use the parse that we have from last time to snag the key and store it in the database. We also want to store the title of the video, because we want users to be able to identify what’s next. To do this, it requires a change in the model. After adding the desired fields, models.py should look like

Now we need to change the view where the urls are uploaded to gather and store this additional information. Youtube has a api where we are able to request a bunch of the information about a video, without having to screenscrape the actual website. This url, if given the video’s identifying key, will return json that we can then parse.

If you visit /update, you should see the new json response with the extra data. With this new info, we have to change the javascript to display the titles, instead of the urls, as well as have queues to play the videos. Only after we have played and unplayed lists will we want to think about actually playing the videos as each of the functionalities are separate. The following code should be wrapped in the jquery document ready function.

The extra logic here is checking to see the state of the video. If it’s been played, we add the extra class so we can do some styling. If it hasn’t, we also want to check to see if we’ve put it in the unplayed queue yet, so we don’t keep throwing in values.

With the queues correct, we now need to deal with the youtube api. Luckily, their documentation is very well done and adding this functionality is almost copy / paste. The reference for this is found at https://developers.google.com/youtube/iframe_api_reference. The first step is to add an element where the player will go. Depending on your design, the only important thing is that there is a div with id yt-player, or anything that you want to name it.

In your javascript file, we now want to add the following code, which is from the api website mentioned above.

The only difference from the website is that we added an onStateChange event to the player. This allows us to dynamically set the next video based on the queues we set up from the update function above. Note that all this should also go in the jquery document ready function. Changing onYouTubeIframeAPIReady function to the syntax above allows this to work.

After this, you should be able to load a page and see the queue all pretty with working sync from a different tab. The one issue is that if we want to add another link from the tab, the whole queue starts over. One standard way to work around this is to intercept the submit, and use ajax to post the url. But since we want to add a search functionality, we’ll pass on that, and next time, see how to do the search and use ajax there.

As always, if something doesn’t make sense, shoot me an email with the question / comment and I’ll be more than happy to reply.

(Originally posted July 21, 2013)

Last time we added a database you the app for persistance, and we added very primitive url testing.

Now we want to go ahead and allow for automatic updating of the list of the queue of youtube links. One way to implement this is by using something called web sockets. Looking around, we can see that there are a few python libraries where we might be able to perform this task. That being said, it would be way more complicated than the method we’ll be using and when starting out, there’s no reason to go overly complicated.

The method that we want to use is called long polling. Basically, we use a client side ajax call to get the updated list. This function will then be called at set intervals so that everyone on the page has the same results.

The first thing we’re going to do is write the route that returns a json list of the queue. At the top of the views.py file, we need to add an import to the flask line — jsonify. It should now look like

Jsonify is a function from flask that does the heavy lifting and makes it easier to return json with the correct headers so we don’t have to deal with that. Then we want to add the route to the bottom of the file.

When we loop over the list from the database, we need to cast the results to string since they are of a type defined by sqlalchemy. If we don’t, jsonify will raise a type error since it doesn’t know how to serialize the link. After this, we can go to /update and hopefully see the json returned for all the links that had been uploaded.

With that complete, we now want to go and add the javascript to the client. Before we write the code, we need to add a few links to javascript files in the templates. These links should go at the end of the body tag in layout.html.

The first is a link to jquery file, the javascript library we’re going to use for the ajax call. It’s extremely prevalent and widely used. The second is for the js file that we’re about to write. Open up a file called ytp.js in static/js directory and put the following code inside.

This code waits for the document to load completely, and then, at the interval set by the number (in milliseconds) at the bottom of function, performs the function. The function itself is an ajax call to /update. When the data returns, it clears the queue list, which is now wrapped in a div with class “queue-list”, and for each of the returned links, appends it.

If you fire up the server now, and you should see that nothing changed, but if you keep an eye on the terminal which logs the requests, we can see that every 8 seconds, there is a get request for the updated list. To test it out, you can open another tab at localhost:5000 and input a url in that tab, then quickly switch to the original one and watch as the list is now updated with the new link!

Next time we’re going to start working on using the youtube api to get titles and other info about the links since we want to see information about what is coming next, not just a link.

(Originally posted July 20, 2013)

After having the most basic site, we now want to add a database to the application so we can store the urls that have been uploaded. First we need to do a little refactoring. As it is, our app is just a single file, and that won’t cut it as the code base and functionality grow. The first thing is to move app.py to __init__.py.

That’s fine, but now we’re also going to want to move the actual routes out of __init__.py just to keep things tidy and since we don’t really want app logic in __init__. I’ll include the __init__.py and views.py file in their (relatively small) entirety in a little so there is no confusion.

We also want to make sure that we have room in the directory above to have a script that runs the shell and server.

To be able to run this, you need

This should allow you to run the app now. So the file system now is

youtubeparty/

manage.py

youtubeparty/

__init__.py

templates/

static/

Now we want to add a database to the app. The first step is to download Flask-SQLAlchemy, the most widely used ORM. We’re going to use this because of how large the community is, as well as the fact that it allows connections to many different databases. Make sure your virtualenv is activated, and

Now we want to create a simple models.py in the same directory as the __init__.py which has the app in it. Here is what models.py looks like

Line 2 at the top actually finds __init__.py and imports from there. This part is a little confusing, so I’m going to include the entire __init__.py so there isn’t any confusion.

Views.py looks like

With this setup, you should be able to go to the top level directory, with manage.py, and run

and see the server start successfully. There should be some extra messages about the database around the standard ‘listening on 5000′ message. This means that the database is connected because of that last line in models.py Now that we have the database (hopefully for you guys) set up, we want to test it out. We should be able to type a string in and see it in the same manner as we had at the end of the last post. This time now, if we stop and restart the server, the data persists!

The next thing we want to do is to check to make sure that the string that was inputted is a valid youtube url. To do that, we want to use the python module, urlparse. After performing the check, the route in views.py becomes

This is a very naiive approach for more than a few reasons. It doesn’t check to see if the url actually is valid, and just having the url there doesn’t tell the users much about what’s coming next. Both of these issues however require use of the youtube api, so I’ll wait until a later post to go over that.

Hopefully you’re up to date on all this. Email for any questions at all — I like helping people out. Next up, adding real time support, since having to reload would defeat the purpose.