Youtube Together 3 — Ajax Updating

(Originally posted July 21, 2013)

Last time we added a database you the app for persistance, and we added very primitive url testing.

Now we want to go ahead and allow for automatic updating of the list of the queue of youtube links. One way to implement this is by using something called web sockets. Looking around, we can see that there are a few python libraries where we might be able to perform this task. That being said, it would be way more complicated than the method we’ll be using and when starting out, there’s no reason to go overly complicated.

The method that we want to use is called long polling. Basically, we use a client side ajax call to get the updated list. This function will then be called at set intervals so that everyone on the page has the same results.

The first thing we’re going to do is write the route that returns a json list of the queue. At the top of the views.py file, we need to add an import to the flask line — jsonify. It should now look like

1
from flask import Flask, render_template, url_for, request, redirect, jsonify

Jsonify is a function from flask that does the heavy lifting and makes it easier to return json with the correct headers so we don’t have to deal with that. Then we want to add the route to the bottom of the file.

1
2
3
4
5
@app.route('/update')
def update():
queue = db.session.query(YTUrl)
    queue_list = [str(q) for in queue.all()]
return jsonify(queue=queue_list)

When we loop over the list from the database, we need to cast the results to string since they are of a type defined by sqlalchemy. If we don’t, jsonify will raise a type error since it doesn’t know how to serialize the link. After this, we can go to /update and hopefully see the json returned for all the links that had been uploaded.

With that complete, we now want to go and add the javascript to the client. Before we write the code, we need to add a few links to javascript files in the templates. These links should go at the end of the body tag in layout.html.

1
2
<script src="http://code.jquery.com/jquery.js"></script>
<script src="static/js/ytp.js"></script>

The first is a link to jquery file, the javascript library we’re going to use for the ajax call. It’s extremely prevalent and widely used. The second is for the js file that we’re about to write. Open up a file called ytp.js in static/js directory and put the following code inside.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
$(document).ready(function() {
(function update() {
    setInterval(function(){
    $.ajax({
        url:'/update',
        type:"get",
        dataType: "json",
        success: function(response) {
            $('.queue-list').children().remove();
            for (q in response['queue']) {
                var item = '<div class="row">' + response['queue'][q] + '</div>';
                $('.queue-list').append(item);
            }
        },
    });
    }, 8000);
})();
});

This code waits for the document to load completely, and then, at the interval set by the number (in milliseconds) at the bottom of function, performs the function. The function itself is an ajax call to /update. When the data returns, it clears the queue list, which is now wrapped in a div with class “queue-list”, and for each of the returned links, appends it.

If you fire up the server now, and you should see that nothing changed, but if you keep an eye on the terminal which logs the requests, we can see that every 8 seconds, there is a get request for the updated list. To test it out, you can open another tab at localhost:5000 and input a url in that tab, then quickly switch to the original one and watch as the list is now updated with the new link!

Next time we’re going to start working on using the youtube api to get titles and other info about the links since we want to see information about what is coming next, not just a link.

Advertisements

Youtube Together 2 — Adding Database

(Originally posted July 20, 2013)

After having the most basic site, we now want to add a database to the application so we can store the urls that have been uploaded. First we need to do a little refactoring. As it is, our app is just a single file, and that won’t cut it as the code base and functionality grow. The first thing is to move app.py to __init__.py.

1
2
cp app.py __init__.py
rm app.py

That’s fine, but now we’re also going to want to move the actual routes out of __init__.py just to keep things tidy and since we don’t really want app logic in __init__. I’ll include the __init__.py and views.py file in their (relatively small) entirety in a little so there is no confusion.

We also want to make sure that we have room in the directory above to have a script that runs the shell and server.

1
2
3
4
5
6
7
from flask.ext.script import Manager, Shell, Server
from youtubeparty import app
manager = Manager(app)
manager.add_command('runserver',Server())
manager.add_command('shell', Shell())
manager.run()

To be able to run this, you need

1
$ pip install Flask-Script

This should allow you to run the app now. So the file system now is

youtubeparty/

manage.py

youtubeparty/

__init__.py

templates/

static/

Now we want to add a database to the app. The first step is to download Flask-SQLAlchemy, the most widely used ORM. We’re going to use this because of how large the community is, as well as the fact that it allows connections to many different databases. Make sure your virtualenv is activated, and

1
$ pip install Flask-SQLAlchemy

Now we want to create a simple models.py in the same directory as the __init__.py which has the app in it. Here is what models.py looks like

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
from youtubeparty import app
from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy(app)
class YTUrl(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    url = db.Column(db.String(80))
    def __init__(self, url):
        self.url = url
    def __repr__(self):
        return self.url
db.create_all()

Line 2 at the top actually finds __init__.py and imports from there. This part is a little confusing, so I’m going to include the entire __init__.py so there isn’t any confusion.

1
2
3
4
5
6
7
8
from flask import Flask
app = Flask(__name__)
from models import *
app.config['SQLALCHEMY_DATABASE_URI'= 'sqlite:///datebase.db'
app.debug = True
import youtubeparty.views

Views.py looks like

1
2
3
4
5
6
7
8
9
10
11
12
from youtubeparty import app
from flask import Flask, render_template, url_for, request, redirect
from models import db, YTUrl
@app.route('/', methods=['GET''POST'])
def index():
    if request.method == 'POST':
         url = YTUrl(request.form['youtube-url'])
         db.session.add(url)
         db.session.commit()
         queue = db.session.query(YTUrl)
    return render_template('index.html', queue=queue)

With this setup, you should be able to go to the top level directory, with manage.py, and run

1
$ python manage.py runserver

and see the server start successfully. There should be some extra messages about the database around the standard ‘listening on 5000′ message. This means that the database is connected because of that last line in models.py Now that we have the database (hopefully for you guys) set up, we want to test it out. We should be able to type a string in and see it in the same manner as we had at the end of the last post. This time now, if we stop and restart the server, the data persists!

The next thing we want to do is to check to make sure that the string that was inputted is a valid youtube url. To do that, we want to use the python module, urlparse. After performing the check, the route in views.py becomes

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
from youtubeparty import app
from flask import Flask, render_template, url_for, request, redirect
from models import db, YTUrl
import urlparse
YOUTUBE_URL = 'www.youtube.com'
@app.route('/', methods=['GET''POST'])
def index():
    if request.method == 'POST':
        poss_url = request.form['youtube-url']
        parsed_url = urlparse.urlparse(poss_url)
        if parsed_url[1== YOUTUBE_URL:
            url = YTUrl(poss_url)
            db.session.add(url)
            db.session.commit()
    queue = db.session.query(YTUrl)
    return render_template('index.html', queue=queue)

This is a very naiive approach for more than a few reasons. It doesn’t check to see if the url actually is valid, and just having the url there doesn’t tell the users much about what’s coming next. Both of these issues however require use of the youtube api, so I’ll wait until a later post to go over that.

Hopefully you’re up to date on all this. Email for any questions at all — I like helping people out. Next up, adding real time support, since having to reload would defeat the purpose.

Youtube Together 1 — Start to Link Upload

(Originally posted July 20, 2013)

Having been working with Django for a while now, I felt like now would be a good time to learn a different python framework — Flask. Flask seems to boil all the things down to a fantastic minimal development experience. It seems perfect for quick apis, or apps. Django is fantastic for more complex apps with its large community and many extensions. Flask may be great for that too, it’s just not something I’ve seen right off the bat.

In order to learn Flask, I’ve decided to solve a mini-problem that I’ve encountered — managing music at a party. From my experience, many of the parties have just one computer hooked up to the sound system, and it’s somewhat of a free for all as to who can play the music and what they can play. And since this computer may or may not have a specific song the person wants to play. As a result, most of the music is played straight from youtube. This leads to having a terrible, or even non existant, queue system. The vision of this project is to have people be able to join a “room” where anyone can search youtube and upload a link to a song they want to hear and have it be put into the queue for playing. This is a rough outline and surely the project is able to change paths, but for the beginning, this seems like a good path.

Like all good python projects, this one starts with firing up a virtualenv.

1
virtualenv --distribute youtube-parties

Then

1
source youtube-parties/bin/activate

to activate the environment. Then you’re going to want to move to another directory and

1
2
3
mkdir youtube-parties
cd youtube-parties
pip install Flask

The last line there installs the framework we’re going to use to our pip environment. Now we want to start the app. Obviously we need to begin with the standard hello world app.

1
2
3
4
5
6
7
8
9
from flask import Flask
app = Flask(__name__)
@app.route("/")
  def hello():
  return "Hello World!"
if __name__ == "__main__":
  app.run()

You should see a message saying that there is a local server running on localhost:5000 and if you navigate there you should see “Hello World!”. Very nice and very pointless.

Now to templating. Create a directory for the templates and change to that directory.

1
2
mkdir templates
cd templates/

The templating system Flask uses is called Jinja2. Looking at the docs  and running through a few examples (the ones here), it seems very similar to Django’s templating system and can easily be picked up by those unfamiliar. Now we want to add the two files to template. The first is layout.html, which will be the base of the whole site, and index.html, which will be the base of the site.

layout.html

1
2
3
4
5
6
7
8
9
10
<!doctype html>
<html lang="en">
  <head>
    <title>Youtube for Parties</title>
  </head>
  <body>
    {% block content %}
    {% endblock %}
  </body>
</html>

index.html

1
2
3
4
5
6
7
{% extends "layout.html" %}
{% block content %}
<h1>Youtube Party!</h1>
{% endblock %}

The gist of is simple where the content block in index get put into the content block in layout. Now change app.py to

1
2
3
4
5
6
7
8
9
from flask import Flask, render_template
app = Flask(__name__)
@app.route('/')
def index():
 return render_template('index.html')
if __name__ == '__main__':
 app.run()

and run the app again, navigate to localhost:5000 and see the new html.

The next step is to allow uploads of urls to the site. Right now, since we just want to get something going for us, we don’t need to worry about malicious text or anything, and we’re even going to skip the database for the moment. After changes, the new app.py should look like

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
from flask import Flask, render_template, url_for, request, redirect
app = Flask(__name__)
dumb_db = []
@app.route('/', methods=['GET''POST'])
def index():
  if request.method == 'POST':
  dumb_db.append(request.form['youtube-url'])
  return render_template('index.html', queue=dumb_db)
@app.route('/upload-url', methods=['POST'])
def upload_url():
  dumb_db.append(request.form['youtube-url'])
  return redirect(url_for('index'))
if __name__ == '__main__':
  app.debug = True
  app.run()

index.html

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
{% extends "layout.html" %}
{% block content %}
<h1>Youtube Party!</h1>
{% for q in queue %}
 <div>
   {{q}}
 </div>
{% endfor %}
<form method="post">
  <input type="text" name="youtube-url">
  <button type="submit">Upload</button>
</form>
{% endblock %}

If you restart the app, you should be able to see a form, and if you type things into the form, and submit, you should see a list of the printed above the form. Some notes about the new code.

1) We need some more imports for app.py.

2) dumb_db is just a list where the values from the submit form are appended. If you restart the app, the values go away and you get a new, blank list.

3) The render_tempate call in index has another variable at the end. This passes the value on to the template for use.

4) The app route decorator for index has a method keyword. This allows for a POST method to go through when the default is only GET request. Note that if you didn’t mention GET along with POST, you wouldn’t be able to process a GET request.

5) We added debug = True in the app module at the bottom. This allows for printouts if there are server errors. Very important if you don’t want to use a whole bunch of pdb.

6) In index.html, we loop the queue in a very pythonic way, printing the item each loop

The last thing we want to do for the time is to make the site look halfway decent. In production, you’d want a webserver (nginx or apache) to do this for you. In development though, all you have to do is create a folder called static in the same folder as app.py, and the development server will serve the files automatically at the uri /static/*. Throughout the tutorial, I’m going to be using the css framework Foundation, made by Zurb. The reason I’m doing this is because I’m getting pretty sick of seeing Bootstrap. Bootstrap is fantastic, but I’m looking for a little change up in scenery. Go to http://foundation.zurb.com/ and click the download button. Unzip the file and move the contents to that newly created static/ folder.

To review, we started with nothing, created a virtualenv, added flask, created a first route, added templates, added support for POSTing urls, and marked up the html into something that looks halfway decent.

Next time, we’ll add a database to the url’s have persistance. We’ll also add some error checking to make sure that the links being added are valid.

Python datetimes and Django’s humanize

(Originally posted July 16, 2013)

One of the important things to know when dealing with deadlines is how long you have left to complete them. The complete, but very annoying to deal with, python datetime module is able to calculate these times. But displaying them on a webpage isn’t exactly pleasing to the eye. For example:

5 days, 10:14:57.121807

is what it would look like if you just printed it out. This is fine if your looking for pure information, but not good at all for anything in production since it takes longer than half a second to figure out, and we don’t want users to have to think very hard about something like this. Luckily, Django has template filters in their “humanize” library that converts unpleasing text like that into something human readable. To do this, all you do is throw a filter after the time and you should get the humanized time instead. If I returned the time difference between the final time and the current time,

1
2
time_difference = end_time - cur_time
return time_difference

I should be able to run that through the filter and get the desired format.

{{time_difference|naturaltime}}

Unfortunately, for the code I was using, this wasn’t working. The filter didn’t have any effect on the time object. Time to dig into Django’s source code. After finding the humanize file, the first thing I noticed was

1
2
if not isinstance(value, date): # datetime is a subclass of date
    return value

This check makes sure that the value being passed in the filter is of type date, or datetime. Since I was returning a difference of datetimes, I was actually passing a timedelta object, which would fail this check and return the original value, which was the error I was getting. Like I said, python datetimes are annoying.

To fix this error, I simply had to make sure that my function returned a datetime. Since a datetime – timedelta returns a datetime, I changed my code to

1
2
time_difference = end_time - cur_time
return cur_time + time_difference

And just like that, the filter was working as expected. Kind of a silly error considering that when you use the naturaltime filter, you’re always going to end up with a timedelta, so you should be able to use one off the bat. Either way, humanizing the times is way better than the clunky normal time display.

Random Primary Keys

(Originally posted July 15, 2013)

When you create an object in Django and save it, Django automatically assigns it a primary key (pk) to the object. This is the

There are two ways to do this in general.

One way to do this is to obfuscate the standard, auto-incrementing integer primary key values that Django’s model ORM provides out of the box. One way to do this method is to use something like AES encrypting that is provided up pycrypto. From there, you can create encrypt and decrypt methods that you wrap when you go to retrieve models from the database. Unfortunately, this method requires you remember to call this function every time you access an object. Hopefully you remember, and that would probably get really annoying.

The other method is to create a primary key yourself on object creation, and then use that throughout the app. To do this, you need to override the model’s save method which, when the object is first saved, will generate a random value to use as the primary key, check to see if there is something with the value previously assigned, and if not, use that as the value’s key.

The code is as follows:

1
2
3
4
5
6
7
8
9
10
11
def save(self*args, **kwargs):
    if not self.pk:
        while True:
            self.pk = int(str(uuid.uuid4().int)[0:6])
            try:
                super(Goal,self).save(*args, **kwargs)
                break
            except IntegrityError:
                continue
    else:
        super(Goal,self).save(*args, **kwargs)

I prefer overriding the save method to generate a random id. It’s a one stop definition where you write the code once and don’t have to think about it again, which is always a good thing. Also, it allows you to get a very large number of objects just by changing the length of the slice you use to generate the key.

Simple Django, Celery, RabbitMQ start

(Originally posted July 12, 2013)

Considering email is a big part in reminding people to do things, we need a way to distribute the sending of emails somewhat asynchronously. To do this, the consensus way in Django is to use Celery with RabbitMQ. After sifting through a bunch of different posts on the subject, I decided to write a little about how I got the initial setup going. I’m using a mac with homebrew installed.

The first thing to do is install and get RabbitMQ going.


brew install rabbitmq

With rabbitMQ installed, we now want to get the rabbit server running.

1
sudo rabbitmq-server -detached

will start the server in the background.

To make sure it’s running, run the command,

1
sudo rabbitmqctl status

You should see a long nested bracket output with information regarding the status of the server. If you see that, you can be sure it is running. To stop the server, never just kill the pid, rather run

1
sudo rabbitmqctl stop

and this will stop the server. Note that it’s important to run all these commands with sudo. Otherwise, you might see a message of an erlang dump meaning there was an error.

Celery

1
$ brew install celery

will install celery! We then want to use pip to install the django connector

1
$ pip install django-cellery

Settings.py

Add ‘djcellery’ to installed apps and run (since you’re hopefully using South),

1
$ python manage.py migrate djcellery

We also need to add a few other things to get the django connection finished. In settings.py, add the following lines

1
2
3
4
5
6
import djcelery
djcelery.setup_loader()
CELERY_IMPORTS = ('<project_name>.tasks',)

where <project_name> is the name of your projects. The CELERY_IMPORTS line can actually be the path to any file that has tasks defined, whether it is a tasks.py file or not.

Tasks.py

Conventionally, it tasks for cellery are in a file called tasks.py which can reside anywhere in the project tree. For this simple starter task, the task will run every minute, and print a simple string to the command line. This takes no extra configuration and is simple to see if it is all working.

1
2
3
4
5
6
from celery.task.schedules import crontab
from celery.decorators import periodic_task
@periodic_task(run_every=crontab())
def project_tasks():
    print 'hello world!'

Wherever you put this, just make sure that settings.py has the correct path so that it can find the definition.

Running celery

The last thing to do is to run celery and see the output!

1
python manage.py celeryd -v 2 -B -s celery -E -l INFO

After letting it run for a minute, you should see, along with some other celery information, hello world! being printed!

Obviously this isn’t that impressive, but now you know you’re able to run python code periodically. Now go do something more impressive than hello world.

Developer, Golf on the Mind writer, Packers, Brewers, Bucks fan