My Security Planet » Tags

14 items tagged "Python"

Mozilla Webdev

Mozilla Webdev At PyCodeConf

Posted: September 26th, 2011, 9:57am CDT by Laura Thomson

Tags: Python [edit]

Next week, a number of Mozilla webdevs will be making a pilgrimage to PyCodeConf, put on by the nice people at github, in Miami. I’ll be speaking on the topic of Processing Firefox Crash Reports with Python. Mozilla is also sponsoring the conference. I’m excited that we get to go, because the other talks look awesome.

This brings me to the next point: If you like Python, and use github, you might fit in here. Interested? If you’re coming to the conference, find a Mozilla person and come talk to us about a job.

Knowing what you want can make a huge difference (…with pymongo)

Posted: August 17th, 2011, 2:17pm CDT by Peter Bengtsson

Tags: Python [edit]

So I’ve got a piece of MongoKit code that simply wraps a pymongo query and it looks like this

        times = defaultdict(list)
        for pq in (db.PlayedQuestion.collection
                   .find({'play.$id': {'$in': play_ids}})):
            if pq['time'] is not None:
                if pq['right']:
                    times['right'].append(pq['time'])
                elif pq['answer']:
                    times['wrong'].append(pq['time'])

What I noticed was that is was quite slow. For example, looping over 12,200 documents to get this data out took a big fat 600 milliseconds on average! Note that it’s not dressing things up as Python model instances as the .collection.find() just returns a dict straight from pymongo.

After realising that I actually only need time, right and answer I just set the fields parameter like this:

        times = defaultdict(list)
        for pq in (db.PlayedQuestion.collection
                   .find({'play.$id': {'$in': play_ids}},
                         fields=('time','right','answer')  # the difference
                         )):
            if pq['time'] is not None:
                if pq['right']:
                    times['right'].append(pq['time'])
                elif pq['answer']:
                    times['wrong'].append(pq['time'])

Now, looping over 12,200 documents only takes 200 milliseconds on average. Now we’re talking!

But wait a minute! If I only care about those with a time that isn’t None (null), can’t I include that in the query? Well, the time field is not indexed but it’s usually a good idea to push work onto the database. Let’s try:

        times = defaultdict(list)
        for pq in (db.PlayedQuestion.collection
                   .find({'play.$id': {'$in': play_ids},
                          'time': {'$ne': None}},
                         fields=('time','right','answer')
                         )):
            if pq['right']:
                times['right'].append(pq['time'])
            elif pq['answer']:
                times['wrong'].append(pq['time'])

As it turns out, only 1,575 of my 12,200 documents have a time attribute that isn’t null. So doing it this way takes only about 39 milliseconds on average.

Let’s repeat the same logic and push even more work into the database query instead. Here’s two queries instead of one but this time there’s less python control flow and all done in two queries:

        for pq in (db.PlayedQuestion.collection
                   .find({'play.$id': {'$in': play_ids},
                          'time': {'$ne': None},
                          'right': True},
                         fields=('time',)
                         )):
            times['right'].append(pq['time'])

        for pq in (db.PlayedQuestion.collection
                   .find({'play.$id': {'$in': play_ids},
                          'time': {'$ne': None},
                          'right': False,
                          'answer': {'$ne': u''}
                          },
                         fields=('time',)
                         )):
            times['wrong'].append(pq['time'])

You’d think this would be a lot faster now since the only field it returns is time which is a float point number and takes up significantly less bytes compared to answer which is a unicode string. However this approach takes 63 milliseconds on average which is worse than before. Having said that, now things are getting a bit weird because I only have 1,575 documents to loop over so a large part of that 63 milliseconds is probably spent just setting up and tearing down the query. Next: map/reduce!

As you might have noticed, this query is now ripe for an aggregate instead of this silly for-loop. So, let’s try a map reduce job on it:

from bson.code import Code
map_ = Code("""
 function() {
   if (this.right)
     emit('right', {total:this.time, count:1});
   if (!this.right && this.answer)
     emit('wrong', {total:this.time, count:1});
 }
""")

reduce_ = Code("""
 function r( who , values ){
   var n = { total : 0 , count : 0 };
   for ( var i=0; i<values.length; i++ ){
     n.total += values[i].total;
     n.count += values[i].count;
   }
   return n;
 }
""")

finalize = Code("""
 function f( who , res ){
     res.avg = res.total / res.count;
     return res;
  }
""")
 ...
        result = db.PlayedQuestion.collection.map_reduce(
          map_, reduce_, 'myoutput',
          finalize=finalize,
          query={'play.$id': {'$in': play_ids},
                 'time': {'$ne': None},
                 }
        )
        times = {}
        for reduction in result.find():
            if reduction['_id'] == 'right':
                times['right'] = reduction['value']['avg']
            elif reduction['_id'] == 'wrong':
                times['wrong'] = reduction['value']['avg']

Unfortunately all that juggling and javascript malarkey isn’t actually that performant. This takes 130 milliseconds on average. Bear in mind that the query reduces the collection down to a small number (1,575) which might mean that a map/reduce job simply isn’t the right tool.

In conclusion

I think doing this optimization dance was a bit premature. The collection size is simply too little. Looping over 1.5k lightweight dicts in python is a breeze. Also, knowing what data you need from a query makes a huge difference (200% in my case).

Also, by the time my database is big enough such that silly loops in the drivers don’t work, MongoDB might have the new promised aggregate functions (estimated for version 2.2 (two releases away)) which will do basic operations such as average() natively without the javascript parsing.

Scaling Django to a Global Audience with Playdoh

Posted: July 26th, 2011, 4:38pm CDT by Fred Wenzel

Enclosure: [download]

Tags: Python [edit]

At Open Source Bridge last month, I gave a talk about Mozilla’s Playdoh project, which tweaks the Django web application framework to reach our global audience at Mozilla.

Here’s a second run of that presentation, recorded at the Mozilla offices (video credit: Rainer Cvillink).

It is a high-level overview of what the playdoh project is and what we do to write scalable web applications, that should be understandable even for nontechnical folks who would like to learn about the project. In the future, we’ll follow up with more technical, detailed posts on certain aspects of the framework.

Download video as MP4, WebM, or Ogg.

Quick summary: In an effort to standardize our web development platform across our various projects, Mozilla created the playdoh project. Playdoh is a web application template consisting of Django, plus additional libraries and documentation. Check out the (work in progress) features list, or the full documentation. The source code is available on github, and if you have any questions, feel free to step by #webdev on the Mozilla IRC network.
A Humane Python Test Runner

Posted: April 14th, 2011, 8:40pm CDT by Erik Rose

Tags: Python [edit]
I have a bit of a dot dependency.

As someone who spends a lot of time running tests on his Python code, my mind now self-administers a minute dopamine hit with every dot emitted by my test runner.
```
..........................................................................
..................
----------------------------------------------------------------------
Ran 93 tests in 30.182s
```
Ah, beautiful dots, each heralding a passed test. But dots have their downsides. For example, I dread early failures:
```
......F...................................................................
...................
======================================================================
FAIL: test_human_path (thing.tests.test_utils.UtilsTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/erose/thing/tests/test_utils.py", line 14, in test_human_path
    assert False
AssertionError
----------------------------------------------------------------------
Ran 93 tests in 30.030s
```
After that F, I have to wait another 25 seconds before I get the skinny on the error. Ugh. Sure, I could ask nose to stop after the first failure (--stop), but then I wouldn’t get the chance to recognize patterns evinced by sets of similar failures.

Another problem: once I fix the failure, how do I re-run just that test? The obvious copy-and-paste suggested by my visual cortex doesn’t work:
```
% nosetests thing.tests.test_utils.UtilsTests
----------------------------------------------------------------------
Ran 0 tests in 0.000s
```
The line we actually need requires manual intervention, changing a period to a colon and reordering a few things:
```
% nosetests thing.tests.test_utils:UtilsTests.test_human_path
```
These and several other productivity-sappers drove me to drink (sparkling apple juice) and consider what I could do about them. After all, I had some free time while tests ran—support.mozilla.com has over 1000 of them, taking almost 4 minutes in all. My answer is nose-progressive, a more human-centric test runner that plugs into the nose testing framework. Its goal? Make the best use of your perceptual apparatus and your interaction time to speed up the debugging process.
A Few Tempting Nuggets
Here are a few of nose-progressive’s features that smooth out the testing workflow.
Quicker, Slimmer Tracebacks
nose-progressive shows its tracebacks immediately; you don’t have to wait for the test run to complete. Its output looks like this:
```
FAIL: kitsune.apps.notifications.tests.test_events:MailTests.test_anonymous
       vi +361 apps/notifications/tests/test_events.py
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/unittest.py", line 279, in run
    testMethod()
  File "/Users/erose/Checkouts/kitsune/../kitsune/apps/notifications/tests/test_events.py", line 361, in test_anonymous
    eq_(1, len(mail.outbox))
  File "/Users/erose/Checkouts/kitsune/vendor/packages/nose/nose/tools.py", line 31, in eq_
    assert a == b, msg or "%r != %r" % (a, b)
AssertionError: 1 != 0

ERROR: kitsune.apps.questions.tests.test_templates:TemplateTestCase.test_woo
        vi +494 apps/questions/tests/test_templates.py
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/unittest.py", line 279, in run
    testMethod()
  File "/Users/erose/Checkouts/kitsune/vendor/packages/mock/mock.py", line 196, in patched
    return func(*args, **keywargs)
  File "/Users/erose/Checkouts/kitsune/../kitsune/apps/questions/tests/test_templates.py", line 494, in test_woo
    attrs_eq(mail.outbox[0], to=['some@bo.dy'],
IndexError: list index out of range
```
It strips 3 unnecessary lines off each traceback to make better use of precious terminal space: two lines worth of dividers, plus the “Traceback (most recent call last)” which is totally superfluous after you’ve been using Python for a few days. The bold does a fine job of visually delimiting tracebacks.
Holy Wombats, a Progress Bar
But where are the dots? My thoughts harkened back to the Zope test runner. It had a lovely percentage-done display that told me exactly how much apple juice I had time to drink. Why couldn’t we have that in nose but without madly scrolling my tracebacks-so-far off the screen? Drumroll, please:
```
thing.tests.test_templates:TaggingTests.test_add_new         [===========/  ]
```
That little bar on the right ticks and spins along merrily, indicating the completeness of the test run, while the path on the left names the currently running test. The whole thing lives down at the bottom of the terminal, while tracebacks scroll gaily upwards above it. If you have a lot of errors, tracebacks that push up above the top of the terminal slide helpfully into your scrollback buffer; I sidestepped curses and did the terminal addressing myself so you won’t lose anything.
Editor Integration
Did you notice those funny lines at the tops of the tracebacks?
```
vi +361 apps/notifications/tests/test_events.py
```
Triple-click, copy, and paste that line, and zoom!—you’re plunked down into your editor of choice right at your failing test. It even applies some heuristics to find the stack frame of your actual test rather than placing you unhelpfully in the middle of a nose helper function like eq_(). And you’re by no means limited to vi; nose-progressive obeys your $EDITOR environment variable. So far, it’s known to work with vi, emacs, and BBEdit.
Re-running Failed Tests
In case you want to re-run a specific failing test as above, nose-progressive gives you just what you need. Copy the pathname from this line…
```
FAIL: kitsune.apps.notifications.tests.test_events:MailTests.test_anonymous
```
…and paste it into your invocation of the test runner:
```
% nosetests kitsune.apps.notifications.tests.test_events:MailTests.test_anonymous
```
nose-progressive also gets along famously with nose’s --failed option in case you want to re-run all failing tests.
Getting It
Trying nose-progressive is a matter of two commands:
```
easy_install nose-progressive
nosetests --with-progressive --logging-clear-handlers
```
For more details, including how to use it with Django, check out nose-progressive’s PyPI page. Happy testing!
Python scoping: understanding LEGB

Posted: January 31st, 2011, 11:17pm CST by Paul Craciunoiu

Tags: Python [edit]
Note: this is a repost from my blog. You can find the original post here.
Update: Thanks to Avazu for a clean suggestion and Fred for pointing out I should have indicated a good way to get around this.
Summary
Python scoping fun! Read about LEGB to understand the basics of python scoping.
Beef
I never bothered to read about how python scoping works until I hit this. It’s not exactly something to research until you have issues with it.

I had something like this going on:
```
def func1(param=None):
    def func2():
        if not param:
            param = 'default'
        print param
    # Just return func2.
    return func2

if __name__ == '__main__':
   func1('test')()
```
Note: Actual code was not as straightforward, func2 was actually a decorator. Admittedly, using the same parameter name is not a must, but it’s still a curiosity. I just wanted to fall back to a default value on run-time.

If you try to run this in python, here’s what you get:
```
~ $ python test.py
Traceback (most recent call last):
  File "test.py", line 11, in 
    func1('test')()
  File "test.py", line 3, in func2
    if not param:
UnboundLocalError: local variable 'param' referenced before assignment
```
If you’re curious, you can read about the principles of LEGB. You have to understand a bit about compilers and the AST to get what’s going on behind the scenes. You might think that replacing lines 3-4 with:
```
param = param or 'default'
```
Might work. But no. You can’t assign the same parameter at the local level if the enclosing level defines it. Even this fails:
```
param = param
```
Fun, no?
What to do?
There are a few ways to get around this.
- Assign param outside of func2. This doesn’t work if you need the default value to be dependent on what params func2 receives.
- Use a second variable, param2 inside of func2 (posted below).
Here is the solution suggested by our commenter Avazu:
```
def func1(param=None):
    def func2(param2=param):
        if not param2:
            param2 = 'default'
        print param2
    # Just return func2.
    return func2
```
Read more
- LEGB (stackoverflow)
- Python docs on scope and namespaces.

Ian Bicking: a blog

Silver Lining: More People!

Posted: April 21st, 2010, 10:20pm CDT by Ian Bicking

Tags: Python [edit]
OK… so I said before Silver Lining is for collaborators not users. And that’s still true… it’s not a polished experience where you can confidently ignore the innards of the tool. But it does stuff, and it works, and you can use it. So… I encourage some more of you to do so.

Now would be a perfectly good time, for instance, to port an application you use to the system. Almost all Python applications should be portable. The requirements are fairly simple:
1. The application needs a WSGI interface.
2. It needs to be Python 2.6 compatible.
3. Any libraries that aren’t pure-Python need to be available as deb packages in some form.
4. Any persistence needs to be provided as a service; if the appropriate service isn’t already available you may need to write some code.
Also PHP applications should work (though you may encounter more rough edges), with these constraints:
1. No .htaccess files, so you have to implement any URL rewriting in PHP (e.g., for WordPress).
2. Source code is not writable, so self-installers that write files won’t work. (Self-installing plugins might be workable, but that hasn’t been worked out yet.)
3. And the same constraints for services.
So… take an application, give it a try, and tell me what you encounter.

Also I’d love to get feedback and ideas from people with more sysadmin background, or who know Ubuntu/Debian tricks. For instance, I’d like to handle some of the questions packages ask about on installation (right now they are all left as defaults, not always the right answer). I imagine there’s some non-interactive way to handle those questions but I haven’t been able to find it.
The Magic Sentinel

Posted: December 3rd, 2008, 10:59pm CST by Ian Bicking

Tags: Python [edit]
In an effort to get back on the blogging saddle, here’s a little note on default values in Python.

In Python there are often default values. The most typical default value is None — None is a object of vague meaning that almost screams "I’m a default". But sometimes None is a valid value, and sometimes you want to detect the case of "no value given" and None can hardly be called no value.

Here’s an example:
```
def getuser(username, default=None):
    if not user_exists(username):
        return default
    ...
```
In this case there is always a default, and so anytime you call getuser() you have to check for a None result. But maybe you have code where you’d really just like to get an exception if the user isn’t found. To get this you can use a sentinel. A sentinel is an object that has no particular meaning except to signal the end (like a NULL byte in a C string), or a special condition (like no default user).

Sometimes people do it like this:
```
_no_default = ()
def getuser(username, default=_no_default):
    if not user_exists(username):
        if default is _no_default:
            raise LookupError("No user with the username %r" % username)
        return default
    ...
```
This works because that zero-item tuple () is a unique object, and since we are using the comparison default is _no_default only that exact object will trigger that LookupError.

Once you understand the pattern, this is easy enough to read. But when you use help() or other automatic generation it is a little confusing, because the default value just appears as (). You could also use object() or [] or anything else, but the automatically generated documentation still won’t look that nice. So for a bit more polish I suggest:
```
class _NoDefault(object):
    def __repr__(self):
        return '(no default)'
NoDefault = _NoDefault()
del _NoDefault

def getuser(username, default=NoDefault):
    ...
```
You might then think "hey, why isn’t there one NoDefault that everyone can share?" If you do share that sentinel you run the risk of accidentally passing in that value even though you didn’t intend to. The value "NoDefault" will become overloaded with meaning, just as None is. By having a more private sentinel object you avoid that. A single nice sentinal factory (like _NoDefault in this example) would be nice, though. Though really PEP 3102 will probably make sentinals like this unnecessary for Python 3.0.

Note that you can also implement arguments with no default via *args and **kwargs, e.g.:
```
def getuser(username, *args):
    if not user_exists(username):
        if not args:
            raise LookupError(...)
        else:
            return args[0]
```
But to do this right you should test that len(args)<=1, raise appropriate errors, maybe consider keyword arguments, and so one. It’s a pain in the butt, and when you’re finished the signature displayed by help() will be wrong anyway.
Decorators and Descriptors

Posted: October 24th, 2008, 1:18am CDT by Ian Bicking

Tags: Python [edit]
So, decorators are neat (maybe check out a new tutorial on them). Descriptors are neat but may seem hard (though they hardly take a long time to describe). Sometimes these two things intersect, and this post describes how.

Here’s an example decorator where this comes up. First, we want something that looks like a WSGI application:
```
def application(environ, start_response):
    start_response('200 OK', [('content-type', 'text/html')])
    return ['hi!']
```
But we want to use WebOb, like:
```
from webob import Request, Response
def application(environ, start_response):
    req = Request(environ)
    resp = Response('hi!')
    return resp(environ, start_response)
```
(We don’t use req in this example, but of course you probably would in a real WSGI application)

Now req = Request(environ) is boilerplate, and it’d be nicer to just do return resp instead of return resp(environ, start_response). So let’s make a decorator to do that:
```
class wsgiapp(object):
    def __init__(self, func):
        self.func = func
    def __call__(self, environ, start_response):
        resp = self.func(Request(environ))
        return resp(environ, start_response)

@wsgiapp
def application(req):
    return Response('hi!')
```
If you don’t understand what happened there, go read up on decorators.
Now, what if you want to decorate a method? For instance:
```
class Application(object):
    def __init__(self, text):
        self.text = text
    @wsgiapp
    def __call__(self, req):
        return Response(self.text)
application = Application('hi!')
```
This won’t quite work, because @wsgiapp will call Application.__call__(req) — with no self argument. This is generally a problem with any decorator that changes the signature, because the signature for methods has this extra self argument. Descriptors can handle this. First we’ll have the same wsgiapp definition as we had before, but we’ll add the magic descriptor method __get__:
```
class wsgiapp(object):
    def __init__(self, func):
        self.func = func
    def __call__(self, environ, start_response):
        resp = self.func(Request(environ))
        return resp(environ, start_response)
    def __get__(self, obj, type=None):
        if obj is None:
            return self
        new_func = self.func.__get__(obj, type)
        return self.__class__(new_func)
```
So, to explain:

When you get an attribute from an instance, like Application().__call__, Python will check if the object that was fetched has a __get__ method. If it does, it will call that method and use the result of that method.

This part:
```
if obj is None:
    return self
```
is what happens when you do Application.__call__ — in other words, when get a class attribute. In that case obj (self) will be None, and it will just return the descriptor (it could do something else, like in this example, but usually it doesn’t).

Functions already have a __get__ method. You can try it yourself:
```
>>> def example(*args):
...     print 'got', args
>>> example_bound = example.__get__(1)
>>> example_bound('test')
got (1, 'test')
```
So in the example with wsgiapp we are just changing the decorator to wrap the new bound function instead of the old unbound function. This allows wsgiapp to be compatible with both plain functions and methods. In fact, it would probably be preferable to always call func.__get__(obj, type) (even if obj is None), as then we could also wrap class methods or other kinds of descriptors.
pyinstall pybundles

Posted: October 1st, 2008, 2:14pm CDT by Ian Bicking

Tags: Python [edit]
Update: pyinstall has been renamed to pip (per the suggestions in the comments to this post)

I added pybundles to pyinstall very shortly before I announced pyinstall. I hadn’t actually tried it out that much. Since then I’ve made three more minor releases of pyinstall, and I think the bundle support is working pretty decently.

A .pybundle file is just a bunch of source code, all the source code you need to install some package(s). For instance, for Deliverance I’ve created a bundle file so you can do:
```
$ easy_install pyinstall virtualenv
$ pyinstall.py -E DeliveranceTest/ 
>   http://deliverance.openplans.org/dist/Deliverance-snapshot-latest.pybundle
```
This creates a virtualenv environment in DeliveranceTest/, unpacks all the source from the bundle, and installs all of it. It’s not magical — it still has to compile the source and move the files around — but it does mean just a single download, and the versions of everything that is installed aren’t going to change unless that bundle file is regenerated.

I’ve been thinking about some other features for pybundles, like post installation scripts. All of this has raised a problem though: pyinstall needs to be a two-level command, with commands like:

pyinstall.py install X
pyinstall.py bundle Y
pyinstall.py freeze req.txt
and of course the not-yet-implemented:
pyinstall.py remove Z

But pyinstall.py install does not read well. It’s not too late to rename the package (yet again), or just rename the script. Ideas?
pyinstall: A New Hope

Posted: September 24th, 2008, 12:21am CDT by Ian Bicking

Tags: Python [edit]

Note: pyinstall has been renamed to pip

Ever been frustrated with easy_install? Interested in an alternative? Then you should go over to my work blog and read pyinstall: A New Hope.
A Doctest Wishlist

Posted: July 31st, 2008, 11:11pm CDT by Ian Bicking

Tags: Python [edit]
Lately I’ve been doing most of my testing with doctest, primarily using stand-alone text files. I generally like it (otherwise I wouldn’t be using it), but it does make me frustrated with doctest sometimes. On my wishlist (roughly in order):
- I wish output was always displayed, even when there’s an exception. I see no reason for the current behavior. Really exceptions could be treated like any other output (if ELLIPSIS was on by default).
- I wish you could turn on options like ELLIPSIS from within a doctest, for all expressions. (# doctest: +ELLIPSIS on every line is beyond ugly.)
- <BLANKLINE> is terribly ugly.
- There’s no way of saying "I don’t care what this prints". You can’t do:
```
>>> some_function()
...
```
  because the … is treated like a continuation.
- Plugging in an alternate output checker is kind of tedious, and can’t be done from within a doctest (without horrible hacks).
- I’d like to be able to easily jump into an interactive state from doctest. Maybe pdb can do this, but I’ve never figured that out exactly.
- Getting nose to run .txt files as doctests is really hard, involving a combination of options I always forget.
- There’s no way to abort the doctest. Sometimes I’d like to run some environment checks early on, and be able to stop the test if they fail.
- I wish it was easier to apply to non-Python code. (I’ve adapted it via subclassing for Logo but I wouldn’t do that often.)
- I wish I could copy and paste from doctests to consoles. But I don’t see any solution to this problem.
- The integration with unittest is pretty hacky. Not that I’ve used unittest in years. But some other test frameworks build off this integration.
- python -m doctest sometest.txt doesn’t do what it should do. Instead it runs doctest’s self-tests.

Making a proxy with WSGI and lxml

Posted: July 30th, 2008, 11:52am CDT by Ian Bicking

Tags: Python [edit]

You can use WSGI to make rewriting middleware; WebOb specifically makes it easy to write. And that’s cool, but it’s more satisfying to use your middleware right away without having to think about writing applications that might live behind the middleware.

There’s two libraries I’ll describe here to make that possible: paste.proxy to send WSGI requests out via HTTP, and lxml.html which lets you rewrite the HTML to fix up the links.

To start, we need some kind of middleware that at least is noticeable. How about something to make a word jumble of the page? We’ll use lxml as well:

from lxml import html
from random import shuffle

def jumble_words(doc):
    """Mixes up the words in an HTML document (doesn't touch tags or attributes)"""
    doc = html.fromstring(doc)
    # .text_content() gives the text without tags or attributes,
    # .body is the <body> tag:
    words = doc.body.text_content().split()
    shuffle(words)
    for el in doc.body.iterdescendants():
        # The ElementTree model puts all text in .text and .tail on elements, so that's
        # what we mix up:
        el.text = random_words(el.text, words)
        el.tail = random_words(el.tail, words)
    return html.tostring(doc)

def random_words(text, words):
    """Pulls some words from the list words, with the same number of words in
    the previous `text`"""
    # text can be None, so we need this test:
    if not text:
        return text
    word_count = len(text.split())
    try:
        return ' '.join(words.pop() for i in range(word_count))
    except IndexError:
        # This shouldn't happen, because we should have exactly
        # the right number of words, but just in case...
        return text

from webob import Request

class JumbleMiddleware(object):
    """Middleware that jumbles the words of HTML responses
    """
    # This __init__ and __call__ are the basic pattern for middleware:
    def __init__(self, app):
        self.app = app
    def __call__(self, environ, start_response):
        req = Request(environ)
        # We don't want 304 Not Modified responses, because we mix up the response
        # differently every time.  So we'll make sure all the headers that could call that
        # (If-Modified-Since, etc) are removed with .remove_conditional_headers():
        req.remove_conditional_headers()
        # This calls the application with the request, and then returns a response; this
        # is the typical pattern for response-modifying middleware using WebOb:
        resp = req.get_response(self.app)
        if resp.content_type == 'text/html':
            resp.body = jumble_words(resp.body)
        return resp(environ, start_response)

Well, you don’t really need to jumble up your own pages, right? Much more fun to jumble other people’s pages. Enter the proxy. Here’s a basic proxy:

from paste.proxy import Proxy
# We use this to make sure we didn't mess up anything with JumbleMiddleware;
# the validator checks for many WSGI requirements:
from wsgiref.validate import validator
import sys

def main():
    proxy_url = sys.argv[1]
    app = JumbleMiddleware(
        Proxy(proxy_url))
    app = validator(app)
    from paste.httpserver import serve
    serve(app, 'localhost', 8080)

if __name__ == '__main__':
    main()

If you look at the full source the command-line is a bit fancier, but it’s all obvious stuff.

OK, so this will work, but the links will often be broken unless the server only gives relative links. But you can rewrite the links using lxml…

import urlparse

class LinkRewriterMiddleware(object):
    """Rewrites the response, assuming the HTML was generated as though based at
    `dest_href`, and needs to be rewritten for the incoming request"""

    # The normal __init__, __call__ pattern:
    def __init__(self, app, dest_href):
        self.app = app
        if dest_href.endswith('/'):
            dest_href = dest_href[:-1]
        self.dest_href = dest_href

    def __call__(self, environ, start_response):
        req = Request(environ)
        # .path_info (aka environ['PATH_INFO']) is the path of the request
        # (URL rewriting doesn't really have to care about query strings)
        dest_path = req.path_info
        dest_href = self.dest_href + dest_path
        # req.application_url is the base URL not including path_info or the query string:
        req_href = req.application_url
        def link_repl_func(link):
            link = urlparse.urljoin(dest_href, link)
            if not link.startswith(dest_href):
                # Not a local link
                return link
            new_url = req_href + '/' + link[len(dest_href):]
            return new_url
        resp = req.get_response(self.app)
        # This decodes any possible gzipped content:
        resp.decode_content()
        if (resp.status_int == 200
            and resp.content_type == 'text/html'):
            doc = html.fromstring(resp.body, base_url=dest_href)
            doc.rewrite_links(link_repl_func)
            resp.body = html.tostring(doc)
        # Redirects need their redirect locations rewritten:
        if resp.location:
            resp.location = link_repl_func(resp.location)
        return resp(environ, start_response)

Then we rewire the application:

app = JumbleMiddleware(
    LinkRewriterMiddleware(Proxy(proxy_url), proxy_url))

Now there’s a fun little proxy for you to play with. You can see the code here.

Governance

Posted: May 5th, 2008, 11:08pm CDT by Ian Bicking

Tags: Python [edit]

It occurred to me… Django is something like a dictatorship… or maybe an oligarchy. At first it seems like Pylons is the same… but no. Pylons is clearly feudal. I lord over Paste, WebOb, FormEncode. Mike Bayer lords over Mako and SQLAlchemy. Ben lords over Routes, Beaker, and Pylons.

I suppose in all cases there is a certain amount of democracy, because there are no serfs, and any individual is free to travel to any kingdom they like. Well, at least among the open source kingdoms. Without citizenship, and with no exclusiveness of ownership, with even property having largely disappeared, I suppose it’s inevitable that traditional metaphors of control and governance don’t really make sense.
PyCon Talks

Posted: March 21st, 2008, 5:11pm CDT by Ian Bicking

Tags: Python [edit]

I gave my talk at PyCon this year on HTML processing in Python. It seemed to be well received, but I did see some comments on the web from people who wanted more technical content. The presentation I ended up giving was really more about HTML and its place and advantages compared with XML and XHTML. Basically I decided to talk about why you want to process HTML, instead of getting too much into how. I talked a little bit about how, but if you had hoped to hear much technical substance then certainly it would have been disappointing. (As to the slide question, I will get them up, but I want to assemble at least a little of the material that motivated it, since slides alone aren’t that useful.)

I’m not sure what to do for talks like these. In 30 minutes it is hard to go into much depth about a subject. And I’m not sure what the purpose would be. If you want to learn to use a tool you should read the documentation, sit down with a computer, and give it a try. You certainly shouldn’t come to a talk. So I’ve tried to avoid the technical details, and instead try to make people want to learn the "how" on their own. This was the goal of my WSGI talk last year as well.

That said, the whole talk format was unsatisfying to me at PyCon. Lightning Talks are great (or at least, can be great — and I thought on Sunday when all the sponsors were gone, they were great). But 30 minute is too much time for just presenting an idea, and too little time (and a poor format) for presenting advanced content. And that slides are built into the format just makes it worse.

I wish the Open Spaces had been more functional at PyCon. I couldn’t find the ones I wanted and felt conflicted between them and talks (in retrospect of course there shouldn’t have been conflict). I didn’t have a single successful Open Space experience this year, despite trying a couple times.

While the organization of Open Spaces could be improved, I don’t think Just In Time Planning is going to work at PyCon. And I don’t think Open Spaces have to be JIT. What makes them most interesting isn’t that they are totally ad hoc, but that they aren’t just one person talking to a bunch of other people. I’d have been very happy to lead some more extensive "talk" about HTML and XML processing in Python, probably starting out with 10-15 minutes of introduction and survey and then going completely to discussion, Q/A, or wherever the attendees wanted to take it. I would expect a smaller group of people (I guess), as it would be a larger time commitment and attendance would feel less casual than just showing up at a normal talk. Maybe it could be clearly setup so that it was something like 10-15 minutes of introduction, 30-35 minutes of talking, and then a clearly planned structure for people to stop talking (moving on to another topic) or stick around to talk more in a smaller and more intimate space and group. Figuring out where to continue the discussion should not be left until the moment everyone has to choose whether to stay or leave. You’d have to be careful not to let it degenerate into people just asking questions of only personal interest. I’m pretty comfortable generalizing people’s overly specific questions. People who are less comfortable doing that might benefit from a moderator to help them guide the discussion.

This format seems like much less work for me as a speaker (and the stress of speaking greatly infringes on my enjoyment of the conference), it seems more likely to fit the interests of the attendees, and I think it can provide benefit for a much wider range of interests and experience levels.

Anyway, an idea. I’m not sure how things should be structured at PyCon next year, but I’m pretty sure the current talk format isn’t it. A few talks, sure: The State Of X talks can work well, as do talks about ideas and experience instead of talks about tools and libraries. But I think one track could be sufficient.

My Security Planet » Tags » Python

Feeds

14 items tagged "Python"

Posted: September 26th, 2011, 9:57am CDT by Laura Thomson

Tags: Python [edit]

Posted: August 17th, 2011, 2:17pm CDT by Peter Bengtsson

Tags: Python [edit]

Posted: July 26th, 2011, 4:38pm CDT by Fred Wenzel

Enclosure: [download]

Tags: Python [edit]

Posted: April 14th, 2011, 8:40pm CDT by Erik Rose

Tags: Python [edit]

Posted: January 31st, 2011, 11:17pm CST by Paul Craciunoiu

Tags: Python [edit]

Posted: April 21st, 2010, 10:20pm CDT by Ian Bicking

Tags: Python [edit]

Posted: December 3rd, 2008, 10:59pm CST by Ian Bicking

Tags: Python [edit]

Posted: October 24th, 2008, 1:18am CDT by Ian Bicking

Tags: Python [edit]

Posted: October 1st, 2008, 2:14pm CDT by Ian Bicking

Tags: Python [edit]

Posted: September 24th, 2008, 12:21am CDT by Ian Bicking

Tags: Python [edit]

Posted: July 31st, 2008, 11:11pm CDT by Ian Bicking

Tags: Python [edit]

Posted: July 30th, 2008, 11:52am CDT by Ian Bicking

Tags: Python [edit]

Posted: May 5th, 2008, 11:08pm CDT by Ian Bicking

Tags: Python [edit]

Posted: March 21st, 2008, 5:11pm CDT by Ian Bicking

Tags: Python [edit]