My Security Planet » Tags

24 items tagged "Programming"

Git-as-sync, not source-control-as-deployment

Posted: February 14th, 2012, 2:30am CST by Ian

Tags: Programming [edit]

I don’t like systems that use git push for deployment (Heroku et al). Why? I do a lot of this:

$ git push deploy
... realize I forgot a domain name ...
$ git commit -m "fix domain name" -a ; git push deploy
... realize I didn't do something right with the database setup ...
$ git commit -m "configure database right" -a ; git push deploy
... dammit, I didn't fix it quite right ...
$ git commit -m "typo" -a ; git push deploy

And then maybe I’d actually like to keep my config out of my source control, or have a build process that I run locally, or any number of things. I’d like to be able to test deployment, but every deployment is a commit, and I like to commit tested work. I think I could use git rebase but I lack the discipline to undo my work so I can do it correctly. This is why I don’t do continuous commits.

There’s a whole different level of weirdness when you use GitHub Pages as you aren’t pushing to a deployment-specific remote, you are pushing to a deployment-specific branch.

So I’ve generally thought: git deployment is wrong.

Then I was talking to some other people at Mozilla and they mentioned that ops was using git for simply moving files around even though the stuff they were deploying was itself in Mercurial. They had a particular site with a very large number of files, and it was faster to use git than rsync (git has more metadata than rsync; rsync has to look at everything everytime you sync). And that all seemed very reasonable; git is a fine way to sync things.

But I kind of forgot about it all, and just swore to myself as I did too many trivial commits and wrote too many meaningless commit messages.

Still… it isn’t so hard to separate these concerns, is it? So I wrote up a quite small command called git-sync. The basic idea: copy the working directory to a new location (minus .git/), commit that, and push the result to your deployment remote. You can send modified and untracked files, and you can run a build script before committing and push the result of the build script, all without sullying your "real" source control. And you happen to have a nice history of deployments, which is also nice.

I’ve only used this a little bit, but I’ve enjoyed when I have used it, and it makes me feel much better/clearer about my actual commits. It’s really short right now, and probably gets some things entirely wrong (e.g., moving over untracked files). But it works well enough to be improved (winkwinknudgenudge).

So check it out: [https:]

My Unsolicited Advice For PyPy

Posted: April 4th, 2011, 10:23am CDT by Ian

Tags: Programming [edit]

I think the most interesting work in programming languages right now is about the runtime, not syntax or even the languages themselves. Which places PyPy in an interesting position, as they have put a great deal of effort into abstracting out the concept of runtime from the language they are implementing (Python).

There are of course other runtime environments available to Python. The main environment has and continues to be CPython — the runtime developed in parallel with the language, and with continuous incremental feedback and improvement by the Python developer community. It is the runtime that informs and is informed by the language. It’s also the runtime that is most easy-going about integrating with C libraries, and by extension it is part of the vague but important runtime environment of "Unix". There’s also Jython and IronPython. I frankly find these completely uninteresting. They are runtimes controlled by companies, not communities, and the Python implementations are neither natural parts of their runtime environments, nor do the runtimes include many concessions to make themselves natural for Python.

PyPy is somewhere different. It still has a tremendous challenge because Python was not developed for PyPy. Even small changes to the language seem impossible — something as seemingly innocuous as making builtins static seems to be stuck in a conservative reluctance to change. But unlike Jython and IronPython they aren’t stuck between a rock and a hard place; they just have to deal with the rock, not the hard place.

So here is my unsolicited advice on what PyPy-the-runtime should consider. Simple improvements to performance and the runtime are fine, but being incrementally better than CPython only goes so far, and I personally doubt it will ever make a big impact on Python that way.

PyPy should push hard on concurrency and reliability. If it is fast enough then that’s fine; that’s done as far as I’m concerned. I say this because I’m a web programmer, and speed is uninteresting to me. Certainly opinions will differ. But to me speed (as it’s normally defined) is really really uninteresting. When or if I care about speed I’m probably more drawn to Cython. I do care about latency, memory efficiency, scalability/concurrency, resource efficiency, and most of all worst cases. I don’t think a JIT addresses any of these (and can even make things worse). I don’t know of benchmarks that measure these parameters either.

I want a runtime with new and novel features; something that isn’t just incrementally better than CPython. This itself might seem controversial, as the only point to such novel features would be for people to implement at least some code intended for only PyPy. But if the features are good enough then I’m okay with this — and if I’m not drawn to write something that will only work on PyPy, I probably won’t be drawn to use PyPy at all; natural conservatism and inertia will keep me (and most people) on CPython indefinitely.

What do I want?

Microprocesses. Stackless and greenlets have given us micro-threads, but it’s just not the same. Which is not entirely a criticism — it shows that unportable features are interesting when they are good features. But I want the next step, which is processes that don’t share state. (And implicitly I don’t just want standard async techniques, which use explicit concurrency and shared state.)
Shared objects across processes with copy-on-write; then you can efficiently share objects (like modules!) across concurrent processes without the danger of shared state, but without the overhead of copying everything you want to share. Lack of this is hurting PHP, as you can’t have a rich set of libraries and share-nothing without killing your performance.
I’d rather see a break in compatibility for C extensions to support this new model, than to abandon what could be PyPy’s best feature to support CPython’s C extension ecosystem. Being a web programmer I honestly don’t need many C modules, so maybe I’m biased. But if the rest of the system is good enough then the C extensions will come.
Make sure resource sharing that happens outside of the Python environment is really solid. C libraries are often going to be unfriendly towards microprocesses; make sure what is exposed to the Python environment is solid. That might even mean a dangerous process mode that can handle ctypes and FFI and where you carefully write Python code that has extra powers, so long as there’s a strong wall between that code and "general" code that makes use of those services.
Cython — it’s doing a lot of good stuff, and has a much more conservative but also more predictable path to performance (through things like type annotation). I think it’s worth leaning on. I also have something of a hunch that it could be a good way to do FFI in a safe manner, as Cython already supports multiple targets (Python 2 and 3) from the same codebase. Could PyPy be another target?
Runtime introspection of the runtime. We have great language introspection (probably much to the annoyance of PyPy developers who have to copy this) but currently runtime introspection is poor-to-nonexistant. What processes are running? How much memory is each using? Where? Are they holding on to resources? Are they blocking on some non-Python library? How much CPU have they been using? Then I want to be able to kill processes, send them signals, adjust priorities, etc.

And I guess it doesn’t have to be "PyPy", but a new backend for PyPy to target; it doesn’t have to be the only path PyPy pursues.

With a runtime like this PyPy could be an absolutely rocking platform for web development. Python could be as reliable as, oh… PHP? Sorry, I probably won’t win arguments that way ;) As good as Erlang! Maybe we could get the benefits of async without the pain of callbacks or Deferreds. And these are features people would use. Right now I’m perceiving a problem where there’s lots of people standing on the sidelines cheering you on but not actually using PyPy.

So: I wouldn’t tell anyone what to do, and if someone tries this out I’ll probably only be on the sidelines cheering you on… but I really think this could be awesome.

Update: there’s some interesting comments on Hacker News as well.

Javascript on the server AND the client is not a big deal

Posted: March 30th, 2011, 5:09pm CDT by Ian

Tags: Programming [edit]

All the cool kids love Node.js. I’ve used it a little, and it’s fine; I was able to do what I wanted to do, and it wasn’t particularly painful. It’s fun to use something new, and it’s relatively straight-forward to get started so it’s an emotionally satisfying experience.

There are several reasons you might want to use Node.js, and I’ll ignore many of them, but I want to talk about one in particular:

Javascript on the client and the server!

Is this such a great feature? I think not…

You only need to know one language!

Sure. Yay ignorance! But really, this is fine but unlikely to be relevant to any current potential audience for Node.js. If you are shooting for an very-easy-to-learn client-server programming system, Node.js isn’t it. Maybe Couch or something similar has that potential? But I digress.

It’s not easy to have expertise at multiple languages. But it’s not that hard. It’s considerably harder to have expertise at multiple platforms. Node.js gives you one language across client and server, but not one platform. Node.js programming doesn’t feel like the browser environment. They do adopt many conventions when it’s reasonable, but even then it’s not always the case — in particular because many browser APIs are the awkward product of C++ programmers exposing things to Javascript, and you don’t want to reproduce those same APIs if you don’t have to (and Node.js doesn’t have to!) — an example is the event pattern in Node, which is similar to a browser but less obtuse.

You get to share libraries!

First: the same set of libraries is probably not applicable. If you can do it on the client then you probably don’t have to do it on the server, and vice versa.

But sometimes the same libraries are useful. Can you really share them? Browser libraries are often hard to use elsewhere because they rely on browser APIs. These APIs are frequently impossible to implement in Javascript.

Actually they are possible to implement in Javascript using Proxies (or maybe some other new and not-yet-standard Javascript features). But not in Node.js, which uses V8, and V8 is a pretty conservative implementation of the Javascript language. (Update: it is noted that you can implement proxies — in this case a C++ extension to Node)

Besides these unimplementable APIs, it is also just a different environment. There is the trivial: the window object in the browser has a Node.js equivalent, but it’s not named window. Performance is different — Node has long-running processes, the browser might. Node can have blocking calls, which are useful even if you can’t use them at runtime (e.g., require()); but you can’t really have any of these at any time on the browser. And then of course all the system calls, none of which you can use in the browser.

All these may simply be surmountable challenges, through modularity, mocking, abstractions, and so on… but ultimately I think the motivation is lacking: the domain of changing a live-rendered DOM isn’t the same as producing bytes to put onto a socket.

You can work fluidly across client and server!

If anything I think this is dangerous rather than useful. The client and the server are different places, with different expectations. Any vagueness about that boundary is wrong.

It’s wrong from a security perspective, as the security assumptions are nearly opposite on the two platforms. The client trusts itself, and the server trusts itself, and both should hold the other in suspicion (though the client can be more trusting because the browser doesn’t trust the client code).

But it’s also the wrong way to treat HTTP. HTTP is pretty simple until you try to make it simpler. Efforts to make it simpler mostly make it more complicated. HTTP lets you send serialized data back and forth to a server, with a bunch of metadata and other do-dads. And that’s all neat, but you should always be thinking about sending information. And never sharing information. It’s not a fluid boundary, and code that touches HTTP needs to be explicit about it and not pretend it is equivalent to any other non-network operation.

Certainly you don’t need two implementation languages to keep your mind clear. But it doesn’t hurt.

You can do validation the same way on the client and server!

One of the things people frequently bring up is that you can validate data on the client and server using the same code. And of course, what web developer hasn’t been a little frustrated that they have to implement validation twice?

Validation on the client is primarily a user experience concern, where you focus on bringing attention to problems with a form, and helping the user resolve those problems. You may be able to avoid errors entirely with an input method that avoids the problem (e.g., if a you have a slider for a numeric input, you don’t have to worry about the user inputing a non-numeric value).

Once the form is submitted, if you’ve done thorough client-side validation you can also avoid friendly server-side validation. Of course all your client-side validation could be avoided through a malicious client, but you don’t need to give a friendly error message in that case, you can simply bail out with a simple 400 Bad Request error.

At that point there’s not much in common between these two kinds of validation — the client is all user experience, and the server is all data integrity.

You can do server-side Javascript as a fallback for the client!

Writing for clients without Javascript is becoming increasingly less relevant, and if we aren’t there yet, then we’ll certainly get there soon. It’s only a matter of time, the writing is on the wall. Depending on the project you might have to put in workarounds, but we should keep those concerns out of architecture decisions. Maintaining crazy hacks is not worth it. There’s so many terrible hacks that have turned into frameworks, and frameworks that have justified themselves because of the problems they solved that no longer matter… Node.js deserves better than to be one of those.

In Conclusion Or Whatever

I’m not saying Node.js is bad. There are other arguments for it, and you don’t need to make any argument for it if you just feel like using it. It’s fun to do something new. And I’m as optimistic about Javascript as anyone. But this one argument, I do not think it is very good.

WebTest HTTP testing

Posted: April 2nd, 2010, 5:27pm CDT by Ian Bicking

Tags: Programming [edit]

I’ve yet to see another testing system for local web testing that I like as much as WebTest… which is perhaps personal bias for something I wrote, but then I don’t have that same bias towards everything I’ve written. Many frameworks build in their own testing systems but I don’t like the abstractions — they touch lots of internal things, or skip important steps of the request, or mock out things that don’t need to be mocked out. WSGI can make this testing easy.

There’s also a hidden feature here: because WSGI is basically just describing HTTP, it can be a means of representing not just incoming HTTP requests, but also outgoing HTTP requests. If you are running local tests against your application using WebTest, with just a little tweaking you can turn those tests into HTTP tests (i.e., actually connect to a socket). But doing this is admittedly not obvious; hence this post!

Here’s what a basic WebTest test looks like:

from webtest import TestApp
import json

wsgi_app = acquire_wsgi_application_somehow()
app = TestApp(wsgi_app)

def test_login():
resp = app.post('/login', dict(username='guest', password='guest'))
resp.mustcontain('login successful')
resp = resp.click('home')
resp.mustcontain('<a href="/profile">guest</a>')
# Or with a little framework integration:
assert resp.templatevars.get('username') == 'guest'

# Or an API test:
def test_user_query():
resp = app.get('/users.json')
assert 'guest' in resp.json['userList']
user_info = dict(username='guest2', password='guest2', name='Guest')
resp = app.post('/users.json', content_type='application/json',
body=json.dumps(user_info)
assert resp.json == user_info

The app object is a wrapper around the WSGI application, and each of those methods runs a request and gets the response. The response object is a WebOb response with several additional helpers for testing (things like .click() which finds a link in HTML and follows it, or .json which loads the body as JSON).

You don’t have to be using a WSGI-centric framework like Pylons to use WebTest, it works fine with anything with a WSGI frontend, which is just about everything. But the point of my post: you don’t have to use it with a WSGI application at all. Using WSGIProxy:

import os
import urlparse

if os.environ.get('TEST_REMOTE'):
from wsgiproxy.exactproxy import proxy_exact_request
wsgi_app = proxy_exact_request
parsed = urlparse.urlsplit(os.environ['TEST_REMOTE'])
app = TestApp(proxy_exact_request, extra_environ={
'wsgi.scheme': parsed.scheme,
'HTTP_HOST': parsed.netloc,
'SERVER_NAME': parsed.netloc})
else:
wsgi_app = acquire_wsgi_application_somehow()
app = TestApp(wsgi_app)

It’s a little crude to control this with an environmental variable ($TEST_REMOTE), but it’s an easy way to pass an option in when there’s no better way (and many test runners don’t make options easy). The extra_environ option puts in the host and scheme information into each request (the default host WebTest puts in is http://localhost). WSGIProxy lets you send a request to any host, kind of bypassing DNS, so SERVER_NAME is actually the server the request goes to, while HTTP_HOST is the value of the Host header.

Going over HTTP there are a couple features that won’t work. For instance, you can pass information about your application back to the test code by putting values in environ['paste.testing_variables'] (which is how you’d make resp.templatevars work in the first example). It’s also possible to use extra_environ to pass information into your application, for example to get your application to mock out user authentication; this is fairly safe because in production no request can put those same special keys into the environment (using custom HTTP headers means you must carefully filter requests in production). But custom environ values won’t work over HTTP.

The thing that got me thinking about this is the work I’m doing on Silver Lining, where I am taking apps and rearranging the code and modifying the database configuration ad setup to fit this deployment system. It would be really nice having done that to be able to run some functional tests, and I really want to run them over HTTP. If an application has tests using something like Selenium or Windmill that would also work great, but those tools can be a bit more challenging to work with and applications still need smaller tests anyway, so being able to reuse tests like these would be most useful.

More Sentinals

Posted: April 2nd, 2010, 1:26pm CDT by Ian Bicking

Tags: Programming [edit]

I’ve been casually perusing Paradigms of Artificial Intelligence Programming: Case Studies in Common Lisp. One of the things I am noticing is that Lisp traditionally has a terrible lack of sentinels: special objects denoting some kind of meaning. Specifically in Common Lisp the empty list and false and nil are all the same thing. As a result there’s all these cases where you want to distinguish false from empty, especially when false represents a failure of some sort. In these AI examples, usually a failure to find something, while in many cases the empty list could mean "the thing is already found, no need to look". But there’s also lots of other examples when this causes problems.

More modern languages usually distinguish between these objects. Python for instance has [], False and None. They might all test as "falsish", but if you care to tell the difference it is easy to do; especially common is a test for x is None. Modern Lisps also stopped folding together all these notions (Scheme for example has #f for false as a completely distinct object, though null and the empty list are still the same). XML-RPC is an example of a language missing null… and though JSON is almost the same data model, it is a great deal superior for having null. In comparison no one seems to care much one way or the other about making a strong distinction between True/False and 1/0.

These are all examples of sentinels: special objects that represent some state. None doesn’t mean anything in particular, but it means lots of things specifically. Maybe it means "not found" in one place, or "give me anything I don’t care" in another. But sometimes you need more than one of these in the same place, or None isn’t entirely clear.

One thing I noticed while reading some Perl 6 examples is that they’ve added a number of new sentinels. One is *. So you could write something like item(*) to mean "give me any item, your choice". While the Perl tendency to use punctuation is legend, words work too.

I wonder if we need a few more sentinel conventions? If so what?

Of course any object can become a sentinel if you use it like that, None isn’t more unique than any other object. (None is conveniently available everywhere.)

Any seems useful, ala Perl’s *. But… there’s already an any available everywhere as well. It happens to be a function, but it’s also a unique named object… would it be entirely too weird to do obj is any? And there’s very few cases where the actual function any would be an appropriate input, making it a good sentinel.

The Web Server Benchmarking We Need

Posted: March 16th, 2010, 6:23pm CDT by Ian Bicking

Tags: Programming [edit]

Another WSGI web server benchmark was published. It’s a decent benchmark, despite some criticisms. But it benchmarks what everyone benchmarks: serving up a trivial app really really quickly. This is not very useful to me. Also, performance is not to me the most important differentiation of servers.

In Silver Lining we’re using mod_wsgi. Silver Lining isn’t tied to mod_wsgi (applications can’t really tell), and we may revisit that decision (mostly because of memory concerns), but it is a deliberate choice. mod_wsgi is one of the few multiprocess WSGI servers, and it manages its children (the same way Apache manages all its children). So if a child stops responding, it gets taken out of the pool and killed (brutal efficiency! Or at least brutal terminology). Child processes are also recycled, guarding against memory leaks or other peculiarities. Sometimes these kinds of things are dismissed for covering up bugs, but (a) production is a lousy time to learn about bugs, (b) it’s like a third tier of garbage collection, and (c) the bugs you are avoiding are often bugs you can’t fix anyway (for instance, if your mysql driver leaks memory, is that the application developer’s fault?)

I wish there was competition among servers not to see who can tweak their performance for entirely unrealistic situations, but to see who can implement the most fail-safe server. We’re missing good benchmarks. Unfortunately benchmarks are a pain in the butt to write and manage.

But I hope someone writes a benchmark like that. Here’s some things I’d like to see benchmarked:

A "realistic" CPU-bound application. for i in xrange(10000000): pass is a reasonable start.
An application that generates big responses, e.g., "x"*100000.
An I/O bound application. E.g., one that reads a big file.
A simply slow application (time.sleep(1)).
Applications that wedge. while 1: pass perhaps? Or lock = threading.Lock(); lock.acquire(); lock.acquire(). Wedging in C and wedging in Python are different, so a bunch of different kinds of wedging.
Applications that segfault. ctypes is specially designed for this.
Applications that leak memory like a sieve, e.g., global_var.extend(['x']*10000).
Large uploads.
Slow uploads, like a client that takes 30 seconds to upload 1Mb.
Also slow downloads.
In each case it is interesting what happens when something bad happens to just a portion of requests. E.g., if 1% of requests wedge hard. A good container will serve the other 99% of requests properly. A bad container will have its worker pool exhausted and completely stop.
Mixing and matching these could be interesting. For instance Dave Beazley found some bad GIL results mixing I/O and CPU-bound code.
Add ideas in the comments and I’ll copy them into this list.

The hardest part of writing this is not the applications (they are simple). One annoyance is wiring up the applications, but handily Nicholas covers that well in his benchmark. You also have to make sure to clean up, as many servers will not exit cleanly from some of the tests. Another nuisance is that some of these require funny clients. These aren’t too hard to write, but you can’t just use ab. Then you have to report.

Anyway: I would love it if someone did this, and packaged it as repeatable/runnable code/scripts. I’ll help some, but I can’t lead. I’d both really like to see the results, and in my ideal world people writing servers would start using these benchmarks to make their servers more robust.

What Does A WebOb App Look Like?

Posted: March 12th, 2010, 5:50pm CST by Ian Bicking

Tags: Programming [edit]

Lately I’ve been writing code using WebOb and just a few other small libraries. It’s not entirely obvious what this looks like, so I thought I’d give a simple example.

I make each application a class. Instances of the class are "configured applications". So it looks a little like this (for an application that takes one configuration parameter, file_path):

class Application(object):
def __init__(self, file_path):
self.file_path = file_path

Then the app needs to be a WSGI app, because that’s how I roll. I use webob.dec:

from webob.dec import wsgify
from webob import exc
from webob import Response

class Application(object):
def __init__(self, file_path):
self.file_path = file_path
@wsgify
def __call__(self, req):
return Response('Hi!')

Somewhere separate from the application you actually instantiate Application. You can use Paste Deploy for that, configure it yourself, or just do something ad hoc (a lot of mod_wsgi .wsgi files are like this, basically).

I use webob.exc for things like exc.HTTPNotFound(). You can raise that as an exception, but I mostly just return the object (to the same effect).

Now you have Hello World. I then sometimes do something terrible, I start handling URLs like this:

@wsgify
def __call__(self, req):
if req.path_info == '/':
return self.index(req)
elif req.path_info.startswith('/view/'):
return self.view(req)
return exc.HTTPNotFound()

This is lazy and a very bad idea. So you want a dispatcher. There are several (e.g., selector). I’ll use Routes here… the latest release makes it a bit easier (though it could still be streamlined a bit). Here’s a pattern I think makes sense:

from routes import Mapper

class Application(object):
map = Mapper()
map.connect('index', '/', method='index')
map.connect('view', '/view/{item}', method='view')

def __init__(self, file_path):
self.file_path = file_path

@wsgify
def __call__(self, req):
results = self.map.routematch(environ=req.environ)
if not results:
return exc.HTTPNotFound()
match, route = results
link = URLGenerator(self.map, req.environ)
req.urlvars = ((), match)
kwargs = match.copy()
method = kwargs.pop('method')
req.link = link
return getattr(self, method)(req, **kwargs)

def index(self, req):
...
def view(self, req, item):
...

Another way you might do it is to skip the class, which means skipping a clear place for configuration. I don’t like that, but if you don’t care about that, then it looks like this:

def index(self, req):
...
def view(self, req, item):
...

map = Mapper()
map.connect('index', '/', view=index)
map.connect('view', '/view/{item}', view=view)

@wsgify
def application(req):
results = map.routematch(environ=req.environ)
if not results:
return exc.HTTPNotFound()
match, route = results
link = URLGenerator(map, req.environ)
req.urlvars = ((), match)
kwargs = match.copy()
view = kwargs.pop('view')
req.link = link
return view(req, **kwargs)

Then application is pretty much boilerplate. You could put configuration in the request if you wanted, or use some other technique (like Contextual).

I talked some with Ben Bangert about what he’s trying with these patterns, and he’s doing something reminiscent of Pylons controllers (but without the rest of Pylons) and it looks more like this (with my own adaptations):

class BaseController(object):
special_vars = ['controller', 'action']

def __init__(self, request, link, **config):
self.request = request
self.link = link
for name, value in config.items():
setattr(self, name, value)

def __call__(self):
action = self.request.urlvars.get('action', 'index')
if hasattr(self, '__before__'):
self.__before__()
kwargs = req.urlsvars.copy()
for attr in self.special_vars
if attr in kwargs:
del kwargs[attr]
return getattr(self, action)(**kwargs)

class Index(BaseController):
def index(self):
...
def view(self, item):
...

class Application(object):
map = Mapper()
map.connect('index', '/', controller=Index)
map.connect('view', '/view/{item}', controller=Index, action='view')

def __init__(self, **config):
self.config = config

@wsgify
def __call__(self, req):
results = self.map.routematch(environ=req.environ)
if not results:
return exc.HTTPNotFound()
match, route = results
link = URLGenerator(self.map, req.environ)
req.urlvars = ((), match)
controller = match['controller'](req, link, **self.config)
return controller()

That’s a lot of code blocks, but they all really say the same thing ;) I think writing apps with almost-no-framework like this is pretty doable, so if you have something small you should give it a go. I think it’s especially appropriate for applications that are an API (not a "web site").

Configuration management: push vs. pull

Posted: March 10th, 2010, 7:45pm CST by Ian Bicking

Tags: Programming [edit]

Since I’ve been thinking about deployment I’ve been thinking a lot more about what "configuration management" means, how it should work, what it should do.

I guess my quick summary of configuration management is that it is setting up a server correctly. "Correct" is an ambiguous term, but given that there are so many to configuration management the solutions are also ambiguous.

Silver Lining includes configuration management of a sort. It is very simple. Right now it is simply a bunch of files to rsync over, and one shell script (you can see the files here and the script here — at least until I move them and those links start 404ing). Also each "service" (e.g., a database) has a simple setup script. I’m sure this system will become more complicated over time, but it’s really simple right now, and I like that.

The other system I’ve been asked about the most about lately is Puppet. Puppet is a real configuration management system. The driving forces are very different: I’m just trying to get a system set up that is in all ways acceptable for web application deployment. I want one system set up for one kind of task; I am completely focused on that end, and I care about means only insofar as I don’t want to get distracted by those means. Puppet is for people who care about the means, not just the ends. People who want things to work in a particular way; I only care that they work.

That’s the big difference between Puppet and Silver Lining. The smaller difference (that I want to talk about) is "push" vs. "pull". Grig wrote up some notes on two approaches. Silver Lining uses a "push" system (though calling it a "system" is kind of overselling what it does) while Puppet is "pull". Basically Silver Lining runs these commands (from your personal computer):

$ rsync -r <silverlining>/server-files/serverroot/ root@server:/
$ ssh root@server "$(cat <silverlining>/server-files/update-server-script.sh)"

This is what happens when you run silver setup-node server: it pushes a bunch of files over to the server, and runs a shell script. If you update either of the files or the shell script you run silver setup-node again to update the server. This is "push" because everything is initiated by the "master" (in this case, the developer’s personal computer).

Puppet uses a pull model. In this model there is a daemon running on every machine, and these machines call in to the master to see if there’s any new instructions for them. If there are, the daemon applies those instructions to the machine it is running on.

Grig identifies two big advantages to this pull model:

When a new server comes up it can get instructions from the master and start doing things. You can’t push instructions to a server that isn’t there, and the server itself is most aware of when it is ready to do stuff.
If a lot of servers come up, they can all do the setup work on their own, they only have to ask the master what to do.

But… I don’t buy either justification.

First: servers don’t just do things when they start up. To get this to work you have to create custom images with Puppet installed, and configured to know where the master is, and either the image or the master needs some indication of what kind of server you intended to create. All this is to avoid polling a server to see when it comes online. Polling a server is lame (and is the best Silver Lining can do right now), but avoiding polling can be done with something a lot simpler than a complete change from push to pull.

Second: there’s nothing unscalable about push. Look at those commands: one rsync and one ssh. The first is pretty darn cheap, and the second is cheap on the master and expensive on the remote machine (since it is doing things like installing stuff). You need to do it on lots of machines? Then fork a bunch of processes to run those two commands. This is not complicated stuff.

It is possible to write a push system that is hard to scale, if the master is doing lots of work. But just don’t do that. Upload your setup code to the remote server/slave and run it there. Problem fixed!

What are the advantages of push?

Easy to bootstrap. A bare server can be setup with push, no customization needed. Any customization is another kind of configuration, and configuration should be automated, and… well, this is why it’s a bootstrap problem.
Errors are synchronous: if your setup code doesn’t work, your push system will get the error back, you don’t need some fancy monitor and you don’t need to check any logs. Weird behavior is also synchronous; can’t tell why servers are doing something? Run the commands and watch the output.
Development is sensible: if you have a change to your setup scripts, you can try it out from your machine. You don’t need to do anything exceptional, your machine doesn’t have to accept connections from the slave, you don’t need special instructions to keep the slave from setting itself up as a production machine, there’s no daemon that might need modifications… none of that. You change the code, you run it, it works.
It’s just so damn simple. If you don’t start thinking about push and pull and other design choices, it simply becomes: do the obvious and easy thing.

In conclusion: push is sensible, pull is needless complexity.

Why toppcloud will not be agnostic

Posted: February 10th, 2010, 12:41am CST by Ian Bicking

Tags: Programming [edit]

I haven’t received a great deal of specific feedback on toppcloud (update: renamed Silver Lining), only a few people (Ben Bangert, Jorge Vargas) seem to have really dived deeply into it. But — and this is not unexpected — I have already gotten several requests about making it more agnostic with respect to… stuff. Maybe that it not (at least forever) require Ubuntu. Or maybe that it should support different process models (e.g., threaded and multiple processes). Or other versions of Python.

The more I think about it, and the more I work with the tool, the more confident I am that toppcloud should not be agnostic on these issues. This is not so much about an "opinionated" tool; toppcloud is not actually very opinionated. It’s about a well-understood system.

For instance, Ben noticed a problem recently with weird import errors. I don’t know quite why mod_wsgi has this particular problem (when other WSGI servers that I’ve used haven’t), but the fix isn’t that hard. So Ben committed a fix and the problem went away.

Personally I think this is a bug with mod_wsgi. Maybe it’s also a Python bug. But it doesn’t really matter. When a bug exists it "belongs" to everyone who encounters it.

toppcloud is not intended to be a transparent system. When it’s working correctly, you should be able to ignore most of the system and concentrate on the relatively simple abstractions given to your application. So if the configuration reveals this particular bug in Python/mod_wsgi, then the bug is essentially a toppcloud bug, and toppcloud should (and can) fix it.

A more flexible system can ignore such problems as being "somewhere else" in the system. Or, if you don’t define these problems as someone else’s problem, then a more flexible system is essentially always broken somewhere; there is always some untested combination, some new component, or some old component that might get pushed into the mix. Fixes for one person’s problem may introduce a new problem in someone else’s stack. Some fixes aren’t even clear. toppcloud has Varnish in place, so it’s quite clear where a fix related to Django and Varnish configuration goes. If these were each components developed by different people at different times (like with buildout recipes) then fixing something like this could get complicated.

So I feel very resolved: toppcloud will hardcode everything it possibly can. Python 2.6 and only 2.6! (Until 2.7, but then only 2.7!). Only Varnish/Apache/mod_wsgi. I haven’t figured out threads/processes exactly, but once I do, there will be only one way! And if I get it wrong, then everyone (everyone) will have to switch when it is corrected! Because I’d much rather have a system that is inflexible than one that doesn’t work. With a clear and solid design I think it is feasible to get this to work, and that is no small feat.

Relatedly, I think I’m going to change the name of toppcloud, so ideas are welcome!

toppcloud and Django

Posted: February 5th, 2010, 2:50am CST by Ian Bicking

Tags: Programming [edit]

I wrote up instructions on using toppcloud (update: renamed Silver Lining) with Django. They are up on the site (where they will be updated in the future), but I’ll drop them here too…

Creating a Layout

First thing you have to do (after installing toppcloud of course) is create an environment for your new application. Do that like:

$ toppcloud init sampleapp

This creates a directory sampleapp/ with a basic layout. The first thing we’ll do is set up version control for our project. For the sake of documentation, imagine you go to bitbucket and create two new repositories, one called sampleapp and another called sampleapp-lib (and for the examples we’ll use the username USER).

We’ll go into our new environment and use these:

$ cd sampleapp
$ hg clone http://bitbucket.org/USER/sampleapp src/sampleapp
$ rm -r lib/python/
$ hg clone http://bitbucket.org/USER/sampleapp-lib lib/python
$ mkdir lib/python/bin/
$ echo "syntax: glob
bin/python*
bin/activate
bin/activate_this.py
bin/pip
bin/easy_install*
" > lib/python/.hgignore
$ mv bin/* lib/python/bin/
$ rmdir bin/
$ ln -s lib/python/bin bin

Now there is a basic layout setup, with all your libraries going into the sampleapp-lib repository, and your main application in the sampleapp repository.

Next we’ll install Django:

$ source bin/activate
$ pip install Django

Then we’ll set up a standard Django site:

$ cd src/sampleapp
$ django-admin.py sampleapp

Also we’d like to be able to import this file. It’d be nice if there was a setup.py file, and we could run pip -e src/sampleapp, but django-admin.py doesn’t create that itself. Instead we’ll get that on the import path more manually with a .pth file:

$ echo "../../src/sampleapp" > lib/python/sampleapp.pth

Also there’s the tricky $DJANGO_SETTINGS_MODULE that you might have had problems with before. We’ll use the file lib/python/toppcustomize.py (which is imported everytime Python is started) to make sure that is always set:

$ echo "import os
os.environ['DJANGO_SETTINGS_MODULE'] = 'sampleapp.settings'
" > lib/python/toppcustomize.py

Also we have a file src/sampleapp/sampleapp/manage.py, and that file doesn’t work quite how we’d like. Instead we’ll put a file into bin/manage.py that does the same thing:

$ rm sampleapp/manage.py
$ cd ../..
$ echo '#!/usr/bin/env python
from django.core.management import execute_manager
from sampleapp import settings
if __name__ == "__main__":
    execute_manager(settings)
' > bin/manage.py
$ chmod +x bin/manage.py

Now, if you were just using plain Django you’d do something like run python manage.py runserver. But we’ll be using toppcloud serve instead, which means we have to set up the two other files toppcloud needs: app.ini and the runner. Here’s a simple app.ini:

$ echo '[production]
app_name = sampleapp
version = 1
runner = src/sampleapp/toppcloud-runner.py
' > src/sampleapp/toppcloud-app.ini
$ rm app.ini
$ ln -s src/sampleapp/toppcloud-app.ini app.ini

The file must be in the "root" of your application, and named app.ini, but it’s good to keep it in version control, so we set it up with a symlink.

It also refers to a "runner", which is the Python file that loads up the WSGI application. This looks about the same for any Django application, and we’ll put it in src/sampleapp/toppcloud-runner.py:

$ echo 'import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()
' > src/sampleapp/toppcloud-runner.py

Now if you want to run the application, you can:

$ toppcloud serve .

This will load it up on http://localhost:8080, and serve up a boring page. To do something interesting we’ll want to use a database.

Setting Up A Database

At the moment the only good database to use is PostgreSQL with the PostGIS extensions. Add this line to app.ini:

service.postgis =

This makes the database "available" to the application. For development you still have to set it up yourself. You should create a database sampleapp on your computer.

Next, we’ll need to change settings.py to use the new database configuration. Here’s the lines that you’ll see:

DATABASE_ENGINE = ''           # 'postgresql_psycopg2', 'postgresql', 'mysql', 'sqlite3' or 'oracle'.
DATABASE_NAME = ''             # Or path to database file if using sqlite3.
DATABASE_USER = ''             # Not used with sqlite3.
DATABASE_PASSWORD = ''         # Not used with sqlite3.
DATABASE_HOST = ''             # Set to empty string for localhost. Not used with sqlite3.
DATABASE_PORT = ''             # Set to empty string for default. Not used with sqlite3.

First add this to the top of the file:

import os

Then you’ll change those lines to:

DATABASE_ENGINE = 'postgresql_psycopg2'
DATABASE_NAME = os.environ['CONFIG_PG_DBNAME']
DATABASE_USER = os.environ['CONFIG_PG_USER']
DATABASE_PASSWORD = os.environ['CONFIG_PG_PASSWORD']
DATABASE_HOST = os.environ['CONFIG_PG_HOST']
DATABASE_PORT = ''

Now we can create all the default tables:

$ manage.py syncdb
Creating table auth_permission
Creating table auth_group
Creating table auth_user
Creating table auth_message
Creating table django_content_type
Creating table django_session
Creating table django_site
...

Now we have an empty project that doesn’t do anything. Let’s make it do a little something (this is all really based on the Django tutorial).

$ manage.py startapp polls

Django magically knows to put the code in src/sampleapp/sampleapp/polls/ — we’ll setup the model in src/sampleapp/sampleapp/polls/models.py:

from django.db import models

class Poll(models.Model):
    question = models.CharField(max_length=200)
    pub_date = models.DateTimeField('date published')

class Choice(models.Model):
    poll = models.ForeignKey(Poll)
    choice = models.CharField(max_length=200)
    votes = models.IntegerField()

And activate the application by adding ’sampleapp.polls’ to INSTALLED_APPS in src/sampleapp/sampleapp/settings.py. Also add ‘django.contrib.admin’ to get the admin app in place. Run manage.py syncdb to get the tables in place.

You can try toppcloud serve . and go to /admin/ to login and see your tables. You might notice all the CSS is broken.

toppcloud serves static files out of the static/ directory. You don’t actually put static in the URLs, these files are available at the top-level (unless you create a static/static/ directory). The best way to put files in there is generally symbolic links.

For Django admin, do this:

$ cd static
$ ln -s ../lib/python/django/contrib/admin/media admin-media

Now edit src/sampleapp/sampleapp/settings.py and change ADMIN_MEDIA_PREFIX to ‘/admin-media’.

(Probably some other links should be added.)

One last little thing you might want to do; replace this line in
settings:

SECRET_KEY = 'ASF#@$@#JFAS#@'

With this:

from tcsupport.secret import get_secret
SECRET_KEY = get_secret()

Then you don’t have to worry about checking a secret into version control.

You still don’t really have an application, but the rest is mere "programming" so have at it!

WebOb decorator

Posted: May 22nd, 2009, 9:35pm CDT by Ian Bicking

Tags: Programming [edit]

Lately I’ve been writing a few applications (e.g., PickyWiki and a revisiting a request-tracking application VaingloriousEye), and I usually use no framework at all. Pylons would be a natural choice, but given that I am comfortable with all the components, I find myself inclined to assemble the pieces myself.

In the process I keep writing bits of code to make WSGI applications from simple WebOb -based request/response cycles. The simplest form looks like this:

from webob import Request, Response, exc

def wsgiwrap(func):
    def wsgi_app(environ, start_response):
        req = Request(environ)
        try:
            resp = func(req)
        except exc.HTTPException, e:
            resp = e
        return resp(environ, start_response)
    return wsgi_app

@wsgiwrap
def hello_world(req):
    return Response('Hi %s!' % (req.POST.get('name', 'You')))

But each time I’d write it, I change things slightly, implementing more or less features. For instance, handling methods, or coercing other responses, or handling middleware.

Having implemented several of these (and reading other people’s implementations) I decided I wanted WebOb to include a kind of reference implementation. But I don’t like to include anything in WebOb unless I’m sure I can get it right, so I’d really like feedback. (There’s been some less than positive feedback, but I trudge on.)

My implementation is in a WebOb branch, primarily in webob.dec (along with some doctests).

The most prominent way this is different from the example I gave is that it doesn’t change the function signature, instead it adds an attribute .wsgi_app which is WSGI application associated with the function. My goal with this is that the decorator isn’t intrusive. Here’s the case where I’ve been bothered:

class MyClass(object):
    @wsgiwrap
    def form(self, req):
        return Response(form_html...)

    @wsgiwrap
    def form_post(self, req):
        handle submission

OK, that’s fine, then I add validation:

@wsgiwrap
def form_post(self, req):
    if req not valid:
        return self.form
    handle submission

This still works, because the decorator allows you to return any WSGI application, not just a WebOb Response object. But that’s not helpful, because I need errors…

@wsgiwrap
def form_post(self, req):
    if req not valid:
        return self.form(req, errors)
    handle submission

That is, I want to have an option argument to the form method that passes in errors. But I can’t do this with the traditional wsgiwrap decorator, instead I have to refactor the code to have a third method that both form and form_post use. Of course, there’s more than one way to address this issue, but this is the technique I like.

The one other notable feature is that you can also make middleware:

@wsgify.middleware
def cap_middleware(req, app):
    resp = app(req)
    resp.body = resp.body.upper()
    return resp

capped_app = cap_middleware(some_wsgi_app)

Otherwise, for some reason I’ve found myself putting an inordinate amount of time into __repr__. Why I’ve done this I cannot say.

Treating configuration values as templates

Posted: April 20th, 2009, 1:11am CDT by Ian Bicking

Tags: Programming [edit]

A while back I described fassembler and one of the things I liked in it is how the configuration works. It uses a conventional declarative INI-style but also allows arbitrary code, so that defaults can be based on each other.

Here’s a basic example of a default configuration:

[some_app]
port_offset = 10
port = {{int(section.DEFAULT['base_port'])+int(port_offset)}}

Then if another configuration file defines base_port then this will all resolve. You can do this in Python, but you don’t get sections, and you have to define everything in just the right order. So while base_port will probably be defined in a deployment-specific configuration, it has to be defined before these other derivative settings are defined. On the other hand, you want deployment-specific configuration to take precedence… so there’s really no good ordering.

Anyway, the implementation really isn’t that hard. I use Tempita as the templating language because, well, I wrote it, and because it’s simple and appropriate for small strings. For the configuration parsing, ConfigParser will do.

Here’s what the basic code looks like in ConfigParser:

from ConfigParser import ConfigParser
from tempita import Template

class TempitaConfigParser(ConfigParser):

    def _interpolate(self, section, option, rawval, vars):
        ns = _Namespace(self, section, vars)
        tmpl = Template(rawval, name='%s.%s' % (section, option))
        value = tmpl.substitute(ns)
        return value

Actually instead of using tempita.Template, we could just do eval(rawval, {}, ns), it would just require a lot more quoting (every value would have to be a valid Python expression). Either with that or Tempita the implementation of _Namespace will look the same.

Here’s a simple implementation:

from UserDict import DictMixin

class _Namespace(DictMixin):
    def __init__(self, config, section, vars):
        self.config = config
        self.section = section
        self.vars = vars

    def __getitem__(self, key):
        if key == 'section':
            return _Section(self)
        if self.config.has_option(self.section, key):
            return self.config.get(self.section, key)
        if vars and key in self.vars:
            return self.vars[key]
        raise KeyError(key)

   def __setitem__(self, key, value):
       if self.vars is None:
           self.vars = {key: value}
       else:
           self.vars[key] = value

We’ve introduced a magic variable section, which is used to refer to other sections. It looks like this:

class _Section(object):
    def __init__(self, namespace):
        self._namespace = namespace

    def __getattr__(self, attr):
        if attr.startswith('_'):
            raise AttributeError(attr)
        return _Namespace(self._namespace.config, attr,     self._namespace.vars)

With these I think you get many of the benefits of using Python code as your configuration format, while still having the benefits of a more declarative approach to configuration, one that allows for forward and backward references.

A full implementation has several more things than I show here, but you can see the full example in my recipes. It also has an example of using INITools instead of ConfigParser to give more accurate filenames and line numbers when there is an exception, while otherwise using the same interface.

Woonerf and Python

Posted: January 16th, 2009, 1:09pm CST by Ian Bicking

Tags: Programming [edit]

At TOPP there’s a lot of traffic discussion, since a substantial portion of the organization is dedicated to Livable Streets initiatives. One of the traffic ideas people have gotten excited about is Woonerf. This is a Dutch traffic planning idea. In areas where there’s the intersection of lots of kinds of traffic (car, pedestrian, bike, destinations and through traffic) you have to deal with the contention for the streets. Traditionally this is approached as a complicated system of rules and right-of-ways. There’s spaces for each mode of transportation, lights to say which is allowed to go when (with lots of red and green arrows), crosswalk islands, concrete barriers, and so on.

A problem with this is that a person can only pay attention to so many things at a time. As the number of traffic controls increases, the controls themselves dominate your attention. It’s based on the ideal that so long as everyone pays attention tothe controls, they don’t have to pay attention to each other. Of course, if there’s a circumstance the controls don’t take into account then people will deviate (for instance, crossing somewhere other than the crosswalk, or getting in the wrong lane for a turn, or the simple existance of a bike is usually unaccounted for). If all attention is on the controls, and everyone trusts that the controls are being obeyed, these deviations can lead to accidents. This can create a negative feedback cycle where the controls become increasingly complex to try to take into account every possibility, with the addition of things like Jersey barriers to exclude deviant traffic. At least in the U.S., and especially in the suburbs or in complex intersections, this feeling of an overcontrolled and restricted traffic plan is common.

Copenhagen retail street

So: Woonerf. This is an extreme reaction to traffic controls. An intersection designed with the principles of Woonerf eschews all controls. This includes even things like curbs and signage. It removes most cues about behavior, and specifically of the concept of "right of way". Every person entering the intersection must view it as a negotiation. The use of eye contact, body language, and hand signals determines who takes the right of way. In this way all kinds of traffic are peers, regardless of destination or mode of transport. Also each person must focus on where they are right now, and not where they will be a minute from now; they must stay engaged.

So, I was reading a critique of Python where someone was saying how they missed public/private/protected distinctions on attributes and methods. And it occurred to me: Python’s object model is like Woonerf.

Python does not enforce rules about what you must and must not do. There are cues, like leading underscores, the __magic_method__ naming pattern, or at the module level there’s __all__. But there are no curbs, you won’t even feel the slightest bump when you access a "private" attribute on an instance.

This can lead to conflicts. For example, during discussions on installation, some people will argue for creating requirements like "SomeLibrary>=1.0,<2.0", with the expectation that while version 2.0 doesn’t exist, so long as you install something in the 1.x line it will maintain compatibility with your application. This is an unrealistic expectation. Do you and the library maintainer have the same idea about what compatibility means? What if you depend on something the maintainer considers a bug?

Practically, you can’t be sure that future versions of a library will work. You also can’t be sure they won’t work; there’s nothing that requires the maintainer of the library to break your application with version 2.0. This is where it becomes a negotiation. If you decide to cross without a crosswalk (use a non-public API) then okay. You just have to keep an eye out. And library authors, whether they like it or not, need to consider the API-as-it-is-used as much as the API-they-have-defined. In open source in particular, there are a lot of ways to achieve this communication. We don’t use some third party (e.g., a QA team or language features) to enforce rules on both sides (there are no traffic controls), instead the communication is more flat, and speaks as much to intentions as mechanisms. When someone asks "how do I do X?" a common response is: "what are you trying to accomplish?" Often an answer to the second question makes the first question irrelevant.

Woonerf is great for small towns, for creating a humane space. Is it right for big cities and streets, for busy people who want to get places fast, for trucking and industry? I’m not sure, but probably not. This is where a multi-paradigm approach is necessary. Over time libraries have to harden, become more static, innovation should happen on top of them and not in the library. Some times we create third party controls through interfaces (of one kind or another). I suppose in this case there is a kind of negotiation about how we negotiate — there’s no one process for how to build negotiation-free foundations in Python. But it’s best not to harden things you aren’t sure are right, and I’m pretty sure there’s no "right" at this very-human level of abstraction.

pyinstall is dead, long live pip!

Posted: October 28th, 2008, 12:59pm CDT by Ian Bicking

Tags: Programming [edit]

I’ve finished renaming pyinstall to its new name: pip. The name pip is a acronym and declaration: pip installs packages.

I’ve also added a small feature intended for Google App Engine users, allowing you to zip and unzip packages in an environment. For instance:

$ pip zip --list
In ./lib/python2.5/site-packages:
  No zipped packages.
  Unzipped packages:
    paste  (98 files)
    pygments  (64 files)
    tempita  (7 files)
    weberror  (31 files)
    webob  (22 files)
    webtest  (9 files)
    nose  (43 files)
    setuptools-0.6c9-py2.5.egg  (43 files)
    simplejson  (28 files)
$ pip zip webob
Zip webob (in ./lib/python2.5/site-packages/webob)

Right now this doesn’t work well with egg directories (i.e., packages installed with easy_install), though that shouldn’t be too hard to resolve. pip install itself does not install packages into egg directories (it does install eggs, which is to say it installs all the egg metadata and works fine with pkg_resources).

I don’t really use buildout myself, but I would like to throw it out there that I think someone should create a pip recipe as an alternative to zc.recipe.egg. There’s not really a stable programmatic API in pip at this point, but with no consumers of the API it feels like premature design to settle on something now — integrate with pip and we can figure out what that stable API should be. If you integrate buildout, probably another useful feature would be an option have to pip freeze write the packages out to a setting in your buildout.cfg.

Firefox Security Tool Kit - Now FF v3 compatible

Posted: June 21st, 2008, 10:06am CDT by Disenchant

Tags: Programming [edit]

I’ve been asked to make my Firefox Security Tool Kit (FSTK) compatible to the new Firefox version 3 and here it is

Download and/or Install the FSTK v3.1 now

Additionally to the new version compatibility changes, I replaced the Extension “User Agent Switcher” with “Modify Headers“. All other extensions are the same as described here.

Choosing a License

Posted: May 5th, 2008, 6:48pm CDT by Ian Bicking

Tags: Programming [edit]

I thought I’d take some time to talk about licensing.

Licensing is something that F/OSS programmers talk about a lot. There’s two major categories of licenses:

The GPL, aka Copyleft. You must distribute source with your application, and users get full rights to the source code, including any additions you may make.
Permissive licenses, X/MIT, BSD, etc. These let you do pretty much everything.

There’s also the LGPL, which vaguely applies GPL-like terms to the original code, but is not "viral". LGPL originally meant "Library GPL" but was renamed to "Lesser GPL" because people automatically used it for libraries without considering what the license actually enforced.

How much do these licenses matter? How should a person choose between these options?

First, the LGPL. It has specific phrasing that makes some sense for C code, to distinguish between extending the library itself and using the library. It doesn’t make much sense for other runtime environments. Some people have tried to clarify its meaning in other environments, but I’m not sure if a clarification like that means much. The ambiguity seems like the worst of all worlds. People who are afraid of the GPL won’t use the software anyway. People who act in bad faith can satisfy the terms of the license through trickery. You don’t get any practical protection over a permissive license, and you get most of the stigma of the GPL. The GPL has some success stories, places where source was actually released due to the terms of the license. I don’t know of any similar success stories for the LGPL. If you don’t want to use the GPL, just use a permissive license.

Then the question: GPL or permissive licensing?

For some time I’ve chosen a permissive license for my code. But I’m not really advocating that, and every so often I throw out a little GPL code. Underneath this is something of a disinterest in licensing. I don’t think it means much. If specifics of your license matters then something has gone wrong. People are leeching code, and/or the community isn’t providing enough benefit to encourage participation. I don’t believe that software has much intrinsic worth, and treating it like property doesn’t make that much sense to me. Licensing treats software as property, which is why I see the relevance of licensing as a kind of disfunction. But there is the outside chance that it’s really just a big project, or that the project is being participated in by rivals. But nothing I work with is like this, and it’s pretty uncommon generally, and anyway those projects all have their licensing pretty much figured out.

The GPL does seem to serve a constructive purpose when rivals have to cooperate in some fashion. It makes a kind of demilitarized zone where everyone has to work for the collective good. But even this is a sign that you can’t trust the participants. This is somewhat reasonable, because you can’t actually trust large corporations with faceless programmers working on the code. But even that’s a kind of disfunction, because you can at least kind of trust large corporations with programmer employees who have a public face. The individual programmers are going to be very uncomfortable participating in any kind of Machiavellian conspiracy. Good open source projects are a coming together of individuals. Institutions are not effective participants. There are however scaling issues with individual participation: particularly that only a minority of developers are actually inclined to participate in this way. An institution can pay someone to work on something they don’t care very deeply about, and that person can still do useful work. But they won’t be part of the community.

Outside of this mediation of rivalries I’m not really sure what the GPL provides. It protects users, because the terms of the GPL ensure that users get code. It doesn’t actually ensure that code is made public, though it does give developer employees leverage to make their work public. There’s some urban legends about the viral aspect of the GPL, but that’s really just bullshit. Code doesn’t magically get relicensed just because there’s a little GPL in the mix. The GPL police won’t show up at your door and confiscate your computers. The only people who make those claims are GPL-haters like Microsoft. Who should you trust: the FSF or Microsoft? If you answered "Microsoft" then you need to get your head screwed on straight.

Unfortunately there is a stigma about the GPL, and there are competent developers that avoid all GPL software. I find this frustrating, especially since I don’t actually care about licensing, and so this adds a point of contention that I don’t really care to pay attention to. Some people in this situation then become GPL haters, because they feel like it’s all GPL’s fault. But the GPL didn’t create the licensing situation — proprietary software started this. Direct your hate where it belongs. The GPL ain’t it.

If my libraries were GPL I doubt I could leverage that to make other people open source their code. But I know I would alienate some people, and as a result I choose a permissive license because it’s just strategically advantageous.

Applications I think are different strategically. You don’t just swap an application in and out like you might a library. If you choose to use an application, then often the licensing is a secondary choice. The licensing ultimately only really applies to extensions. Libraries also have a different pattern of acceptance. Stubborn GPL haters can in effect veto a library in many projects (this is because stubborn GPL haters are viral). But in an application their influence is much smaller. Developers concerns like architecture and choice of libraries are not what drives an application. An application is driven by its functionality. If you have a useful application then licensing tends to be a secondary concern. The licensing tends to define the community in some sense as well, and as a result there’s a kind of opt-in consensus. And I think the GPL in these cases really does create an environment of collaboration around extensions. It has a real benefit.

This seems to be true empirically as well. GPL’d libraries don’t tend to get very popular. People will do crazy-big projects just because of licensing issues. GPL’d Applications seem to do quite well, with examples like WordPress, Plone, The Gimp, and even the Linux kernel, which is closer in structure to an application than a library.

I haven’t studied the GPL v3 very closely, but it seems useful to me. Software-as-a-service has the potential to make the GPL irrelevant (especially for the kind of code I write), and could put the GPL in the same category as the LGPL where it retains its stigma without offering any practical advantages. Version 3 makes the permission/GPL choice seem useful, but so far I’ve never made any truly conscious choice between version 2 and 3.

If you are thinking about choosing a license, or thinking about choosing software based on the licensing, maybe these thoughts will be helpful in your own thinking. And please don’t GPL-hate in the comments, I’m not interested and if you feel the need then go write your own post.

Update: I wrote a second article to expand on some thoughts: The GPL and Principles

Firefox Security Tool Kit - FSTK v3.0

Posted: April 1st, 2008, 5:50am CDT by Disenchant

Tags: Programming [edit]

About one year ago, I’ve created a bundle which contained several different extensions for Firefox related to web application security testing. Today I’d like to release version 3.0 of the Firefox Security Tool Kit or short FSTK, which will make your browser a (nearly) full-fledged webappsec pentesting tool.

Download and/or Install the FSTK v3.0 now

By installing it to your Firefox, you’ll get all of the following extensions:

allcookies
Version: 1.2
By: Bernard Maison
dumps ALL cookies to cookies.txt file
For Firefox versions 1.5 to 2.0.0.*
Homepage: [allcookies.mozdev.org]

Cert Viewer Plus
Version: 1.2
By: Kaspar Brand
Certificate viewer enhancements: PEM format view, file export
For Firefox versions 1.5.0.4 to 3.0.0.*
Homepage: [https:]

Firebug
Version: 1.05
By: Joe Hewitt
Web Development Evolved
For Firefox versions 1.5 to 2.0.0.*
Homepage: [www.getfirebug.com]

Firecookie
Version: 0.0.5
By: Jan Odvarko
Cookie manager for Firebug. Firebug has to be installed in order to use this extension.
For Firefox versions 2.0 to 3.0b5pre
Homepage: [www.janodvarko.cz]

Fire Encrypter
Version: 3.0
By: Ronald van den heetkamp
Encryption, Decryption, and Hashing program.
For Firefox versions 1.5 to 2.0.0.*
Homepage: [www.jungsonnstudios.com]

FoxyProxy
Version: 2.7.1
By: LeahScape, Inc.
Premier proxy management for Firefox
For Firefox versions 1.5 to 3.0b4pre
Homepage: [foxyproxy.mozdev.org]

Live HTTP Headers
Version: 0.13.1
By: Daniel Savard
View HTTP headers of a page and while browsing.
For Firefox versions 0.8 to 2.0.0.*
Homepage: [livehttpheaders.mozdev.org]

Selenium IDE
Version: 1.0b1
By: Shinya Kasatani
Record, edit and play Selenium tests
For Firefox versions 1.5 to 3.0b3
Homepage: [www.openqa.org]

Smart Middle Click
Version: 0.5
By: spiers
Middle click for javascript links.
For Firefox versions 1.5 to 2.0.0.*
Homepage: [spiers.free.fr]

Tamper Data
Version: 9.8.1
By: Adam Judson
View and modify HTTP/HTTPS headers etc.
For Firefox versions 1.0 to 2.0.0.*
Homepage: [tamperdata.mozdev.org]

User Agent Switcher
Version: 0.6.10
By: Chris Pederick
Adds a menu and a toolbar button to switch the user agent of the browser.
For Firefox versions 1.0 to 2.0.0.*
Homepage: [chrispederick.com]

Web Developer
Version: 1.1.4
By: Chris Pederick
Adds a menu and a toolbar with various web developer tools.
For Firefox versions 1.0 to 2.0.0.*
Homepage: [chrispederick.com]

Hope you enjoy it

Selenium Remote Control

Posted: March 26th, 2008, 3:14pm CDT by Disenchant

Tags: Programming [edit]

A working colleague of mine showed me a tool, which I think is really cool for automating tasks during web application security test, even if it’s not made especially for that. The tool I’m talking about is called Selenium Remote Control and licensed under the Apache 2.0 License. Let’s have a look at the description of Selenium:

Selenium Remote Control (RC) is a test tool that allows you to write automated web application UI tests in any programming language against any HTTP website using any mainstream JavaScript-enabled browser.

This is exactly what Selenium is/does

I can especially recommend the Selenium IDE, which’s a Firefox extension that can record all actions of your browser, replay them and also export the whole thing into HTML, Java, C#, Perl, PHP, Python and Ruby.

This is, how the Selenium IDE looks like:

I don’t want to say more on this, just give it a try and have fun

Runtime vs. Test time

Posted: February 6th, 2008, 10:39pm CST by Ian Bicking

Tags: Programming [edit]

Defenders of static typing have long claimed that it helps them avoid bugs, the compiler providing a constant test of code consistency. Agile practitioners counter that tests do the same thing and much more.

I find myself inclined to something more in the middle. I don’t care for static typing, but I think programming by contract has a lot of validity, is more powerful and helpful than static typing, and works just as well in a dynamically typed language. Programming by contract is more a principle than a particular technology. Unit testing is not xUnit; that’s just one way to do it. You can do unit testing perfectly well without any framework or special infrastructure at all. Similarly programming by contract just means: state and check your up-front expectations in a piece of code, and state and check our expectations for what that code returns or does. This can be as simple as “assert isinstance(n, int)“ in your code.

Programming by contract is not enough to produce quality code. While the contract is checked at runtime that only means that exceptions are raised early and bad or inconsistent data does not propagate into the system. But it only works if you go through the code paths to invoke all those contracts.

Here’s where functional tests come in. A functional test does stuff. In a web application a functional test does a simulated HTTP request and gives you the result. Generally they are high-level and go through the entire application stack. This is in contrast to unit tests that test just one piece of code.

Proponents of unit tests say that functional tests are problematic. They often give confusing output, so that you have to work harder to debug problems when you are invoking so much code. You have combinatorial problems where it can be difficult to run tests that really get to all branches of the code. Functional tests can be slow.

Of course no one really claims you shouldn’t have a mixture of both functional and unit tests for your code, this is more a question of what the best balance is. Functional tests without runtime checks in your code are a problem. They can be fragile because people look for every detail of how the application responds to identify how specific features function. But every detail that is verified is a detail that might change and require updates to the tests.

With runtime tests functional tests can be much more compact. You can do things like simply verify that a page renders, or that code completes without an exception. If you have some confidence that it won’t be complete wrong — wrong as in without any exception but with errors — then simply exercising to code can be sufficient. In addition to getting more use out of your functional tests, you also have some assurance that production code will act in a sane way — that is, if it is broken, the code will act broken. Given a good deployment situation, the turnaround on fixing deployed bugs can be much shorter.

Easy String to Hex in Shellscripts

Posted: January 21st, 2008, 5:39am CST by beNi

Tags: Programming [edit]

echo "/etc/passwd" | cut -s -d ' ' -f 2- | tr -d ' |n'

Easy String to Hex in Shellscripts

Posted: January 21st, 2008, 5:39am CST by beNi

Tags: Programming [edit]

echo "/etc/passwd" | cut -s -d ' ' -f 2- | tr -d ' |n'

How to Create a Gigabyte Testfile with dd?

Posted: December 28th, 2007, 7:53am CST by beNi

Tags: Programming [edit]

And again, the whole task is as simple as typing that into your terminal:

dd if=/dev/zero of=test.file bs=1000 count=0 seek=$[1000*1000*1]

which will create a one (1) gigabyte big test file named “test.file” in the current Directory.

Fix your PHP Code without changing it

Posted: December 27th, 2007, 2:03am CST by Disenchant

Tags: Programming [edit]

What is it about?
This blog posting describes a way on how you can patch security problems and real vulnerabilities in your PHP code, when you’re not allowed to change the code or for example if this is just not possible for any reason. What you’ll learn here is something like virtual patching, as we know it from web application firewalls but we don’t change the code of the already existing application in any way and we don’t even need new components like such an application firewall. Does this this sounds like black magic to you? Don’t worry, it’s not complicated at all

What is it not about?
This blog posting will not show you for example how to fix all vulnerabilities you can expect in your PHP code nor is the described technique working against every kind of security problem. If you have the possibility to change the existing code, this is for sure the way to go and not what I’m writing in this posting. This is really only a technique which you can use when you can’t change the existing code.

Please also be aware, that I don’t give you any guarantee, that your PHP web applications are secure after you’ve done what’s written in this posting.

Skills you need
I tried to write everything as clear as possible but it’s good if you’ve got some basic knowledge of the PHP programming language because I won’t describe my simple code examples.

So let’s start with the main content

The php.ini
First we should talk about the so called php.ini, which will be our interface between the fixes and the already existing source code. The php.ini is the configuration file for PHP and you can see it’s actual configuration by simply have a look at the file or by creating a PHP script like the following:

<?php
phpinfo();
?>

Get your own php.ini on shared Servers
(If you’ve got your own dedicated server, this might not be relevant to you)
This subsection now shows you, how you can work with the php.ini even if you’re application is hosted on a shared server and you don’t have access to the main php.ini file. As a first step, you have to create a file with the name .htaccess, which means that it’s a htaccess file. Under Unix like systems this is not a problem at all but under Windows systems you’ll get an error when you try to rename a file to “.htaccess”, so with the following two steps, you can also create such a file under Windows:

1. Open Notepad or any other text editor.
2. Go to Save As, and chose “.htaccess” for the file name including the quotes.

Now you can upload the file into your wwwroot directory and add the following line to the .htaccess file, when your php.ini file can be found at /home/disenchant/php.ini:

suPHP_ConfigPath /home/disenchant

Now, the new defined php.ini will be used instead of the one, your web hoster has set up globally.

The auto_prepend_file Option
So let’s go into the part of the php.ini which is interesting for us, it’s the auto_prepend_file option. Just add a line to your php.ini like the following, so that it points for example to the PHP file we made before to display the PHP info page or just create a smaller file with just <?= "test" ?> in it.

auto_prepend_file = /abc/xyz.php

As you can see, if you now have a look at any of your PHP scripts in your web browser, the auto_prepend_file option in our php.ini has included our script at the beginning of every file. This is exactly what we need because this way, we can interact with the existing code, even if we never changed something in the original source code. So let’s go on and fix some vulnerable code.

The vulnerable Code
First we need some vulnerable PHP. The following code is vulnerable to an attack type called “Cross-Site Scripting“.

<?php
echo $_GET['var'];
?>

We have now to implement an input encoding functionality for the variable called var (of course this can also be done for all variables or just some pre-defined variables). Let’s have a look at how to write a simple patch for this vulnerability.

The Patch
If we can change the original source code, we would probably use something like the following:

<?php
echo htmlentities($_GET['var'], ENT_QUOTES);
?>

This will replace (all) characters which can be dangerous or better say, can be used for executing malicious script code, with their equivalent HTML entity. This makes this piece of code secure but in our situation, we won’t be able to change the code and that’s why we need the technique I described in this posting, over the php.ini. Instead of modifying the existing source code, we create a new file like this:

<?php
$_GET['var'] = htmlentities($_GET['var'], ENT_QUOTES);
?>

Because of the auto_prepend_file option we set in our php.ini, this code will fix our vulnerability

Final words
As you’ve seen, this technique can become very helpful under some circumstances and of course I showed you just the basic use of it and you can go much further and for example implement login stuff, session management, white listings and so on through this.

Permalink for 'mybeNi_websecurity/2007/12/20/How_to_rescue_your_Xorg_Server_in_a_%e2%80%9cworst_case%e2%80%9d_scenario_'

How to rescue your Xorg-Server in a “worst case” scenario?

Posted: December 20th, 2007, 11:51am CST by beNi

Tags: Programming [edit]

Several years ago, I was a Linux beginner as like most people were at one point in their lives.

I had just installed SuSe 9.0 for the first time, and the problem was that my ancient ATI Radeon 9000 (an old graphics card) didn’t stop bitching around: I just couldn’t manage to install and configure it properly, and ATI didn’t help their users, too.

So we had no real working drivers except several tutorials on “How to Configure your Xorg-Server with an ATI Card” …

I tried to follow them step by step, but I always made a tiny mistake which f*cked up my xorg.conf and thereby my whole system.

It was horrible because I wasn’t able to recover it via console those days due to the famous lack of Linux skills.

Again, if you’re running Ubuntu or Debian, use dpkg-reconfigure xserver-xorg in order to re-setup your graphic devices and monitor etc.

It’s an easy guided process and the developers even proposed that they’ll make it a lot easier in the future releases of Ubuntu!

My Security Planet » Tags » Programming

Feeds

24 items tagged "Programming"

Posted: February 14th, 2012, 2:30am CST by Ian

Tags: Programming [edit]

Posted: April 4th, 2011, 10:23am CDT by Ian

Tags: Programming [edit]

Posted: March 30th, 2011, 5:09pm CDT by Ian

Tags: Programming [edit]

Posted: April 2nd, 2010, 5:27pm CDT by Ian Bicking

Tags: Programming [edit]

Posted: April 2nd, 2010, 1:26pm CDT by Ian Bicking

Tags: Programming [edit]

Posted: March 16th, 2010, 6:23pm CDT by Ian Bicking

Tags: Programming [edit]

Posted: March 12th, 2010, 5:50pm CST by Ian Bicking

Tags: Programming [edit]

Posted: March 10th, 2010, 7:45pm CST by Ian Bicking

Tags: Programming [edit]

Posted: February 10th, 2010, 12:41am CST by Ian Bicking

Tags: Programming [edit]

Posted: February 5th, 2010, 2:50am CST by Ian Bicking

Tags: Programming [edit]

Posted: May 22nd, 2009, 9:35pm CDT by Ian Bicking

Tags: Programming [edit]

Posted: April 20th, 2009, 1:11am CDT by Ian Bicking

Tags: Programming [edit]

Posted: January 16th, 2009, 1:09pm CST by Ian Bicking

Tags: Programming [edit]

Posted: October 28th, 2008, 12:59pm CDT by Ian Bicking

Tags: Programming [edit]

Posted: June 21st, 2008, 10:06am CDT by Disenchant

Tags: Programming [edit]

Posted: May 5th, 2008, 6:48pm CDT by Ian Bicking

Tags: Programming [edit]

Posted: April 1st, 2008, 5:50am CDT by Disenchant

Tags: Programming [edit]

Posted: March 26th, 2008, 3:14pm CDT by Disenchant

Tags: Programming [edit]

Posted: February 6th, 2008, 10:39pm CST by Ian Bicking

Tags: Programming [edit]

Posted: January 21st, 2008, 5:39am CST by beNi

Tags: Programming [edit]

Posted: January 21st, 2008, 5:39am CST by beNi

Tags: Programming [edit]

Posted: December 28th, 2007, 7:53am CST by beNi

Tags: Programming [edit]

Posted: December 27th, 2007, 2:03am CST by Disenchant

Tags: Programming [edit]

Posted: December 20th, 2007, 11:51am CST by beNi

Tags: Programming [edit]