13620 items (0 unread) in 75 feeds
Watchfire Application Security Insider
sirdarckcat
Google Online Security Blog
Switch/Twitch
mybeNi websecurity
cat slave diary
Alex's Corner
Billy (BK) Rios
Chris Shiflett
christian's weblog
GNUCITIZEN
The Spanner
hackademix.net
Ruby on Rails Security Project
It's a shampoo world anyway
ha.ckers.org web application security lab
Yet Another Infosec Blog
PHP Security Blog
Sunnet Beskerming Security Advisories
Disenchant's Blog
Sylvan von Stuppe
Larholm.com
Jeremiah Grossman
Anurag Agarwal - Application Security Evangelist
Ivan Ristic
omg.wtf.bbq.
Tactical Web Application Security
.:Computer Defense:.
tssci security
Plynt Penetration Testing Blog
Security Retentive
Schneier on Security
Rational Survivability
terminal23
1 Raindrop
hiredhacker.com
Matasano Chargen
Michael on Security - Musings on Information Security
Fortify blog
GDS Security Blog
Vodun.org
Mark Curphey - SecurityBuddha.com
Liminal states
Security Ripcord
IBM Internet Security Systems Frequency X Blog
extern blog SensePost;
Zero in a bit
What is persona.org?
Persona.org is where you manage your online identity in a safe and simple way!
Why should I use persona.org?
We don’t sell your personal information to third parties and we enable you to connect and control your online services by using different identities.
Where can I find out more information?
Follow updates at [https:] or the code on Github at [https:].
Two months ago, the former Flux team had an offsite where we played with lasers.
But I also gave a talk on browsers internals. This is just an introduction but it should give web developers a better understanding of this black box that we work with everyday.
Slides are also available. Since I’m not smart enough to hack on browsers, I heavily rely on other people’s writings to document myself. Take a look at those resources to get into the details.
If you speak French, you can watch the original version of the talk that I gave at Paris-Web last year.
Last night we launched http://l10n.mozilla.org
Like all major Open Source projects (for example, Gnome, KDE) the place to go if you want to contribute with translation work is to go to l10n.domain.name.
The codename for the project is “Elmo”. code here and alongside the new domain, we also launched a major re-write of the overall design and with some new significant feature additions. In particular the team pages which is hopefully the place to start. If you’re in a team (aka. a locale, a language) you now get a neat and organized dashboard. For example, the Russian locale team page. Now, you get everything (hopefully) you need in one single page broken down by project (for example Thunderbird), Bugzilla bugs and web translations.
At the moment this new site’s purpose isn’t perfectly tuned. It’s a powerful dashboard for getting deep into actual translations and statistics about code changes and sign-offs. It’s also a landing page or portal for all potential aspects of language contributions. Having this new re-design and a fresh wind blown into it will hopefully spur us to innovate and let it evolve the way it needs to evolve.
Live* from New York, it’s Friday Morning!
* Recorded with no audience. Laugh track may be used.
This week’s special guest on Better Know a WebDev, Michael Kelly!
What do you do at Mozilla?I work on engagement campaigns, like Firefox Flicks and Firefox Affiliates. I also work on the snippets service, which powers the small messages you see on about:home in Firefox. Most of my work is in Python on the backend side of our Django-based sites.
Any fun side projects you’re working on?
Most of my side projects are mediocre javascript games. I also have a few smaller things on my Github account.
How did you get started in web development or programming?I got started with programming when I was around 8 years old, when my brother got a copy of ZZT installed on our home computer. ZZT is awesome! Not only does it come with a set of cool games to play, it lets you make your own games and program objects to do all sorts of cool things, like teaming up against the player, sending the player on a quest, or controlling a role-playing-game simulator.
I got into web development specifically when I was a teenager and learned HTML. I would eventually learn PHP and start making interactive sites in my spare time, which landed me a job for my Computer Science department in college working on their website.
How did you get involved with Mozilla?Unlike a lot of Mozillians, I didn’t start off as a contributor. I got involved with Mozilla when they showed up at a job fair at my university. They were the only web-focused company there, and I rushed to start talking to them about the side projects I was working on. That landed me an interview, which led to a job after I graduated a year ago.
What’s a funny fail or mistake story you can share?Once, while interning at Electronic Arts, I was working on a feature that made a database query once for every page across the Sims 3 website. We had set it up to be heavily cached, but I still managed to take down the entire Sims 3 site for a day. Oops!
Turns out, my database query, which we thought was running directly on the database, was actually running through Hibernate, the ORM we used on the site. This was frightfully slow, and caused all of our database connections to freeze up due to the massive amount of traffic the site got. Luckily, all that caching helped free up the site in the afternoon, and we were able to push a fix that night.
I’m very proud of how Mozilla was able to get together with other companies across the internet and join the blackout on January 18th to protest SOPA and PIPA. My part to play was to create the snippet that blacked out about:home and sent our message to more than 30 million Firefox users.
The coolest part was that we were able to get this snippet and the blackout pages for other Mozilla sites ready in the space of a day. It was awesome to see how quickly everyone was able to mobilize in order to help stop SOPA and PIPA from censoring the web.
What’s coming up that you’re excited about?I’m really excited about Gladius, a Mozilla-developed 3D game engine for the browser. It provides a lot of the plumbing needed to get a game off the ground in Javascript. I’d love if there were a similar effort going on for 2D game development as well, given the positive response we got to BrowserQuest.
Other cool things to watch for are the Gamepad API, Pointer Lock API, and Full Screen API, all of which are helping make open web-based gaming a reality.
What question do you wish you’d been asked?
I wish I’d been asked about what games I am currently playing. 
The answer, by the way, would be “The Longest Journey” by Funcom, “Minecraft” by Mojang (of course you never really stop playing Minecraft), and (in 2 days when it releases) “Fez” by polytron.
Next time we rewrite all of our websites, what will the platform be?
We use Graphite in Web Development to track metrics on our sites. For the Marketplace and Addons team, it has been a real eye opener into the state of our sites.
One thing that I’ve wanted to add is information on when deployments occur as these are often critical periods that can trigger changes in the graphs. Sure enough, this has all been done before and covered in a blog post. Following on from that blog post, here’s how I added this for the Marketplace.
First, I added a management command to django-statsd that allows you to easily send a ping to Graphite for a specific key. The value of the key is set to be a UNIX time stamp. Once that was in place I altered our deployment scripts so that on each update they send a ping with the key “update”:
python2.6 manage.py statsd_ping --key=update
Then each time a server is deployed that key gets added to Graphite.
Next, I altered our graphs so that they show this ping as a vertical line. That’s using the command drawAsInfinite, for example:
drawAsInfinite(stats.timers.addons-dev.update.count)
Our development server updates on every commit. In other words, it continually deploys all the time. This is the graph for the development server, and you can see the large number of small commits we had this afternoon (each commit is indicated by a pink line):
A more realistic view is our production server, where you can see our Thursday afternoon push in context (each push is a pink line):
Adding this data is crucial to helping the Marketplace get closer to continuous deployment in the coming months.
We use Graphite in Web Development to track metrics on our sites. For the Marketplace and Addons team, it has been a real eye opener into the state of our sites.
One thing that I’ve wanted to add is information on when deployments occur as these are often critical periods that can trigger changes in the graphs. Sure enough, this has all been done before and covered in a blog post. Following on from that blog post, here’s how I added this for the Marketplace.
First, I added a management command to django-statsd that allows you to easily send a ping to Graphite for a specific key. The value of the key is set to be a UNIX time stamp. Once that was in place I altered our deployment scripts so that on each update they send a ping with the key “update”:
python2.6 manage.py statsd_ping --key=update
Then each time a server is deployed that key gets added to Graphite.
Next, I altered our graphs so that they show this ping as a vertical line. That’s using the command drawAsInfinite, for example:
drawAsInfinite(stats.timers.addons-dev.update.count)
Our development server updates on every commit. In other words, it continually deploys all the time. This is the graph for the development server, and you can see the large number of small commits we had this afternoon (each commit is indicated by a pink line):
A more realistic view is our production server, where you can see our Thursday afternoon push in context (each push is a pink line):
Adding this data is crucial to helping the Marketplace get closer to continuous deployment in the coming months.
If you’re like me, you enjoy doing lots of quick CRUD (create/read/update/delete) applications that are highly experimental, highly volatile and require low commitment until you know where you want to go with it.
Having worked with various web frameworks (Django, Rails, Sinatra, webpy, Struts, Express, Flask, and probably some other ones I’ve missed), I find that as a coder, one needs a nice balance of getting code up and running on a local machine and not having to organize and plan too much when in “play mode”. By “play mode”, I mean MTAWGSBWATL a.k.a. “Maybe This App Will Go Somewhere But Worry About That Later”.
This kind of mentality is very useful for both beginners and those who are more advanced in their programming career. Think of it as taking a bunch of clay and molding it until you feel that you have the right fit. The last thing the experimental coder needs is to think about is the “ideal organization” of their code and more about getting it running on a local webserver as soon as possible.
I find two frameworks quite efficient at getting this done – Sinatra and Flask. In this post, I’ll be specifically talking about Flask, BrowserID and MongoDB.
Now a quick summary of these three things:
Why would one use these for an application, especially one that is prototypical? First, because we don’t want to over-engineer our code right now. Second, because we don’t want to write our own or maintain an authentication platform that requires us to either handle anything beyond a user’s session, uses a third-party service (such as Twitter or Facebook) which some people may not feel comfortable in signing in with and we want to get things done as fast as possible. Finally, because there’s no point in planning a relational database with something like MySQL or PostgreSQL when you may not even know what you are designing. You may not even know what you’re doing! So keep it simple and to the point.
For my examples, I started with Lexicrypt. Lexicrypt is a fun, small project that generates encrypted messages as images. It took me about a week to get the first iteration up and running on a publicly accessible server. Your mileage may vary – if you’re experienced in Python, this may take you about the same amount of time or less. If this is your first time, don’t worry about taking longer – you’ll eventually find your comfort zone and your modus operandi for developing personal scaffolds of your projects.
Shortly after I created Lexicrypt, I decided to make a small photo service that enables you to sign in, upload photos, tag and favourite them. Instead of starting from scratch I took the scaffold of Lexicrypt, popped out the encryption functionality and replaced it with the photo functionality. This enabled me to go from idea to deployment for the first iteration within a weekend!
To view the source for these projects and get an idea of their similarities and differences you can check out the Github repositories at [https:] and [https:].
As for your rapid prototyping endeavours, good luck and continue to code and code some more. Remember, the process is not only a way for you to learn how to code more efficiently and collaborate with others as you gain more experience, but it is also a way for you to learn more about yourself in the development of your application.
Here is a video of me giving a lightning talk on the subject:
If you’re like me, you enjoy doing lots of quick CRUD (create/read/update/delete) applications that are highly experimental, highly volatile and require low commitment until you know where you want to go with it.
Having worked with various web frameworks (Django, Rails, Sinatra, webpy, Struts, Express, Flask, and probably some other ones I’ve missed), I find that as a coder, one needs a nice balance of getting code up and running on a local machine and not having to organize and plan too much when in “play mode”. By “play mode”, I mean MTAWGSBWATL a.k.a. “Maybe This App Will Go Somewhere But Worry About That Later”.
This kind of mentality is very useful for both beginners and those who are more advanced in their programming career. Think of it as taking a bunch of clay and molding it until you feel that you have the right fit. The last thing the experimental coder needs is to think about is the “ideal organization” of their code and more about getting it running on a local webserver as soon as possible.
I find two frameworks quite efficient at getting this done – Sinatra and Flask. In this post, I’ll be specifically talking about Flask, BrowserID and MongoDB.
Now a quick summary of these three things:
Why would one use these for an application, especially one that is prototypical? First, because we don’t want to over-engineer our code right now. Second, because we don’t want to write our own or maintain an authentication platform that requires us to either handle anything beyond a user’s session, uses a third-party service (such as Twitter or Facebook) which some people may not feel comfortable in signing in with and we want to get things done as fast as possible. Finally, because there’s no point in planning a relational database with something like MySQL or PostgreSQL when you may not even know what you are designing. You may not even know what you’re doing! So keep it simple and to the point.
For my examples, I started with Lexicrypt. Lexicrypt is a fun, small project that generates encrypted messages as images. It took me about a week to get the first iteration up and running on a publicly accessible server. Your mileage may vary – if you’re experienced in Python, this may take you about the same amount of time or less. If this is your first time, don’t worry about taking longer – you’ll eventually find your comfort zone and your modus operandi for developing personal scaffolds of your projects.
Shortly after I created Lexicrypt, I decided to make a small photo service that enables you to sign in, upload photos, tag and favourite them. Instead of starting from scratch I took the scaffold of Lexicrypt, popped out the encryption functionality and replaced it with the photo functionality. This enabled me to go from idea to deployment for the first iteration within a weekend!
To view the source for these projects and get an idea of their similarities and differences you can check out the Github repositories at [https:] and [https:].
As for your rapid prototyping endeavours, good luck and continue to code and code some more. Remember, the process is not only a way for you to learn how to code more efficiently and collaborate with others as you gain more experience, but it is also a way for you to learn more about yourself in the development of your application.
Here is a video of me giving a lightning talk on the subject:
Welcome to another thrilling installment of your favorite, and only, recurring series on this blog: Better Know a WebDev!
This is a special week, as we’re featuring someone who just moved from web development into a product role with our Add-ons team. Give a big welcome and congratulations to Greg Koberger!
For the past year and a half, I’ve been a web developer working on the Mozilla Add-ons site. I wrote both front- and back-end code, and every once in a while I managed to sneak in a little bit of design work. I recently traded in vim for Google Docs, however, and made the switch to a more product-centric role. I still get to work with the same people on the same projects, however I’ll be writing specs and designing flows rather than writing code.
Any fun side projects you’re working on?One of most recent (and relevant) mini side projects is a Firefox Quine. It’s not a true quine, since it loads some external assets (like jQuery). However, the concept is the same: if you were to copy and paste the code that makes up the logo, it would output itself. We’re currently getting the design printed on t-shirts.
How did you get started in web development or programming?I was lucky—my parents always made sure I had a good computer. I started by tinkering with HyperCard, and eventually decided to try making websites. I still remember how asking my parents for my first hosting account went: “Why do you need other people to see it? Can’t you just play with it yourself on your computer?” Luckily they finally agreed, and I started teaching myself PHP and JavaScript.
How did you get involved with Mozilla?While in college, there was an alumni talk about working at Mozilla. I missed the talk, however I emailed the guy after since I was curious. He encouraged me to apply, and I flew out to California to interview. I fell in love with the company. The people I interviewed with were awesome, and I was amazed by the company culture. Everyone just wanted to make the web a better place.
What’s a funny fail story or mistake you can share?Back when I first started at Mozilla, I managed to delete all my private keys. So, some of the other webdevs set up a tissue-box-based “Computer School Fund” for me. I made $20 for deleting my private keys, so I like to think I got the last laugh.
Back in November, awareness of SOPA was just starting to pick up. The day before the initial hearing, a bunch of us decided to try to raise awareness. It was incredibly last minute, but we knew we had to do something. We could get the message out via the snippet on about:home, which serves about 300MM requests a day. However, we had nowhere to link to. SOPA was new, and there wasn’t a definitive source with a few easy-to-digest bullet points. Plus, we couldn’t really send millions of hits to a website without giving them fair notice. So, we made our own. I sketched it out on paper and wrote out the copy. Then, as I designed it in Photoshop, my fellow webdev Potch sat next to me and wrote the code. We didn’t even have time to share files; he just watched what I was doing and wrote the code. It took us about an hour—which is all we had, since it still had to be deployed and tested. We couldn’t have gotten it out without help from all the different teams at Mozilla—everyone from legal to QA to IT to webdev worked late to make it happen. It was a huge success. We got the word about SOPA out to millions of Firefox users within 24 hours. We were able to use the same site a few months later for the SOPA blackout.
What’s coming up that you’re excited about?I’m really excited about my new product management role. I love new challenges. It’s a great time to start this job, since a lot is going on with AMO right now. “Marketplace” (apps and add-ons combined, with the ability to charge) is going to be a huge project in the upcoming months. I can’t wait to see what our team can make in the upcoming months.
What software do you use day-to-day for the work you do?My main machine is a 13″ Macbook Air running OS X Lion. I use vim for writing code, which all ends up on GitHub. I rely heavily on Firefox and Firebug for testing and debugging. I spend a good amount of time in Photoshop, mocking things up before I start coding. I couldn’t survive without either Rdio or Pandora playing in the background.
There are lots of ways to measure the performance of a site. On addons.mozilla.org we measure a few of them: how long it takes to render pages, the cache performance and responsiveness for content delivery networks globally. But with the advent of the navigation timing API in Firefox 7 we’ve been able to add the most important measurement of all – the actual performance in the browser.
Along with a few other Mozilla sites, we’ve been using pystatsd and Graphite for a while. This produces useful graphs of the site health and performance. For example, this graph plots HTTP responses we serve out from Python (this is not the amount of traffic to our site which is much larger).
To facilitate using the timing API we wrote a module between statsd and Django called django-statsd that, amongst other things, provides a way to interface between the browser and Python backend. The library has support for boomerang, a great front end timing library. I quickly wrote a much simpler service called stick, which sends a small subset of timings that we want.
Using stick is straightforward – once you’ve setup your Python backed and included your JavaScript, you call it in your page. For example:
stick.send('https://to.your.site/record')
The API returns with timestamps for each of the events. To give us meaningful numbers, we then subtract the timestamp for each of the events from navigationStart so we can tell the timings relative to that start point. And here’s the end result in our Graphite server:
For internal sites we collect 100% of the data. However, on our live site we get quite a lot of traffic, so we’ve rolled this out by collecting only 1% of traffic. We’ll start increasing this percentage once we are confident this won’t be causing problems. We’ll also be adding it to the about:addons page which gets quite a bit of traffic.
We’ll be tweaking things to ensure we pass through the right numbers and interpret them correctly. The amount of data is probably too low to be useful, right now, but I can’t help but wonder what the peak around 12.30pm today was…
For most people who work with computers for a living, holidays are a time for family… and providing endless tech support. As I was huddled over a half working computer, someone asked what an add-on was. I spend most of my time working with and on add-ons — so the concept seems obvious to me. As I found out this past Christmas, it’s not to everyone. My otherwise-fairly-computer-literate family had a hard time understanding where the browser ended and websites such as Facebook or Google started — so explaining the concept of add-ons was not nearly as easy as I thought it would be.
While over 85% of people using Firefox have at least one add-on installed, most people still don’t really know what an add-on is or the possibilities they provide. To fix this, the add-ons group teamed up with Mozilla Creative and an outside firm, Poke, to create a short video explaining what add-ons are and what they can do.
After seeing the video, we knew we had to do something awesome with Popcorn.js.
Watch The Video
On the page, click “Learn More” to play the video.
I recommend watching it in Firefox, since we interact with the browser in ways we couldn’t with Chrome. You can also view the video directly inside Firefox, by going to Tools > Add-ons > Get Add-ons > Click “Learn More” / “Learn More About Add-ons”.
Popcorn.jsFor anyone unfamiliar, Mozilla’s Popcorn.js makes it easy to interact with video online using JavaScript. For anyone who has worked with jQuery or a similar library, it’s a breeze to pick up. Popcorn.js is the code part of Mozilla’s Popcorn, which is a larger project that aims to supplement video and audio with rich web content.
Setting up Popcorn.js is incredibly simple:
var pop = Popcorn('#promo-video');
Popcorn.js has a plugin called code, which lets you run arbitrary code when the video hits certain points.
pop.code({'start': start, 'end': end, 'onStart': function(){…}, 'onEnd': function(){…});
That’s basically all you need to know to get started with Popcorn.js. Everything else we did used regular JavaScript.
Like I mentioned, we only used one of the many plugins. Popcorn.js is incredibly powerful, and our video barely scratched the surface of what Popcorn.js plugins can do.
Showing Add-onsWe started by making add-ons fade in on the right side of the video when they are mentioned.
Note: You will only see this in the video if your browser window is wide enough.
We’re off to a good start. So far, we have add-ons showing up when they’re mentioned in the video. Now, wouldn’t it be cool if the video could interact with Firefox?
At one point the video talks about Personas, which are basically skins or backgrounds for Firefox. Since it would be incredibly annoying if any site you visited had the ability to skin your Firefox, the ability to trigger a preview is whitelisted to just the Mozilla add-ons sites.
Note: You can add other sites to this whitelist by going into Preferences > Security > Exceptions.
We thought it would be fun to style the browser with the Persona being mentioned — especially since we’re the only site that is allowed by the browser to do this. We simply fire an event when the video hits a certain point. To make sure the Personas are displayed instantly, we pre-load the Personas when you first click “Play”.
Later in the video, we see a plane fly across the screen. The papers and other items on the desk start to blow away as this happens. If you scroll down a bit, the add-ons below the video also start to flutter and are eventually caught in a JavaScript-y wind and blown away.
This is done using -moz-animation and -moz-transform: matrix(). Each add-on is given a class of “flutter” (starting at a staggered interval), and the add-ons flutter back and forth until they eventually “blow away”.
This isn’t Firefox-only; I just left out the Webkit code in the example for the sake of brevity.
In the next scene, both the backdrop in the video and the add-ons on the site drop back into place.
During the scene with the cameras, the page flashes as though a picture with a flash was being taken. The elements are simply hidden, then faded back in.
When the video talks about how add-ons can translate a page, some of the text on the page is translated along with the text in the video. (Note: This only works if you have fewer than three add-ons installed or you watch using this link. Otherwise, the top bar is not displayed.)
We learned a few things along the way. We used popcorn-complete.js, which we didn’t realize loaded a few CSS, JS and SWF files off of popcornjs.org’s servers when the file was loaded. We mistakenly DOS’d popcornjs.org, which in turn made the add-ons site load slowly. Thanks to Jon Buckley and Rick Waldron, we were able to figure out what was going on and push a fix fairly quickly.
I also owe Chris and Krupa a big hug when I get back to the Mozilla office — they helped me out a ton by finding and fixing a bunch of bugs as we were pushing to production, since I was an unfortunate 30,000 feet in the air as the video was going live.
Looking AheadI have no doubt that Popcorn.js is going to become huge as HTML5 video takes off. News sites will use it to show relevant links and tweets, fashion sites will link to online stores as a celebrity is shown wearing a dress or shirt, video sites will provide closed captioning and annotations, media sites will use it to sync viewing across multiple computers — the possibilities are endless. I cannot wait to see what Popcorn powered videos people make next.
Watch The Video
On the page, click “Learn More” to play the video.
Since the add-ons site is open source, be sure to view the code if you’re interested.
As for my family: not all is lost. The next morning I got an excited text from my cousin, who had watched the video. “Guess what I just did! I installed one of those picture addons on the top!” Sure, it pains me a bit that a Justin Bieber Persona now graces her Firefox — but hey, it’s a step in the right direction.
I’ve been working on a Django project that is going to be squeezed in as an external application under Pootle which is our tech of choice for the Mozilla Verbatim site which is where all awesome translations happen for Mozilla web projects. (translations for Firefox, Thunderbird, etc. is done by another project which I’ll blog about more in the next couple of weeks).
In Pootle, there are “Profiles” and that’s where the name and quick stats on contributions is stored. However, what I needed to do was to actually go through every “Suggestion”, “Submission” and “Review” and figure out who has contributed what on what project and language. Every such model has an overridden object manager that always joins on the “Users” model thus causing a ‘INNER JOIN’ on every simple data query. Thus this would build up an accurate page of who has contributed to what on the fly.
There are 14 projects, 99 languages, 847 users, 37,681 suggestions (and reviews) and lastly 127,226 submissions. The task is to group these first by language and then by project and lastly for each project, list the contributors names in alphabetical order. After some getting used to the data model and where things are located this wasn’t difficult. I built a prototype that visually represents this and showed it to the stakeholder who wanted this built.
The only problem is that due to the way the Django ORM and Pootle does this is that this takes on average: 11 seconds and involves 2,544 SQL queries. Yikes!
After some optimization (helped by my colleague Rob Hudson’s wonderful django-debug-toolbar) I was able to turn 11 seconds into 10 milliseconds (ie. one thousand times faster) and reduce the number of queries from 2,544 to 7! All it takes is just having to lessen some of the convenience of Django’s otherwise super practical ORM and get closer to the “SQL of things”. The code doesn’t express anything in raw SQL and everything is in Python with some very specific loops.
Why 7 queries and not 1, you might wonder. The reason is that the data I need to get to is spread across 7 different tables. Because some of them are large and joins are difficult what I do instead is pick only exactly what I want and need and nothing more. The logic kinda goes like this:
I think the lesson here is that if your structures are big but not so big that you need a map/reduce job then getting closer to the raw SQL queries and doing the structure juggling in memory, you don’t need to replace your relational databases with NoSQL or typed raw SQL. Django’s ORM is fantastic. It’s slow because it chooses to be convenient but if it needs to be fast it’s just a few slight API calls away. If you’re curious, check out the code on Github.
This is the second in a series of posts, focusing on issues around open sourcing your Django site and data privacy in Django.
You’ll end up with production data in your Django database and that will likely contain different kinds of data such as: configuration data, required basic data (categories for example), collected data and personal user data. There’s a couple of reasons for taking that production data and copying it off your production servers:
For the first case, it’s nice to prepare a minimal copy of the database that contains key data. For example, for those wanting to develop or contribute to addons.mozilla.org we have Landfill by Wil Clouser.
Django comes with a nice facility for loading data, fixture dumping and loading. This can be used to pull data out of your database and then reload it. However the built in Django dumpdata dumps all the records for your model (depending upon your default object manager). That might not be what you want for this scenario. So a useful utility for dumping just the records you want is provided Django fixture magic written by Dave Dash.
A standard dumpdata looks like this:
manage.py dumpdata users.UserProfile
And will dump every UserProfile. By contrast:
manage.py dump_object users.UserProfile 1
Will just dump the UserProfile with primary key of 1. Django fixture magic also has a few other useful things such as merging and reordering fixtures.
This allows you to trim a set of fixtures from your live database down quickly. Then developers or contributors can load the key parts of the database that they need from those fixtures.
Anonymising the databaseSending the production database downstream to internal developers or internal test sites is a pretty common use case. This process does not require a complete clean of the database, but it does require some cleaning of database. If you stored credit card data, for example, you’d never want to copy that off your production database.
At Mozilla we use an anonymising script, written by Dave Dash again. There are few options: to truncate, nullify, randomize or selectively delete. The format is a simple YAML file, for example:
tables:
users:
random_email: email
nullify:
- firstname
This is a snippet from the config script for addons.mozilla.org.
When the IT copies the databases down from production, this script is run against the database. Ensuring that when us developers access the backups to investigate certain issues, we’ll be getting the bits we want and not the bits that might expose user data.
In the next blog post we’ll look at logs and tracebacks.
Surely you’re convinced too now that developing for the mobile web is important. It certainly is for us here at Mozilla. Mobile web development isn’t just about making a ultra-slim stripped down version with no images or Javascript widgets. It’s also about taking regular full powered web applications and making stepwise adjustments to make sure they work responsively on all mobile devices which means everything from tablets to regular cell phones over 3G.
There are numerous good guides such as this for Designing for Mobile Devices and there are great add-ons for Firefox such as the User Agent Switcher. And don’t miss Jason’s wonderful blogs from the archive part 1, part 2, part 3 and part 4. But ultimately, nothing beats actually testing it on a real physical device.
That’s where localtunnel (by Jeff Lindsay) comes in. What it does is that it binds a local port (e.g. port 8000) on your desktop to a real domain name (e.g. xyz123.localtunnel.com) all over SSH. So, you start your local Django server locally and then start a local tunnel and after waiting 1-2 seconds you get a real domain name to type into your iPhone, Android, Nokia smartphone, whatever and you can test away as much as you like.
Granted, you can use it not just for mobile device development but for anything such as telling a colleague over IRC to visit your site running locally on your laptop. And also, if you are in an office where you connect to the same local WiFi with your laptop as you do with your mobile device then you can just use a local IP address. But that’s often not the case especially when working with people in different locations.
An added, yet strange, benefit with using localtunnel to test your mobile device is that it is naturally slow. That’s because the traffic has to go via SSH over the network and your local network won’t be as fast as a real data center. However, in real life, the network connection will be slow when using a mobile device so it’s important to see how your mobile web app behaves with a limited bandwidth.
Before pushing our CSS and JavaScript assets to our CDN, we run them through jingo-minify to concat and minify the files, as well as cache bust them and any resources (such as images) contained inside them. Turns out, this was by far the slowest part of our push process — it took between 80 and 160 seconds for addons.mozilla.org (AMO) assets. It wasn’t a huge priority, since most of the time this happens in the background and nobody really notices. However, I wanted to see how fast I could get it.
Note: All times are based on AMO being compressed on a MacBook. On our build server, those 80 seconds took about 160 seconds. Most times are averages and/or estimates, and are based on an initial 80 seconds.
This was the low hanging fruit. YUICompressor is slow. We managed to cut off over half our original time by switching to UglifyJS for JavaScript and clean-css for CSS, both of which are written in Node. This switch means less errors (we were getting a lot with YUI), smaller files and much faster build times — and none of us are sad to see our Java dependency disappear. (I benchmarked all our options before we chose them. If you’re interested, you can run them yourself.)
Unlike YUICompressor, UglifyJS builds an abstract syntax tree, compresses the AST, then converts the whole thing back to (minified) JavaScript. So far, this has meant that UglifyJS has been much more resilient against our sometimes-misformed JavaScript. Missed semicolons no longer take down the whole site’s JavaScript. The build times are about the same by my benchmarks, but the results are far superior. [JS Benchmark results]
Clean-css works much like YUI (lots of regex), which is a bit unfortunate. However, it does so much faster — 10x as fast, to be exact. [CSS Benchmark results]
Original Time: 80s
New Time: 37s
Difference: -54%
We were fetching the hash (originally the git commit id, now hash of the actual file) for the same files fairly often. By sticking the already-computed hashes into a dict, we were able to shave off a small but significant amount of time.
Original Time: 37s
New Time: 35s
Difference: -11%
When I first wrote cache busting for individual images in our CSS files, I used the image’s most recent git commit as the cache-busting hash. As it turns out, your script gets really slow when you’re looking up the history for dozens of images using GitPython. So, I switched to using a hash of the file. The end result is the same — the hash is only updated if the image is changed. However, using file hashes instead of git commits shaved a good chunk of time off our build process.
Original Time: 35s
New Time: 23s
Difference: -34%
I got down to 23 seconds, and was stuck. The biggest time sink in the process was the minifying of files, which I couldn’t do much about without touching the UglifyJS or clean-css code. Even if I wanted to, I doubted I could do much to speed either of them up. After thinking about it for a bit, I realized that there was no reason for us to be minifying every file each time. We only needed to minify the files that had actually changed. So, jingo-minify now writes the concatted file out to a temporary location, and compares it to the preexisting concatted file. If they’re the same file, we have no need to re-minify.
Since we push to preview every time anyone commits and most commits don’t modify more than one or two JS or CSS files (if any), we can save between 15 and 17 seconds each time.
Original Time: 23s
New Time: 6s
Difference: -74%
So, that’s how I managed to speed up our asset compression by over an order of magnitude — from 80 seconds down to about 6. (Note: Time is based on the script on a MacBook. On the slower build server, it’s closer to 160 seconds down to 9.) Hopefully this means much smoother pushes for AMO, SUMO and anyone else using jingo-minify to compress their assets.
This is the first in series of posts, focusing on issues around open sourcing your Django site and data privacy in Django.
A lot of people focus on open sourcing their Django libraries, but at Mozilla we open source the entire site. Releasing your entire source code can lead to a few problems, firstly let’s look at your Django settings file.
Separating your settings
All Django sites come with a settings.py file. This file contains some key settings that you should not be releasing to the general public, such as database configuration and the secret key. The simple way to do this is have another file that contains the secret settings and then import it from settings.py. For example include this at the bottom of your settings.py:
All your sensitive settings can now kept in that local file on your server and should not be published as part of your site code. This file will override the base settings file.
There are plenty of other examples on different ways to do this. You can do it in manage.py, or turn your settings.py into a folder that Python can import. Make sure that you ignore your settings_local.py file in your source control.
If you add in new settings, make sure you add them into the main settings.py file. Even if they are just empty strings, lists or whatever, it will mean that when you call settings.SOME_KEY in your code, you won’t have to cope with the setting not being present. There’s nothing more tedious than writing lots of getattr code to cope with that.
One downside of doing this is that you might not be sure what your settings are on the server. At Mozilla only the system administrators who manage and deploy our servers can see the contents of that file. But it’s still helpful to check the settings on the server. For that we wrote a settings page that lists them out.
Django helps by providing a method that lists the settings, but obscures those really sensitive values:
On addons.mozilla.org we require an account to have certain privileges before showing the page. But even if that did get broken into, you wouldn’t know our SECRET_KEY or anything very useful. Here’s how that page looks:
Now that you’ve got your settings files ready, you can confidently open source your Django project safe that you won’t be leaking any key data.
In the next blog post we’ll look at scrubbing personal data from your database.
Happy Ada Lovelace Day! This day is celebrated by writing about a woman in technology who inspired you.
For Ada Lovelace Day this year, I want to honor Sheila Howell, a computer science academic at my alma mater, RMIT University. I first met Sheila in the early 90s in CS280 – Software Engineering. In that course I learned how to design and estimate systems. She taught systems thinking, a skill I have used over and over again and use to this day. I remember an exercise where we had to interview “users” (academics with scripts) about the current system, and design something better. That has stayed with me: the best way to learn something is to do it. The other key thing I learned is that there’s more to a solid system than the code.
Sheila went on to become the Head of School of Computer Science, and I went on to teach CS280, funnily enough, among other things. She was an excellent Head of School: liked by everyone but supremely good at getting things done.
She tried to retire but didn’t manage it, coming back to run the African Virtual University project at RMIT. This was a fantastic program that aimed to help African Universities set up and teach Computer Science programs.
I believe Sheila is still teaching at RMIT to this day.
I’d like to say thank you to Sheila: you inspired and influenced my career more than you’ll ever know.
One of the cool things about working at Mozilla is you get to help more than just yourself or your boss or your company… sometimes you get to help the community.
PyLadies is a mentorship group with a focus on helping more women become active participants and leaders in the Python open-source community. Following the pattern of groups like Ubuntu Women and PHP Women, PyLadies aims to provide a friendly support network for women and a bridge to the larger Python world. Both men and women are encouraged to participate.
PyLadies requested sponsorship to help more women attend DjangoCon and it felt like a good thing for the Python community. We helped send 8 women to DjangoCon – female Django developers who would not have otherwise attended (and in some cases presented) at the conference.
The results have been amazing. Some of the women who attended on the grant gave talks for their first time, while others are now inspired to submit talk proposals to PyCon 2012.
Here are details on how the money was spent.
Building the open web isn’t just about using and producing open source software; it’s also about building an open and welcoming community.
Julie Deroche seems to do it year after year. I’m not sure what her secret is, but we get the most amazing interns in the world.
Matt, Tanay and Adrian worked on important projects for Mozilla this summer and made a difference for millions of people. They were all exposed to healthy doses of Python, scalability, security and webdev best practices. I’m excited to see what they accomplish next with what they’ve learned.
In fact, before they even finished their internships Matt and Tanay won 2nd place in the first annual World Series of Hack for creating MunchyLunchy. How awesome is that?
The Mozilla internship program here is special. Interns get to:
This summer has been great thanks to Matt, Tanay and Adrian. I’ll follow up with another post about their individual projects once they’ve done their final presentations.
For now, I wanted to thank them for an awesome summer. Thanks for your hard work, making us all laugh and for making the web a better place.
Web Development at Mozilla started with two people and has become a worldwide team of 40 of the best web developers in the world. Growing so rapidly can present a number of challenges for leadership, engineers and the rest of the company. This is my attempt at sharing some lessons so you can handle growth a little better.
Have something to fight forIt starts at the heart. Mozilla is a unique company that has made things easier for us to grow well. Some things that will help you:
Goal setting and expectations are huge. First, you have to follow three general rules about goals. A good goal should be:
Second, spend time explaining what success means for your team and yourself. John Wooden said, “Peace of mind, attained only through self-satisfaction, in knowing you made the effort to do the best of which you are capable.” Which is another way of saying you may fail, but don’t fail for lack of effort. Check out his TED talk.
Here is how I’ve tried to define success (taken proudly from John Wooden):
That’s it. Here’s why:
Results are still important — but you don’t get high quality results without high quality effort. If your talent is strong and you get your team’s best effort then your team is a success. The results will take care of themselves.
Build a culture people want to be a part ofIt starts with you, but each person on the team needs to set a great example and build a healthy culture. Here are some general rules:
From a management perspective, there are some things you should try to do on a regular basis that is the foundation for a healthy culture:
I haven’t written any books on organizational development or corporate philosophy. If I ever do, I’d say the most important ingredients in successful growth are:
Calibration is what I’ve heard it called most often. But until you find what you were looking for it’s tough to judge exactly what that is. This is the case with your first hire.
My advice here is to keep trying. If you take 6 months, so be it. Your first hire is the hardest for a reason — it’s a foundation piece for the rest of your team’s future. Don’t skimp.
Tackle problems head onJenny has issues with Bill and comes to you for guidance. Your choices:
Along the way, as the team grows or things change you’re inevitably going to lose some folks. Don’t take it personally, but here are some things to think about:
Sometimes losing people is a necessity. Along the way you’ll make bad hires or have bad contractors. You need to know when to pull the plug. If you fail to do this, you’ll lose the trust of your team by empowering people who don’t pull their fair share. On a personal level, if you let things drag out, you hurt individuals by not helping them address their own issues and get better. Sometimes letting them go is the only way to really help them. Doing so takes courage, due diligence and resolve.
Take hiring seriouslyManagers understand the true cost of hiring the wrong person or letting things slide. The team should, too. On a very basic level, hiring should be up there as one of your team’s priorities if you intend to grow well:
Seems cheesy, but little things do matter. As you grow, here are some ways to make a big team feel small:
My hope is that all this helps you somehow. It’s not a magical guide to doing things right — more an explanation of a thought process and way of approaching things that worked for us.
If you didn’t read anything else, just care about people — really care about them — and see the good in them. If you can do that and work hard, nothing can stop you.
I’ll leave you with some photos I took along the way. Enjoy.
July 2006
(just kidding)
Yesterday, we pushed out Socorro 2.1, which changes the way we calculate crashes/ADU. We’ve now started including content and plugin crashes in these crash counts. This means that when you look at graphs on crash-stats.mozilla.com today, you’ll see an uptick in crashes. This is not a “crash spike” but a change in the way we represent the data, as requested by the CrashKill team.
To give some background, the crashes/ADU metric counts the number of crashes per 100 average daily users. It’s often used as a metric for stability of releases.
In version 2.2, which we are working on at the moment, we’ll be changing the way we aggregate data for beta and release versions. Right now, as an artifact of pre-rapid-release days, all beta crashes are counted under the same version regardless of whether it’s beta 1, beta 7, etc. This is turning out to be quite challenging to build when you realize that a build (and hence crash reports from a build) doesn’t know which beta it is.
My colleague Robert Kaiser has blogged about these changes more extensively, focusing in particular on what this means for data interpretation. He’s written a fantastic blog post – please jump on over there and read it for a lot more context.
While we were at Open Source Bridge we had a great time hanging out and learning with other hackers from around the open source world.
Rainer Cvillink, Video Producer at Mozilla, filmed a video where we talked about our experience at the conference and what we got out of it. Enjoy!
Thanks to our friends at OSB for putting on a great conference. Keep rocking it!
We decided to sponsor Read the Docs in a roundabout fashion.
Documentation is important. It helps us understand what it is we’re building, and since everything we do is open source it helps others understand how to use our tools or contribute to us.
Before the Great Python Revolution at Mozilla, we were documenting things in a wiki, and relying on good habits to keep documentation up to date. Good habits are hard to maintain.
Once we started using python extensively, writing things in Sphinx seemed natural. Since a lot of our projects were in GitHub, we wrote a lot of scripts to automatically build our docs and push them into Github Pages. This also suffered from hard to maintain good habits.
We eventually stumbled upon Read The Docs. An open source project that automatically takes your documentation and builds it and hosts it. Slowly we started pushing more and more of our projects to Read the Docs. Suddenly writing documentation started to pay off. Our results were immediately visible, and documentation now became a priority.
So we decided to support Read the Docs by giving them a build server. The architecture at Read the Docs makes it so that they aren’t dependent on our server, but since it’s there it will get use, and it will make building documentation a lot faster and easier. Our hope is that more open source projects will put time and effort into documentation, thus making them easier to use.
So have a look at this amazing project and start writing some docs.
Mozilla is a global non-profit organization that has one mission: to make the web better. The Mozilla web development team is hiring a number of passionate individuals to build web products that promote and support this mission.
Web Product EngineersWe’re looking for 2 individuals who are generalists that have worked the front lines as a web developer, scoped out APIs, architected thorough back-end solutions for web platforms, and sweated bullets at the command prompt during production deployments. The ideal candidates now want to apply that wisdom as a technical project manager for multiple projects, to help bring teams together to craft and deploy scalable web products.
Django DevelopersWe’re hiring 8 individuals who spend significant portions of their life in Github, lay the smack down on co-workers who commit code without tests, and whiteboard algorithms for fun in their spare time. The ideal candidates have significant Python and Django experience, and are excited about the idea of contributing to open source projects like Playdoh.
Django ConsultantsYou may already be happily employed as a consultant, and are looking to put your energies into web products that will reach millions of individuals worldwide. We need individuals and companies to turn to when we simply have too many projects to handle with our in-house staff. Please contact Mike Alexis, our Web Program Manager, at malexis at mozilla.com to put your name in his hat.
Whatever your role, if you’re looking for a job you’ll love and interested in working with fun, brilliant people, we’d like to hear from you. Thanks!
This is the second in a series of four posts on different approaches to mobile web development. For an overview of what it means to be mobile friendly, check out the previous post.
The Good
The first option I’ll discuss is the one that is the most popular by far: use user-agent detection to route users on phones to a separate mobile site, typically at m.example.com. In a nutshell, this technique uses server-side logic to solve all three goals of mobile web development at once – if the user’s browser looks like it’s on a phone, you serve them mobile content, formatted for their phone and optimized for speed.
Conceptually simple, this is the easiest option to add to an existing site, especially if you are using a CMS or Web application that supports templates. Since only the mobile-specific content, styles, and scripts will be sent to mobile users, this method also provides for the best performance out of any of the other options presented here. Finally, it also allows for completely different user experiences on desktop and mobile – they’re two different sites, after all!
The Bad
Unfortunately, this approach is not without its drawbacks. For starters, you are now maintaining two different pages for every page on your site that you would like to expose to mobile users. If you are using a CMS, is possible to arrange your site templates in a way that minimizes this duplication, but any time that there is a difference between the mobile and desktop templates is a potential source of complication in your code. Similarly, this increases the implementation time of any new site features, since you must now code two sets of front-end logic.
Even more important than that, though, is the fact that user-agent detection is inherently flawed, and anything but future-proof. Every time a new browser comes out you must adjust your algorithm to accommodate it. And false positives are particularly scary – it could be embarrassing to serve desktop users your mobile site accidentally.
When it is Right to Choose This Option 
Firstly, if your target audience includes users on older or low-end feature phones it is worth noting that you may need to employ this strategy to some degree no matter what. This is because the default browsers on some feature-phones do not support consumption of the same markup you would code a website targeted at desktop with, but instead understand formats like XHTML-MP or the older WML.
This aside, in my mind there is one case where this strategy really shines over other methods. If the functionality you would like to provide to your users on mobile devices is extremely different from that on a desktop, then using Separate Sites is simply likely to be the most practical choice. This is because you have the option of sending completely separate HTML, JavaScript, and CSS to phones and PCs.
Another case where you may be forced to use this approach is if you cannot, for whatever reason, modify your existing desktop site, and need to have a 100% separate mobile site. Though it’s not ideal, at least you have this option.
Examples
Most of the major web apps you see in the wild have chosen this path, including Facebook, YouTube, Digg, and Flickr. In fact, we picked this strategy for the mobile versions of addons.mozilla.org (AMO) and support.mozilla.org (SUMO). If you’d like to see the source code behind an example of this approach in action, feel free to check out the github repository for AMO or SUMO.
Next Week
If you’re unhappy with the drawbacks of the Separate Sites technique, stay tuned for next week, and we’ll take a look at one alternative – Responsive Design.
On April 29th, 2011, the Web Production team presented some of the changes that have been made recently to help us improve the way we build and launch websites at Mozilla. The talk was focused on bringing Mozilla employees up to speed with these changes. Topics covered included:
Your browser does not support the video element.
The processes described are intended for new projects run by the Web Production group. The process documentation is still being added to and updated, and can be found at:
[https:]
For more information about the Web Production group, please visit:
[https:]
tl;dr
There is a gap between the Mozilla organization and its community. Mozilla community members have ideas in which they would like to create sites / services that should be built by Mozilla. However, Mozilla employees do not have the bandwidth to bring those solutions to fruition.
This is a proposal for Mozilla to create an incubator program that would enable Mozilla community members to create sites and services under a separate brand. This program would detail a clear path for how successful community-incubated projects can be officially adopted by the Mozilla organization.
Mozilla Branches
Mozilla has a few entities that serve as umbrellas for various mission-driven endeavors:
Mozilla Labs is an incubator of sites and services driven by Mozilla employees. Examples include Sync and Raindrop.
Mozilla Community Services is a set of approved website-building packages that enable a community member to build out a website that serves their local Mozilla community. Examples include the Mozilla Hungarian Community Site and the Mozilla Catalan Community Site.
Drumbeat is a community of open web innovators who want to share their sites and services with others and to connect with like-minded individuals. Examples include Hackasauraus and School of Webcraft.
The Problem
The Mozilla organization promotes openness, innovation and participation on the Internet. It is a non-profit organization with over 40,000 community members that volunteer their time to help further the Mozilla mission. When Firefox 4 launched, it did so with the support of 82 languages, all of which had been localized by volunteer translators around the world.
The Mozilla community is vast, and has a number of problems it faces in organizing itself to support the Mozilla mission with the best of its ability. It has identified a number of these problems, and has approached the Mozilla organization with these solutions. However, the Mozilla organization does not have available staff members to bring these ideas to fruition.
In the past, Mozilla community members would just create their own site / service as a solution to this problem. But we’ve recently started to increase the requirements for Mozilla-branded endeavors in order to protect the security and privacy of people who are users of these products.
For example, for each website there are multiple teams that will review each website before it is launched. The Web Development team will review all code written by both internal and external developers. The Legal team reviews privacy policies and collected data to ensure that the users rights are clearly stated and guaranteed. The Security team will test the site for XSS and CSRF attacks and SQL injections, and has put a bug bounty into effect for existing websites. The QA team ensures that all aspects of the site are in working order, and creates automated tests to periodically review the state of the site.
Recently, the community-developed project Mozilla Reps was launched, and then unfortunately it was promptly taken offline. It used Mozilla branding on the site but it had not gone through this review process by the internal Mozilla Legal, Web Dev, IT, QA and Security teams.
For a community member who wants to develop a Mozilla-branded site / service, there’s currently no program that shows a clear path to becoming an official Mozilla product, and there are too many hoops to jump through in order to launch a new site / service. But we need to find a way to enable community members to incubate an idea for Mozilla with a low barrier to entry.
Community Incubation
Mozilla needs to create a new incubator program that will enable individuals to create a new site / service that should eventually by adopted by the Mozilla organization, while clearly distinguishing that this solution is not an official Mozilla endeavor.
This incubator needs a separate brand name, and one which inherently details an evolutionary path towards becoming an official Mozilla endeavor.
One of the analogies that we have been tossing around is based on frogs, where a young frog is called a tadpole, but there is an assumed evolutionary path towards its becoming a frog when it grows into adulthood. Austin King‘s idea is to call this the Lizard Incubator, where a lizard is the most easily identifiable representation of a young dinosaur.
This separate brand will enable the community members to quickly boot up sites / services without having to pass through the Mozilla gauntlet to get that site / service approved. They can use the badges associated with this brand to label the site / service as an experiment that could potentially have security flaws or privacy concerns, an acknowledgement that any user of the site should use that site / service at their own risk, meanwhile safeguarding the Mozilla brand.
This incubator could cover any new site or service that is created by a Mozilla community member, and could even be a Mozilla employee’s pet project.
There are a few side projects that were developed by Mozilla employees, like Pulse and Badger, that could stand to run through a separate brand until fully developed enough to become an official Mozilla project.
And a good example of a community-led project that would have benefited from this branding is David Tenser‘s Phoenix support site, which provided the impetus for SUMO.
With this separate brand, the community will once again be able to be true incubators of Mozilla solutions, unshackled by the confines of Mozilla regulations.
Incentives
With the new Incubator program, what the Mozilla community lacks is a real incentive to develop new sites and services under this brand, because it won’t be branded as an official Mozilla site / service. But, we must communicate to the community members what the incentive is to solve the problems they are facing as Mozilla community members.
The program will need to identify a clear path to graduation, where a community-built site / service will be adopted as an official Mozilla site / service.
I’ll propose this idea: On a quarterly basis, a Mozilla committee would review all community-built sites / services that have been branded as being part of the incubator program. The committee would identify all of the sites / services that were successful endeavors, whether that’s seen as a site / service that was a brilliant solution to an identified problem or a project that was adopted and used by other community members. At the end of the nomination process, the committee would agree on 2 to 3 sites / services that would be brought into the Mozilla organization as official Mozilla sites / services.
For example, a community member might build a successful website that was nominated and accepted by the Mozilla committee for adoption. In this case, the Mozilla Web Development team would drive the process of writing specifications for that site and would commence rebuilding that site atop Playdoh. They would engage the product owners to help oversee the development of the site, and shepherd the site through the review process by Mozilla teams.
Upon the official launch of the site, the original team who built the community site / service would be flown into California, and would be introduced to the rest of the Mozilla community at the company meeting in the Mountain View office on Monday morning. These individuals would be celebrated as true innovators for the Mozilla organization.
And if there are particular members of the original community team who stood out, those individuals would be considered as people Mozilla could bring aboard as full-time employees.
This blog post is the outcome of a series of lightning talks and pub sessions during the Mozilla all-hands gathering April 4th through April 8th, 2011, and is the evolution of the work David Boswell has done around the Mozilla family of brands and Austin King’s Mozilla Beta program blog post.
A few months ago we founded an experimental new “rapid web development” team at Mozilla. Its purpose was to take care of a growing number of short- to medium-term web projects at Mozilla. We nicknamed it the “Flux” team (for its ability to maintain a higher throughput of projects while reducing the overhead of individual projects).
The team focused on two areas in particular:
The experiment proved successful: Together with a number of internal and external groups, Flux successfully released more than a dozen projects of varying size around the release of Firefox 4 and Firefox 4 mobile.
But along the way it became apparent that the two focus areas mentioned above are different and challenging enough to stand on their owns, so we rearranged the group accordingly, spawning the Web Productions team (led by Ryan Snyder) and the (new) Flux Development Team (led by me, Fred Wenzel).
The web productions team will continue to run Mozilla-internal web projects, but has also branched out into fostering community-sourced projects and similar endeavors.
In the Flux Development group, I am working with a team of (so far) three excellent engineers (Dave Dash, James Long, and Anthony Ricaud) on web development for various parts of the Mozilla community.
The Flux team will be working on:
Say hi to the new Flux and Web Producers teams!
As readers of my blog posts know, Socorro is Mozilla’s crash reporting system. All of the code for Socorro (the crash catcher) and Breakpad (the client side) is open source and available on Google Code.
Some other companies are starting to use Socorro to process crashes. In particular, we are seeing adoption in the gaming and music sectors – people who ship connected client software.
One of these companies is Valve Software, the makers of Half Life, Left 4 Dead, and Portal, among other awesome games. Recently Elan Ruskin, a game developer at Valve, gave a talk at the Game Developers Conference about Valve’s use of crash analysis. His slides are up on his blog and are well worth a read.
If you’re thinking about trying Socorro, I’d encourage you to join the general discussion mailing list (or you can follow it on Google Groups). It’s very low traffic at present but I anticipate that it will grow as more people join.
Later in the year, we plan on hosting the first inaugural Crash Summit at Mozilla, where we’ll talk about tools, crash analysis, and the future of crash reporting. Email me if you’re interested in attending (laura at mozilla) or would like to present. The event will be open to Mozillians and others. I’ll post updates on this blog as we develop the event.
Perhaps you’ve encountered a new Mozilla Webdev beast… the project Steward. What is a Steward? What can I expect from them?
Web development work is done in three different places:
The role of a steward is to make sure a project is technically sustainable. Webdev is often the first point of entry into all the technical aspects of Mozilla Corporation’s systems. Our goal is to facilitate this process and make sure a project is sustainable for us, QA, operations, and security.
How?
The Webdev team has documented many of our standards and guidelines. A steward shares these as well as QA and security’s standards with the development team.
We provide code review. We’ll make sure we can install and run the app once, so it’s ready for QA and Operations. We look at what infrastructure is required.
We help in filing all the various bugs and navigating Mozilla’s process.
So What?Our team has inherited many projects that aren’t sustainable. These projects may have had serious security issues, or were written using a framework that our team didn’t support. These had to be rewritten or are still limping along. With our new web development guidelines, we share what the acceptable infrastructure, programming environments, and practices are today.
We cannot accept arbitrary software into our eco-system without slowing all of Mozilla down.
Ask a web developer to build a feature “Member since X” widget on their current toolkit and it will be a routine task. Asking them to build this widget on top of Tiki-wiki plus Drupal site which does authentication and profiles via SOAP (on Tuesdays, other days it’s done via XML-RPC). Development will take 5x as long and will be buggy. It will take QA much longer to test. It will take operations much longer to deploy. Security will find many issues with the basics of the project (even if the new feature is issue free) resulting in several weeks of work for all teams. Wait… all you wanted was the “member since” widget. Yikes.
Why Stewards?
Mozilla’s power to weight ratio is amazing. Our entire team is under 30 people while Mozilla has over 100 websites. Our flux team is 4 people strong with 17 projects in flight.
Stewards are Mozilla’s leverage.
Some projects have 0 hours of official Webdev staff allocated. By assigning a Steward to a project in need, we can be there for the high-level decisions and help make successful, sustainable projects.
Thoughts?It is a new role and we’re improving the process via feedback from projects.
So I’ll admit I sat through the entire event on two liveblogs. I’ve invested a bunch of time in webOS dev, and I really, really like the platform. This event would say a lot about the future direction of webOS. Hitting a home-run with the press would go a long way to ensuring the platform is around for a long time. So I was pretty interested.
My reaction, after chewing on it for an evening, is mixed at best. Some stuff was really encouraging, but vague information and poor, confusing messages continue to plague HP/Palm.
New Hardware demos: (B)All the tech looks solid. The interaction between devices is impressive. The tablet looks great, the Pre 3 and Veer hit their markets solidly, and all 3 look powerful enough to run webOS properly.
This would be a slam-dunk A if we didn’t get the vague release timeframes that have plagued Palm in the past. “Summer” (in the northern hemisphere) gives us a timeframe between 3-7 months from now. And if we learned anything from the Pre launch, people forget quickly how awesome your stuff looked in a press event a couple months ago. Even on the short end, this seems contrary to HP’s proclamations that “when HP makes announcements, it will be getting ready to ship”. Based on past experience, it’s hard to believe HP/Palm will ship these devices in short order. I’d love to be wrong, though.
Also, no carriers were announced. That’s disappointing, and may indicate problems HP is having getting interest after the poor performance of the Pre and Pixi.
New Software Partnerships: (B)Solid stuff. Skype is a big one, as is Kindle. Both are key apps that webOS has suffered without. The publisher partnerships are interesting, but it remains to be seen whether the concept apps turn into long-term publishing relationships. If Enyo comes through with its potential to be a cross-platform development framework, targeting an app for webOS will likely make more sense for big name third parties.
Support for older devices: (F)HP really dropped the ball here. Not only did they go back on a promise to bring webOS 2.0 to the Pre and Pixi, they seemed unprepared with the news. It seems clear that not everyone was on the same page, and some Palm/HP employees weren’t even aware of this. In addition, no clear plan to make it up to loyal webOS users was presented.
This would be less of an issue if carriers had been announced. Folks stuck with a Sprint Pre would at least have a clear upgrade path that gets them on the 2.0 train. For now, it’s hard for them to be optimistic.
My personal experience (which might get me in trouble for saying, but c’est la vie): webOS 2.x runs pretty well on the Pre Plus. In some ways, the apparent performance is superior to 1.4.5. I’ve heard it runs poorly on the original Pre, though.
Developer Info: (D)I was surprised at how short the developer presentation was last night. Enyo is exciting, no doubt – it’s a very solid framework that is far more flexible than Mojo, and allows for much better development workflows & utilization of superior dev tools.
So that’s awesome, but it also means that if you’ve invested a lot of time in Mojo dev (like me), you can just toss that out. There is no upgrade or conversion path. It’s completely different, and you’ll be writing your app from scratch. And whether Enyo can run on 1.4.5 devices (packaged with the app itself) is up in the air, so if you want to support the users you have now, you now have two codebases to maintain.
@webosinternals on Twitter asked a lot of questions last night about dev topics that remain unanswered. I’ll c+p here:
Even worse, devs I’ve spoken with expect to be let down by HP/Palm – they expect to get unclear, vague answers. Or silence.
Fundamentally, I have to wonder how much effort HP is putting into webOS dev relations. I’ve dealt with the folks there and they’re stand-up people, but my general impression is that they’re hamstrung and unable to communicate clearly to webOS devs. This is the time dev relations needs to be at its strongest, most forward, and clearest. Making folks rely on casual Q&A at social events, or catching Twitter posts is not how it should be done. If HP really wants a top notch dev program, they need to step up and fix this ASAP, or risk alienating their dev core.
If HP can get their communication problems sorted out and start being what I’d call “aggressively transparent” with its development community, I think we’ll get back on the right track. Here’s hoping.
As per my previous blog post, we migrated Socorro to the new data center in Phoenix successfully on Saturday.
This has been a mammoth exercise, that sprang from two separate roots:
Around that time we also had difficulty with Socorro stability. This spawned the creation of a master Socorro Stability Plan, with six key areas. I’ll talk about each of these in turn.
Improve stabilityHere, we solved some Hbase related issues, upgraded our software, scheduled restarts three times a week and, most importantly, sat down to conduct an architectural review.
The specific HBase issue that we finally solved had to do with intra-cluster communication. (We needed to upgrade our Broadcomm NIC drivers and kernel to solve a known issue when used with HBase. This problem surfaces as the number of TCP connections growing and growing until the box crashes. Solving this removed the need for constant system restarts.)
Architectural improvementsWe collect incoming crashes directly to HBase. We determined that as HBase is relatively new and as we’d had stability problems, that we should get Hbase off the critical path for production uptime. We rewrote collectors to seamlessly fall back to disk if HBase was unavailable, and for them optionally to use disk as primary. As part of this process, the system that moves crashes from disk to HBase was replaced. It went from single threaded to multi-threaded, which makes playing catchup after an HBase downtime much much faster.
We still want to put a write through caching layer in front of HBase. This quarter, the Metrics team is prototyping a system for us to test, using Hazelcast.
Build processWe now have an automated build system. When code is checked into svn, Hudson notices, creates a build and runs tests. This build is then deployed in staging. Now we are on the new infrastructure, we will deploy new releases from Hudson as well.
Improved release practicesWe have a greatly improved set of release guidelines, including writing a rollback plan for every major release. We did this as part of the migration, too. Developers now have read only access to everything in production: we can audit configs and tail logs.
The biggest change here, though, is switching to Puppet to manage all of our configurations. Socorro has a lot of moving parts, each with its own configuration, and thanks to the fine work by the team, all of these configurations are automatically and centrally managed and can be changed and deployed at the push of a button.
Improved insight into systemsAs part of the migration, we audited our monitoring systems. We now have many more nagios monitors on all components, and have spent a lot of time tuning these. We also set up ganglia and have a good feel for what the system looks like under normal load.
We still intend to build a better ops dashboard so we can see all of these checks and balances in one place.
Move to bigger, better hardware in PHXThis one is the doozy. Virtually all Socorro team resources have been on this task full time since October, and we managed to subsume half of the IT/Ops team as well. I’d like to give a special shout-out here to NetOps, who patiently helped with our many many requests.
It’s worth noting that part of the challenge here is that Socorro has a fairly large number of moving parts. I present, for your perusal, a diagram of our current architecture.
I’ve blogged about our architecture before, and will do so again, soon, so this is just a teaser.
One of the most interesting parts of the migration was the extensive smoke and load testing we performed. We set up a farm of 40 Seamicro nodes and used them to launch crashes at the new infrastructure. This allowed us to find network bottlenecks, misconfigurations, and perform tuning, so that on the day of the migration we were really confident that things would go well. QA also really helped here because with the addition of a huge number of automated tests on the UI, we knew that things were looking great from an end user perspective.
The futureWe now have a great infrastructure and set of processes to build on. The goal of Socorro – and Webtools in general – is to help Firefox ship as fast and seamlessly as possible, by providing information and tools that help to build the best possible browser. (I’ll talk more about this in future blog posts, too.)
ThanksI’ll wrap up by thanking every member of the team that worked on the migration, in no particular order:
Good job!
On Saturday, January 22, starting at 9am PST, we will proceed with the Socorro datacenter migration.
BackgroundAs most people who touch Socorro know, we have been close to capacity on the current systems in San Jose for some months. Due to space and power considerations, we chose to move to a set of shiny new machines in the Phoenix datacenter. As well as each box being significantly higher specced, we now have a lot more of them:
We’ve done a lot of work to make Phoenix superior in other respects, which I’m really excited about too. All configurations and releases are done automatically through puppet. We have added a great deal more monitoring and instrumentation to the system (with nagios and ganglia) which have allowed us to measure and monitor during our smoke tests, and will continue to give us better insights into the system in production.
For the last few weeks we have been running various tests on our setup in PHX. All QA tests pass and we have run millions of crashes through the system in smoke, load, and component failure tests.
ProcedureThe basic procedure we will follow on Saturday is:
During the migration we will continue to collect crashes, but processing and crash-stats will be unavailable. We anticipate a downtime of approximately 8 hours, but it may be longer. (In an absolute worst case scenario it would take about that again to roll back to San Jose, once we noticed an unsolvable problem.) We’ll keep San Jose static for a couple of weeks, and then it will become our test environment.
After the migration I’ll blog more about what we learned during this process. If you have any questions in the meantime, please send me an email.
David Boswell spoke at the Mozilla Summit in July, 2010 about the issues we faced with the growing collection of over 100 websites under the Mozilla umbrella. One of the issues he mentioned is that some of the websites no longer have product owners, while other sites no longer served the purpose for which they were originally intended. Some sites were created for campaigns that ran 2 to 3 years ago, and while we within the Mozilla organization know that those campaigns are no longer relevant, website visitors won’t necessarily be aware of that fact when they visit the website.
From the perspectives of the web development and security teams, we also faced security and privacy concerns from the fact that some of our website codebases were old and had not been updated for some time. While some of the websites required security updates, with the current list of priorities on the table (namely launching and supporting Firefox 4) we simply could not justify spending the time to patch any sites that were not immediately relevant to Mozilla’s current mission.
David organized a taskforce to handle the various website concerns that affected the organization, and I offered to spearhead an effort (with the help of Fred Wenzel) to wrangle abandoned websites. Our goals were to:
The taskforce decided to create the Mozilla Website Archive. The Website Archive is primarily a blog that is used to document each website that has been identified as an abandoned / outdated site and nominated for retirement. Each blog post will contain a screenshot of the site, an explanation of its purpose and when it was retired. During this process we:
Looking back, one of the lessons that that the Taskforce has learned is the fact that after a website launches, it will require organizational support for the remainder of its existence. Far too often we launch websites without consideration of the resources necessary to support future development, security updates, privacy concerns and eventually retiring the website.
Moving forward we plan to include considerations for future support in each of our site specifications before launch. Additionally, we will outline the end-of-life plan for each site. The end-of-life process could be initiated upon the date which its campaign has been completed, or upon determination that the site no longer supports the Mozilla mission or has not achieved its intended goals.
Note: this is a repost from my blog. You can find the original post here.
This is my first post to the Mozilla WebDev blog. Woot, exciting!
Table of contents: Demo
In this post, we’ll go through how to get AJAX uploads to work with Django, including:
Note: I’m planning to add upload progress. If you can’t wait for that post (understandably), there are several ways to go about it:
First, we’ll look at how the server handles files sent to it.
Model
I created an app called upload with an ImageAttachment model, like so:
from django.conf import settings
from django.contrib.auth.models import User
from django.contrib.contenttypes.models import ContentType
from django.contrib.contenttypes import generic
from django.db import models
class ImageAttachment(models.Model):
"""A tag on an item."""
file = models.ImageField(upload_to=settings.IMAGE_UPLOAD_PATH)
thumbnail = models.ImageField(upload_to=settings.THUMBNAIL_UPLOAD_PATH)
creator = models.ForeignKey(User, related_name='image_attachments')
content_type = models.ForeignKey(ContentType)
object_id = models.PositiveIntegerField()
content_object = generic.GenericForeignKey()
def __unicode__(self):
return self.file.name
This represents an image attached to a piece of content (using a generic foreign key). Pretty basic stuff. The form is ridiculously simple:
Form
from django import forms
class ImageUploadForm(forms.Form):
"""Image upload form."""
image = forms.ImageField()
View (uploading image, saving to disk)
The view is a bit more complicated, so I won’t go into the details. But you can have a look at the entire app and contact me if you have questions. Basically, the view does the file upload as you see in Django’s documentation. The function create_image_attachment deals with the part about saving a file to disk.
There is also a task for generating thumbnails, which is offloaded from the web server thread to improve performance. If you don’t need that, you can just call generate_thumbnail directly, it’s defined here.
Client sideNow, the magical JavaScript!
We’re using jQuery on SUMO, so I wrote two jQuery extensions:
<input type="file"> in a <form> and creates an <iframe> to which that form posts. To get around Django’s csrf protection, it also copies the csrfmiddlewaretoken hidden input into the form. You can’t clone a file input for security reasons (nor can you change or access its value), so you need to wrap it in a form.
input<type="submit"> in a <form> and creates an <iframe> to which that form posts.
These two pretty much summarize the process:
wrapDeleteInput()
To degrade gracefully, you want to post the file input to whatever view you’re including it to. And you can just do something like:
def some_view(request):
# ...
# NOJS: upload image
if 'upload_image' in request.POST:
upload_images(request, obj)
# ...
Thanks for reading, hope it helps!
Having recently come from the Plone world to join Mozilla, I am in the delicious but fleeting position to compare two major Python web frameworks with some pretension of familiarity. Through a series of articles focusing on specific features, I will compare the Zope family of frameworks (as they are used in Plone) with the Django framework, which is gaining popularity at Mozilla and currently runs support.mozilla.com and addons.mozilla.com. My hope is that users of and contributors to each framework will learn from the mistakes and triumphs of the other, and everyone will come out with a broader apprehension of the possibilities in this space.
Zope and Django have vastly different design philosophies, but they both come from a common—yet surprising—place: the newspaper industry. Django was born of an internal content management system at the Lawrence Journal-World; Zope, from a classified ad search system for the InfiNet consortium. It could be just dumb luck, but perhaps the quick deadline cycles and sheer volume of daily papers have a natural tendency to produce agile software for managing large amounts of content. Today, we’ll have a look at each framework’s facilities for storing and accessing that content: Django’s object–relational mapper and the Zope Object Database.
Django: relational to its coreDjango assumes a relational datastore at its very lowest levels, and that assumption percolates unashamedly up to its public API. All the typical DBs are supported—PostgreSQL, MySQL, SQLite, and Oracle—and more are available from third parties. To store data in Django, one first defines a schema in a Pythonic manner…
class Hamster(Model):
first_name = CharField(max_length=50, db_index=True)
middle_name = CharField(max_length=50)
last_name = CharField(max_length=50)
…which Django automatically turns into a DB table. These “model” objects are then created or fetched explicitly:
# Make a new Hamster:
Hamster.objects.create(first_name='Aloysius', last_name='Fearsomepants')
# Get the Hamster with the given first and last names:
h = Hamster.objects.get(first_name='Mister', last_name='Fluffypants')
Table-wide queries happen via so-called “managers” like the above objects one, special objects typically stored as attributes on the pertinent model classes. Individual row operations, meanwhile, are performed directly on instances, where attributes generally map to DB fields:
# Set Mister Fluffypants' middle name:
h.middle_name = 'Fuzzy'
Saves are done explicitly:
# Save the Hamster. This sends an UPDATE query.
h.save()
Django hijacks Python’s keyword argument syntax in a creative way to express more complex concepts such as substring matching and cross-table joins. Note the double underscores in the following:
hamsters = Hamster.objects.filter(last_name__startswith='F')
If we had added a self-referencing mother attribute to the Hamster class, we could even join the table to itself and find all the hamsters whose mothers’ last names contain “pants”…
hamsters = Hamster.objects.filter(mother__last_name__contains='pants')
…or whose mother is a specific hamster:
hamsters = Hamster.objects.filter(mother=some_specific_hamster)
In each of the previous three examples, hamsters is a QuerySet, a lazy representation of a DB query. QuerySets can be combined in set-mathy ways—unions, intersections, exclusions—and even limited using Python’s slice syntax, all without hitting the DB. If a framework is going to expose relational semantics, this is a beautiful, natural way to do it. Relational datastores are a mature technology, and SQL is ubiquitous; by mapping Python closely to SQL, a learning requirement and a complex, leaky abstraction layer are both avoided.
The Zope Object Database (ZODB), on the other hand, eschews any concept of schema or explicit reference to a storage layer at all, instead pursuing transparent object persistence über alles. The ZODB is essentially a hierarchy of pickled Python objects. There is a single root object, and others hang off it via hashable keys, just like a Python dictionary (though in practice, most ZODB make use of B-trees rather than hashes to allow better concurrency).
The hamster example might look like this in ZODB parlance:
# Make a new Hamster:
root['mister-fluffypants'] = dict(first_name='Aloysius', last_name='Fearsomepants')
First, notice that there is no schema: any pickleable Python object can be stored directly. In this case, I store a plain dict just to make a point, but one typically stores things that inherit from various provided base classes so the ZODB can notice changes. This schema-less-ness is fantastic for prototyping, but it does ultimately allow cruft to accumulate in the DB, leading to branching code and annoying database-walking migrations as data formats change.
To avoid having to walk the world to find an object, one uses the ZCatalog, ZODB’s canonical indexing solution:
# Get the Hamster with the given first and last names:
hamsters = catalog.searchResults(first_name='Mister', last_name='Fluffypants')
The ZCatalog is in no way integral to the ZODB; it rides on top, storing its indexes as objects in the database just like everyone else. The ZODB has no particular hooks that update ZCatalog; instead, client code is responsible for notifying it when an object is changed. Zope’s object modification and creation event hooks come in handy here, but it is not uncommon, when using third-party Plone add-ons for instance, for indexes to get out of sync and to require a database-walking rebuild.
ZCatalog has a much smaller API than Django’s managers, as it cannot leverage a whole RDBMS underneath. Typically, a simple query like the above will suffice, or data resides at a predictable path in the object hierarchy. If not, one must write Python code to walk the DB and find what’s needed. Similarly, there is no widely supported protocol for accessing the ZODB from non-Python languages, a continual challenge that leads to a lot of ad hoc XML-RPC or REST interfaces (though wsapi4plone fills many simple needs).
The ZODB provides ACID properties, though it’s light on the C: referential and type consistency are the application’s responsibility. It sports multiversion concurrency control and is reasonably fast, churning through several hundred transactions per second on a modest laptop.
ConclusionI’ve had a bit of a persistence odyssey these past 15 years: from custom serializers to bare SQL, from writing my own ORMs to wrestling the almost unusably ambitious peak.storage, and finally spending 4 years in the Zope world. After all that, I must admit that Django’s object–relational mapper impresses me greatly. By coupling storage and modeling concerns, it gains simplicity. This is a good tradeoff, since relational storage is mature and its limitations known—if you need a different kind of store, go somewhere else (though there are rumblings of making Django work atop key-value stores). It’s also easy on the eyes: its one-liner field definitions are about the same number of tokens one would need to initialize state in a new plain-Python object, making relational persistence almost a break-even proposition.
Zope retains a slight advantage for reckless, pedal-to-the-metal prototyping, and it’s more natural than a relational store for representing hierarchies, polymorphism, and inheritance relationships. It requires discipline, since there is no constraint-enforcing DB to keep you honest, but nothing beats it for transparency and for feeling at home in a Python program. Transparency, of course, is a double-edged sword: when it stops being obvious when you’re performing a linear scan or even doing DB access at all, performance can go out the window for the unwary. But such are the hazards of leaky abstractions, and both Django and Zope carry this warning.
If you want to learn more about Django’s modeling and storage framework, check out its Model and QuerySet documentation. For an introduction to the ZODB, see the ZODB Tutorial. I also expose some hard-to-find facts about ZODB’s on-disk format in chapter 10 of Plone 3 for Education.
Come back next time for a comparison of Django and Zope’s templating or component coupling systems!
As JavaScript codebases become larger and more complex, we start to miss the niceties we take for granted in general purpose programming languages. It would be really great to have a JavaScript module system. The CommonJS community has created a standard that Jetpack and Node.js use.
Let’s start a conversation around JavaScript modules and toolkits.
A module or package system does a couple things, but the main benefits come from:
I’ve been test driving RequireJS and Juicer; it’s a good combo providing the following benefits:
RequireJS is a JavaScript library which provides a CommonJS modules implementation and the ability to load individual JavaScript files. It’s optimized for in-browser use, unlike many of it’s CommonJS siblings.
A common performance tip is to concatenate scripts and minify them to reduce bandwidth and total number of HTTP requests. As applications mature, it’s easy to repeat yourself or include too much code in your minified file. Breaking your JS up into modules allows you to do dependency management and only include relevant code.
Editorial Note: It’s important to know which code is in which file when explaining this stuff… so I’ve included the filename in the code snippet. It isn’t actually important to how the RequireJS or Juicer systems work.
Here is a hypothetical example of JavaScript organized into RequireJS modules:
//file: search/form.js
require.def('search/form', [], function () {
return {
initialize: function () {
$('input[name=q]').focus();
}
};
});
// file: anonymous_visitor.js
require.def('anonymous_visitors', ['search/form'], function (search) {
search.initialize();
});
So the top level script anonymous_visitor.js uses a function declared in the search module. Perhaps this website has a minimal amount of JavaScript for anonymous visitors, but loads a different, more heavy weight script, for recognized visitors. By breaking up the code, the search module could be used a second time from another top level script.
There are multiple ways to use require. If a module doesn’t have any dependencies, then you can use this format:
//file: search/form.js
require.def('search/form', {
initialize; function () {
$.('input[name=q]').focus();
}
});
So far, I’ve found it best to use the “module, dependencies array, function” format even if a module doesn’t depend on any other modules. This allows a module to grow without having to change the last parameter to def from an object literal to a function. It increases consistency in the code and reduces the amount of boilerplate you have to memorize.
There is one variation. In top level JavaScript files, instead of an object literal, we return a function. This function runs once, right after the dependencies have been loaded. Really this code should be thin and just kicking things off. Think of this top level file as the “application” script and the others as reusable components.
By default, RequireJS can load code over the network. It’s got some advanced features here, but I’m not using them for by default. Imagine we’ve taken a 3,000 line JavaScript file and break it into 30 components. The overhead of 30 separate HTTP requests for JavaScript files kill our performance?
Not if we take our top level application scripts like anonymouse_visitor.js and create a minified script with all of their dependencies included. RequireJS will only try to load modules over the network, if it has not seen their definition. If we make a single big old JavaScript file out of all the relevant bits, then no HTTP requests for .js files are needed.
This is where Juicer comes in. It is a command line application for packaging up JavaScript and CSS. Django, Ruby on Rails and other systems often ship with a similar asset packager.
For JavaScript, Juicer will:
For CSS it will:
And many other handy features to improve website performance. These steps happen once at build time.
To Juicify our earlier example code, we do the following:
//file: anonymous_visitor.js
/**
* @depend search/form.js
*/
require.def('anonymous_visitors', ['search/form'], function (search) {
search.initialize();
});
The only change is the @depend and the full filename relative to the current JavaScript file.
Then running the command juicer merge anonymous_visitor.js will produce anonymous_visitor.min.js, after running JSLint against each dependency and top level script.
As a module depends on more modules you just add them to the 2nd argument and add parameters to the function (which is the 3rd argument).
During development, you probably don’t want the code minified, so you can pass juicer –min '' and it will just concatenate your files. JSLint is optional, so if you’ve had your fill of curmudgeonry for the day, then skip it with glee.
CSS is similar and doesn’t invent any new syntax. You just use an empty top level CSS file filed with @import statements. Example:
/* file anonymous_visitor.css */ @import url(search.css); @import url(general.css); @import url(registration.css);
Juicer currently has no verify step for CSS. Read the docs on the merge command for both JS and CSS as there are lots of handy flags.
Feature Request
The only major flaw with this setup is that Juicer uses it’s own convention for capturing dependencies with the @depend doc tag. You may have noticed that once we added Juicer to the mix, we started violating the DRY principal. With @depend search/form.js and require.def('appname', ['search/form'], ... we’re saying the same thing twice. if we rename form.js, we’ll have to update the code in two places.
Juicer should be enhanced to understand the RequireJS standard and load files from it. It should use an optional -d or --document-root, just like it already does for CSS. Then @depend. would be obsolete.
I should note that the RequireJS project does have a Juicer-like tool called Optimize. I liked Juicer (and some other asset packagers I tested) more than RequireJS’s Optimize, YMMV. Perhaps it’s because Optimize uses Google Closure. Perhaps it’s because I like small focused tools. If you choose Optimize instead of Juicer, then you don’t have to repeat yourself and can skip using @depend. tags.
RequireJS and Juicer give you the building blocks for a better codebase. However, they don’t give you any guidance on how to structure your CSS and JavaScript. You’ll still need to do performance testing or guesstimating to know weather you should have 2 top level scripts across your 17 different screens, or 5 top level scripts with 11 lazy loaded components… Performance tuning and architecture aint easy!
jQuery
I’m treating jQuery (and other third party libraries) as part of the platform. I don’t worry about wrapping them in RequireJS style modules nor verifying them (as JSLint hates jQuery). If you want to be consistent, some of the work has already been done for you in making jQuery compatible with RequireJS. Instead I just use an @depend to track these 3rd party dependencies.
I’ve been using this setup for about a week and I really like how it improves the code without tying me to one web development framework.
This week, we pushed version 2.2 of support.mozilla.com (SUMO). The main feature in this release was our new Support Questions feature, replacing the old Firefox Support Forum.
The Support Questions, designed by the inimitable Chris Howse, take advantage of a number of HTML5 and CSS3 features to improve the look and experience: rounded corners, background gradients, placeholder and accept attributes, and more.
We are also using a number of server-side technologies now available to us to improve performance. From the database layer—we moved from MyISAM to InnoDB tables and improved the schema—to the views—we are using Celery to offload non-real-time tasks, like email notifications and thumbnail generation, from the web server thread—we have built a system designed for speed, which we hope our users will feel.
So check out the new Support Questions on SUMO. If you’re already a SUMO contributor, we hope you’ll be as excited about the new platform as we are. If you’re not, this is a great time to start! As always, we welcome feedback and contributions. You can find all the source code on Github.
In recent blog posts I’ve talked about our plans for Socorro and our move to HBase.
Today, I’d like to invite community feedback on the draft of our plans for Socorro 2.0. In summary, we have been moving our data into HBase, the Hadoop database. In 1.7 we began exclusively using HBase for crash storage. In 1.8 we will move the processors and minidump_stackwalk to Hadoop.
Here comes the futureIn 1.9, we will enable pulling data from HBase for the webapp via a web services layer. This layer is also known as “the pythonic middleware layer”. (Nominations for a catchier name are open. My suggestion of calling it “hoopsnake” was not well received.)
In 2.0 we will expose HBase functionality to the end user. We also have a number of other improvements planned for the 2.x releases, including:
Full details can be found in the draft PRD. If you prefer the visual approach you can read the slides I presented at the Mozilla Summit last month.
Give us feedback!We welcome all feedback from the community of users – please take a look and let us know what we’re missing. We’re also really interested in feedback about the best order in which to implement the planned features.
You can send your feedback to laura at mozilla dot com – I look forward to reading it.
Paul Osman, Dan Mills, and I recently attended the Federated Social Web Summit and a commonly mentioned problem was discovery in a de-centralized environment. How do you get website A to talk to website B without a centralized service to coordinate? This post takes a stab at solving this problem with a progressive enhancement that works on Firefox 3.5 and later.
Here is a short screencast to describe an example problem and showing the solution in action.
Mo Federation Mo Problems
The basic issue is that due to web browser’s same origin policy,http://bob.example.com cannot know that Alice uses http://alice.status.net as her preferred Social Networking application for OStatus interactions. This isn’t an issue on today’s monoculture social networking web, but as we move to embrace website diversity, we need a way to discover URLs and other details that are personalized to the current user.
So what’s in the magic sauce in the video, you ask? Two JavaScript methods: registerProtocolHandler and postMessage.
After Alice logs in on http://alice.status.net, the following JavaScript code runs:
if ('navigator' in window &&
'registerProtocolHandler' in window.navigator) {
navigator.registerProtocolHandler(
"ostatus", "http://alice.status.net/follow?uri=%s", "OStatus");
}
When Alice visits http://bob.example.com/bob and clicks Subscribe, an <iframe> is created with its URL pointed to ostatus://bob.example.com/bob/follow. Note the scheme in the URL is ostatus and not http.
$(window).bind('message', function(event) {
$('input[name=profile]').attr('value', event.originalEvent.data);
});
$('body').append("<iframe id='ostatus_follow_frame' src='ostatus://bob.example.com/bob/follow'></iframe>");
Firefox invokes Alice’s protocol handler and the actual request is for http://alice.status.net/follow?uri=ostatus%3A%2F%2Fbob.example.com%2Fbob%2Ffollow. Alice is an anonymous user to http://bob.example.com/bob, but http://alice.status.net/follow receives all of her normal cookies and other headers, so alice.status.net knows she is Alice.
This handler page then uses the postMessage API to communicate safely with bob.example.com to notify it that Alice’s username is alice@alice.status.net.
<html>
<body>
<script>
//This could talk to http://bob.example.com instead of '*' for better security
window.parent.window.postMessage('alice@alice.status.net', '*');
</script>
</body>
</html>
With this last piece in place, Alice is free to subscribe to Bob without remembering arcane credentials, urls, or codes. The two federated websites are able to automatically coordinate how Alice will follow Bob.
Protocol DesignA note on the protocol design: This is a rough prototype to show the design pattern and how it solves the problem. I’ve spent only a few minutes designing the ostatus:// protocol, it is not a standard. Note: There is an existing server to server protocol OStatus 1.0 Draft 1, but this demo client side protocol isn’t a part of that spec currently.
We have to be careful with creating new URI scheme names. We should be thoughtful with protocol designs as we want to make extensible designs that can be given new capabilities without breaking old clients.
There are at least two pieces to play with: The protocol url and then the postMessage interaction. You can design very simple interactions or complex dances back and forth between site A and B via the iframe. Protocols should probably include a version number, etc.
Another design decision is the granularity of protocols: Should we have a new scheme follow:// that handles only one method? Or an ostatus:// scheme that bundles several methods? How many methods should ostatus:// include in it’s initial specification?
This demo is a simple one way communication in the iframe, but could be two way with negotiations and taking protocol versioning into account, much like HTTP is an extensible protocol (content negotiation, very few assumptions, etc).
This demo implements two actions: test and follow. This is enough of the OStatus protocol to kick the tires on the idea, there are other useful actions that would be baked into the real ostatus protocol handler.
UX ProblemA Google search for registerProtocolHandler shows 26,600 results, so it isn’t widely used or discussed currently. Why?
In the video we show what happens if the user doesn’t have an ostatus protocol handler registered. There is no way to query this ahead of time. You can detect the failure and show some help describing how to get an OStatus enabled service, etc. At the end of this post I’ll sketch out some UX improvements to make this less painful.
It would be nice if there was a navigator.isProtocolRegistered('ostatus') that returned true if one or more handlers were registered. This would help solve this issue. Update: dmose points out Bug#440620 which suggests a protocolRegistered function.
This federated website discover technique really improves the user experience for Firefox 3.5+ users. No other browsers implement registerProtocolHandler, but that’s okay as this is a progressive enhancement. Once they do, the awesome sauce will start flowing for them too. The Chromium project is also working on implementing this feature.
Assuming good UX norms can be built around how to use registerProtocolHandler-then-postMessage, It’s quite possible that this technique can be used to solve the following hard problems:
A lot of smart people have done great work in solving this problem. Other solutions that depend on a centralized service, such as Meebo’s XAuth can still be used as fallbacks… Progressive enhancement FTW.
Smoothing out the GotchasThere is some work to be done around the best user workflow. Below we will use localStorage to remember where we are at in registering a protocol handler so we don’t bug the user over and over for a missing protocol handler.
Here is a sketch at fixing registerProtocolHandler gotchas. We keep track of the protocol handler by testing it at various stages and caching the answer across page loads. Something much better can be created per website to help ease adoption and explain to the user why they see the protocol dialog.
var isProtocolRegistered = 'unknown';
if ('localStorage' in window && window['localStorage'] !== null) {
if ('ostatus_handler_state' in localStorage) {
isProtocolRegistered = localStorage['ostatus_handler_state'];
} else {
// first time... initialize
localStorage['ostatus_handler_state'] = isProtocolRegistered;
}
if ('unknown' == isProtocolRegistered) {
if ('navigator' in window &&
'registerProtocolHandler' in window.navigator) {
// Hack url below would be Status.net JS or PHP emitting the url instead of hardcoding to the user alice
navigator.registerProtocolHandler("ostatus",
"http://alice.status.net/follow?uri=%s",
"OStatus");
localStorage['ostatus_handler_state'] = 'shown';
}
}
var testProtocol = function() {
$(window).bind('message', function(event){
localStorage['ostatus_handler_state'] = 'working';
});
$('body').append("<iframe id='ostatus_test_frame' src='ostatus://alice.status.net/test'></iframe>");
var frame = $('#ostatus_test_frame');
}
if ('shown' == localStorage['ostatus_handler_state']) {
// Is the handler still working?
localStorage['ostatus_handler_state'] = 'optout';
setTimeout(testProtocol, 5000);
} else if ('working' == localStorage['ostatus_handler_state']) {
// At one time we had a working protocolHandler... is it still working?
// This handles the case were user removes handler via browser preferences
// We'll want to try to registerProtocolHandler again on next page load
localStorage['ostatus_handler_state'] = 'unknown';
setTimeout(testProtocol, 5000);
}
}
}
Your Turn
What do you think? Can we use registerProtocolHandler and postMessage (repo for short) as the new plumbing for federating web applications? If so, these primitives can be the building blocks of your future protocol designs.
Editors NoteA big thanks to korge (Wilson Lee), Paul Osman, and Ryan Snyder for editing and feedback on this post.
This week, Firefox 3.6.4 was released with out-of-process plugin support. This means that when plugins crash, it doesn’t take the browser with them.
But please remember: it’s still important to keep your plugins up-to-date. Out-of-date plugins can be a security risk.
In a previous blog, I talked about the blocklist service in Firefox and its role in keeping users informed and up-to-date with regards to plugins. It looks something like this:
As a follow-up to that post, I walked through the update process for Firefox. Here were my testing steps:
Adobe and Mozilla both need to make this process easier for users. In summary, this is how it went:
Safe to say that there are many ways we can improve this process:
Protecting users from plugin crashes is a great thing, and I’m looking forward to seeing the plugin update experience becoming just as awesome.
On Thursday June 10 Socorro (including [crash-stats.mozilla.com]) will be down for maintenance from roughly 10am – 2pm Pacific time.
We will be releasing 1.7 which involves switching over crash report storage from NFS to HBase, the Hadoop database.
Please let us know if you have any questions or problems.
Firefox has a blocklist service that protects users from malicious or faulty plugins and extensions. We’ve used this sparingly in the past but due to the success and popularity of Firefox we’ve seen more and more activity on the blocklist than ever before.
The most difficult part of the blocklist service has been deciding when to actually use it. Our policy outlines some general guidelines, but it’s not so simple when millions of users are involved because you also have to consider how you could potentially affect user experience. We have to weigh security and stability with user happiness.
In the past, being proactive with the service has been tough. Take, for example, the time we blocklisted a plugin and add-on as requested by Microsoft. Another example would be the blocking of a relatively hidden Java plugin or ancient versions of QuickTime.
In most cases we prevented an expected user interaction from being used. Whether it’s Flash on YouTube, QuickTime for movie previews or Java for applets — when users can’t do what they want to do, it’s a really negative experience. We don’t want to prevent users from doing what they want to do on the web. On the other hand, we don’t want users to be vulnerable to exploits caused by outdated plugins.
So while we were upset to know that what we did ruined the browsing experience for some people, we also knew that what we were trying to do was right and helped considerably more people than it hurt. I am proud of this — at least the idea that we are willing to do something unpopular because it’s the right thing to do. I do believe, however, that we can manage to keep users safe and happy simultaneously.
Keep people safe and happy? What a wonderful challenge.
Working on the blocklist, we’ve learned:
The threat plugins pose to users will not go away, and we will continue to fight to keep users safe. We can fight smarter, though. The blocklist service will always be here, and we’ll use it when we need to. But increasing awareness about plugins and how to keep them up to date is a much more positive and proactive approach.
A few projects cover this initiative:
With these projects, I am confident we will offer a better experience for users; keeping the web happy and safe at the same time.
Get InvolvedIf you’d like to know more and get involved we’d love to hear from you:
Achtung, Mozilla developers, a new library is heading your way!
While Mozilla’s web tools have been Open Source since the beginning, it’s become somewhat of a recent tradition for us Mozilla web devs to release web application components as reusable libraries (mostly in Python, and for the Django web framework).
Today I am introducing django-mozilla-product-details — quite a mouthful, but the name says it all: It’s a library providing up-to-date Mozilla product information to be consumed by Django projects. The data can be periodically pulled from the Mozilla SVN server using a Django management command.
From the documentation:
Mozilla Product Details is a library containing information about the latest versions, localizations, etc. of Mozilla products (most notably Firefox, Firefox for mobile, and Thunderbird).
(…)The data source of Mozilla Product Details is a PHP library kept on the Mozilla SVN server, and was originally written so it could be included into PHP projects via an SVN external. A simple
svn upwould fetch the latest data when it became available.In the meantime, the Product Details library received an additional JSON feed, allowing non-PHP projects to consume the data. If, however, the consumer is not kept in SVN like the library is, there is no easy way to keep the data up to date.
For Django projects, this app solves that problem.
To learn how to import and use the library, please read the detailed README file.
If you are looking into using the library in your project, or if you have any questions or concerns, please contact us or open an issue on github.
By the way: The geeks among you will be interested to know that we made an effort to go as easy as possible on the client as well as server bandwidth, even when the update task is frequently run. Over on my personal blog fredericiana, I wrote a blog post outlining the algorithm behind the library, and the rationale behind it.
Since this is Mozilla, it’s easy for us to focus on the client-side half of web development: we are a browser company, after all. But most of us on the web dev team spend most of our time working on the server-side, using tools like Python, PHP, Django, Kohana, Hadoop, MySQL, Memcache, Redis, RabbitMQ, and more.
Are there server-side topics or tools you’d like to know more about? Or learn about how we use various tools, or handle scaling our sites from the server side?
Let us know in the comments, and we’ll try to write about the things we do that are interesting to you.
As noted on our security blog, we’ve just pushed out a major update to the plugin check page and service.
The two core ideas are:
Les Orchard has created a backend to the plugin finder service. We’ve added another input to the call named ‘detection’ which will allow us more flexibility in how we match known releases to OS / Product / Version / Plugin / Plugin Version combos. More news at 11, but he’s built the core pieces for a self-service plugin release application.
The FrontendWe updated the JavaScript client to support ‘modern browsers’ as well as IE.
But IE 8 is a modern browser!
Hmm, well it doesn’t have a navigator.plugins object. Other popular and recently released desktop browsers *do* have this feature. Heck, even some phone’s browsers have it.
Breaking News: The platform preview of IE 9 has a working navigator.plugins object! So IE 9 fits the modern browser category… Congrats to the IE 9 team! We’ll make sure the page works by the time IE 9 ships, filed Bug#566003.
Our plugin detection uses one of three strategies:
If your goal is to protect as many users from as many known plugin vulnerabilities as possible… Strategy 3 doesn’t scale. Strategies 1 and 2 are dynamic and (in the best case) plugin agnostic. As new plugins come onto the market, the plugin finder service has to be updated, but no new code has to be written and shipped.
This is why IE plugin detection is limited.
Strategy 2 is the cleanest… and only supported by Firefox 3.6+. We would be pleased as punch if other browser vendors would create a version property. The Plugin Check page and other pieces of code that do plugin detection, will become more accurate.
We’re really excited about supporting all browsers and that is what Strategy 1 buys us. When a vendor has put a useful version number in the description or name, then it’s possible for our page to help Safari, Opera, or Chrome users understand their plugin versions better.
Geeky Aside:Fly in the ointment, even for Firefox 3.6+ we currently will use methods #1 and #2 depending on what’s best for detecting the most accurate version for the most popular plugins. Why is nothing every simple?
What can browser vendors do?
Please implement the navigator.plugins[x].version property. This exposes an explicit plugin version number from the vendor.
It will keep your users safer. This and other security tools can detect vulnerable versions easier and more accurately.
What can plugin vendors do?At a minimum, please put your full version number in the plugin description field. Also make this as exact as possible, include the build number etc. 1.1.2.9282 is better than 1.1.2. Bonus points, expose your version numbers in the version property, even on Linux builds of your plugin.
Why?We’ve built the plumbing and populated it with some popular plugins versions. Our next major release will be focused on building a self-service plugin release management app, so that vendors can populate the data for the backend API.
We’ve seen a huge spike in traffic to getpersonas.com due to our prompted update for Firefox 3.0 and 3.5 users to upgrade to 3.6. Some users may have seen a degradation in site speed and reliability during the past 12 hours.
Our amazing IT team has been hard at work keeping the site up and has fixed most of the performance issues. We also turned off search functionality temporarily to alleviate load. We’ll turn it back on as soon as we can, thanks for your patience!
We’ve released another Python library: Bleach, for sanitizing HTML and automatically linking URLs in text, that we’re using on addons.mozilla.org and support.mozilla.com. It’s based on html5lib, which we chose because of its foundation in web standards and its active development community. Bleach is available on Github and PyPI.
Bleach uses a tree-walking approach to automatically link URLs that I think is pretty interesting. I wrote a short post outlining the method.
Normally I try to chew on an idea for a post for a few days; it lets me sort out my thoughts and form some kind of thesis. I’m totally not doing this here, though, so I should preface this with a note that I could be completely off-base. But I don’t think so.
Discussion about how we interact with computers heated up recently with the introduction of the iPad. Lots of nerdy types (myself included) were frustrated that Apple had introduced not a tablet “computer,” but a big iPod Touch. They’re both computers, of course, but the way we interact with them is different: the modern computer interface uses a multitasking windowing motif, and the iPod/iPad interface is fullscreen and single-task focused.
As a Nerdy Power User, I am well-versed in how to navigate a multitasking interface, and for the most part I understand how and why it works the way it does. I, in fact, enjoy learning about the intricacies of these kinds of systems. So when I use a single-task interface like that of the iPod Touch, I frequently bash my noggin against the barriers it imposes. Copying a URL from the web browser to my Twitter client takes orders of magnitude longer than it would on OS X or Windows, for example.
What I’ve learned from interacting with most computer users, though, is that they do not give a rat’s ass about how computers work. They want to accomplish certain tasks, and will do this in the way that is most sensible and direct for them. And the way they end up accomplishing these tasks within the multitasking window motif is typically not the way I would do it.
The recent fiasco on ReadWriteWeb, where a RWW article became the first Google result for “facebook login,” is a classic example of this. And, unfortunately, so is the reaction of most Learned Computer Fellows: one of mockery and derision, admonishing the confused users for being stupid, incompetent, or lazy.
I’ll admit that I took some glee when I first saw the numerous comments on the article; I love a humorous clusterfuck as much as the next guy. But seeing some of the reactions by the Very Smart Computer People, I began to realize that We Are Not Getting It. Consider:
Isn’t this really a failure of Google? How did it become so easy to game search engine results that an article about Facebook and AOL became the first result for ‘facebook login,’ instead of the obvious thing people are actually looking for?
How is it the fault of the users when we present them with multiple, barely-differentiated text fields within the same window. Is it really surprising that they don’t understand the differences between each? And is it surprising that they choose to use the one which works with more natural language, rather than entering syntactically-unnatural domain names?
There is LOADS of anecdotal evidence that most users simply use search engines as a sort of natural language CLI. Shouldn’t we be designing interfaces that work in the way most natural for the majority of users?
These people have better things to do with their days than tweaking out the spacing in their browser toolbars. A computer for them is a utility. One that is increasingly complex, and one that is used because it’s the only option for accomplishing certain things – not because it’s a good option.
It’s kind of like the Photoshop Problem: when people want to crop a picture, we give them Photoshop. Photoshop is a behemoth application with nearly every image editing and touchup function imaginable, and it is terribly complex. Now Photoshop is an impressive tool, but only a very tiny percentage people need the power it offers. The vast majority just want to crop their ex-husband from the photo and let their friends look at it. But even iPhoto, the poster child for Apps So Easy Your Grandparents Can Use Them, continues to pile on features and complexity.
When folks need an elevator, we should give them an elevator, not an airplane. We’ve been giving them airplanes for 30 years, and then laughing at them for being too stupid to fly them right.
I think we’re the stupid ones.
UpdateAs I said at the start, I wrote this piece a bit off the cuff, so upon further review I think I could have made it a bit clearer. First, a couple great rebuttals I read:
I posted a comment on Phil Crissman’s blog, which I think explains a bit further what I’m thinking, and addresses the notion that some learning may still be required. To copy and paste myself:
I certainly don’t think that the computer can become (anytime soon) a magic box that determines our whims, nor do I think that people shouldn’t have to learn some things.
What I do think is that the current interface modern OSes on computers provide is simply overwhelming for most users, to the point that it’s very challenging to learn how to accomplish tasks without a very significant investment of time. Driving would be a good example of a task that does require investment of time, but is not so overwhelming that the vast majority of people fundamentally get it wrong: you don’t see people steering with their feet, or accelerating and braking with the radio. I’d argue that modern computer interfaces, in a rush to offer flexibility and capability, make it possible to steer with your hands, feet, teeth, and knees — and don’t make it particularly clear which one is best.
Update 2Some more responses:
Feel free to forward me others; I think I’ve given up trying to track them down for now.
Yesterday, the Mozilla webdev team released a few reusable libraries that are helping us scale out the new Django-based version of addons.mozilla.org.
So you’ve built your web application, and people are putting data in it, but how do they get that data back out? Often, they find data by searching. MySQL provides a searchable full-text index, but it has several limitations:
Fortunately, a search system, like Sphinx, allows you to overcome these limitations and provide powerful, flexible, and fast search solutions.
Sphinx, in particular, is easy to set up and use. It powers searches on both Addons and Support. We honestly didn’t shop around, because we found Sphinx did what we needed very well.
Because Sphinx is external to MySQL, it doesn’t care about things like what table engine you use, so you can perform fast, full-text searches on InnoDB tables. It also takes the load of searching off the database, leaving it free to retrieve all the documents your users can find. And because Sphinx finds its data via SQL queries, indexes can collate data in several tables.
How fast does Sphinx index data? For both Addons and Support, indexing is under two minutes, which includes on the order of 10^4 documents on Addons, and on the order of 10^5 on Support. This speed means we could easily double or triple the quantity of data before we’d need to rethink our search strategy.
Sphinx typically collects data by running SQL queries, so if you can write a query to get your data–possibly joining data from several tables, or small, extra queries for many-to-many relationships like tags–you’re done. If your data isn’t in a SQL database, you can feed it data in XML. If you have lots of data, or lots of old data that doesn’t change, you can break this into smaller indexes, or just look at what’s changed, and merge the data together.
The index includes all the text you select and several attributes you can define, which you can use to filter results, or to retrieve the document from the database.
Running searches from your application is very flexible. PHP, Python, Ruby, and others have ports of the search API. You can also search via a SQL interface. There’s even a command line search binary for testing. The API can do some pretty advanced things, like sorting, filtering, settings offsets and result limits, or even building excerpts for your results.
Search is an important part of data-driven web applications, and Sphinx is a powerful ally in providing search capabilities. If you’ve been relying on FULLTEXT indexes, I encourage you to try a search system like Sphinx. And if you need help, come find us in #webdev on irc.mozilla.org.
Early snapshot of the Plugin Directory
Most web browsers, including Firefox, offer the ability to install third-party plugins. These plugins enhance your browser by enabling it to handle new kinds of content—i.e. video, audio, even 3-D games and whole application platforms.
Unfortunately, plugins can also introduce crashes and security vulnerabilities. And even worse: Since there’s no automatic update mechanism across plugins, it’s easy to install a one and forget it’s even there. Unless you return regularly to a plugin’s download page, you may never think to update it.
With this in mind, we launched a page a few months ago to detect outdated installations of the Adobe Flash plugin, encouraging Firefox users to download an update. Soon after, we expanded the effort by launching a general Plugin Check page to cover more plugins used with Firefox.
But, since it’s our mission to make the web a better place, we want to go even further: We want to expand this Plugin Check service to cover as many browsers and plugins as possible.
As it turns out, that’s a tall order: Competing browsers offer entirely different models of collecting information on installed plugins, and plugins themselves reveal inconsistent information from one to the next. This makes a common detection scheme across browsers and plugins somewhat challenging.
Additionally, there’s no single fresh source of information about plugins to check once the installed versions have been detected. For example, while the PluginDoc site at mozdev.org has existed for years, it tends to be out of date because it’s largely maintained manually by just a few people. And, if the information was up to date, there’s no web-based service to use for automated update checks.
So, to address these issues, the Webdev team has started work on building a new Plugin Directory. We’re in the very early stages yet, but we’re hoping to make progress on goals like the following:
We can use help on all of the above as the project progresses, so stay tuned for more details. In the meantime, you can peek in on the progress so far by visiting the project wiki page, where you can find notes, demos, and source code:
If getpersonas.com seems a little bit snappier today, there’s a reason. Last night we released a new version of the website that utilizes a Content Delivery Network (CDN) that makes browsing getpersonas.com even faster.
What’s a CDN, you ask? To put it simply, a CDN moves content that normally would be served from a website in one location and stores it all around the world, making it faster for you, the user to download it. We’re using a CDN to store images, JavaScript and CSS for getpersonas.com.
Another neat trick we are using to make the site faster is far future Expires headers. What this does is tell the browser to cache our files and not re-download them every time you visit our site, saving you bandwidth and time.
This release also contains numerous bugfixes, view the full buglist if you want to know more.
Special thanks to Stephen Donner, Krupa Raj, Jeremy Orem and Matthew Zeier for their hard work in making this release possible.
Are you up for the challenge?
This week Mozilla launched the security quiz on Facebook. We encourage you to take the quiz and see how much you know about web security!
Similar to our plugin checker, the security quiz is a part of our larger effort to raise awareness about web security.
Help us spread the word and make the web safer for everyone. And don’t forget to check your plugins!
Following up on the Flash Detection on the What’s New page, we are developing an upgrade to the Plugin Finder Service (PFS2).
We could use your help! Please hit our testing server’s Plugin Check. We will be able to capture information about plugins and help fill-out the PFS2 database. See an issue? Look through current bugs and leave feedback in Bugzilla.
Screenshot of plugin detection
Some plugins don’t expose a good version number in the description, but can be detected by instantiating the plugin. We’re using Eric Gerds’ PluginDetect for this type of plugin.
On the other hand, kudos go to Microsoft’s Silverlight team for the following information: name=”Silverlight Plug-In” description=”3.0.40818.0″. That’s exactly what we need to identify when a Plugin has fallen out of date. If a vulnerability is discovered and published against 3.0.40818.0, we can alert the user to pick up the newest version.
It’s very fast and easy for us to detect your release version, when the proper information is provided by the plugin. Doing so is a win for you and your users. We’ll be encouraging Firefox users to keep their plugins updated to the latest and greatest. This means better distribution and lower support costs for you. We’re contacting many vendors right now to make this happen.
Firefox 3.6 is going to be adding enhancements to the way Plugin information is exposed to JavaScript. We’re looking forward to how this will simplify this task.
Interested in the code under development? Check out PFS2 server, PFS2 client and of course Mozilla.com where it will eventually live.
Update 10/3 @12:50 PDT: Thanks to everyone who has filed bugs! Additionally, here is the list of Plugin states, copy, and links. This is going to change, based on your feedback, but I think it will help the discussion.
| Status | Copy | Button | Link |
|---|---|---|---|
| Unknown Plugin | N/A we don’t display anything | N/A | N/A |
| Current | You’re Safe | Learn More | Vendor URL |
| Old or Unknown Release | Potentially Vulnerable | Update | Vendor URL |
| Old with Known Exploit | Update Now | Vulnerable | Vendor URL |
| Current with Known Exploit | Vulnerable No Fix | Disable Now | #disable-now |
I think the consensus is that the copy for Current and Old send the wrong message.
Folks who got the fancy lad Palm® Pre™ on opening day found the App Catalog chock full of superapps. One of them is Spaz, a ground-up rewrite of the award-winning desktop application I created for the Adobe AIR platform using pure HTML+JavaScript. Getting Spaz on webOS™ has been a big undertaking for me, and the process has certainly had it’s ups and downs. But I’m proud of the fact that we shipped a truly open source, transparent app on the first day of a new platform.
But much like Spaz on the desktop, this is not a revenue generator for the project. Everyone who works on the Spaz project is a volunteer, and we rely on motivated folks to help us make cool software. It doesn’t need to be a huge commitment — even just getting involved in the discussion and coming up with ideas is a big help. But here’s where we need help the most:
CodingWhile I intend to be the lead on Spaz for webOS™ for a while, I really could use help — I’m working in my free time, and that’s pretty limited with a family and a day job. The full source code of Spaz for webOS™ is available on GitHub, and people who want to help make new features happen are encouraged to check it out.
Even if you’re not well-versed in webOS™ coding, if you have a good JavaScript background (or don’t and are just interested in developing one), we need help to build up SpazCore, our component library that drives Spaz for webOS™, and will drive all our projects in the future. Hacking on SpazCore requires no knowledge of webOS™.
SupportWe have a brand new, very awesome support site at [spaz.tenderapp.com] which is sponsored by the awesome folks at ENTP. We need to direct people there, and we need to get involved in helping them there. We also should build up the FAQ/Knowledge base. In addition, identifying folks on Twitter who are trying Spaz or having issues and pointing them to the support site is very important.
AdvocacyDo you like Spaz? Why? First off, let us know — I’d like to build a repository of positive mentions like that. In addition, telling people about Spaz and encouraging them to check it out is great. Hwoever, this needs to be done in a nice, non-abusive, non-spammy way, and we always need to respect other applications and their developers. THIS IS NOT A COMPETITION — it’s making people aware of quality alternatives to closed, non-transparent software.
Design/UII’m an opinionated fellow and rather controlling of how the Spaz apps look, but I also know there are better designers and UX experts than me. If you’re interested in helping with design and UI work on Spaz — including additional themes — I’d love to hear your ideas.
What About the Desktop?You are not forgotten. Spaz’s desktop client is long in the tooth, and we have Big Plans for a full rewrite based on SpazCore. That’s long-term though, and in the interim work is still being done on the current Spaz codebase, including adding multi-account support and improving filtering capabilities. We also intend to transition away from AIR to the Titanium platform.
So if you want to stop bitching in your Twitter account and actually make things better, here’s your change. Drop me a line at spaz@funkatron.com and we’ll sort out the best way for you to kick ass.
I have these five promo codes from Dreamhost. They expire in three days. Apparently you get super-hot deals with them, like:
I would not sign up for 5 years with any shared host. However, I do think DH is pretty decent for small-ish sites that can work with shared hosting. I’m well aware that they’ve had a couple embarrassing public flubs in the past couple years, but I generally think they’re good folks who try to do right by their customers. Some will certainly disagree, of course. 8)
I still maintain my old account there for low-cpu, high-bandwidth/storage stuff. They do quite well with PHP hosting, and can handle Ruby and Python setups with a little effort. They’ll let you do crazy stuff like compile your own PHP if you are fine with using FastCGI, and have a nice GUI to setup SVN repos. Not many shared hosts will do the same at a similar price point.
Anyway, here are the codes. They’re first come, first serve. Yes, I make money if you sign up with one of them. I will not be hurt if you don’t use them.
I’ve been working at Purdue for over 6 years now, and I have to say that I’ve enjoyed it immensely. While the experience can vary from department to department, in general it’s a fairly low-pressure environment with a lot of opportunities to learn and develop new skills for motivated folks. Purdue offers great benefits, and the Greater Lafayette area is a pretty cool, decently progressive environment considering the size – US News & World Report recently picked West Lafayette one of the “Best Places to Retire” in the US.
A job just opened up in the Math department for a “Web Programmer/Analyst”. This position calls for strong PHP, PostgreSQL and MySQL skills; Linux and Unix fu (you should be able to admin the web stack); good communication skills (you’ll be working with a number of different people with varying tech backgrounds); and plenty of independent motivation.
I’m not involved in the hiring process at all, but I’m friends with those who are, so if you have specific questions, I can route them to the right person. Please email me directly – do not post questions in the comments.
When Google AppEngine was announced, a lot of sturm und drang went through the PHP community because it only supported Python. Of course, the same thing went through the Ruby community. The Perl community, not so much, because they’re used to getting ignored.
Anyway, I didn’t really give a crap, because Python is a cool language, and I’ve been interested in learning new languages and frameworks lately. PHP is cool and all, but having done it for 9 years, you kinda look for new challenges. Plus, getting hung up on one language is a sure way to be a boring dude who gets caught up in lame religious language wars.
I’ve actually toyed a little with Python here and there, but never dove in on a significant project. I’m someone who really likes whitespace and indenting rules and such, so those aspects of the language never bothered me. Otherwise, Python is a good fit for me philosophy-wise, because it’s totally the anti-Perl, lacking goofy magic characters and 1000 ways of doing the same thing. I can’t stand languages where “~^s” means “find all numbers that start with the letter ‘s’, add them, multiply them by 75, and then post the result to Twitter”. I’m not freakin’ R2D2 here.
So anyway, I started working on a little side project with Alex Payne on GAE. It’s been an enjoyable experience to work with a new language and framework, and there’s a lot to like about what GAE offers. It also has some rough edges, though, in some areas you’d be surprised by.
The good The webapp frameworkThe built-in framework for handling requests is webapp, a nicely straightforward system for mapping requests to handlers.
class MainPage(webapp.RequestHandler):
def get(self):
self.response.out.write('Sup dogg?')
class FooPage(webapp.RequestHandler)
def get(self):
self.response.out.write('This is the foo page')
def main():
wapp = webapp.WSGIApplication([
('/', MainPage),
('/foo', FooPage)
],
debug=True)
wsgiref.handlers.CGIHandler().run(wapp)
So we define a class to handle a specific URL request, and define methods for GET or POST or whatever HTTP commands inside those classes. Then in the main function we map URLs to those handler classes in the webapp.WSGIApplication constructor, and then start it up.
I like this because it’s simple and easy to follow. I could read the code before I’d written any myself and get what’s going on. There’s no magic happening here; no auth-mapped stuff or relationships established because I happened to name something the right way and put it in the right directory. Convention is good, and we have conventions here, but the conventions are clear and obvious.
Django templatingDjango has a really nice templating system. GAE lets you use it outside of the whole Django stack. This is cool. For example:
# define some values to plug into the template
tpl_data = {
'page_title':"This is an awesome page",
'body':"Here's the page body, sucker"
}
# get the template path
path = os.path.join(os.path.dirname(__file__), 'index.html')
#render and output the template
self.response.out.write(template.render(path, tpl_data))
Here’s a super simple index.html template:
<html>
<head>
<title>{{ page_title.striptags }}</title>
</head>
<body>
{{ body }}
</body>
</html>
Google User Auth
Writing a user system is a pain. Making people set up new accounts to use your site is yet another barrier to experiencing your amazing new Google sellout project site. You could presumably do these things if you want in GAE, but most of the time that seems kinda stupid. GAE offers you very easy integration with Google user accounts, so you can use that as your authentication system. Since everyone and their mom has a Gmail account, it’s a pretty good bet many or most of your users will be able to skip account setup.
PricingGAE is free for your first 500MB of storage and “around” 5 million pageviews. That’s plenty to host a smallish app or do initial testing on something you want to get big (because big == profit, right?). If you push past that, the pricing is in-line with similar services from Amazon. That means it’s totally reasonable to set up an app just for goofing around and learning.
PythonPython is a cool language: easy to pick up and quite powerful. It doesn’t fit some tasks as well as PHP IMHO, but it does a ton of stuff well, and its large community means there is lots of support and library development. Some stuff it does really well includes:
Recently the GAE team released the Google App Engine Launcher, an app for OS X that handles launching the development server and deploying your app to Google’s servers. You can of course script all this stuff yourself on the command line, but for folks less comfortable with that, this is a godsend. I actually did use the CLI before, but I don’t touch it now because the GAEL is just so fly. Single-click deployment is especially awesome.
Leveraging Google’s InfrastructureThey gots some big computers, rite? You’d certainly think so, and they probably have better uptime than your $5 shared hosting account. We hope.
The lame Datastore query limitations & primitive text searchThe Datastore API is GAE’s database system. It has some similarities to traditional relational DBs like MySQL and PostgreSQL, but it is fundamentally different. It stores data objects as entities instead of rows, and properties instead of columns. Objects are described by model classes, defined in Python:
class Page(db.Model):
title = db.StringProperty(required=True)
keywords = db.StringProperty()
date_added = db.DateProperty(required=True)
date_updated = db.DateProperty(required=True)
body = db.StringProperty()
Models are to entities as classes are to objects, then (I think).
The object approach has some cool flexibility, like expando models that let you add arbitrary properties to entities (very unlike rows in a relational database).
You can do SQL-like queries against the Datastore with GQL. It’s a lot like SQL, but a lot more basic. In fact, this is a summary of the entire language:
SELECT * FROM <kind>
[WHERE <condition> [AND <condition> ...]]
[ORDER BY <property> [ASC | DESC] [, <property> [ASC | DESC] ...]]
[LIMIT [<offset>,]<count>]
[OFFSET <offset>]
<condition> := <property> {< | <= | > | >= | = | != } <value>
<condition> := <property> IN <list>
<condition> := ANCESTOR IS <entity or key>
This lets you do basic queries, but it’s missing a lot of powerful stuff present in SQL. Most notably there is no equivalent to the LIKE comparison operator. That means there is no way to do partial string matches!
Okay, so you can do kludgy prefix matching, and there is an as-of-yet undocumented whole word search using the SearchableModel class, but those are primitive compared to the string matching capabilities in MySQL. Personally I would’ve thought that the search company would have given GAE really awesome text search capabilities out of the box, but for now folks have to roll their own hacks.
I guess now I have an idea of why Gmail’s search can’t match partial strings.
No “beginner” docsGAE’s docs are aimed at experienced developers who have already built web apps. Most of it is reference, and it’s a little light on examples. If you’ve not used Python before, here’s the entire introduction form the GAE docs:
For more information about Python, see the Python website and the Python documentation.
I think Python is easy to pick up, but GAE doesn’t even bother pointing out some good tutorials for beginners. Compare this to The Django Book or The Definitive Guide to Symfony, both of which aren’t really for beginners, but do a good job of helping them get up to speed.
I’m early in my development experience with GAE, so this is by no means comprehensive. Google has already demonstrated that the GAE feature set will be growing, having introduced memcache and image manipulation APIs just a week ago. I’m sure the documentation will be improved and supplemented by third parties in short order, and the issues with text search will almost certainly be addressed soon – it’s an obvious deficiency.
For now, I’m having a lot of fun working with GAE. It’s a solid platform with a lot of promise, and it offers a lot of learning opportunities for folks who want to explore web app dev with Python.