Not logged in | Login

My Security Planet » Tags » Socorro

Feeds

13620 items (0 unread) in 75 feeds

WebSecurity

Security

SEO

SEO

Tools

Tools

Opensourcetesting.org News

12 items tagged "Socorro"

Mozilla Webdev

Moving Socorro to HBase

Posted: July 26th, 2010, 4:33pm CDT by Laura Thomson

Tags: Socorro [edit]

We’ve been incredibly busy over on the Socorro project, and I have been remiss in blogging. Over the next week or so I’ll be catching up on what we’ve been doing in a series of blog posts. If you’re not familiar with Socorro, it is the crash reporting system that catches, processes, and presents crash data for Firefox, Thunderbird, Fennec, Camino, and Seamonkey. You can see the output of the system at [crash-stats.mozilla.com]. The project’s code is also being used by people outside Mozilla: most recently Vigil Games are using it to catch crashes from Warhammer 40,000: Dark Millenium Online.

Back in June we launched Socorro 1.7, and we’re now approaching the release of 1.8. In this post, I’ll review what each of these features represents on our roadmap.

First, a bit of history on data storage in Socorro. Until recently, when crashes were submitted, the collector placed them into storage in the file system (NFS). Because of capacity constraints, the collector follows a set of throttling rules in its configuration file in order to make a decision about how to disseminate crashes. Most crashes go to deferred storage and are not processed unless specifically requested. However, some crashes are queued into standard storage for processing. Generally this has been all crashes from alpha, beta, release candidate and other “special” versions; all crashes with a user comment; all crashes from low volume products such as Thunderbird and Camino; and a specified percentage of all other crashes. (Recently this has been between ten and fifteen percent.)

The monitor process watched standard storage and assigned jobs to processors. A processor would pick up crashes from standard storage, process them, and write them to two places: our PostgreSQL database, and back into file system storage. We had been using PostgreSQL for serving data to the webapp, and the file system storage for serving up the full processed crash.

For some time prior to 1.7, we’d been storing all crashes in HBase in parallel with writing them into NFS. The main goal of 1.7 was to make HBase our chief storage mechanism. This involved rewriting the collector and processor to write into HBase. The monitor also needed to be rewritten to look in HBase rather than NFS for crashes awaiting processing. Finally, we have a web service that allows users to pull the full crash, and this also needed to pull crashes from HBase rather than NFS.

Not long before code freeze, we decided we should add a configuration option to the processor to continue storing crashes in NFS as a fallback, in case we had any problems with the release. This would allow us to do a staged switchover, putting processed crashes in both places until we were confident that HBase was working as intended.

During the maintenance window for 1.7 we also took the opportunity to upgrade HBase to the latest version. We are now using Cloudera’s CDH2 Hadoop distribution and HBase 0.20.5.

The release went fairly smoothly, and three days later we were able to turn off the NFS fallback.

We’re now in the final throes of 1.8. While we now have crashes stored in HBase, we are still capacity constrained by the number of processors available. In 1.8, the processors and their associated minidump_stackwalk processes will be daemonized and move to run on the Hadoop nodes. This means that we will be able to horizontally scale the number of processors with the size of the data. Right now we are running fifteen Hadoop nodes in production and this is planned to increase over the rest of the year.

Some of the associated changes in 1.8 are also really exciting. We are introducing a new component to the system, called the registrar. This process will track heartbeats for each of the processors. Also in this version, we have added an introspection API for the processors. The registrar will act as a proxy, allowing us to request status and statistical information for each of the processors. We will need to rebuild the status page (visible at http://crash-stats.mozilla.com/status) to use this new API, but we will have much better information about what each processor is doing.

We will freeze on 1.8 later this week, and expect release in about two weeks’ time.
Socorro: Mozilla’s Crash Reporting System

Posted: May 19th, 2010, 2:49pm CDT by Laura Thomson

Tags: Socorro [edit]
Recently, we’ve been working on planning out the future of Socorro. If you’re not familiar with it, Socorro is Mozilla’s crash reporting system.

You may have noticed that Firefox has become a lot less crashy recently – we’ve seen a 40% improvement over the last five months. The data from crash reports enables our engineers to find, diagnose, and fix the most common crashes, so crash reporting is critical to these improvements.

We receive on our peak day each week 2.5 million crash reports, and process 15% of those, for a total of 50 GB. In total, we receive around 320Gb each day! Right now we are handicapped by the limitations of our file system storage (NFS) and our database’s ability to handle really large tables. However, we are in the process of moving to Hadoop, and currently all our crashes are also being written to HBase. Soon this will become our main data storage, and we’ll be able to do a lot more interesting things with the data. We’ll also be able to process 100% of crashes. We want to do this because the long tail of crashes is increasingly interesting, and we may be able to get insights from the data that were not previously possible.

I’ll start by taking a look at how things have worked to date.
History of Crash Reporting

The data flows as follows:
- When Firefox crashes, the crash is submitted to Mozilla by a part of the browser known as Breakpad. At Mozilla’s end, this is where Socorro comes into play.
- Crashes are submitted to the collector, which writes them to storage.
- The monitor watches for crashes arriving, and queues some of them for processing. Right now, we throttle the system to only process 15% of crashes due to capacity issues. (We also pick up and transform other crashes on demand as users request them.)
- Processors pick up crashes and process them. A processor gets its next job from a queue in our database, invokes minidump_stackwalk (a part of Breakpad) which combines the crash with symbols, where available. The results are written back into the database. Some further processing to generate reports (such as top crashes) is done nightly by a set of cron jobs.
- Finally, the data is available to Firefox and Platform engineers (and anyone else that is interested) via the webui, at [crash-stats.mozilla.com]
Implementation Details
- The collector, processor, monitor and cron jobs are all written in Python.
- Crashes are currently stored in NFS, and processed crash information in a PostgreSQL database.
- The web app is written in PHP (using the Kohana framework) and draws data both from Postgres and from a Pythonic web service.
Roadmap
Future Socorro releases are a joint project between Webdev, Metrics, and IT. Some of our milestones focus on infrastructure improvements, others on code changes, and still others on UI improvements. Features generally work their way through to users in this order.
- 1.6 – 1.6.3 (in production)
  The current production version is 1.6.3, which was released last Wednesday. We don’t usually do second dot point releases but we did 1.6.1, 1.6.2, and 1.6.3 to get Out Of Process Plugin (OOPP) support out to engineers as it was implemented.
  
  When an OOPP becomes unresponsive, a pair of twin crashes are generated: one for the plugin process and one for the browser process. For beta and pre-release products, both of these crashes are available for inspection via Socorro. Unfortunately, Socorro throttles crash submissions from released products due to capacity constraints. This means one or the other of the twins may not be available for inspection. This limitation will vanish with the release of Socorro 1.8.
  
  You can now see whether a given crash signature is a hang or a crash, and whether it was plugin or browser related. In the signature tables, if you see a stop sign symbol, that’s a hang. A window means it is crash report information from the browser, and a small blue brick means it is crash report information from the plugin.
  
  If you are viewing one half of a hang pair for a pre-release or beta product, you’ll find a link to the other half at the top right of the report.
  
  You can also limit your searches (using the Advanced Search Filters) to look just at hangs or just at crashes, or to filter by whether a report is browser or plugin related.
- 1.7 (Q2)
  We are in the process of baking 1.7. The key feature of this release is that we will no longer be relying on NFS in production. All crash report submissions are already stored in HBase, but with Socorro 1.7, we will retrieve the data from HBase for processing and store the processed result back into HBase.
- 1.8 (Q2)
  In 1.8, we will migrate the processors and minidump_stackwalk instances to run on our Hadoop nodes, further distributing our architecture. This will give us the ability to scale up to the amount of data we have as it grows over time. You can see how this will simplify our architecture in the following diagram.
  
  With this release, the 15% throttling of Firefox release channel crashes goes away entirely.
- 2.0 (Q3 2010)
  You may have noticed 1.9 is missing. In this release we will be making the power of Hbase available to the end user, so expect some significant UI changes.
  
  Right now we are in the process of specifying the PRD for 2.0. This involves interviewing a lot of people on the Firefox, Platform, and QA teams. If we haven’t scheduled you for an interview and you think we ought to talk to you, please let us know.
Features under consideration
- Full text search of crashes
- Faceted search: start by finding crashes that match a particular signature, and then drill down into them by category.
  Which of these crashes involved a particular extension or plugin? Which ones occured within a short time after startup?
- The ability to write and run your own Map/Reduce jobs (training will be provided)
- Detection of “explosive crashes” that appear quickly
- Viewing crashes by “build time” instead of clock time
- Classification of crashes by component
This is a big list, obviously. We need your feedback – what should we work on first?

One thing that we’ve learned so far through the interviews is that people are not familiar with the existing features of Socorro, so expect further blog posts with more information on how best to use it!
How to get involved
As always, we welcome feedback and input on our plans.

You can contact the team at socorro-dev@mozilla.com, or me personally at laura@mozilla.com.

In addition, we always welcome contributions. You can find our code repository at
[code.google.com]

We hold project meetings on a Wednesday afternoon – details and agendas are here
[https:]
Socorro Moves to New Hardware

Posted: May 15th, 2009, 3:49pm CDT by K Lars Lohn

Tags: Socorro [edit]

What has two quad core 3GHz 64bit CPUs, sixteen gigs of RAM and makes the Socorro users happy? That would be the new hardware that the Socorro system moved to during a six hour operation on Thursday night. The new hardware was recommended by the folks from the aptly named PostgreSQL Experts, Inc after an intense week of consultation and analysis in March earlier this year. After auditing our existing system of hardware and software, it was apparent that we were woefully underpowered for what we were trying to do. While simply tuning PostgreSQL helped in the interim, a more powerful platform was clearly in order.

Before we deployed the new hardware, we had to take several steps to tame our voracious use of disk space. In the previous week, we removed the archived dumps from the database. They were rarely ever accessed but took up the lion’s share of our disk space. By migrating them to file system storage, we made a three hundred gig database migration onto new hardware into a migration of only sixty gig.

While there may be a need for tuning over the next week, Socorro users should have a much accelerated experience using the Socorro Web site.

Many thanks to aravind for shepherding this project through IT, chizu in IT for his db cloning/replication scripting/tweaking and jberkus from PostgreSQL Experts for his superior navigation skills and a steady hand at the PostgreSQL tiller.
Crash Reporter Homepage Reskin

Posted: March 2nd, 2009, 2:02pm CST by aking

Tags: Socorro [edit]
The crash reporter has been given a new look, and the homepage has a new Dashboard.

Our UX Engineer Neil Lee has applied some simplifications to the query form. This redesign was focused on the homepage and global navigation.

Another new feature is that MTBF and Top Crashers By Signature can be exported in CSV format. In the future, as we want to slice and dice different reports, it should be trivial to add this feature to other reports.

In addition I’ve fixed a handful of issues:
- 428110 - Quick and dirty changes to speed up crash analysis
- 478043 - Make ‘is exactly’ the default choice
- 479256 - Clarify labels to be Date Processed
- 470524 - Crash signatures not indented
- 479460 - Bad Unicode in User Comments
- 479447 - report/list with no results has JS error
We would love your feedback. Check out some recently filed bugs or send us some feedback and file a new Socorro bug.

We’ve had some known issues around MTBF and Top Crashes by Signature in the last month and are working on fixing these issues. The upside is that SeaMonkey is now in MTBF.
Socorro Partitioning Rolled Back

Posted: February 2nd, 2009, 12:29pm CST by morgamic

Tags: Socorro [edit]
This Thursday and Friday we attempted to push updates to re-partition our crash report database and optimize the reporting tool to take advantage of it. This was the deployment of bug 432450 and a fix for bug 444749, among others.

Our first attempt suffered from a network timeout, which required an eleven hour restore and re-run. The re-run, done Friday, was done using a socket connection but would have required an additional 1-3 days of downtime, which was well outside our originally announced window. Consequently, the database was rolled back to its contents as of 6:55PM PDT, January 29. Reports have since resumed processing.

We plan on doing the following:
- Set up a complete replica of production to test this process end-to-end. Our dry runs were done on a staging database that was roughly 1/5 the size. We anticipated a scaling of O(n), but in practice on the production server, we got performance more inline with O(n^2). So we did not expect the full extent of timeouts or how much downtime would be needed. This will be avoided in future updates and we are setting up a stage database from a recent dump (once we gather the hardware for it).
- Push a now+ partitioning script. The work done in bug 432450, on top of a complex migration script for old data, has logic for handling new partitions automatically which benefits new reports. Since we don’t want to keep adding to our old database schema, we will push these updates so that new reports are properly partitioned. Pros - in a week or two, things will be speedy and we aren’t going to struggle with timeouts. Cons - we aren’t migrating the last 4 weeks. We will not see a performance increase when querying data older than the date of the repartitioning.
We would like to push the partitioning script (without migration of old data) on Thursday. We will announce when it will be as soon as we know.

Long term, we are already in the process of seeking additional resources to help examine our database configuration and systems architecture. We will have more updates on that process in the future.

Our team wants this work deployed as much as everyone else. Thanks to everyone for their patience as we work through these issues.
Socorro Database Partitioning is Coming

Posted: January 20th, 2009, 3:20pm CST by K Lars Lohn

Tags: Socorro [edit]
How big can a table in a database get? Well, the answer varies by database, for most modern databases, the answer is “really huge”. That’s what we’ve going in Socorro, some honking big tables. Queries can get slow on big tables. Sure you can add indexing to prevent having to scan the whole thing for the most common queries, but you can’t index every column without slowing performance elsewhere and using up ever more disk space. Indexes can get really huge, too.
- tables divided into sub tables
- queries optimized to use smaller subtables
- ad hoc queries sped up
- summary queries ignore irrelevant data
- smaller indexes
- simplified data retirement
- fresh lemon scent*
Partitioning, a feature offered by many RDMSs, is a trick to help manage titanic tables. You take a table and break it up into smaller tables, each containing a part of the whole. Say we have a data set called ‘reports’. Rather than storing them all in one table, we store them in several smaller tables. The data in each smaller table share a common trait. For example, for reports from Week 1, Week 2, and Week 3 would each have their own table. The master table ‘reports’ physically has no data in it at all. The database knows that when we reference ‘reports’ it means the union of all the smaller tables. If we do a query on the ‘reports’ table and we ask for reports from January, the database is clever enough to just look to the weekly sub-tables for January instead of looking at all the sub-tables.

We’re currently converting the Socorro database into a partitioned database (okay, technically it was already partitioned, but its partitioning was degenerate and didn’t work properly). We’re testing a Python script that is going to take the ‘reports’ table, along with the associated ‘frames’ and ‘dumps’ tables, and start breaking them into little one week chunks. Unfortunately, because of the massive size of the tables, we cannot afford to have two copies of the data in the database at the same time. The chunking of the data will be destructive. After a week of data is copied into a new partition, that corresponding week of data will be deleted from the original table.

The database, the Socorro Web App, breakpad crash processing and aggregate analysis will be down during the migration process. However, data collection will not be down: we’re not going to lose new crash data.

If we were to chunk the entire dataset, the migration process is estimated to take more than twenty hours. As a compromise, we’re going to chunk only the most recent four weeks of data and leave the rest as a single oversize partition. This will significantly reduce the time that migration takes and, therefore reduce the down time. We can get away with this because most of the aggregate reports only look at the most recent few weeks of data anyway.

Another advantage to partitioning is in the retirement of old data. In the future, we’re probably only going to keep at most one hundred twenty days of history. Any more than that and our storage needs would require its own building. To get rid of the old data, all we need to do is delete the oldest partitions. That action is fast because it doesn’t even require looking at indexes or scanning tables.

Partitioning is going to allow Socorro to scale much more smoothly. At the same time, it will make our aggregate reporting much more efficient.

This repartitioning process will happen within the next week. We will announce the scheduled down time in advance. And be assured, because of the file system changes what we made during our last big Socorro update, data collection will not be down while we’re repartitioning.

* also available in fresh wintergreen and sparkling pumpkin
Top Crashers By Url and MTBF

Posted: December 30th, 2008, 5:06pm CST by aking

Tags: Socorro [edit]
Working with ss and chofman, we’ve created 2 new types of reports: a Top Crashers by Url and a Mean Time Before Failure (MTBF).

Given the current state of performance of the non-report parts of Socorro’s webapp, most of the thought and time have gone into the backend piece of these reports. You can read about the ReportDatabaseDesign on the project’s wiki.
Top Crashers by URL

On which websites do our browser builds crash the most? Which curses do our users hurl at us when this happens?

This report uses the optional url feild of a crash report to answer this question. It has two modes byurl and bydomain. You can read more about the details on TopCrashersByUrl. Crashes which have a comment, include the comment and a link to the actual crash report. Don’t worry, personal details have been removed, we don’t tie a specific user to a specific url.

We will be putting links into Socorro to these new reports, with the work neilio is doing, but for now here are various links.

We’ve enabled top crashers by URL for Firefox 3.0.5, 3.1b2, 3.1b3pre, and 3.0.6pre. Each of these link to “by domain” breakdowns, so 3.0.6pre has a link to this by domains view.
MTBF

Is this new release more crashy than previous releases?

Squeaking in before New Year’s Eve’s MFBT comes the MTBF report. It is a graph of the average number of seconds a release runs before crashing. Details are at MeanTimeBeforeFailure on the wiki.

We’re running MTBF reports for 14 releases:

Firefox major, milestone, and development releases.

Thunderbird milestone, and development releases. (No Milestone releases in Socorro yet)

Coming Soon: SeaMonkey

These reports are for a release in general as well as stats for Mac and Win, allowing for drilling down into OS. Several frontend enhancements to this report are coming.

, Product and versions in these reports include:
- Firefox 3.0.4
- Firefox 3.0.5
- Firefox 3.1a2
- Firefox 3.1b1
- Firefox 3.1b2
- Firefox 3.0.4pre
- Firefox 3.0.5pre
- Firefox 3.0.6pre
- Firefox 3.1b3pre
- Firefox 3.1b2pre
- Thunderbird 3.0a3
- Thunderbird 3.0b1
- Thunderbird 3.0b1pre
- Thunderbird 3.0b2pre
I’ve gotten a good dose of feedback on tweaks to make and bugs to fix, but hopefully you’ll find these new reports useful. Tomcat has already mentioned augmenting his list of urls to populate his test automation for 3.1 (using spider to test most popular urls) with the urls in these reports.
Socorro wireframes

Posted: December 4th, 2008, 11:52am CST by nlee

Tags: Socorro [edit]
As part of our ongoing work on the Mozilla crash reporting system (codenamed “Socorro”) a redesign of the entire interface is in the works, and I have some preliminary wireframes to share for feedback and discussion.

My personal goals with this redesign are to make working with crash data more efficient, and to make each screen as useful and as intuitive as possible. This project is an interesting challenge, however, as there are not a lot of publicly-available examples of crash reporting systems to use as a baseline.
“Home” page
Current page: [crash-stats.mozilla.com]

Currently when you go to the Socorro home page you’re dumped right into search. This makes a lot of sense, but there could be a lot more useful data right up front. This wireframe tries to incorporate more of a “dashboard” approach with top crashers for release versions and other pertinent information.

Home page, default configuration

Home page with advanced filters visible

Some key changes in this wireframe:
1. Search is now called “Filter” as that’s really what you’re doing.
2. The filter options have been cleaned up a bit, and more specific filters are now grouped under Advanced and hidden by default.
3. The Advanced Filters toggle will remember if it was opened or closed, so if you regularly access any of these options they will be easily accessible.
4. There is a new “top crashers” widget that shows the top 3-5 reported crashes for current release versions.
5. The boxes underneath the filter / top crashers widget are for other chunks of data such as top crashers for development versions, mean time before failure, top URLs that cause crashes, etc.
6. The versions filter auto-fills with just the versions available for the selected product, to help keep the number of options down.
7. The “Mozilla Developers” button at the top right-hand corner opens a jump menu that lists all of the Mozilla developer web sites / tools. I think it’s kind of silly the various Mozilla developer sites aren’t linked together and this navigational tool addresses that deficiency.
Top Crashers
Current page: [crash-stats.mozilla.com]

At the moment the existing page is nothing more than a jumping point to link you to the various product versions. The redesigned wireframe tries to float up crash report information for more commonly used product versions while whittling the number of versions down to a more sensible number.
Individual Crash Signatures
Example: Crash reports in mozcrt19.dll@0×1838a

The redesign does away with the tabbed interface currently in use and brings everything onto one page for quicker access. The top left-hand box displays either the number of crashes by operating system (for release versions) or the number of crashes by build (for development versions).
Give us your comments, your feedback, your huddled criticism
As I mentioned these are quite preliminary and I’m very interested to hear your comments and whether you feel these wireframes are headed in the right direction.

Redesigning a tool such as Socorro is rather challenging as there aren’t many existing examples of good design in this area (or many publicly available examples at all for that matter). I also had to keep in mind that whatever is created needs to work for both Mozilla’s specific requirements as well as be generic enough to be adaptable by others, which makes this redesign even more tricky.

I spent quite a bit of time looking at similar or parallel systems such as network and web hosting dashboards to get some ideas for how this information could be displayed and these wireframes incorporate some of my discoveries.

Beefs, bouquets, comments, suggestions?
Socorro Project Update

Posted: November 20th, 2008, 11:33pm CST by morgamic

Tags: Socorro [edit]
There has been a lot going on for the Mozilla crash reporting system. Here are some quick links for project activity:
Upcoming changes:
- Socorro site redesign - Neil and Austin are working with Sam and Chris to work through crash analysis to understand how the system is used to track down the cause of certain crashes. Structure, navigation and layout will be redone to best accommodate common uses.
- Socorro database repartitioning and migration - Lars is working hard on refactoring our partitioning to better support the kinds of queries we make. The end result is better long-term maintainability and improved performance.
- New reports - MTBF reports, URL reports - Austin is working on two new report types to provide better information for crash analysis.
Current issues:
- Query timeouts - this usually happens when a user does a complex query (either no product selected or a search over a period longer than 4 weeks).
- There is no dialog saying the app is working and connections time-out leading a user to simply hit refresh to try to get where they want to go. However, many of the queries taxing the system were non-specific queries over 3-month periods. In some cases these jobs would timeout in the reporter but continue for 30 minutes or more in the database. To prevent this from happening we pushed a revision to the search form to limit query breadth.
- Search engines were crawling Socorro so we added a robots.txt file to prevent this from happening
Learn more:
- Join us in #breakpad @ irc.mozilla.org
- Checkout the code and read the documentation to setup, install and hack on Socorro.
Crash Stats updated with Flot

Posted: October 27th, 2008, 11:03am CDT by aking

Tags: Socorro [edit]

As we headed closer and closer to Día de Muertos, do you ask “where can I mingle with the dead (processes)”? Why at Mozilla’s crash reporter tool, you’ll find Firefoxen, SeaMonkeys, and Thunderbirds who have crossed over and yet still linger…

Last week we updated the plotting library we use for graphs to use Flot and we fixed a handful of Socorro webapp bugs. Flot is a good choice where we have simple graphs to display. Below are two graphs from the crash report listing page: An new crashes by OS bar chart for production builds, and an updated crash by build, for development builds.

Also using Flot, we’ve revamped the server status page, so that we can see the health of the Socorro system. Socorro is one half to it’s sibling Breakpad which, together, provides an Open Source client and server crash reporting system. Let’s see if crash-stats is happy…

This graph is powered by a new cron job which checks the server status based on several statistics related to processing crash reports. Another cron job we’ve worked on is topcrashers.

In coming weeks we’ll be bring new life to the crash reporter web app and improving this unique and important part of how the Mozilla dev and qa teams work. We will be focusing on making it easier to use by providing new reports, easier navigation, and better visibility into crashes. What do you want to see in your crash reporting tool?
Socorro Database Migration

Posted: September 22nd, 2008, 12:22pm CDT by morgamic

Tags: Socorro [edit]

In our last post we described one of our Q4 goals as:

work with PostgreSQL to update and optimize our hardware and software configurations

Aravind has gone ahead and migrated the Socorro database to its new hardware and we’ve upgraded PostgreSQL to 8.3 in order to take advantage of performance optimizations that went into the most recent release. We’ve also updated the configuration so xlogs and logs are written to a different data store than the main database.

Unfortunately this required a full migration over the weekend, which required some downtime. This morning we are running a full vacuum which will also take a considerable amount of time to finish. There is no ETA, but we will post an update when it is completed.

Update: We have postponed the full vacuum/analyze until a maintenance window later this week — it has taken way too long to complete and can’t afford to lock tables for this long.
The Crashing Edge

Posted: September 17th, 2008, 7:45pm CDT by morgamic

Tags: Socorro [edit]
This will be the first of a weekly blog about the crash reporting system — much like the other *edge blogs out there.

In short, we’ve rewritten most of the system to accomodate throughput that is more than 10 times the projected traffic. It is not because Firefox 3 is crashing more — we are seeing increased traffic because:
- Client-side throttling has not been effective
- Overall number of users has increased
- Mac Intel builds now submit crashes properly
What have we been up to?
First, two bug lists:
- Bugs closed between the last blog post and now
- Socorro 0.6 — our next milestone, targeted for early October
Here are some issues that have been addressed in the last 4-6 weeks (not all of which had bugs):
- server-side throttling now possible and configurable
- improved separation between monitor and processor jobs
- updated collector now saves unprocessed crashes grouped by hour
- mysterious death of main monitor thread resolved
- collector, monitor, processor now use a common config
- SVN structure simplified, no longer required to use /dist or /scripts/dist-* scripts to copy things inside SVN
- Python/Pylons reporter replaced with PHP/Kohana reporter
- web application now clustered with memcache support
- removal of SQLAlchemy from all tiers of the system
Known Issues
Known issues are mostly listed in the 0.6 buglist. If you know of a problem that isn’t already filed please file it to help us keep track of things. Pressing issues with the reporter are:
- database bottleneck
- inefficient queries
- corrupted summary tables for topcrashers
What’s Next?
Our goals as we move into Q4 2008 are:
- work with PostgreSQL to update and optimize our hardware and software configurations
- move expensive aggregate queries to cron jobs, summary tables
- move focus from performance and maintenance to feature development (e.g. new reports, graphs)
- fully document the system to make sure it is easy to contribute or learn about how this works
In the next week, we will be focusing on documentation and making the reporter usable. As we resolve our scaling issues, if anybody is interested in revamping some of the UI, you are welcome to jump in. Find us in #breakpad on IRC.