Summary
=======
On January 16th, 2011 at 10PM PST Heroku was notified of a
security vulnerability by David E. Chen, a long-time customer.
We deployed a fix to our production environment the following
day, January 17th, 2011 at 2pm PST.
We have done extensive analysis and have no reason to suspect
this vulnerability was exploited. However, we believe it is
important to let the community know about the situation and what
we are doing to prevent similar issues in the future. As a
precaution, we are working with add-on providers to change all
credentials. We also recommend that users should change any
manually set credentials in their apps as well.
Details on Vulnerability
========================
The vulnerability was a window through which an unauthorized
user could potentially gain read-only access to an app’s deployed
code and configuration variables.
We confirmed the vulnerability, determining that it was
introduced on December 28th. The underlying bug was fixed on
Monday January 17th, the day after we learned about it. It is no
longer possible to exploit this vulnerability. We do not believe
that any customer data was accessed or changed. We have
thoroughly audited our logs for that period and have found no
evidence that anyone exploited this vulnerability.
Consistent with best practices for security incidents, to
minimize the risk of a 0-day exploit, we waited 5 days to notify
the community and work with our add-on providers on a
precautionary mitigation plan.
Precautionary Actions
=====================
We believe it is important to take all prudent steps to ensure
the safety of apps. Heroku uses environment variables to provide
configuration information to apps. These variables often include
things like database passwords, API tokens, and credentials that
are used to access add-ons or other third-party services.
Although there is no evidence that these were compromised, we are
taking additional steps to protect users.
Actions Heroku Is Taking
————————
Our add-on partners have been notified of the problem and advised
to update the credentials for all Heroku apps. You can track the
status of credential changes for all add-on providers
at http://status.heroku.com/20110116-credentials. We expect all
add-on credentials to be updated within the next week.
We have already started rolling credentials for all Heroku hosted
PostgreSQL databases and expect to complete the update
this weekend.
The process of updating credentials will require restarting all
apps. While we do not expect any apps will have issues with the
update, if you do run into any issues please open an urgent
support ticket at <http://support.heroku.com>.
Actions App Developers Must Take
——————————–
Some apps may make use of hard-coded credentials in either their
source code or manually set configuration variables. As a
precautionary measure, we recommend that you update these
credentials.
Some examples of hard-coded credentials may include:
* Amazon RDS credentials - http://docs.heroku.com/amazon_rds#changing-your-credential
* Amazon S3 credentials - http://docs.heroku.com/s3#updating-your-s3-credentials
* Heroku username and password, often used by automatic scaling plugins. Visit <https://api.heroku.com/account> to change your password.
We have enabled advanced releases (http://docs.heroku.com/releases)
on all apps for free for the next 2 weeks, providing rollback
capabilities and a log of changes made to your app. To use,
update to the latest gem (`sudo gem update heroku`) and run
`heroku releases`.
If you need help or have further questions about the incident,
contact us at <http://support.heroku.com/>
Preventative Changes
====================
We are making several changes to our process and technology
architecture in an effort to prevent this type of security
regression in the future. First, we have introduced automated
regression testing to specifically check for permission issues.
Second, we have expanded our security audit review process for
all changes on the platform. Third, we are increasing the
frequency of both internal and external security reviews to help
ensure that we are continually following the industry best
practices. Finally, we are testing a new environment for
isolating customer processes from one another that will provide
a second layer of protection beyond filesystem permissions.
Reporting Security Issues
=========================
We want to thank David E. Chen for his contribution to our
community by helping us to identify this issue and working with
us to resolve it. Heroku is committed to continued improvements
to our trust and transparency. Any individuals who believe
they’ve identified a security issue within Heroku should contact
us at security@heroku.com
Sincerely
- Heroku Security Team
================================
This email was sent to mark@curphey.com.
If you do not wish to receive service notices you can unsubscribe:
http://lists.heroku.com/t/r/u/ydtlujy/tkolylyh/