Support all your favorite nonprofits with a single donation.Donate safely, anonymously & monthly, in any amount. It's a smarter way to give online. Learn more
The Sunlight Foundation uses technology and ideas to make government transparent and accountable.
The Sunlight Foundation is:
A think-tank that develops and encourages new policies inside the government to make it more open and transparent.
A campaign to engage citizens in demanding the policies that will open government and hold their elected officials accountable for being transparent.
An investigative organization that uses the data we uncover to demonstrate why we need new policies that free government data.
A grant-giving institution that provides resources to organizations using technology to further our mission and create community
An open source technology community that revolves around the Sunlight Foundation’s core mission
Last week, a flock of Sunlighters flew north to join fellow Pythonistas in the community's annual celebration of hacking in the language we love. As a bonus, this year's conference was held in Montreal, giving us a chance to learn some Français Québécois, sample the finest poutine and hang out with local Montrealer and OpenNorth founder James McKinney. As usual, we learned a lot from the excellent talks, and had a great experience at the PyCon development sprints.
The State of the Python Community is Strong
The Python community continues to earn its reputation as an inclusive and eclectic one, where hackers, researchers, newcomers and cutting-edge developers are both teaching and learning from each other. We're happy that Sunlight Labs has attended for many years, and we're better off for the experiences that we have at such an exciting conference.
The tone was set early this year. An uplifting keynote from John Perry Barlow, former songwriter for the Grateful Dead and co-founder of the Electronic Frontier Foundation, who seemed to be speaking directly to us at times:
"...I dream of a day where the right to know is understood to be a natural human right that extends to every being on the planet who is governed by anything. The right to know what its government is doing, and how, and why. And I depend on people like you, who are building the plumbing."
Other keynotes that stood out to us were Fernando Pérez's talk on the amazing IPython project and Jessica McKellar's talk on the Python community's role in encouraging a diverse population of American kids to engage with computer science in meaningful ways. These talks had a natural synergy, because IPython has given both teachers and researchers new tools with which to communicate and document good code practices as well as subject matter in fields like math, science and statistics. When Kaitlin Devine set up Python training for Sunlighters, her course made good use of the IPython notebook as a tool for engaging those new to coding.
The Open Spaces at PyCon, an unconference going on in parallel to the scheduled talks, also featured examples of excellent community building. The Geekfeminism and feminist hackerspace meetup, Trans* Pythonistas and Allies meetup, and Regional PyCon organizer meetup are examples of conscious efforts to build a broader community that is welcoming to traditionally under-represented groups, and share ideas about the future.
Other sessions at Open Spaces included the "Birds of a Feather" (BoF) meetups. At the Data Workflow BoF, we saw some new offerings in shareable data visualizations from plotly, a video analysis tool used at Mozilla for product testing and [lifelines], a survival analysis package developed by Cameron Davidson-Pilon (more here). At the Vim users BoF we met up with our PyCon buddy and fellow DC developer Matt Boehm, developer of the excellent unstack, a plugin to help decode Python stack traces. A few other excellent plugins to brag about to your Emacs frenemies: vimdeck for terminal-based slide presentations, vim-ipython for integration with IPython, and syntastic for all your code highlighting and linting needs.
Open Data Sprinting at Pycon
Finally, we're happy to report that there was not one, but two Open Gov / Open Data BoFs! We organized one on Saturday night and also stopped in to talk with a separate one the following day. The second BoF was organized by Sarah Bird of Aptivate, who was working on Open Data Comparison a project comparing metadata across different open sources of government contracting data. We all decided to sprint together, using a shared hackpad and collaborate where we could.
During the sprints, James tried out some of the new devops software that was generating buzz at PyCon, including Ansible and Vagrant. Lindsay prepared FARA data for mapping. Paul made some new converts, both to Hy and to government transparency. Daniel worked on tests for the python-sunlight package, which is a wrapper for Sunlight's APIs.
This year's sprints also saw the beginning of Bob's influence-usa project as a community-driven effort. Michael Mulich contributed data validation and import scripts to our work on federal lobbying disclosure. Aaron Rothenberg lent a hand in identifying the availability of campaign finance data in various states, and wrote some initial scraping code for Ohio. Finally, we were lucky enough to receive some advice from Catherine Devlin, whose ddl-generator project (still under development) will be of immense use to projects like influence-usa and the unitedstates org. Because the ideal output of our transformation scripts is json text files, having a tool that suggests SQL create and insert statements will target a general need for RDBMS users that want to ingest that data.
Our favorite talks
So many talks, so little ability to exist in multiple rooms simultaneously! There were some times when choosing between two co-occurring talks was difficult, but thankfully PyVideo had most talks posted very promptly -- even the same day in many cases. Here are our staff picks:
Paultag's Hy talk
Coming in at an obvious #1 is our very own Paul Tagliamonte's "Getting Hy on Python". Call us biased, but Paultag commanded a very engaging talk despite difficult material and technical difficulties. Many attendees rushed the stage to discuss Hy, a Lisp that compiles to Python, with its creator afterwards.
- Software Engineering Research for Hackers: Bridging the Two Solitudes: A fantastic introduction from Tavish Armstrong to falsifiable statistical research that aims to find conclusive evidence to many claims made about developer productivity and organizational decisions.
- Hello Physical World: A Crash Course on the Internet of Things: Katherine Scott makes a Skinner box using Python, some simple electronics, and a RaspberryPi
New Tools for Python Developers
- What is coming in Python packaging: This talk was planned as a sneak peek, but ended up as a year-in-review, because development proceeded so swiftly on projects like PyPI, pip, devpi, and twine.
- Fan-in and Fan-out: The crucial components of concurrency: The asyncio module in the standard library of Python 3.4 introduces single-thread concurrency to python. This talk rocked Bob's world.
- The State of Crypto in Python: Drew says this is "almost too good to be true," and recommends checking out the recipe layer of cryptography.
- Designing Django's Migrations: The most significant addition to Django since 1.0 is the addition of database migrations in soon-to-be-released Django 1.7, and James learned a lot from this overview.
How-to's and Surveys
- Subprocess to FFI: Memory, Performance, and Why You Shouldn't Shell Out: Christine Sprang talked about why CFFI, a "Foreign Function Interface for Python calling C code." Basically, you can wrap a C library while only writing a small amount of C surrounded by Python. She also explained how subprocess can be bad.
- Postgres Performance for Humans: The ever-informative Craig Kerstiens advises us not to run our own Postgres server, but tells us what to do if we have to.
- See Docs Run. Run, Docs, Run!: Co-sprinter Catherine Devlin survey's the state-of-the-art in Python documentation, including a demo of dexy, a new and extremely powerful platform for thoroughly blurring the line between code and documentation.
- Developing Flask Extensions: Great introduction to Flask and how-to for creating your own extensions
- Character encoding and Unicode in Python: How to (╯°□°）╯︵ ┻━┻ with dignity: Paul called this a great overview of the topic, outstanding talk for anyone wanting to learn about handling Unicode properly in Python
- Django for Web Designers and Front End Developers: None of us actually caught this talk, but we've heard good things and look forward to checking it out in a special screening with our intrepid design team.
Computer Science Concepts
- All Your Ducks In A Row: Data Structures in the Standard Library and Beyond: How Python uses memory and why the list object is the most dangerous one in the standard library! Very accessible intro to Python internals from Brandon Rhodes
- Computer science fundamentals for self-taught programmers: Lindsay (a recent migrant from our reporting group to Labs) appreciated this introduction to Big O notation, linked lists, and search algorithms. Bob, originally a double major in English and linguistics, has added it to his reading list for this weekend.
- Garbage Collection in Python: How CPython and pypy handle GC, and how you can configure an alternative in pypy. A bit more advanced, but a good intro to some low-level optimization concerns.
- Pickles are for Delis, not Software: Alex Gaynor's talk was, as expected, educational and engaging. Alex tells us why we shouldn't rely on the pickle object, with examples from pickletools.dis, a tool that shows the innards of Python's native serialization object.
Our Favorite Posters
I have a confession, something I don’t usually say in public: I enjoy looking at datasets on the weekend. I love playing with data and I love seeing what developers can create with data.
I am a data geek. It’s safe to admit now, since I’ve already married and started a family.
But my love for data is only one reason I have devoted my career to the open data movement. A bigger reason is that I believe open data can actually improve democracy.
That’s a pretty big statement and I know there are some doubters out there. But I’ve seen enough examples to know it’s true. The really exciting part is that we are still in the very early days of open data.
Even today, most of the cities opening their datasets are large and have money to invest in technology and try things. There are some smaller, leading cities, such as Palo Alto, Calif., but so far they are the outliers.
It’s not that cities don’t want to open up data. Rather, they either simply feel it’s out of reach for one reason or another, or they cannot yet measure the impact of an open data program.
Open data must be made simple
At Junar our main focus has been on making open data so simple that a lot of small and midsize cities – cities that aren’t usually on the leading edge and don’t think they have budgets to embrace open data – can have it. Junar is a cloud-based data platform that enables businesses, governments and other organizations to free their data in order to drive new opportunities, collaboration and transparency. The platform works by collecting, enhancing, publishing, sharing and analyzing data.
Think about the democratization of content on the web. What made that possible was the emergence of low-threshold, easy-to-use content platforms. If I ask you to go to the web and publish content of 140 characters or less, it’s obvious which tool you would use and that tool is so simple. If I ask you to create a blog, you know how to create an account and start doing that. It’s easy.
But if I ask you to go publish information that fulfills the definition of open data – usable format, easily sharable, proper structure, enriched metadata, APIs, etc. – what is the application that comes to mind? Probably none, because until very recently there was no easy way to publish open data.
But it is already getting easier and there will come a day when it gets easier still. When it does, we’ll really start to see the transformation of democracies.
How data transformed a city in Argentina
I’ve seen this first-hand while participating in an open data project in Bahía Blanca, a port city and a center for petrochemicals in my native Argentina, where not all things are transparent, so to speak.
A few years ago, some smart, brave people leveraged open data to shed light on official corruption. A new mayor came in, determined to change things and he hired a great CIO.
They started an open data portal that I can tell you has completely changed the status quo and helps the government engage citizens on a different level. They published all kinds of data – even some salary information – and they created an API that allows apps to send out a lot of data about the environment. They brought all this data to life and it is now bringing a lot of applications to life.
Now, you see a lot of young people in Bahía Blanca (a city of about 300,000) sensing that they can change government and this is generating an effect across Argentina.
Other seeds of progress
Even as open data is transforming government, other seeds are beginning to take root. In the United States, we’re starting to see hard evidence that open data is having profound economic effects.
Joel Gurin’s recently released an Open Data 500 study at New York University which shows that a “data economy” is, in fact, taking shape and that data spurs economic development wherever it is set free.
It’s amazing to think that it is all starting with the transformation of democratic governments. In this case, government is leading the pack. Many other organizations – NGOs, academia, corporations, media companies – are soon to follow, once they discover easy ways to open up valuable data they have compiled.
And that will be very good news for data geeks like me.
Diego May is the co-founder of Junar, a global company with offices in Dallas, Silicon Valley and Latin America. Junar provides a cloud-based open data platform that enables innovative organizations worldwide to quickly, easily and affordably make their data accessible to all. Using the Junar platform, initial datasets can be published in a few weeks, providing greater transparency, driving collaboration and citizen engagement and freeing up precious staff resources.
Interested in writing a guest blog for Sunlight? Email us at email@example.com
Welcome to this week's review of notable deleted tweets caught and archived by the Sunlight Foundation's Politwoops project. We start with a deletion from the official account of Rep. Henry Cuellar, D-Texas, that said, "Flying over beautiful #Colombia" with the image seen to the right. There are no similar tweets on his official Twitter account or mention of a trip to Colombia. It could just be a vacationing staffer who is mistakenly logged into the member's account, but I have yet to receive a response from his office.
Champ Edmunds, a Republican U.S. Senate candidate in Montana who you may remember from last week's Politwoops roundup where he discovered a backpack full of money while airing out the scene of a clogged toilet, makes another Politwoops appearance this week when he deleted, "My dark green 2002 Dodge Crew Cab truck with a REVDUP plate has been stolen! If you see it, please call the Msla PD or MCSO. Thanks! #theft." The tweet was deleted after 7 hours and while I hoped the truck was located, I noticed the Missoula Police Department tweeted, "Stolen from 2000 blk Mullan Rd: 2002 Dodge Ram 1/2 ton, dk grn in color w/rusted rear pass door. MT Lic "REVDUP". Call 9-1-1 if you see it." I called both the Edmunds campaign and the Missoula Police Department about the status of the REVDUP truck case, but was not able to find someone who knew what happened.
In what was likely a copy-paste mix up, the official Twitter account for Rep. Bennie Thompson, D-Miss., deleted a link to the YouTube embedded to the left after being up for just 4 seconds. There are no other mentions of this video or other Vine compilations from his official House account.
Finally, Gov. Susana Martinez, R-N.M., deleted a retweet of a Buzzfeed reporter who linked to a Mother Jones cover story about Gov. Martinez that included profanity-laden audio recordings. The now deleted retweet said "RT @mckaycoppins: I wonder what % of prominent pols would come off looking good when secret recordings of them were selectively leaked." She deleted the retweet after 14 seconds and replaced it with a tweet saying "Stand with me and show the D.C. liberal media their desperate attacks have no place in NM http://bit.ly/1eQf7Bq #StandwithSusana #NMPol." The deletion caught by Politwoops was picked up by Mother Jones and Slate.
- A number of healthcare industry groups are teaming up to issue new recommendations to increase price transparency in the sector. (Washington Post)
- President Obama has attended 373 fundraisers during his time in office. (Government Executive)
- Jakarta, Indonesia is holding an open data challenge and working to get its city managers to use open data to make better decisions. (Future Gov)
- Only 1 in 200 citizens in Pakistan bothers to file their income tax return. In an attempt to shame the other 199, officials are publishing a directory of taxpayers' details. (Tech President)
- The French antitrust authority persuaded Nespresso to open its espresso machine to third-party pod providers. A similar story may be headed in the opposite direction in the United States, where Keurig is planning to lock their system to outside pod-makers. (Ars Technica)
State and Local News
- Maryland passed a law that will set up a state Council on Open Data, comprised of 37 government, academic, and private-sector leaders. The law also mandates more data releases. (Government Technology)
- The Department of Housing and Urban Development is planning to post salary data for officials running America's 4,000 public housing authorities by mid-May(Government Executive)
At Sunlight we spend a lot of time trying to make sense of who has a say in the policy making process, and whose perspective is being heard. Time and time again we find that well-organized corporate interests are far more involved in the process than ordinary citizens, or the underfunded groups that seek to represent them.
Now, a recent paper — forthcoming in Perspectives on Politics by political scientists Marty Gilens of Princeton and Benjamin Page of Northwestern — has provided some really striking empirical evidence that the kinds of imbalances we have observed anecdotally in our work at Sunlight are actually systematic features of modern American democracy. The preferences of economic elites and business interests, according to Gilens and Page, significantly shape policy outcomes — and the preferences of average citizens simply don’t.
Gilens and Page looked at 1,779 high salience policy issues between 1981 and 2002. They compared whether these specific policies were adopted or not dependent on the preferences of median income citizens and wealthy individuals, measured through public opinion surveys and the activities of mass-based and business-oriented interest groups.
They found that the percentage of average citizens who favored a policy being enacted had no demonstrable effect on the likelihood that the policy would pass. The top chart in the figure below demonstrates this finding: As the percentage of citizens supporting a policy increases, the likelihood that that policy will pass does not increase – the line is horizontal.
Conversely, the second chart indicates that elites are very influential: As the percentage of economic elites favoring a policy increases, the likelihood that the policy will be adopted increases as well.
The final chart in the figure below shows that policies are unlikely to be adopted when many powerful interest groups are aligned against them.
As Gilens and Page summarize their findings: “Economic elites and organized groups representing business interests have substantial independent impacts on U.S. government policy, while mass-based interest groups and average citizens have little or no independent influence.”
The mechanisms behind the outsized influence of elites are still not well known. Of course at Sunlight, we think lobbying and campaign finance are pretty important parts of this story. One way to think about the elite domination story is to consider the vectors of influence that elites have over policy makers. In our "1% of the 1% study," we found that just 31,000 people contributed 28% of all the (traceable) money in the 2012 cycle. It’s a good guess that these folks have the ears of the politicians whose campaigns they fund. As Sen. Chris Murphy, D-Conn., noted while speaking about raising money from big donors, “I talked a lot more about carried interest inside of that call room than I did in the supermarket.”
This is really important work, and provides empirical backing to how we have suspected politicians respond to conflicting policy preferences and pressures. This work suggests at least as many interesting research questions as it answers, and we look forward to seeing where Gilens and Page take it.
A few days ago, a broader coalition of European civil society organizations launched a campaign to make some noise around the influence of big businesses on EU institutions. The timing seems perfect as the upcoming EU elections could create some more serious buy-in from candidates for an effective lobbyist registry. Fixing the current system — which is broken in so many ways — has been on the table for quite a while now, without any success.
So what's the problem? As a new study shows, a vast majority of banks and financial lobby groups working to influence EU banking regulations are not registered within the EU's volunteer lobbyist system, and other industries are notoriously opaque too. As a result, EU citizens have no idea how many laws that affect their lives directly have the fingerprints of lobbyists all over them – from climate to public health to data protection – or how their MEPs (Members of the European Parliament) are influenced by the agenda of big business.
The new Politics for People website – launched as the centerpiece of the broader civic campaign – provides an easy way for EU citizens to directly contact their MEPs through social media channels. People can urge their politicians running for re-election to sign a pledge, and support the interest of the broader public as opposed to the voice of those with money. The website also features case studies explaining how excessive industry lobbying impacts upon the daily lives of ordinary EU citizens, and how the "fire power" of big business can impede meaningful reform.
A lot of prominent MEPs have already signed the pledge, which is a truly encouraging sign. Two days ago, the European Parliament issued a resolution upon the European Commission (the executive body of the European Union) to prepare a legislative proposal for a mandatory register by 2016. It also asks the Commission to, in the meantime, introduce ambitious measures to encourage lobbyists to join the register — for instance, by limiting the number of meetings with unregistered lobbyists.
For our friends living across the pond: you can take action now!
The 25 House members who have announced plans to retire at the end of this year and who aren't seeking higher office have money to burn.
An analysis of campaign finance reports filed this week with the Federal Election Commission shows that the House short-termers have more than $13 million combined in their collective campaign kitties. Number one on the list, compiled using Sunlight's Real-Time FEC tracker: Rep. Dave Camp, a Michigan Republican who is currently chairman of the tax-writing House Ways and Means Committee, a popular target for lobbyists' contributions.
We've already detailed the money that parting lawmakers can keep: Campaign contributions to leadership PACs can — and are — appropriated for any use imaginable, as 60 Minutes reporter Steve Kroft has documented. But House members generally have far larger amounts of cash stashed in their regular campaign committees. FEC regulations and the rules of the House strictly limit the uses of this money. Basically departing members can:
- Retire debts
- Refund contributions
- Contribute to other campaigns
- Pay to continue operating their committees
The latter two options provide plenty of opportunities for politicians who might want to keep their options open for continuing their careers, or take positions where maintaining cordial relations with politicians might be key.
In fact, the House committee with the most cash on hand — $4.5 million — belongs to the chancellor of the University of Massachusetts at Lowell — Marty Meehan, who left his congressional seat in 2007. A look at the campaign finance report filed this week by the former Democratic representative shows that he spent $67,500 making donations to politicians and political campaigns, most of them in Massachusetts, but not all: Meehan gave $5,000 to the gubernatorial campaign of Rep. Allyson Schwartz, D-Pa., and made smaller contributions to the campaign committees of Sens. Ben Cardin, D-Md., Jeanne Shaheen, D-N.H., and Jack Reed, D-R.I. Marty Meehan for Congress also made donations to a number of Massachusetts charities, with the largest — $12,000 — going to the Marty Meehan Educational Foundation, and another $10,000 going to the Urban League of Springfield (Mass.).
Another former Massachusetts representative also makes the top 20 list of House committees with the most cash on hand: No. 15 is Citizens for Joe Kennedy 1988, with $2.5 million. That would not, obviously, be the current Rep. Joe Kennedy III, D-Mass., who was born in 1980. The Kennedy in question is the congressman's father, Joe Kennedy II, who is now an advocate for affordable energy. His filing this week to the FEC reported some $30,000 in operating expenses and no contributions of any sort — not even to his son's campaign.
Both the Meehan and Kennedy committees continue to register "contributions" in the form of interest on their bank accounts. You can see a complete list of all the House candidates' campaign committees, ranked by cash on hand, here.
- A new study from CREW found that companies that voluntarily disclose their political spending don't always do a particularly good job. (Roll Call)
- Many Americans electronically file their taxes, but only 21 Senators electronically filed their campaign finance reports. Senator Jon Tester (D-MT) is pushing to require all Senators to e-file, which could save $500,000. (Public Integrity)
- The Office of Information and Regulatory Affairs updated its meetings database, making it easier to search through information about meetings with non-government officials. So far, the update only applies to recent meetings. (Center for Effective Government)
- Dave Camp (R-MI), the retiring chairman of the powerful House Ways and Means Committee, could be looking at a plum spot on K Street if he decides to move across town after he leaves office. (The Hill)
- Zimbabwe's President Robert Mugabe, himself not known as a paragon of good governance, lashed out at Nigeria for perceived corruption in that country recently. (Global Voices)
- A growing government watchdog in Armenia is approaching its anti-corruption work with a variety of high and low tech tools. (Open Society Foundations)
State and Local News
- Louisville is planning to add more than 90 new datasets to its website in the coming days and another 173 in the future. The moves comes after a report, requested by Louisville Mayor Greg Fischer, found that the city has 414 datasets. (Louisville Courier-Journal)
- Opinion: A recent court decision that exempted public business conducted on private devices from California Public Records Act lacks common sense. "When public officials conduct public business, their constituents get to watch, regardless of the platform." (Los Angeles Times)
- ICANN and Global Internet Governance: Challenges and Opportunities in 2014. Georgetown University. Thurs. 4/17. 7:00 - 8:30 pm. 3600 N Street NW, Mortara Building, Washington, DC.