Support all your favorite nonprofits with a single donation.Donate safely, anonymously & monthly, in any amount. It's a smarter way to give online. Learn more
The Sunlight Foundation uses technology and ideas to make government transparent and accountable.
The Sunlight Foundation is:
A think-tank that develops and encourages new policies inside the government to make it more open and transparent.
A campaign to engage citizens in demanding the policies that will open government and hold their elected officials accountable for being transparent.
An investigative organization that uses the data we uncover to demonstrate why we need new policies that free government data.
A grant-giving institution that provides resources to organizations using technology to further our mission and create community
An open source technology community that revolves around the Sunlight Foundation’s core mission
Information about money in elections needs to be open and transparent -- this is a mantra you’ve heard before. But, here’s something new: in Japan, there may not be enough money in elections to make them competitive.
In Sunday’s elections, the Liberal Democrats (LDP) retained their majority in parliament by a landslide, securing another four years in office for Prime Minister, Shinzo Abe. However, the sweep of the self-proclaimed referendum of the current economic policy, “Abenomics,” doesn’t necessarily indicate a glowing endorsement of either the policy or the leadership. If anything, the victory was borne of a lack of opposition and voter apathy. Not only were parties caught completely off guard by the snap elections, but the rigid election structure in Japan makes it difficult for parties to even compete for seats.
After only two years in office, Shinzo Abe dissolved the Diet, Japan’s lower house of parliament, and announced that elections would be held on December 14. This surprise left opposition parties scrambling and ultimately, the second largest contending party, the Democrats, only fielded 198 candidates compared to the LDP’s 352. Without ample time to prepare, opposition parties were already left at a distinct disadvantage to the leading party -- and that’s before any campaigning even began.
The rules for engagement for campaigning are uncharacteristically strict in Japan compared to other countries we’ve looked at so far. The campaign period is just 12 days and during that time, parties are very limited in the scope of their campaigning. Advertising is tightly controlled and heavy shows of signage are not common. Candidates are limited to displaying signboards in their own allotted and equal-sized spot. Like many other countries, the state grants parties access to public media outlets free of charge, but they are restricted to a limited amount of airtime equally distributed across all parties and only last year approved online campaigning. In general, TV ad spending in Japan is very low compared to countries that inundate voters with negative ads, like the U.S. So, parties took to the streets to get the word out. Literally. They drove around in trucks screaming their candidates’ names. Already burdened by little to no name recognition, nascent and opposition parties have few platforms to voice their positions.
The structure of political finance regulations in Japan can also make it difficult for opposition parties to gain a foothold into the political system. As we’ve seen in other countries, major political donors are not likely to fund new parties with little influence on policy. So, these parties often rely on public funds to get started. In some places, like Albania, the state even provides “start-up cash” to new parties. In order to eligible to receive public funding in Japan, parties must already hold seats in parliament and have received at least 2% of the vote in a recent election. The amount of media airspace allotted also decreases when a party puts forth less than 12 candidates. These provisions give little promise to parties working to get off the ground.
It’s clear that these rigid regulations are designed to make campaigns more fair by giving parties an equal opportunity to showcase their platforms -- and certainly, there’s merit in that -- but it also can reinforce the power of existing positions. In exit interviews, those who actually showed up to the polls on Sunday cited a lack of laudable alternatives as the main reason they cast a vote in favor of the Liberal Democrats. Perhaps if there was an opportunity for new ideas to infiltrate the political system and to use campaign methods fit for 2014, more people would have shown up in the first place.
Today's meeting of the Legislative Branch Bulk Data Task Force produced some major news. Specifically, there will be a lot of new bulk legislative data in the new year.
The newly renamed Government Publishing Office (GPO) has been making bills from the House of Representatives available for bulk download in XML format for nearly two years. Now it appears the Senate will finally be joining the fun.
Even better, both chambers are closer than ever to releasing bill status information in XML. That addition was promised sometime next year.
Not to be out done, the House Clerk's office announced a number of additions to its XML offerings including better member and committee data as well as information on nominees and elections.
Those are only a few of the changes announced yesterday, for a full run down we recommend this blog post from the Congressional Data Coalition.
Sunlight, along with a number of other of dedicated transparency advocates, has been pushing for these and other changes for a long time now. We have been closely following the activities of the Bulk Data Task Force and are thrilled that they are continuing to move towards making legislative data available in bulk and machine readable formats. We are particularly excited that the Senate has decided to join the House in these efforts and hope that they continue to engage and improve their data.
In a move that could make it easier for the public to identify who's behind TV advertising aimed at influencing elections and legislation, the Federal Communications Commission on Thursday proposed requiring cable and satellite television providers, as well as radio stations, to begin posting political ad contracts online.
The FCC's notice of proposed rulemaking, which would expand a two-year-old online database of ads purchased on broadcast stations, comes in response to a request by the Sunlight Foundation, the Campaign Legal Center and Common Cause. The organizations are represented by the Georgetown University Law School's Institute of Public Representation.
It could add tens of thousands of radio stations and hundreds of cable systems to a database that gives journalists and the public detailed information about the spending, target audiences and principals of committees trying to influence public opinion.
In its 51-page notice, the FCC agreed with Sunlight and its allies about the public benefits in making political ad buy information available online. "Adding cable, satellite television, and broadcast and satellite radio political file material to the existing television online database would facilitate public access to disclosure records for all these media and allow the public to view and analyze political advertising expenditures more easily in each market as well as nationwide," the commission wrote.
There will also be benefits for radio and TV providers as well as the taxpayers, the FCC document argues: "By taking advantage of the efficiencies made possible by digital technology, we intend to make information that cable and DBS providers and broadcast and satellite radio licensees are already required to make publicly available more accessible while also reducing costs both for the government and the private sector."
Publication of the notice in the Federal Register launches a 60-day period during which the public can comment on the proposed regulations. The FCC is proposing a phased-in approach, with larger stations and cable providers on a faster timetable to begin posting online. The smallest outlets -- those with audiences under 1,000 -- would be exempt from the online posting requirement. Sunlight favors the speediest possible implementation of the rule, so that the maximum amount of information will be available to the public in time for the 2016 elections.
In 2012, the first presidential election following the Supreme Court's landmark Citizens United ruling, Sunlight built two tools, Political Ad Sleuth and Ad Hawk, to help track the enormous sums of money that outside interest groups began spending to influence political campaigns. In the 2014 election cycle, outside committees and organizations spent more than $1.1 billion, compared to some $3 billion in expenditures by candidates and political parties, according to data compiled by Sunlight's Real-Time Federal Campaign Finance tracker.
But outside interest groups spent more than all the candidates combined in 28 House and Senate races this year. The Wesleyan Media Project has estimated that TV ad spending topped $1 billion in 2014's congressional and gubernatorial races.
Since July of this year, all of the nation's nearly 2,000 broadcast television stations have been required to post their political ad contracts online. But political advertisers are increasingly turning to cable. Currently, anyone who wants to look at those ads has to go physically to view them at a cable provider's billing office during their business hours, greatly limiting the accessibility of the information.
Also currently difficult to access: contracts for political advertising on radio, an important medium for political advertising, where the messages often are substantially different from those on TV. Ethnic radio has been a notoriously popular venue for political dirty tricks. The FCC order would make it much easier for the public to monitor those ads and determine who is behind them.
Another section of the FCC brief requests suggestions on how its database can be made more accessible to consumers. The Sunlight Foundation has consistently advocated that filings be required in electronic, machine readable formats for improved accuracy and ease of data analysis. Currently, broadcasters use non-standardized, PDF forms (essentially, images), which require laborious data entry for analysis.
(Contributing: Sean Vitka)
The Federal Communications Commission has begun asking the public to weigh in on whether radio, cable and satellite TV operators should report political ads the way that broadcast TV already does. But for a clear demonstration of just how bad existing data is, the FCC can look to a U.S. Government Accountability Office report released earlier this year.
The government's spending watchdog gave up even trying to track radio ad spending by issue, concluding there just wasn't a credible source covering 2007 through 2012.
The point of the report, which was released in September as "Broadcast Television and Radio: Disclosure Requirements for Broadcasted Content" was relatively obscure: track down advertising aired by TV and radio stations promoting their own interests. Though the stations disclose political ads aired by others, they aren't required to report ads they air on their own behalf, or to air ads with opposing viewpoints.
The proposed rulemaking announced Thursday wouldn't close the loophole the GAO looked at or alter the rules of reporting. Instead it would compel radio as well as cable, satellite television, and satellite radio to report put their files online via an FCC site.
In spite of the legal ambiguity, many TV stations do disclose their own in-house ad buys--but the GAO was unable to say how many because the records are such a mess, and can be thrown out after just two years.
Instead, the GAO ended up paying $30,000 in taxpayer funds to Kantar Media, an ad-tracking company, for records of TV ads on topics investigators thought would be relevant, according to a copy of the contract requested from GAO. Only the TV data was good enough to use--the report gave up on even tracking radio at all "due to limitations in public and private data sources."
The first obstacle to using the public data on TV advertising was the time the GAO wanted to examine. The report set out to look at ads aired between 2007 and 2012. But broadcast TV files need only be kept for two years. Moreover, an order to collect these files from the biggest stations in the country began only in August of 2012; prior to that these files were only kept on paper at individual station offices.
When GAO researchers looked further into the issue, things looked worse. The TV files are required for political issue ads "relating to any political matter of national importance, including a national legislative issue of public importance." But there's no official guide as to what issues meet this criteria. The GAO report noted "stakeholders we spoke with, including FCC and NAB, told us that broadcasters are responsible for making determinations on a case-by-case basis."
Even after paying $30,000 for data on ads in four subject areas, GAO still couldn't produce reliable cost estimates of the ads. That's in spite of the fact that broadcast TV stations make public the cost of every political ad spot they air--something their lobby, the National Association of Broadcasters, has complained bitterly about.
For radio, less data is available, and the GAO was unable to even find cost estimates. The best information again came from Kantar Media's Campaign Media Analysis Group ('CMAG'')--but it had holes. "We determined that CMAG’s radio data were not comprehensive enough for our purposes because these data cover fewer stations in fewer markets than the television data and do not include cost estimates, among other reasons."
It was another year full of encouraging news on the open data front in states and municipalities across the country. New open data policies were approved in municipalities of all sizes from coast to coast, existing open data programs matured and sparked new innovations, and there were numerous other open government wins as a result of advocacy efforts. Here's a look back at the year's highlights.
A growing number of strong open data policies
2013 was a landmark yearfor the adoption of open data policies, and 2014 built on that momentum. There are now 50 open data policies at the state and municipal level across the country. There were 19 new state and municipal policies adopted this year, and four existing open data policies were updated and improved. All five of the largest cities in the U.S. now have open data policies, and small and mid-sized cities are increasingly joining the movement for open data. Bloomington, Ill., with a population of fewer than 78,000; Jackson Mich., with a population under 34,000; and Williamsville, N.Y., with a population of fewer than 6,000, all adopted open data policies in 2014, showing that smaller communities can and should embrace the benefits open data can bring.
The communities adopting open data are finding ways to build on the success of those who came before them in this space. Boston's new open data policy, for example, encourages the Chief Information Officer overseeing implementation to work with people across departments and issue additional guidance, building a strong foundation for moving forward with buy-in across government. Pittsburgh's new open data policy follows more than half of the best practices for open data.
Even governments with open data policies already in place are seizing the opportunity to iterate and make improvements. Washington, D.C., the first local government we identified as having an open data policy, updated its policy this year with an executive directive from the mayor. The city made several important improvements, and though it still has plenty of room for growth, we hope it will continue to iterate its policy to include more best practices.
Other places have worked on updating their guidance supporting existing open data policies. NYC updated its open data plan that complements the policy, committing to releasing more data but also prompting some questions. Montgomery County, Md., released an implementation plan that outlines the prioritization process used for releasing datasets and shares insights on the steps taken to complete an inventory of datasets. Philadelphia showed its open data growth, too, with the release of a new Strategic Plan outlining how it will approach its inventory and prioritization process, accompanied by a revamped public-facing method for tracking all of the progress.
Widespread transparency wins
There has been open government progress this year at the state and municipal level beyond open data policies, too. New York City is beating the feds to modernizing transparency by putting public information online. We looked into Philadelphia's updated lobbying portal and found improvements since our last reviewthat are resulting in more open data. We kept a watchful eye on California's vote on the public records law, which came after a failed attempt last year to roll back public records requirements at the municipal level. The approval of the amendment to the state's Constitution means municipalities will have to continue to comply with the state's public records law. That change could open the door for broad open data movement across the state, too, if open data is written into the public records law.
There is certainly no lack of momentum among California groups hoping to see the state improve its open data. We joined a coalition of groups calling for open data improvements to California's influence data portal, Cal-Access. Improvements to that system could help shine more light on campaign finance, lobbying, and ethics disclosures data.
California wasn't alone in tackling bad policies this year. In a huge win for transparency, an examination of the openness of executive orders in all 50 statesled to a deeper look into the case of the disappearing executive orders in Georgiaand, ultimately, the restoration of those orders online.
We also examined limited liability donations and corporate registry access in Texas and Washington, finding that states vary drastically when it comes to how they deal with disclosing information about campaign donations.
Several efforts this year built on work from previous years. We continued our initial exploration of how federal levers might help municipalities be more open with their financial data. We gathered a group of people to ask the federal agency collecting a significant amount of municipal financial data to consider open data best practices after wondering whether the agency could play a role in that openness. We first became interested after receiving an encouraging comment from the agency around open data recommendations.
Federal open data improvements are poised to help with this issue, too. The passage of the DATA Act earlier this year will help improve the transparency of state and municipal grants from the federal government. It's an important step toward better transparency of state and municipal finances.
Beyond financial data, we explored other specific kinds of open data. We took a look at the progress of open legislative data at all levels of government around the country. Opening up legislative data is helping people access more information about proposed bills and existing law, empowering them to be more informed participants in government processes.
We explored the intersection of open data with other areas that can support open government, too. Our efforts throughout 2014 included advocating for getting serious about protecting access to public emails, describing what the federal government could learn from the state and municipal electronic-filing efforts, looking at the need for updating public records tools to include more open data concepts, exploring the intersection of police accountability and open data, showing how government- and community-run public records portals are helping bring open data concepts to the public records process, examining how proactive release of public records could lower costs, and sharing recommendations for strengthening open meetings with open data.
We took a deep look at the intersection of public records, records management, and open data, including the future of storing and sharing records, addressing the costs of turning public records into open data, the need to balance records management and open data when it comes to liabilities, how good records management helps open data. We pointed out that open data is the next iteration of public records -- which means public records advocates and open data advocates should be working together, leveraging their skills, insights, and shared interests to create improvements
Building resources to support the movement
Increased open data adoption and improved open government practices have come about through research, advocacy, and the sharing of best practices. Sunlight has developed and continues expanding on a wide variety of resources to help with these efforts.
Education about the benefits of open data is part of what is helping drive forward adoption. Our Impacts of Open Data document highlights the range of potential benefits to government and the public from open data, including increasing transparency, empowering accountability, enhancing efficiency and leading to cost savings, improving service quality, and increasing public participation. The document is also available in Spanish, thanks to the work of Sunlight friend and open government advocate Iris Palma
Of course, open data policies and programs cannot achieve their full potential without being strongly written and implemented. Sunlight's Open Data Policy Guidelines, originally crafted in 2012, were revamped this year to help with those processes. The corresponding examples for best practices -- including narrative examples and policy language -- were expanded to reflect the range of approaches to crafting and implementing strong open data policies. The updated guidelines now include a look at the different kinds of oversight authorities, the need for a balance test to appropriately safeguard sensitive information while releasing as much as possible, prioritization processes for data release, and more.
Building on the foundation of the Guidelines, the Open Data FAQ page answers specific questions about open data policy drafting and implementation, aiming to help the processes be as informed as possible.
To help inspire action and engagement around open data and tools for exploring open data, we crafted a civic toolkit to help engage the next generation of leaders.
The US City Open Data Census, a collaboration between Sunlight, Open Knowledge, and Code for America, is another resource for staying informed about open data around the country. It tracks the technical openness of datasets in cities where a non-government community member has volunteered to do that assessment.
And, of course, we've been continually updating and expanding resources including the open data map listing all of the U.S. open data policies, the GitHub repository for additional resources, and our Sunlight Cities Tumblr, highlighting fun, important, and innovative open data visualizations and stories from around the country. We even refreshed the hub of our local work to help bring all these resources together and make them easy to find.
It's been a great year for open data and open government across the country, and we believe there is always room for improvement. We built upon the great progress made in 2013 to advance open data in states and municipalities of all sizes, helping them craft and implement policies that fit the local context, and we look forward to continuing and growing that work going ahead.
We're sure 2015 will bring further advances for open data and broader open government wins. There are sure to be more challenges, too, and we'll be ready to take those on, working with government staff and officials as well as community advocates and stakeholders to bring greater transparency to local governments.
Despite all of the different promises made by governments to publish public sector information or make government data open by default — whether through the Open Government Partnership, the G8 Open Data Charter, the EU Digital Agenda or country-level commitments — many transparency and open data groups around the world are struggling with governments that are reluctant at best, obstructive at worst.
In many contexts, it seems that governments know more about their citizens than citizens know about their administration. One important step forward in resetting the default to open is to create public, comprehensive listings of all information holdings; however, we have seen very few examples of strong index and audit requirements so far. Recently, the government in the Netherlands has welcomed the advocacy efforts of the Open State Foundation for a government-wide data inventory and has agreed to publish an index in 2015.
The flow of information has changed
If you think that information collected and paid for by the government belongs to us all, think again. Despite the launch of new open data projects, initiatives and portals — and despite various types of government declarations and action plans — the unlocking of public service information is not happening fast enough. One might wonder how many more research reports on the economic value and societal benefits of open data need to be published.
Information still flows from government to citizens, from professors to students and from media to consumers. At the same time, there are generations that do not need to be told what is good for them. They can investigate, publish and organize themselves. The way they interact with authority is completely different from previous generations, but the structures of government have not changed yet.
Information is a right, not a privilege
To reverse this, one needs to understand not only how government works, but also how to influence it. Open data allows people to see how their governments operate; public officials need to stop building applications and start sharing the data they hold. When governments build apps, they continue to dictate what people need to know from their own perspectives. This makes these government bodies not necessarily more transparent or open.
The call for open data is an attempt to confront a closed system from the outside. Open data raises questions about public institutions, how they work, how they deal with information and their public tasks. It raises questions about what data is being collected, used, maintained, managed and released. From outside the government, it is difficult to know what information the government holds. Often, public officials first want to know what you will do with the data and exactly what the economic or societal benefits you will create for them. Some governments have invested in research to help them decide what data to open. Others have organized speed dating sessions between the private sector and some government data holders. However, access to government information is a right, not a privilege for the included few.
Let us know what you have
A year ago, we sent a letter to all government departments in the Netherlands, explaining the need to unlock government information and the benefits of machine-readable, government-wide data. We are often asked which datasets we would like to see opened. So, despite well-publicized benefits, there are two major unknowns: We do not know what data the government holds, and we do not know all possibilities and opportunities that occur when particular datasets are opened.
In order to help prioritize what datasets can or cannot be made public, the first step is to know what type of information the government collects and produces. Therefore, we asked all departments to create an data inventory and publish the results, preferably in a machine readable format. Ronald Plasterk, minister of Interior in the Netherlands, who also serves as the coordinating minister for open data, agreed to have such a government-wide data inventory, sending a recent letter to the Open State Foundation confirming that the Dutch government sees open data is an important policy instrument. The results will be published in the first quarter of 2015. We realize this is only the beginning, but it still represents a major milestone for both our country and the whole open government agenda.
Interested in writing a guest blog for Sunlight? Email us at email@example.com
During the final weeks of his last term in the House, retiring congressman Jim Moran, a Democrat who still represented Virginia's eighth district, negotiated jobs with a lobbying firm, an energy company and a nonprofit with ties to the government of a country that has been accused of serious human rights violations, including the recent jailing of an investigative journalist.
Moran disclosed his job hunt to the House Ethics Committee. Members can seek outside employment while still in office, but must disclose negotiations with prospective employers to the committee. Members who must recuse themselves from official matters due to a potential conflict or the appearance of a conflict of interest must make public disclosures to the House Clerk's Office. Moran recused himself from matters concerning Washington law firm and lobby shop McDermott, Will and Emery, Homeland Fuels LLC, and an organization listed as the "Assoc. of Friends of Azerbaijan" but is actually known as the Assembly of Friends of Azerbaijan, or AFAZ.
Kemal Oksuz, president of AFAZ, confirmed that the group had been talking to Moran. "This is true that retiring Congressman Jim Moran and Assembly of the Friends of Azerbaijan entered the negotiations for employment," he wrote in an email, adding, "He will provide consulting and advising services to AFAZ in its activities in the United States."
According to the group's website, those activities include fostering "friendship, understanding and cooperation between the United States and Azerbaijan." Last February, the group worked with the Embassy of Azerbaijan to help organize an event on Capitol Hill commemorating the anniversary of the Khojaly massacre, where more than 160 Azerbaijanis were killed during the Nagorno-Karabakh war. According to AFAZ, 20 members of the House and Senate — including Moran — and 60 congressional staffers attended the event.
Oksuz also organized a conference in Baku in 2013 that featured a dizzying array of state and federal legislators, and former White House officials, most of whom did not report the trip on House ethics disclosures according to the Houston Chronicle.
It doesn't appear that Moran joined Oksuz and his colleagues on that junket, but he did visit Baku in November 2013, courtesy of the Humpty Dumpty Institute according to his personal financial disclosure. Moran also served on the Congressional Azerbaijan Caucus.
AFAZ, currently based in Houston, has not registered with the Justice Department under the Foreign Agent Registration Act, though Oksuz says they will after they relocate to Washington in early 2015.
According to the Houston Chronicle, AFAZ "operates as a U.S.-based public relations arm of SOCAR," which is in turn wholly controlled by the Azerbaijani government. The nonprofit lists Rauf Mammadov, director of SOCAR America, as its treasurer.
The Washington Business Journal recently reported that SOCAR America bought a DuPont Circle office building, and that it plans to expand its U.S. operations and influence. The building purchase and Moran's employment negotiations are another indication of Azerbaijan's efforts to up the nation's influence profile in Washington.
At home, the country's track record on human rights has not been sterling.
Nongovernmental watchdog groups, including Human Rights Watch, have repeatedly called out human rights abuses in the Central Asian nation, citing the "Azerbaijani government’s systematic crackdown on human rights defenders and other perceived government critics."
On Dec. 5, just days before Moran said his negotiations with AFAZ began, the country arrested Khadija Ismayilova, who reports for Radio Free Europe/Radio Liberty, an international broadcasting service funded by the U.S. government. Ismayilova, who exposes government corruption, has faced attacks before.
Moran could not be reached for comment.
Update: Over e-mail on Thursday, Oksuz added:
AFAZ is neither lobbying firm nor energy company. It is 501 (c) 6, non-profit organization certified by IRS. During Khojaly Tragedy, 613 people had been killed brutally by Armenians, over 1200 we tortured and 155 remain still missing. AFAZ is not representing either the Azeri Goverment or the Embassy in DC. So I have no idea whether or not AFAZ is supposed be registered in the Dept of Justice. I need to consult with our legal counsel.
He later added:
Furthermore, AFAZ is not the US Arm of Socar.
Questions regarding AFAZ's sources of funding did not receive an immediate reply. This post will be updated if and when we receive a response.
The debate over net neutrality is anything but neutral. That’s abundantly clear from the intense response and questions unleashed by Sunlight’s latest analysis of public comments on the Federal Communications Commission’s proposal to regulate Internet traffic.
Our commitment to transparency, to open data and to the informed use of that data prompted us to do this follow up to address some of the concerns and reaction:
- A number of groups on the pro-net neutrality side of the debate are telling us that they submitted far more comments than we found in the download from the FCC. As we pointed out in our initial post, there’s a big discrepancy between the number of comments the FCC says it received and the number we were able to find in the files the agency released to the public.
- The conservative group that appears to have generated the vast majority of comments in the second set of comments we analyzed said we confirmed it “won” the comment period. In fact, as we were careful to point out in both our first and second post, these numbers cannot be read the same way as a baseball score. That’s partly because of data noise (more on that below) and partly because of the way those numbers were generated, both factors which we went to some pains to elucidate in our post.
Also important to underscore: Sunlight, a nonpartisan nonprofit that advocates for more transparency, undertook this analysis in service of that mission. Our aim is to lift the veil on how government decision makers are influenced, who is doing the influencing and what, if any, special interests or agendas might be behind those efforts.
Now, to the data.
As of now, it looks like there’s a major discrepancy between the numbers of comments the FCC reported receiving and the number we actually found in files they released to the public. This is something we pointed out in our earlier posts, but since it has become an issue, let’s be crystal clear: At this point, there’s a difference of 1,124,656 between what the FCC is reporting and what we counted in the files the agency provided.
Moreover, groups such as Battle For the Net (Free Press, Fight for the Future and Demand Progress) and ColorOfChange.org insist they sent the FCC far more comments than we were able to find in the data released.
Fight for the Future, in particular, disagreed with the counts in our analysis, claiming that one of its form letter campaigns produced 367,000 comments in the FCC’s dump. Upon further examination, we believe Fight for the Future didn’t actually count the number of distinct documents from its campaign that occurred in the dump, but rather did only a rudimentary full-text search for key phrases from the campaign to see how many times those phrases occurred. This failed to account, however, for the fact that some comments are duplicated (that is to say, occur more than once with identical text, submitter and unique ID number) within the data, likely because of sloppy exporting processes on the part of the FCC. Indeed, after closer inspection to confirm our numbers, we found 96,263 comments that are included more than once in different parts of the export file, which we had correctly excluded from our analysis during our initial data-cleaning process. We thus stand by our numbers as reported, and continue to maintain that they accurately characterize the data as it appears in the dump produced by the FCC.
Whether the FCC’s dump actually includes all documents they received is another story, but figuring out whether there are missing comments and why is a project for the FCC, not Sunlight. Our experience working with these records, however, suggests several possible explanations:
- Counting comments versus counting signatures: Some submissions are single PDF documents containing numerous comments and signatures. They list a number near the beginning that the FCC may be using to perform its counts. That number is the total combined count of (a) comments with signatures, and (b) a separate list that is comprised only of signatures. For our purposes, we were interested in counting comments and did not take signature-only submissions into account. This isn’t to suggest that signature-only submissions shouldn’t be counted, but the focus of our report meant that we discarded them. In many cases, the difference between comment-plus-signature counts and comment-only counts is extreme. In this document from Free Press, for instance, the reported count is 60,269 whereas we identified 14,848 comments. Presumably, the difference (45,421) is attributable to the list of signatures that begins on page 1,240 and continues to the end of the document. Once again, we’re not impugning the count provided by Free Press, but for our project, we explicitly chose only to count comment submissions.
- Faulty import processes for CSV submissions: All of the reported cases of under-representation involve campaigns that opted for CSV submission. It seems possible, especially given what we know about the technical challenges that are already well-documented at FCC.
- Faulty export processes for CSV submissions: Even if the submission process went smoothly, another point of possible failure would be the export process that generated the downloadable bulk files. As we’ve already reported, these files showed inconsistent formats, terrible encoding mistakes and often lacked any record or field delimiters. It’s not unreasonable to suggest that sufficient care was not taken in producing the bulk downloads.
As we pointed out in the beginning of the post, we relied upon FCC data. It’s worth repeating: There’s a big discrepancy between the number of comments the FCC says it received and the number we were able to find in the files released to the public. Moreover, it’s worth reiterating that our purpose was never to disclose who were the “winners” or “losers” of the public comment period on net neutrality. Others may have used our analysis to make such a call, but we never did.
If nothing else, this exercise points out both the value and the shortcomings of government data and suggests that for the sake of decision makers, and the taxpayers who pay them, it might be worth investing in a more modern, reliable way of compiling and reporting this information.