Blog

• Data Literacy for Data Stewards Registration is Open
  by Bob Gradeck on 3/10/2023
  The session is full. If you want to participate, please email us at wprdc@pitt.edu and we'll let you know if a space becomes available.

  What is data literacy and why does it matter? 

  We will be holding our second round of Data Literacy for Data Stewards training workshops this spring. This series emphasizes the importance of thinking critically about creating, applying, and managing data and technology.  In these workshops, participants will learn to ask questions about data and technology, reflect on how their actions and decisions impact people and communities, and identify how data and technology are used to reinforce systems of oppression. They’ll also develop strategies to institutionalize the values of equity, fairness, and justice into the data and technology practices and infrastructures of their organizations.  Our twin pandemics of COVID and structural racism have highlighted the need for data practitioners to expose and dismantle technologies and practices that reinforce and amplify powerlessness, discrimination, and injustice. Increasing data literacy is just one of the ways to protect members of our community. Here are just a few examples of harms caused by data and technology in recent years:  

  
  
  • Algorithms that are used to assess whether someone should be incarcerated before trial; 
  
  
  • Adoption of technologies that direct police to overpoliced communities;  
  
  
  • Use of biased and inaccurate facial recognition technologies that can hold severe consequences for misidentified people;  
  
  
  • Data sharing practices that create risk for our society’s most-vulnerable people; and  
  
  
  • Data systems that render people invisible due to how data about race, ethnicity, and gender identities are captured. 
  
  

  Who are we looking for? 

  This series is designed to bring together people whose lives and communities are impacted by data, along with everyone that has a role in the data life cycle. People involved in the data life cycle include those that invest in, design, and manage data systems, data producers, data stewards, and people that use data to make decisions that impact communities. We’re seeking participants from community and nonprofit organizations, local government agencies, offices of elected officials, colleges and universities, local foundations, and civic institutions.  Thanks to support from a local foundation, there is no cost to participate. 

  What will we cover? 

  We believe that “learning by doing” is a great way to build skills and confidence in data literacy and explore issues of power and privilege in data. Workshops will be structured around a series of interactive, participatory activities where attendees will work together in both small and large groups. Topics covered in this series of peer-learning activities will cover: 

  
  
  • Missing data 
  
  
  • Classification 
  
  
  • Stigmatizing communities  
  
  
  • Data context  
  
  
  • Data visualization 
  
  
  • Dashboard design
  
  
  • Data privacy
  
  
  • Algorithms and surveillance technology
  
  
  • Data management 
  
  
  • Purchasing data and technology
  
  
  • Data governance 
  
  

  We will not provide training in how to use software in this workshop series. All workshops will be delivered in English. 

  What does it mean to participate? 

  In our spring 2023 cohort, we’re looking for up to 40 people to participate in weekly 90-minute virtual workshops as part of our initial 12 workshop series, and may offer an additional in-person or virtual workshop to explore additional topics. While most of the work will take place in the workshop session, there may be one or two optional assignments or self-directed activities included in the series. We plan to start the series the morning of Friday, March 31 with an in-person celebration where participants from the first (fall 2022) and second cohorts will get to know one another. The workshops will then move online starting on Friday, April 14 at 11 am. We will hold sessions nearly every week at that time into July. We’ll send calendar invitations to all participants several weeks before the start of the series. 

  Benefits of participating 

  As a result of participating in this series, you’ll  

  
  
  • Build confidence accessing, using and sharing data  
  
  
  • Make your personal and organizational data practices more inclusive and just. 
  
  
  • Engage in a fun and collaborative learning experience 
  
  
  • Build relationships and connections with others in Pittsburgh 
  
  
  • Share your expertise and experience with peers 
  
  
  • Get help from others with your data challenges 
  
  
  • Receive a certificate that may be suitable for framing. 
  
  

  Who is organizing the training series? 

  The Western Pennsylvania Regional Data Center is organizing this training series. The Regional Data Center is a civic data partnership between the University of Pittsburgh, Allegheny County, and the City of Pittsburgh that works to make information available and accessible and works with partners to apply data for community impact. Through this series, we hope that participants will strengthen their practices, become more-responsible and more-confident stewards of data, and work to dismantle or avoid building oppressive systems. 

  How do I register? 

  Fill out our form or contact the team at the Western Pennsylvania Regional Data Center via email if you’re interested in participating wprdc@pitt.edu. You can also use the email for questions.

  What people are saying 

  Participants in our first cohort offered testimonials about their experience. These are legit actual quotes people offered at the end of the first cohort, and not stuff we made up to try and get you to spend time with us.  

  
  
  • “The Data Literacy for Data Stewards workshop series highlighted the need for an equity focus around data practices. Discussing this important topic with colleagues in a highly interactive, activity-based workshop series allowed me to grow, ask questions, and learn from others in the field and apply what I learned to my future work.”  
  
  
  • “This was an amazing course on data literacy and I feel like I learned a lot but truly appreciated the participatory interaction and process.” 
  
  
  • “I would encourage anyone who even tangentially deals with data in their day-to-day work to take the Data Literacy for Data Stewards training. It provides you with useful insights that change the way you relate to data in a welcoming, participatory environment.” 
  
  
  • “The Data Literacy for Data Stewards series was an engaging and eye-opening experience.  I know that I will bring the insights that I gained to my work going forward.”  
  
  
  Read More
• Data Literacy for Data Stewards Data Governance Workshop
  by Bob Gradeck on 2/10/2023
  The goal of our January 27 workshop on data governance was geared to getting participants familiar with what data governance is, and comfortable and confident making the case for data governance. This was the final workshop in our initial cohort. Special thanks to everyone who participated and were generous of their time to provide feedback on the individual activities and design of the series.In this workshop, Jason Ficorilli of the City of Pittsburgh served as a co-facilitator, for which we were grateful. He presented the City’s definition of data governance as “establishing authority, control, and shared decision-making over the management of data assets.” Jason then took questions about his job at the Department of Innovation and Performance (I&P), where he serves as the primary point of contact for data within I&P and works with teams in other departments to produce data inventories. The audience was able to learn more about how the City is approaching data governance, including: 

  
  
  • This is the first time that the City has comprehensively approached data governance. 
  
  
  • The open data legislation enacted in 2014 provides a mandate for data governance in the City and in related authorities. 
  
  
  • I&P is using this process to establish a basic understanding of how departments manage data for different processes. Some systems have now been fully updated and digitized. Even where that’s the case, there can be a considerable amount of older paper records to manage. 
  
  
  • An important part of the job involves building trust with data stewards and managers in other departments.  
  
  

  To prepare for our main activity, participants were given a listing of the benefits often cited by advocates for data governance, which include:  

  
  
  • More-efficient processes 
  
  
  • Reduced barriers to data access 
  
  
  • More-effective decisions using data 
  
  
  • Improved coordination 
  
  
  • Increased transparency 
  
  
  • Higher-quality and more-consistent data 
  
  
  • Opportunities for accountability 
  
  
  • Reduction in costs/expenses 
  
  
  • Ability to manage risks related to: 
    
    
    
    • Privacy 
    
    
    • Data security
    
    
    • Civil and regulatory liability
    
    
    
    
  
  
  • Improved perception of the benefits of data use
  
  
  • Strategic and sufficient investments in information initiatives 
  
  
  • Enhanced trust in data  
  
  
  • Ability to identify investment opportunities in people and systems 
  
  
  • Able to take advantage of opportunities to embed values into data systems and practices 
  
  
  • Greater sustainability and resiliency 
  
  

  We also shared a partial listing of actions, activities, systems, programs, and processes that fall under the “data governance” umbrella. These include: 

  
  
  • Form a data governance committee  
  
  
  • Establish values, philosophies, decision-making processes and priorities 
  
  
  • Craft a needs assessment 
  
  
  • Develop engagement activities 
  
  
  • Formulate data and documentation standards  
  
  
  • Create data inventories and data catalogs 
  
  
  • Write data management plans  
  
  
  • Craft privacy, security and access policies and protocols 
  
  
  • Draft incident response plans  
  
  
  • Implement data validation and quality control processes  
  
  
  • Define interoperability protocols  
  
  
  • Establish legal frameworks 
  
  
  • Identify staff roles and responsibilities 
  
  
  • Implement training and professional development programs 
  
  
  • Develop procurement guidelines 
  
  
  • Enact long-term preservation policies   
  
  
  • Monitor enforcement and compliance frameworks 
  
  

  Data governance requires a lot of resources in terms of technology investments - and even more importantly - investments in people. Success hinges in assembling the adequate resources for data governance at the city. Our activity asked participants to put themselves in Jason’s shoes (but in a different city). Participants were asked to choose a scenario and develop a plan to engage people that may either be in a position to either strengthen or stall your efforts to enact a strong data governance program.  The scenario we worked on together involved planning for a meeting with their new Mayor.  

  The Mayor is interested in using city data as an asset to measure performance. Trained as a community organizer, she has also mentioned that she’s a bit wary of adopting surveillance technologies that may put more-vulnerable members of the community at risk of harm. She campaigned on enhancing opportunities for community engagement. 

  To prepare for the meeting, participants in the workshop suggested that they:  

  
  
  • Collect stories of how other communities with strong data governance have avoided making mistakes with surveillance technology; 
  
  
  • Conduct research on public safety and transparency in the city;
  
  
  • Uncover different models for including the public in data decisions;
  
  
  • Gather stories of things that went wrong where data governance wasn’t established;
  
  
  • Learn more about how and why the mayor wants to measure performance;
  
  
  • Discover how other cities have measured performance around issues that are of particular interest, including surveillance and community engagement; and 
  
  
  • Ask colleagues in other places about their experience advocating for data governance with their leadership. 
  
  

  After discussing this scenario, participants discussed a second scenario in more depth in breakout groups. They then came back together to have a conversation about the ways that data governance processes and frameworks can be used to institutionalize the values of equity and justice into data practices.  Since this was our final meeting of the first cohort, we conducted a second survey of our participants to learn more about their experience. Several of the questions asked in the closing survey were also asked in our initial survey. A comparison of results suggests that our participants grew in their awareness and understanding of ethical and just data and technology practices and became more confident in their ability to make decisions about data and technology. Assessment of how much Data Literacy for Data Stewards participants agree with the following statements before and after the workshop series: 

  Statement	Pre-workshop average score	Post-workshop average score
  I understand the benefits and challenges that come from the use of data and technology.	4.1	4.3
  I can describe the importance of ethical and just data and technology practices.	3.9	4.6
  I form my own opinions related to the use of data and technology.	3.9	4.3
  I can ethically justify decisions I make when it comes to data and technology.	3.5	4.4
  I am knowledgeable about ethics and justice and am comfortable advising others working in data and technology.	3.1	4.3

  Scale: 1= Totally Disagree, 5=Totally Agree Source: WPRDC  28 respondents completed the initial survey, 20 respondents completed the final survey. We’ll run another cohort of this series this spring. If you’re interested in participating, please let us know in an email (wprdc@pitt.edu), and we’ll let you know when registration opens.  
  Read More
• Data Literacy for Data Stewards Procurement Workshop
  by Bob Gradeck on 1/30/2023
  The goal of our January 20 workshop on procurement was geared to getting participants comfortable and confident in developing questions and guidelines that could be used when purchasing data and technology. There is an increased focus on improving processes used to buy data and technology within government, and people view the process as an important place for preventing harm, and a tool for injecting values like equity and justice in data systems. In our workshop, our primary activity focused on the pre-award component of the procurement process. At the bottom of this document, you’ll find different links that can provide a deeper look in how public agencies are approaching procurement reform before and after the contract is awarded.   In the workshop, attendees discussed several different scenarios related to purchases of data or technology. Here are the instructions they received:  

  The first-ever manager of the newly formed Department of Data and Technology in your city worries obsessively about making the wrong decisions when it comes to buying data and technology. You work in this department. While the organization hasn’t made major mistakes yet, the manager is always reading case studies of how other communities have made decisions that have put vulnerable residents at risk, or decisions that have made it more difficult for departments to operate transparently, efficiently, independently, and in-line with their community’s values.  Your manager is hoping to build a culture where people in the department feel comfortable asking critical questions about technology. They have developed this workshop, which shares some of the most misfortunate use cases that could be ripped from the headlines.  In this workshop, you and your peers will develop questions that could have been asked before harmful data and technologies were purchased. Your director was inspired to draft these scenarios from some of the most-problematic use cases as a guide in what not to do. Through this training, your director hopes that your organization will avoid many of the mistakes that others have already made. 

  One of the scenarios discussed involved the hidden, frequently-exploited labor force behind many of the artificial intelligence (AI) systems in use. Several recent news articles on the topic of AI colonialism told the stories of some of the people that perform important labor in Venezuela and Kenya, and were the inspiration for the following scenario.  

  A vendor contracted with a neighboring City to purchase AI technology that was used to create a dataset of public infrastructure conditions using imagery. A trusted local human rights organization points out that the company relies on frequently exploited workers from Venezuela to tag and process images that are instrumental to the function of the algorithm. These workers typically act as independent contractors, are usually paid very low wages, and lack the power to object when unscrupulous contractors invent ways to not pay these workers for their labor. The human rights organization has asked your mayor to rescind the contract, but doing so may trigger a lawsuit with the company.  What questions could have been asked and what rules and practices could have been made before the contract was awarded that would have prevented this government from buying a product that exploits extremely vulnerable people?  

  In the workshop, participants thought the scenario could have been prevented had the following questions been asked prior to the purchase: 

  
  
  • Who developed the algorithm, and what do we know about them? 
  
  
  • How does the algorithm work? 
  
  
  • Who performs the labor to prepare data for the algorithm? 
  
  
  • What wages were paid to these workers (developers, data workers, etc) and how does it compare to prevailing wages? 
  
  
  • What sorts of guarantees can the company provide that all people who contribute to the algorithm (employees, contractors, etc.) are fairly-compensated and treated with dignity and respect? 
  
  
  • Does the company or do its contractors engage in union-busting activities? 
  
  
  • How can the procurement process be improved to require certifications from potential vendors?
  
  

  Other scenarios in the workshop included: 

  
  
  • Social media platforms that were used to both engage and surveil residents;  
  
  
  • Programs that used potentially biased data to make program eligibility determinations; 
  
  
  • Data systems that couldn’t be easily-modified to adopt more-inclusive data standards; 
  
  
  • Technologies with origins in border enforcement; and  
  
  
  • Vendors that shared data with law enforcement agencies in violation of city policy. 
  
  

  Participants felt strongly that: 

  
  
  • Purchasing decisions will be more just and more effective if people from different backgrounds and people with different experiences are part of the process; 
  
  
  • It’s not enough for processes to provide an opportunity for public input. Residents should be directly involved in making purchasing decisions;  
  
  
  • People without technical backgrounds should have power to ask questions about data and technology; 
  
  
  • We shouldn’t view every problem as one that needs to be solved by technology.  
  
  

  Our first Data Literacy for Data Stewards cohort concludes on January 27 with activities related to data governance and group reflections on the series. Additional Resources 

  
  
  • A guide (and introductory blog post) to de-risk government technology projects produced by the federal government's 18f.
  
  
  • The disconnect between software development and government contracting – blog post by Waldo Jaquith
  
  
  • What Lies Beneath – blog post by Mark Headd 
  
  
  • Resources from the Harvard Kennedy School Government Performance Lab, including:
    
    
    
    • Guidebook for crafting a results-driven request for proposals
    
    
    • Using government procurement to advance equity (report)
    
    
    
    
  
  
  Read More
• Data Literacy for Data Stewards Data Management Workshop
  by Bob Gradeck on 1/17/2023
  The goal of our January 13 workshop on data management plans was geared to getting participants comfortable and confident in asking up-front questions about how data would be created, collected, used, documented, organized, stored, shared, and archived. Data management plans have been a requirement for federal research grants, and academic libraries have developed tools and expertise in documenting important aspects of how research projects handle data throughout the data life cycle. Civic data projects can benefit from community conversations around many of the same discussion prompts used to develop research data management plans. In our data management workshop, we talked about the importance of being intentional about data stewardship. Librarians at the University of Pittsburgh have talked about how the choices we make about data can go a long way toward supporting discoverability, long-term sustainability, security, provenance, usability, interoperability, and efficiency. A data management plan is just one of several structures and frameworks that can help data stewards be better custodians of data. We then asked participants to begin writing a data management plan for a community garden. The plan would have to account for data related to: 

  
  
  • volunteer recruitment and management;  
  
  
  • distribution of crops between volunteers, the community food bank, and those sold at the farmers market;  
  
  
  • fundraising, revenue and expenses; and  
  
  
  • documenting growing operations, which include planting schedules, soil testing, and rainfall. 
  
  

  Our activity asked participants to select one type of data that is important to managing the garden, and have a conversation using the following questions and prompts that are often a part of a data management planning process: Questions that describe the data 

  
  
  • Briefly describe the data. What information is included in this dataset (fields, variables, etc) 
  
  
  • Why does the data exist? What purpose does it serve? 
  
  
  • How is it collected and used? Include information about frequency, people, tools and instruments. 
  
  
  • Who is responsible for capturing this data? What training is provided? 
  
  
  • Who was consulted in the design of this dataset? 
  
  
  • How will you check your work and make sure you’re following the plan? 
  
  

  Questions about documentation, organization, and storage 

  
  
  • Are there any standards that relate to this data? 
  
  
  • What documentation will be created to make the data understandable to others (metadata)? 
  
  
  • What is the file format for this data?  
  
  
  • What file naming conventions are used? 
  
  
  • Where will data be stored? How will it be backed-up? 
  
  

  Questions about access, sharing, and re-use 

  
  
  • Will this data be shared? How and with whom? 
  
  
  • Describe any privacy, ethical, or confidentiality concerns related to this data 
  
  
  • Is there a plan to protect and secure this data as needed? 
  
  
  • Will this data be made publicly available? Where? How will it be licensed? 
  
  
  • Who owns this data? What laws may relate to this data? 
  
  

  Questions about archiving 

  
  
  • Which data has long-term value?  
  
  
  • How will it be archived and preserved? 
  
  
  • How long will the data be kept? 
  
  

  They also suggested that data management plans could add questions that could help institutionalize equitable data practices. Some of the suggestions included: 

  
  
  • How have we attempted to incorporate community ownership of this data into the data system? 
  
  
  • How will you communicate what the data will/should be be used for? 
  
  
  • How are the identities of people most-affected by use of the data reflected in the identities of the data team, and 
  
  
  • Do the ways that data is collected, managed, and use exclude anyone from participating? 
  
  

  Participants felt strongly that data management plans are best if written before any data is collected or any systems are built, and that plans should be regularly updated. They would be eager to engage in more of these conversations about the design of public data systems.  They also saw parallels between data systems and community gardens, in that both needed a lot of planning, maintenance and care, and often rely on “hidden labor” in order to be successful.  In the penultimate workshop of our initial cohort this Friday, we’ll learn about the importance of designing our procurement systems to reinforce equitable data practices. Additional Resources 

  
  
  • Data Management Plan website – Stanford Libraries 
  
  
  • Guide to writing a data management plan – from Pitt’s library system 
  
  
  • Civic Switchboard Project 
  
  
  Read More
• Data Literacy for Data Stewards Algorithms and Surveillance Technology Workshop
  by Bob Gradeck on 1/12/2023
  On Friday, January 6, we held the ninth workshop in the Data Literacy for Data Stewards series.  Our objectives for this workshop included helping participants understand risks and benefits that can come from the use of public algorithms and data collection devices. We also wanted them to get practice asking critical questions about public algorithms, and learn more about the Pittsburgh Task Force on Public Algorithms. In a definition adopted by the Task Force, "a “public algorithmic system” means any system, software, or process that uses computation, including those derived from machine learning or other data-processing or artificial intelligence (AI) techniques, to aid or replace government decisions, judgments, and/or policy implementations that impact opportunities, access, liberties, rights, and/or safety."  While a robot judge is an extreme fictional example of a public algorithm, public agencies are using algorithms in their work. In Allegheny County, algorithms are used to contribute to staff assessments in the foster care system, detect gunshots, connect residents with resources, manage traffic lights, plan transportation routes, and identify people that may have engaged in plagiarism.  According to the Task Force report, here are many potential benefits from the use of algorithms, including:  

  
  
  • They can help solve complex problems 
  
  
  • Decisions may be made more-fairly and with less bias 
  
  
  • Improved service delivery 
  
  
  • Faster response times 
  
  
  • Efficiency increases 
  
  
  • Public agencies are more-responsive to demands and needs 
  
  
  • Costs are reduced. 
  
  

  There are also potential problems that can come from the use of algorithms. These problems include:  

  
  
  • Error 
  
  
  • Biased data 
  
  
  • Non-representative data  
  
  
  • Privacy and freedom are eroded 
  
  
  • Lack of transparency 
  
  
  • Lack of public ownership and control 
  
  
  • Lack of diversity among the developers of algorithms 
  
  
  • Overconfidence in the ability of technology to solve problems 
  
  
  • Inadequate public deliberation and input 
  
  

   All of these problems can enable systemic racism/discrimination and other structural biases to persist, and algorithms can even amplify them. That’s why the Task Force suggests that public agencies adopt a risk-based approach for assessing the impact of the algorithm against the status-quo.  In groups of 3-4 people, participants in the workshop were given a scenario involving a public algorithm, and asked to list the benefits of the algorithm and the risks of using the algorithm. They were also asked to add to and customize a list of questions developed by the Task Force that people can ask about algorithms.  Here’s one of the scenarios that were developed for the workshop. This algorithm was developed to improve tax collections by identifying returns that are highest potential for an audit.  

  Three years ago, East Versailles township contracted with a private vendor to manage tax collection efforts in the hope that an outside company could bring in more revenue than the existing collection process that involved municipal staff. Additional revenue driven by greater compliance would forestall tax increases. After accounting for the cost of the contract, the revenue received by the local government has remained virtually unchanged since the City outsourced this work. The vendor has been growing increasingly worried that the township may decide not to renew the contract with them when it expires in two years. As a result, the company has proposed the use of a proprietary algorithmic tool that, according to marketing materials “uses a range of data from previous returns, public records, and commercial sources (including credit scores)” to flag people whose returns should be audited because the algorithm suggests that they may not have been paying their fair share. Thanks to the township’s new algorithms ordinance, the company must get approval from the commissioners before purchasing the algorithm from the vendor. As a commissioner, you will attend a public hearing on the matter. 

  Here are the benefits, risks, and questions that people in one of the breakout groups had developed. What are some of the benefits that can come from this algorithm? 

  
  
  • Increase revenue 
  
  
  • Catch people who are breaking the law 
  
  
  • Encourage compliance
  
  

  What are some of the risks or harms that could come from the use of this algorithm? 

  
  
  • Inaccuracies 
  
  
  • Bias – could unfairly target some demographics or low-income communities, particularly in use of credit scores/commercial sources 
  
  
  • Privacy risks 
  
  
  • Seem to be based on company/vendor’s financial interest, rather than true public interest 
  
  

  What questions would you ask at a public meeting about this algorithm (please add to, modify, or delete the existing suggestions from the Task Force’s Public Toolkit)? 

  
  
  • What is the government agency trying to achieve with this system? 
    
    
    
    • What is the policy goal? Increasing revenue, catching delinquent tax filers – or increasing vendor’s bottom-line? 
    
    
    
    
  
  
  • Why is an algorithm the most effective way to achieve this policy goal? 
    
    
    
    • Were alternatives considered and what were they?  
    
    
    • Is an algorithm even better than a human at this task? Do you have data showing its effectiveness? 
    
    
    
    
  
  
  • What are the potential social, racial, economic, and privacy harms? 
    
    
    
    • What is the worst harm that could result if this system were inaccurate? 
    
    
    • How will potential risks and harms be mitigated? 
    
    
    • What are safeguards to ensure it doesn’t target low-income residents or specific demographic groups? 
    
    
    
    
  
  
  • Who is funding the system? 
    
    
    
    • And what is cost/benefit analysis? How much will it cost township to institute this system compared to how much revenue it could bring in? (Especially given that vendor’s contract cost is roughly equivalent to additional revenue that vendor has brought in.) 
    
    
    
    
  
  
  • Who will design the system? 
    
    
    
    • Does the team include diverse inputs? 
    
    
    • How are they seeking community input? 
    
    
    • Has this tool been used in other communities, and what changes have been made in response to previous community input/engagement? 
    
    
    • Will the government agency own and maintain access to the data? 
    
    
    • Will the public be able to view their own information? 
    
    
    • How will public even know about the use of this tool? 
    
    
    
    
  
  
  • What data sources are being used? 
    
    
    
    • How are they guarding against biased data and other common issues with data that exacerbate bias in algorithmic systems? 
    
    
    • Do data sources cost additional money, like using credit scores or other commercial sources? 
    
    
    
    
  
  
  • Who is scrutinizing the system? 
    
    
    
    • Is the system auditable? 
    
    
    • Is there continued evaluation throughout the lifecycle of the system? 
    
    
    • How and when will we know the results of those evaluations? 
    
    
    • Will government be able to challenge results or encourage changes to algorithm once in use? 
    
    
    • Will (and how) members of the public be able to challenge results of the algorithm? 
    
    
    • Do people know when and how the algorithm is applied to them? 
    
    
    • Can someone – member of government or member of public - challenge the result? And how? 
    
    
    • Who rules on a challenge?  Can you challenge underlying data used too, like if someone had identity stolen and algorithm is run while they are in process of repairing that? 
    
    
    
    
  
  
  • Who has ultimate decision-making authority? The vendor, the contractor, the government? Who in the government (commissioners or municipal employees)? 
  
  

  As we do each week, we wrapped-up the session asking everyone to list "better" practices that they'll adopt following their workshop. The bulk of responses mentioned the need for public agencies to establish opportunities for people to be active participants in a public decision-making process when considering the adoption of algorithms, and being transparent about algorithms in use or those that may be used. People outside of government identified a role for questioning the use of algorithms, and playing a role in public education about algorithms.This Friday (January, 13 2023), we will discuss data management. Participants will learn more about the importance of documenting their data management practices through the framework provided by a data management plan. Describing the processes and tools we use to create, collect, manage, and share data enables us to be more-intentional in how we build equity and sustainability into our data systems. If you are interested in participating in the next cohort of our Data Literacy for Data Stewards peer learning series starting in the first quarter of 2023, email us at wprdc@pitt.edu and we will let you know when registration is open.  
  Read More
• Data Literacy for Data Stewards Privacy Workshop
  by Bob Gradeck on 12/22/2022
  Special thanks to Adena Bowden for her support in delivering this series and co-authorship of this blog post.Last Friday, December 16, 2022, we hosted the eighth meeting of our workshop series, “Data Literacy for Data Stewards,” where we discussed practices for protecting privacy when sharing data with the public. The outcomes for our discussion included (1) understanding the risks and benefits that can come from public data sharing, (2) becoming familiar with frameworks for balancing risks and benefits, and (3) learning about methods to protect people included in data. Data privacy is a broad topic. Other areas related to data privacy we didn’t cover include personal privacy protection; privacy issues created by extractive data collection, data sharing, surveillance, and data manipulation; risks resulting from data management, data security, and data storage practices; and decisions about collecting and publicly releasing data. To begin thinking about privacy when sharing data, we talked about what kinds of information fall into the category of Personally Identifiable Information (PII). PII is data that can directly identify an individual, like personal identifiers (e.g. name, social security number, credit card number), contact information (e.g. physical address, phone number), and medical data (e.g. insurance ID, fingerprints, signature), or that can identify an individual in combination with other data, like date of birth, gender, age, or race, and communications data. When considering whether to make data public, it is important to assess the potential benefits along with the potential risks or harms that the information may pose to the people included in the dataset. The following graphics from the Open Data Privacy guide by the Berkman Klein Center for Internet & Society at Harvard University provides a guide for data stewards to assess benefits and risks when sharing data:   When thinking about benefits of sharing data, we must ask the following questions: 

  
  
  • What are some ways that there could be personal, economic, or societal benefit from sharing this data?  
  
  
  • Who are potential users of this information? 
  
  
  • What might malicious actors, governments, corporations, and even other members of the community do with the data to harm people and communities that have little power.  
  
  

  We shared a definition of what constitutes low, medium, and high risks from NIST.  

  
  
  • LOW impact means that an individual may face an inconvenience, such as having to change a phone number;
  
  
  • MODERATE impacts can include monetary loss due to identity theft, discrimination, denial of benefits, and possible blackmail;
  
  
  • HIGH impacts include "serious physical, social, or financial harm, resulting in potential loss of life, loss of livelihood, or inappropriate physical detention."  
  
  

  Several options exist for managing risks associated with sharing data, and we used a case study from New York City to talk about the following things that could have been done to protect the identities of people in sharing a dataset of taxi trips: 

  
  
  • Records or fields can be removed to protect sensitive information; 
  
  
  • Data can be generalized e.g. decreasing the accuracy of geographic coordinates or generalizing dates and times; 
  
  
  • Data can be aggregated to protect individual records;
  
  
  • Records can be suppressed where someone might be identified in the data; 
  
  
  • Anonymous identifiers can be created to prevent identification; 
  
  
  • Random noise can be inserted into a dataset for obfuscation; 
  
  
  • Other statistical techniques such as differential privacy and synthetic data can be applied to a dataset to address privacy while releasing data of value. 
  
  

  Participants worked together in breakout groups to apply these frameworks to the following scenario: 

  “You work at the County and you were asked to make a version of the 911 emergency response data publicly available to cut down on the number of Right to Know requests. The burden of responding to each one of those requests can be substantial.”   

  911 Data includes:   

  
  
  • Dispatch date and time;   
  
  
  • Response time;  
  
  
  • Address;  
  
  
  • Coordinates;  
  
  
  • Type of call (structure fire, cardiac arrest, domestic disturbance, overdose, etc.);  
  
  
  • Responding agency;  
  
  
  • Unstructured notes from the dispatcher; 
  
  

  We asked participants to think about what could be gained by sharing this data. They saw the following benefits in its use: 

  
  
  • Saving county staff time in responding to Right to Know requests;  
  
  
  • Promoting government transparency;  
  
  
  • Providing an understanding of community needs and demand for services; 
  
  
  • Informing public service budgets;  
  
  
  • Locating overdose prevention and other crisis services; 
  
  
  • Exploring potential alternatives to policing; 
  
  
  • Investigating response times. 
  
  

  Participants were concerned about the possible harms that could come from sharing this data. These include: 

  
  
  • Further stigmatizing or stereotyping communities and providing justification for disinvestment and discrimination; 
  
  
  • Justification for over-policing; 
  
  
  • Risks to domestic violence callers; 
  
  
  • Disclosure of health conditions; 
  
  
  • Monitoring emergency response to identify opportunities to commit crimes; 
  
  
  • People may be reluctant to call 911 due to fear of exposure; 
  
  
  • Disclosure of LGBTQ+ identities and safe spaces; 
  
  
  • Identifying individuals who placed the calls; 
  
  
  • Exposing people to extortion, phishing attacks, predatory practices, and solicitation of unwanted services; and 
  
  
  • Damaging someone’s reputation. 
  
  

  Participants assessed the risk/benefit of sharing the data as-is using the framework provided in the Open Data Privacy guide .   Participants proposed the following mitigation strategies to protect the privacy of the individuals in the dataset, then re-assessed risks and benefits: 

  
  
  • Generalizing or aggregating addresses/ coordinates (city neighborhoods, blocks, etc.); 
  
  
  • Redacting names/ identifiers from unstructured notes or removing unstructured notes altogether; 
  
  
  • Generalizing or aggregating date and time of the call;  
  
  
  • Generalizing the type of the call; 
  
  
  • Removing sensitive records or redacting calls with small numbers (K-anonymity). 
  
  

  In closing, participants expressed the critical need to protect the privacy of individuals represented in data. The discussion mentioned how important it is to think about who might be affected when data is made public, especially those who are most vulnerable to harm. The group suggested we all consider who is at the table (and who is not) when deciding which information to share and how, and to carefully weigh the benefits against the potential for harm.  For more information about making data publicly available, we encourage you to check out the following resources: 

  
  
  • Open Data Privacy guide;
  
  
  • NIST Guide to Protecting the Confidentiality of Personally-Identifiable Information;
  
  
  • WPRDC Data Guide page on Protecting Privacy;
  
  
  • Data SF Open Data Release Toolkit 
  
  

  Resources covering other aspects of privacy and data protection (just scratching the surface with these links): 

  
  
  • Our Data Bodies Digital Defense Playbook;
  
  
  • Electronic Frontier Foundation;
  
  
  • Future of Privacy Forum;
  
  
  • Privacy resources from the Library Freedom Project;
  
  
  • OCHA Data Responsibility Guidelines 
  
  

  On Friday, January 6, 2022, we will continue our series with a workshop on algorithms and surveillance. Participants will learn techniques for evaluating technology and how to prevent harm from technology that is already in use. If you are interested in participating in the next cohort of our Data Literacy for Data Stewards peer learning series starting in the first quarter of 2023, email us at wprdc@pitt.edu and we will let you know when registration is open.  
  Read More
• Data Literacy for Data Stewards Data Visualization Workshop
  by Bob Gradeck on 12/16/2022
  Note - special thanks to Adena Bowden for co-authoring this post.On Friday, December 9, 2022, we met for the seventh installment of our virtual “Data Literacy for Data Stewards” workshop series to build empathy into our data visualization practices.  Before the workshop, we asked participants to create and re-make a data visualization after reading the “Do No Harm Guide: Applying Equity Awareness in Data Visualization” by Jonathan Schwabish and Alice Feng, published by the Urban Institute. By refining their initial visualization, participants were able to immediately adopt principles from the guide into their practices. We reviewed a few examples at the beginning of the workshop.  We then directed participants to review and discuss a summary of recommendations from page 41 of the “Do No Harm Guide.” Participants expected to use the following recommendations most often: 

  
  
  • Ordering labels purposefully; 
  
  
  • Communicating with people and communities in focus; 
  
  
  • Critically examining the data used in the visualization; 
  
  
  • Using person-first and inclusive language, labels, icons, and shapes; 
  
  
  • Reflect lived experience in the data visualization; 
  
  
  • Considering missing groups; 
  
  
  • Considering alternatives to labeling the “other” catch-all category; 
  
  
  • Considering the needs of the audience. 
  
  

  Attendees planned to include these recommendations into their own work by: 

  
  
  • Centering lived experiences; 
  
  
  • Critically examining all features of data visualizations; 
  
  
  • Ensuring that the labels reflect how people identify themselves;  
  
  
  • Ensuring communities are directly involved in creating the data visualization. 
  
  

  Our group would add the following  advice to the guide's recommendations: 

  
  
  • Making data visualizations more engaging; 
  
  
  • Being mindful of the audience, purpose, and intentions up front; 
  
  
  • Creating data visualizations in a collaborative, iterative process; 
  
  
  • Adopting accessible design, including choice of colors and alt. text; 
  
  
  • Using less-stigmatizing language (e.g. “income qualifying” rather than “low income”); 
  
  
  • Providing multiple ways to interact with the data. 
  
  

  We then critiqued a visualization produced by KFF together through the lens of the “Racial Equity in Data Visualization Checklist” on page 42 of the “Do No Harm Guide.” Some of our observations and notes were that: 

  
  
  • Subgroup labels are not person-centered; 
  
  
  • Using “better,” “worse,” and other comparative language assigns value and can cause harm by implying some people are “less than;”
  
  
  • While the colors didn’t stereotype, the color choices could have been more-accessible; 
  
  
  • Racial subcategories could be ordered alphabetically rather than ordered from the “worst” to “best” performing groups; 
  
  
  • Each individual racial group could have been presented as separate charts; 
  
  
  • The chart centers whiteness, since all other races listed are compared to the white population; 
  
  
  • More context about structural causes of health disparity could be presented. 
  
  

  For more information on creating and consuming data visualizations in a diverse, equitable, and inclusive way, we encourage you to check out the following resources: 

  
  
  • Do No Harm Guide: Applying Equity Awareness in Data Visualization by Jonathan Schwabish and Alice Feng, published by the Urban Institute 
  
  
  • Dear Data, a year-long, analog data drawing project by Giorgia Lupi and Stefanie Posavec 
  
  
  • Storytelling With Data: A Data Visualization Guide for Business Professionals 
  
  
  • ColorBrewer: Color Advice for Maps 
  
  

  This Friday (December 16, 2022), we will discuss data privacy. We will talk about the risks and benefits that can come from data sharing, become familiar with frameworks for balancing harms and positive outcomes, and learn about methods to protect people included in data. If you are interested in participating in the next cohort of our Data Literacy for Data Stewards peer learning series starting in the first quarter of 2023, email us at wprdc@pitt.edu and we will let you know when registration is open.  
  Read More
• Data Literacy for Data Stewards Dashboard Workshop
  by Bob Gradeck on 12/6/2022
  Special thanks to Adena Bowden for co-authoring this blog post. We’re halfway through the first cohort of our twelve-week virtual data literacy series, “Data Literacy for Data Stewards.” Over the past six workshops, we’ve discussed different topics including missing data, stigmatization, context, and data classification systems. This past Friday, December 2nd, we discussed data dashboards.  Dashboards “are a way to aggregate, analyze and visualize various data streams from different sources” (Sadowski, 2021). While dashboards can provide important context about urban spaces in a central interface, dashboards also have implications for how communities are understood, experienced, planned, and managed. It is important that we think critically about such technologies, understanding and acknowledging their limits and the consequences of their use.The outcomes of our discussion on Friday included (1) understanding the power and privilege people have in defining issues and communities through dashboards, (2) developing better practices for designing community dashboards, and (3) becoming more informed consumers of data dashboards.  We started out in breakout groups, asking participants to dissect one of the following four dashboards: 

  
  
  • Charlotte/Mecklenburg Quality of Life Explorer 
  
  
  • Durham Neighborhood Compass 
  
  
  • City of San Diego’s Strategic Plan Dashboard 
  
  
  • Performance Seattle 
  
  

  We asked participants to reflect on some of the following prompts or to add their own observations when thinking about their assigned dashboard: 

  
  
  • Who is centered in the design of the dashboard? 
  
  
  • What is its purpose? 
  
  
  • What level of data literacy is needed to use it? 
  
  
  • How does it approach data context issues? 
  
  
  • How appropriate is the data being used? Is anything missing? 
  
  
  • Does the dashboard highlight structural causes of issues? 
  
  
  • Could use of this dashboard lead to harm? 
  
  
  • What actions might you take as a result of using this dashboard? 
  
  

  Participants noted that both the Charlotte/Mecklenburg dashboard and the Durham dashboard present a more contextual story that portrays the complexities of a community through data without reducing it to management targets and performance metrics. The scope of the data is broad, and data is presented in different ways, including maps, charts, and figures. While these dashboards aren’t perfect, they do allow people using them to get a better sense of what the community is like. The dashboards for San Diego and Seattle are a performance management tool. These two dashboards create the impression that cities can be managed by pulling a lever when monitoring analytics and data visualizations. Participants found these dashboards to be less accessible and useful than the other two. They are not community or resident-centered, but rather focus the design on things that are important to people in power. One participant likened these performance-based dashboards to Lego sets, where the designers seem to think that they can add, remove, and replace blocks to improve communities, In the workshop, people believed that data dashboards can be very useful, but felt that we can take a more-critical approach to designing and using them. They also acknowledged that by reducing the experience of living in the city to data on a dashboard, we lose nuance, context, and human experience.  Participants created a list of practices that we can adopt to design better dashboards  

  
  
  • Ask ourselves, do we need this? 
  
  
  • What is its purpose? 
  
  
  • Keep it simple, especially for dashboards that are designed for residents. 
  
  
  • Keep broad usability as a central guiding principle. 
  
  
  • Understand who the intended audience is and their purpose for using the tool. 
  
  
  • Consider how the information will serve the audience.  
  
  
  • Provide enough context enabling people to understand the data along with its limitations. 
  
  
  • Include input from communities in the design process. 
  
  
  • Provide ways to capture and incorporate community feedback 
  
  

  The group also thought about questions that we should ask frequently to become better dashboard consumers: 

  
  
  • How are people’s lived experiences reflected in the data? 
  
  
  • Who made this dashboard? 
  
  
  • For whom was it created?  
  
  
  • Why was it created? 
  
  
  • Where does the data used to create the dashboard come from, and what do we know about it? 
  
  
  • What was the reason for selecting the data and visualizations featured on the dashboard? 
  
  
  • What isn’t being presented?  
  
  
  • What is the dashboard telling us (intentionally and unintentionally)? 
  
  
  • How will this dashboard impact the community it represents? 
  
  
  • What do people need in order to access and use the dashboard?  
  
  

  For more information on creating and using data dashboards in a way that accurately reflects and contextualizes the range of experiences of people and communities represented in data, we encourage you to check out the following resources: 

  
  
  • Shannon Mattern, “Mission Control: A History of the Urban Dashboard,” Places Journal, April 2014.    
  
  
  • Shannon Mattern, “Interfacing Urban Intelligence,” Places Journal, March, 2015.    
  
  
  • Designing Data Platforms for Action and Influence: Lessons Learned From a Case Study of Five Data Platforms. USC Price (Sol Price School of Public Policy) and USC Annenberg (School for Communication and Journalism). September 2021.  
  
  
  • Kitchin, Rob & McArdle, Gavin. (2016). Urban data and city dashboards: Six key issues. Working Paper 
  
  
  • Sadowski, J. (2021).  ‘Anyway, the dashboard is dead’: On trying to build urban informatics. New Media & Society, 0(0).  
  
  
  • Rob Kitchin, Sophia Maalsen, Gavin McArdle, The praxis and politics of building urban dashboards, Geoforum, Volume 77, 2016, Pages 93-101, ISSN 0016-7185 
  
  
  • Rob Kitchin, Tracey P. Lauriault & Gavin McArdle (2015) Knowing and governing cities through urban indicators, city benchmarking and real-time dashboards, Regional Studies, Regional Science, 2:1, 6-28 
  
  

   This coming Friday (December 9, 2022), we will discuss data visualization. We’ve asked participants to create a data visualization based on a dataset from the Household Pulse Survey and then to remake their visualization after reading the “Do No Harm Guide: Applying Equity Awareness in Data Visualization” by Jonathan Schwabish and Alice Feng published by the Urban Institute. We will discuss practices we can adopt to approach data visualizations through a lens of diversity, equity, and inclusion.  If you are interested in participating in the next cohort of our Data Literacy for Data Stewards peer learning series starting in the first quarter of 2023, email us at wprdc@pitt.edu and we will let you know when registration is open. 
  Read More
• Data Literacy for Data Stewards Classification Workshop
  by Bob Gradeck on 11/23/2022
  Friday, November 18 was week five of twelve of our virtual data literacy series, “Data Literacy for Data Stewards.” During our workshop, we explored the different classification systems we use for data and the values, assumptions, and power hierarchies that are embedded within them. We worked together to explore how we can maximize value and minimize harm from the use of classification systems in data and we co-created a list of practices that we can adopt to develop and use classification systems to advance equity and justice.   To kick off the workshop, participants listed systems that sort, categorize, classify, or group things: 

  
  
  • Library books (Library of Congress or Dewey decimal system)  
  
  
  • Dublin Core Metadata Schema 
  
  
  • Employee hierarchy and payroll 
  
  
  • Material Safety Data Sheets – categorizes different chemicals by how hazardous they are 
  
  
  • Type of appointment scheduled at a medical center 
  
  
  • Relationship types – friend; partner; platonic partner; best friend; acquaintance; etc 
  
  
  • Dishwasher racks 
  
  
  • Grocery store aisles 
  
  
  • Cancer staging system 
  
  
  • International Classification of Diseases (ICD) 
  
  
  • Organizational Maturity 
  
  
  • Types of Kosher Foods 
  
  
  • Body Mass Index (BMI) 
  
  
  • Alcohol Use Disorders Identification Test (AUDIT) scores 
  
  
  • Taxonomy 
  
  
  • Logic Models (inputs, activities, outputs, outcomes, impacts) 
  
  
  • Grammy nomination categories 
  
  
  • Ballot types 
  
  
  • Military Rank 
  
  
  • National Incident Based Reporting System (NIBRS) for crime data 
  
  
  • Geographical Boundaries (neighborhood, council district, fire zones, census tract, ZIP codes) 
  
  
  • Race and ethnicity classifications 
  
  
  • Billing codes  
  
  
  • The Food Pyramid 
  
  
  • Maslow’s Hierarchy of Needs 
  
  

   After this list-building activity, we had a full-group discussion where people shared examples of how classification systems can be helpful, harmful, or reinforce existing hierarchies and power structures. Some thoughts from the group: 

  
  
  • If you don’t measure something, you can’t appropriately address it. Following World War II, France no longer collects data on race, ethnicity, or religion, fearing atrocities of the Nazi occupation could be revisited. As a result, it is impossible to accurately measure the disadvantages experienced by groups of people defined by race, ethnicity, or national origin.  
  
  
  • Some of our classification systems are rooted in dominant hierarchies. The Dewey Decimal System for classifying library materials reflects a white, heterosexual, cisgender, patriarchy of the global north. The system perpetuates colonial mindsets in how we organize and make information accessible, and de-values content provided by non-dominant groups.  Here’s an example of how a group of people from Northeastern University are working to confront the colonial nature of archives through their attempt to Decolonize the Archive. 
  
  
  • Geographic boundaries like Zip codes, municipalities, and voting districts may be useful administrative boundaries, but they may not reflect how residents define their communities. 
  
  
  • Classification systems may hold benefits for some, but may limit others. Movie rating systems protect young audiences from being exposed to age-inappropriate material, but may censor freedom of speech. 
  
  

   When we attempt to bring order to data by cleaning or classifying it, we lose nuance and context. We asked participants to share thoughts and examples of what is lost when we try to bring order and structure to data. Examples include:  

  
  
  • People who do not identify with existing racial, ethnic, or gender categories, such as people who are of a non-binary gender, those that identify as a member of more than one race, or people whose racial identity is excluded as a response option, such as Latinx people, and those of Middle-Eastern or North-African descent. 
  
  
  • Classification systems that result in missing or inaccurate data, such as the Uniform Crime Reporting (UCR) classification system, which captures only the highest-ranking crime in the hierarchy for incidents where more than one crime was reported. Police departments including Pittsburgh’s are now transitioning to the National Incident Based Reporting System (NIBRS), which captures each separate crime tied to incidents with multiple crimes. 
  
  

   Next, participants worked together in breakout groups to co-create lists of questions about classification systems that we can use to challenge dominant power structures and hierarchies. Their questions include: 

  
  
  • Who designed the classification system? What power do they hold? 
  
  
  • If the system classifies people, what power do those being classified hold? 
  
  
  • Why was this classification system created?  
  
  
  • What was the process behind the creation of the classification system?  
  
  
  • Did people captured in the data have the power to design the classification system? 
  
  
  • Is this system public? Is it easy to understand? Is it accessible (language, location) 
  
  
  • Is the system culturally appropriate? 
  
  
  • How will the system be used? How might it be used by others? 
  
  
  • What relationships are described or implied by the schema? 
  
  
  • Are there other ways to classify or represent the data?  
  
  
  • Who benefits from this classification system? Who might be harmed? 
  
  
  • Who or what is included in the system? Who/what might be left out? 
  
  
  • What biases may exist in the system? What unintended consequences might result from using this system? 
  
  
  • How does this classification system affect the perception of the subject matter in question? 
  
  
  • Can self-reported values be captured (e.g., How would you describe yourself? Which do you think is best for describing you?)  
  
  

   Finally, as a full group, we discussed practices that we can adopt to design and use classification systems to advance equity and justice. Attendees would like to adopt these "better" practices: 

  
  
  • Adopt more inclusive policies, involving individuals from each subgroup so as not to cause unintended harm; 
  
  
  • Provide opportunities for feedback when classifying data; 
  
  
  • Pair quantitative with qualitative data; allowing room to provide context and capture nuance 
  
  
  • Investigate and acknowledge potential limitations and bias when using well-established or popular classification systems; 
  
  
  • Consider whether classification systems are necessary. We now have complex and powerful tools and technology that may allow for more nuanced organization of information; 
  
  
  • Consider historical and other contexts;  
  
  
  • Ask: How have we seen similar data classified before? Who was at the table then? Who wasn’t?; What policies are in place that may have led to that? What are the assumptions behind groupings? How do they interface with power dynamics?; 
  
  
  • Think about natural groupings versus non-natural groupings. When collecting data, see if there are natural patterns that emerge that can be used to organize the information; 
  
  
  • Decide whether classification systems will be viewed internally or made public. If we’re putting these schemas out into the public, we must be intentional about quality and ethics; 
  
  
  • Work to shift the balance of power from those who have historically been in power to those who haven’t by emphasizing equity when classifying information. Can historically marginalized groups gain power from categorized data?; 
  
  
  • Note the limitations of classification systems that we're utilizing. For example, record when data is gathered on a male-female binary. 
  
  

   For more information about creating and using classification systems in an ethical and accountable manner, check out the following resources: 

  
  
  • Creating Inclusive Surveys 
  
  
  • Comment Letter on Equitable Data Engagement and Accountability 
  
  
  • Measuring Sex, Gender Identity, and Sexual Orientation 
  
  
  • The Release of the Equitable Data Working Group Report 
  
  

   Next time (Friday, December 2, 2022), we will discuss data dashboards. We will work to uncover and understand the power and privilege people have in defining how communities are framed and represented through dashboards and data analyses.  Participants will develop practices that they can use to ensure that dashboards and data analyses accurately reflect and contextualize the range of experiences of people and communities represented in data.  If you are interested in participating in the next cohort of our Data Literacy for Data Stewards peer learning series starting in the first quarter of 2023, email us at wprdc@pitt.edu and we will let you know when registration is open. 
  Read More
• Data Literacy for Data Stewards Context Workshop
  by Bob Gradeck on 11/14/2022
  Special thanks to Adena Bowden for co-authoring this post.On Friday, November 4, 2022, we met for the fourth installment of our twelve-week virtual data literacy series, “Data Literacy for Data Stewards,” where we talked about the importance of having contextual information about the data that we work with.  Without context, it can be difficult to know who is creating data and how data is created. It can also be hard to identify how data may be biased, inaccurate, or incomplete. Additionally, a lack of context can reinforce the status quo.   To start the session with a fun activity, we presented a few datasets that were stripped of important context (titles, column names, and some fields) and asked participants to try to identify what the datasets were representing. Here’s an example - can you guess what it is?  This dataset is from the NYC Squirrel Census, a multimedia science, design, and storytelling project in New York City’s Central Park. This activity, while fun, also pointed out the importance of providing context around data. Data is connected to the context of a place, and what might look like the same data can come from very different sources where people have very different ways of doing things. Context about data is needed if people are going to evaluate data quality and potential biases and use data to challenge power. Some important questions that can be asked to uncover context about data include:  

  
  
  • Motivation: Why does this data exist, who created it, and what's the history?  
  
  
  • Composition: What's included or represented in the data? What isn't? Is any of the data sensitive? Do any standards, laws, or policies apply to this data? 
  
  
  • Collection Process: How was the data collected or created? Who did the work? What tools were used? Was consent or fair compensation offered?  
  
  
  • Transformations: How was the data generated, processed, cleaned, and labeled? Were any algorithmic tools used?  
  
  
  • Uses: Who uses the data internally and externally? How is it used by people in different contexts? Are there any potential use-cases? 
  
  
  • Sharing: Is the data shared? Is it publicly available? What rules exist around sharing 
  
  
  • Maintenance: How is the data and data infrastructures funded, managed, and maintained? 
  
  

  After being introduced to these types of questions, participants worked together in breakout groups to practice developing interview questions of a data steward from one of the five scenarios presented in the workshop. An interview with someone that knows a lot about the context of a dataset can uncover many important details. We encouraged participants to think about questions whose answers could (1) highlight structural causes of an issue, (2) uncover data quality issues, biases, and limitations, (3) expose power, privilege, and hypocrisy, and (4) enable people to contest unjust systems and claim power. Several of our breakout groups chose the following scenario tied to a 3-1-1 non-emergency service request system. The City Manager in your community has proposed a new budgeting system that allocates a portion of the budget to the number of service requests made to the new 3-1-1 system. 3-1-1 streamlines the reporting processes for non-emergency service requests for service, which include pothole repair, vacant lot cleanup, snow removal, and playground maintenance. As someone who works to provide social services in many communities where people have low incomes, you’re worried about the biases that may result in the use of this data to determine budget allocations through a formula or algorithm. The city council representative in District 10 shares your concerns and has scheduled a call with you and the director of the 3-1-1 program.   Here are some of the questions our participants felt were important to ask the 3-1-1 data steward: Motivation  

  
  
  • Why does this data exist? What is the purpose of 311 service requests? 
  
  
  • How do you currently allocate a budget to address complaints?  
  
  
  • How are you relating the number of 3-1-1 complaints to the budget?   
  
  
  • What do you think the number of service requests tells us about budget needs versus other types of data points?  
  
  

  Composition  

  
  
  • What data is in the 311 data set? (I.e., location, resident/business owner/etc., variables about the person, etc.?) What might be missing or overrepresented? 
  
  
  • How do people know about 3-1-1? Who uses 3-1-1?  
  
  
  • What are the most common types of complaints?  
  
  
  • Where are complaints addressed the fastest? (Is it in higher income neighborhoods?)  
  
  

  Collection Process  

  
  
  • How is the geography of the caller captured? 
  
  
  • How are duplicate requests handled in the system? 
  
  
  • What barriers may exist for people to access and use the 311 system? 
  
  
  • What percentage of tickets are submitted via phone versus online?  
  
  
  • What are the pre-defined categories? What happens with the unique calls and how do they get sorted and included in the data?  
  
  
  • Is there a difference in recording/reporting methods depending on who answers the 3-1-1 call that day?  
  
  
  • How does the data collection differ in well- and under-represented areas? What efforts might have been made to correct imbalances identified? 
  
  
  • Are you planning on making inferences to underrepresented areas based on other data? 
  
  

  Transformations  

  
  
  • Are you able to attribute the number of complaints to a single source? Is this a neighborhood issue or a personal one?  
  
  
  • How are 311 requests distributed across the city? Have calls ever been mapped? 
  
  

  Uses  

  
  
  • Is the data publicly available?  
  
  
  • Are follow-up complaints reported in the data set too? Can they be separated out?  
  
  

  Sharing  

  
  
  • Who can see 3-1-1 data? Are there any publicly facing dashboards or other sharing of the information? 
  
  
  • Are there reports issued on requests received and resolved?  
  
  
  • How many people use this for community development planning?  
  
  
  • How is personal information protected? 
  
  

  Maintenance  

  
  
  • How many complaints have been successfully addressed? And in what categories? 
  
  
  • How often is the data updated? 
  
  
  • Are data audited for quality? How often? 
  
  

  Other 

  
  
  • What additional information will inform the budget allocations? 
  
  
  • How does the 311 data align or not align with actual investments in cleanups, physical infrastructure, etc.? 
  
  

  To end our session, we brainstormed practices we can build to capture important context about data. Here’s what participants shared: Practices for people that have some power or control over data  

  
  
  • Consider how reports will affect different populations (especially those that experience oppression) and the unintended consequences that might result from sharing data 
  
  
  • Involve the community in data collection 
  
  
  • Pay people fairly to collect data and share information 
  
  
  • Practice “using” data before the data collection process is finalized  
  
  
  • Talk with people who will be affected by the decisions made based on the data collected 
  
  
  • Shift power to those who don’t historically have the power   
  
  

  Practices for people that don’t have power or control over data  

  
  
  • Ask about how the data is collected and the reason(s) behind the methodology 
  
  
  • Think of the media format used to capture the data and what biases or barriers that format might introduce 
  
  
  • Advocate for data transparency  
  
  
  • Ask hard questions of folks who do have power over data that can reveal inequities and bias 
  
  
  • Record limitations in data  
  
  
  • Ask if you can be involved in data analysis and dissemination  
  
  
  • When using data without context, ask for context; if you can’t get context, be transparent about the context that is missing and how it impacts the analysis/project 
  
  

  Reading list / sources: 

  
  
  • “Datasheets for Datasets” by Gebru et al. 
  
  
  • Data Feminism by D’Ignazio and Klein 
  
  
  • All Data Are Local by Loukissas 
  
  

  For examples of data guides to capture and share context, we encourage you to check out the following: 

  
  
  • Allegheny County Property Assessments - Property Assessment Data User Guide Allegheny County - WPRDC 
  
  
  • WPRDC Data Guides 
  
  
  • A Guide to Home Mortgage Disclosure Act Data | Urban Institute 
  
  
  • Crime and Punishment in Chicago (archive.org) 
  
  

  This week (November 18, 2022), we will continue our Data Literacy series with a discussion on classification and representation. We'll explore the different classification systems we use for data, and the values, assumptions, and power hierarchies that are embedded within them. We'll work together to explore how we can maximize value and minimize harm from the use of classification systems in data and create practices that accurately represent how people want to be represented in them. If you are interested in participating in the next cohort of our Data Literacy for Data Stewards peer learning series starting in the first quarter of 2023, email us at wprdc@pitt.edu and we will let you know when registration is open. 
  Read More