Algorithmic Accountability is Possible in ICT4D

As we saw recently, when it comes to big data for public services there needs to be algorithmic accountability. People need to understand not only what data is being used, but what analysis is being performed on it and for what purpose.

Further, complementing big data with thick, adjacent and lean data also helps to tell a more complete story of analysis. These posts piqued much interest and so this third and final instalment on data offers a social welfare case study of how to be transparent with algorithms.

A Predictive Data Tool for a Big Problem

The Allegheny County Department of Human Services (DHS), Pennsylvania, USA, screen calls about the welfare of local children. The DHS receives around 15,000 calls per year for a county of 1.2 million people. With limited resources to deal with this volume of calls, limited data to work with, and each decision a tough and important one to make, it is critical to prioritize the highest need cases for investigation.

To help, the Allegheny Family Screening Tool was developed. It’s a predictive-risk modeling algorithm built to make better use of data already available in order to help improve decision-making by social workers.

Drawing on a number of different data sources, including databases from local housing authorities, the criminal justice system and local school districts, for each call the tool produces a Family Screening Score. The score is a prediction of the likelihood of future abuse.

The tool is there to help analyse and connect a large number of data points to better inform human decisions. Importantly, the algorithm doesn’t replace clinical judgement by social workers – except when the score is at the highest levels, in which case the call must be investigated.

As the New York Times reports, before the tool 48% of the lowest-risk families were being flagged for investigation, while 27% of the highest-risk families were not. At best, decisions like this put an unnecessary strain on limited resources and, at worst, result in severe child abuse.

How to Be Algorithmically Accountable

Given the sensitivity of screening child welfare calls, the system had to be as robust and transparent as possible. Mozilla reports the ways in which the tool was designed, over multiple years, to be like this:

  • A rigorous public procurement process.
  • A public paper describing all data going into the algorithm.
  • Public meetings to explain the tool, where community members could ask questions, provide input and influence the process. Professor Rhema Vaithianathan is the rock star data storyteller on the project.
  • An independent ethical review of implementing, or failing to implement, a tool such as this.
  • A validation study.

The algorithm is open to scrutiny, owned by the county and constantly being reviewed for improvement. According to the Wall Street Journal the trailblazing approach and the tech are being watched with much interest by other counties.

It Takes Extreme Transparency

It takes boldness to build and use a tool in this way. Erin Dalton, a deputy director of the county’s DHS and leader of its data-analysis department, says that “nobody else is willing to be this transparent.” The exercise is obviously an expensive and time-consuming one, but it’s possible.

During recent discussions on AI at the World Bank the point was raised that because some big data analysis methods are opaque, policymakers may need a lot of convincing to use them. Policymakers may be afraid of the media fallout when algorithms get it badly wrong.

It’s not just the opaqueness, the whole data chain is complex. In education Michael Trucano of the World Bank asks: “What is the net impact on transparency within an education system when we advocate for open data but then analyze these data (and make related decisions) with the aid of ‘closed’ algorithms?”

In short, it’s complicated and it’s sensitive. A lot of convincing is needed for those at the top, and at the bottom. But, as Allegheny County DHS has shown, it’s possible. For ICT4D, their tool demonstrates that public-service algorithms can be developed ethically, openly and with the community.

Stanford University is currently examining the impact of the tool on the accuracy of decisions, overall referral rates and workload, and more. Like many others, we should keep a close watch on this case.


3 Data Types Every ICT4D Organization Needs – Your Weekend Long Reads

After five years researching the effectiveness of non-profit organizations (NPOs) in the USA, Stanford University lecturer Kathleen Kelly Janus found that while 75% of NPOs collect data, only 6% feel they are using it effectively. (Just to be clear, these were not all tech organizations.)

She suggests the reason is because they don’t have a data culture. In other words, they need to cultivate “a deep, organization-wide comfort level with using metrics to maximize social impact.” Or, in ICT4D speak, they need to be data-driven.

Perhaps NPOs feel that if they start collecting, analysing and using big data, that need will be satisfied. But one cloud server of big data does not a data culture make. While big data can be a powerful tool for development, there are three other data types that could significantly improve the impact of any ICT4D intervention.

Thick data

Technology ethnographer, Tricia Wang, warns us about the dangers of only looking to big data for the answers, of only trusting large sets of quantitative data without a human perspective. She proposes that big data must be supplemented with “thick data,” which is qualitative data gathered by spending time with people.

Big data excels at quantifying very specific environments – like delivery logistics or genetic code – and doing so at scale. But humans are complex and so are the changing contexts in which they live (especially true for ICT4D constituents). Big data can miss the nuances of the human factor and portray an incomplete picture.

As a real-life example, in 2009 Wang joined Nokia to try to understand the mobile phone market in China. She observed, talked to, and lived amongst low-income people and quickly realised that – despite their financial constraints – they were aspiring to own a smartphone. Some of them would spend half of their monthly income to buy one.

But the sample was small, the data not big, and Nokia was not convinced. Nokia’s own big data was not telling the full story – it was missing thick data, which led to catastrophic consequences for the company.

Adjacent data

Sometimes there is value in overlaying data from other sources onto your own to provide new insights. Let’s call this “adjacent data”. Janus provides the case of Row New York, an organization that pairs rigorous athletic training with tutoring and other academic support to empower youth from under-resourced communities.

To measure success, Row started by tracking metrics like the number of participants, growth, and fitness levels. But how could they track determination or “grit” – attributes of resilient people?

They started recording both attendance and daily weather conditions to show which students were still showing up to row even when it was 4C degrees and raining. “Those indicators of grit tracked with students who were demonstrating academic and life success, proving that [Row’s] intervention was improving those students’ outcomes.”

Pinpointing adjacent data requires thinking outside of the box. Maybe reading Malcom Gladwell or Freakonomics will provide creative inspiration for finding those hidden data connectors.

Lean data

Lastly, there is a real risk in just hoovering up every possible data point in the hope that the answers to increased impact and operational efficiencies will emerge. That’s not referring only to the data security and privacy risks related to the sponge approach. Rather, that’s because it’s easy to drown in data.

Most ICT4D initiatives don’t have the tech or the people to meaningfully process the stuff. Too much data can overwhelm, not reveal insights. The challenge is gathering just enough data, just the data we need – let’s call this the “lean data”. When it comes to data, more is not better, just right is better. In fact, big data can be lean. It’s not about quantity but rather selectiveness.

Lean data is defined by the goals of the initiative and its success metrics. Measure enough to meet those needs. When I was head of mobile at Pearson South Africa’s Innovation Lab, we were developing an assessment app for high school learners called X-kit Achieve Mobile.

With the team we brainstormed the data we needed to serve our goals and those of the student and teacher users. We threw in quite a lot of extra bits based on “Hmm, that would be cool to know, let’s put it in a dashboard.”

The company was also preparing to report publicly on its educational impact, so certain data points were being collected by all digital products. Having a common data dictionary and reporting matrix is something worth considering if you’re implementing more than one product.

After building the app we only really used about 20% of all the reports and dashboards. Only as we iterated did we discover new reports that we actually needed. The fact is that data is seductive, it brings out the hoarder in all of us. We should resist and only take what we need

So, perhaps the path to building a data culture is to always have thick data, be creative about using adjacent data, and keep all data lean.

Image: CC by janholmquist

Every Big Data Algorithm Needs a Storyteller – Your Weekend Long Reads

The use of big data by public institutions is increasingly shaping peoples’ lives. In the USA, algorithms influence the criminal justice system through risk assessment and predictive policing systems, drive energy allocation and change educational system through new teacher evaluation tools.

The belief is that the data knows best, that you can’t argue with the math, and that the algorithms ensure the work of public agencies is more efficient and effective. And, often, we simply have to maintain this trust because nobody can examine the algorithms.

But what happens when – not if – the data works against us? What is the consequence of the algorithms being “black boxed” and outside of public scrutiny? Behind this are two implications for ICT4D.

The Data Don’t Lie, Right?

Data scientist and Harvard PhD in Mathematics, Cathy O’Neill, says that clever marketing has tricked us to be intimidated by algorithms, to make us trust and fear algorithms simply because, in general, we trust and fear math.

O’Neill’s 2016 book, Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy, shows how when big data goes wrong teachers lose jobs, women don’t get promoted and global financial systems crash. Her key message: the era of blind faith in big data must end, and the black boxes must be opened.

Demand Algorithmic Accountability

It is very interesting, then, that New York City has a new law on the books to do just that and demand “algorithmic accountability” (presumably drawing on the Web Foundation’s report of the same name). According to MIT Technology Review, the city’s council passed America’s first bill to ban algorithmic discrimination in city government. The bill wants a task force to study how city agencies use algorithms and create a report on how to make algorithms more easily understandable to the public.

AI Now, a research institute at New York University focused on the social impact of AI, has offered a framework centered on what it calls Algorithmic Impact Assessments. Essentially, this calls for greater openness around algorithms, strengthening of agencies’ capacities to evaluate the systems they procure, and increased public opportunity to dispute the numbers and the math behind them.

Data Storytellers

So, what does this mean for ICT4D? Two things, based on our commitment to being transparent and accountable for the data we collect. Firstly, organisations that mine big data need to become interpreters of their algorithms. Someone on the data science team needs to be able to explain the math to the public.

Back in 2014 the UN Secretary General proposed that “communities of ‘information intermediaries’ should be fostered to develop new tools that can translate raw data into information for a broader constituency of non-technical potential users and enable citizens and other data users to provide feedback.” You’ve noticed the increase in jobs for data scientists and data visualisation designers, right?

But it goes beyond that. With every report and outcome that draws on big data, there needs to be a “how we got here” explanation. Not just making the data understandable, but the story behind that data. Maybe the data visualiser does this, but maybe there’s a new role of data storyteller in the making.

The UN Global Pulse principle says we should “design, carry out, report and document our activities with adequate accuracy and openness.” At the same time, Forbes says data storytelling is an essential skill. There is clearly a connection here. Design and UI thinking will be needed to make sure the heavy lifting behind the data scenes can be easily explained, like you would to your grandmother. Is this an impossible ask? Well, the alternative is simply not an option anymore.

Data Activists

Secondly, organisations that use someone else’s big data analysis – like many ICT4D orgs these days – need to take an activist approach. They need to ask questions about where the data comes from, what steps were taken to audit it for inherent bias, for an explanation of the “secret sauce” in the analysis. We need to demand algorithmic accountability” We are creators and arbiters of big data.

The issue extends beyond protecting user data and privacy, important as this is. It relates to transparency and comprehension. Now is the time, before it’s too late, to lay down the practices that ensure we all know how big data gets cooked up.

Image: CC by kris krüg

Across Africa the Feature Phone is Not Dead – Your Weekend Long Reads

Quartz Africa reports that last year feature phones took back market share from smartphones in Africa. The market share of smartphones fell to 39% in 2017 (from 45%), while feature phones rose to 61% (from 55%).

Quartz Africa sees the reasons as likely to be twofold: first, the growth of big markets, like Ethiopia and DR Congo, which until recently have had relatively low penetration. Second, low price.

Transsion, a little-known Chinese handset manufacturer, now sells more phones than any other company in Africa. It’s three big brands outnumber Samsung’s market share there. The devices are cheap and appealing for new users.

The FT reports that Transsion’s phones are specifically designed for the African market: they have multiple sim-card slots, camera software adapted to better snap darker skin tones, and speakers with enhanced bass (seriously). Many of the feature phone models have messaging apps. The batteries remain on standby for up to 13 days!

What does this mean? That you should freeze your flashy new app project? No! There’s no need to stop planning and developing for a smartphone-enabled Africa. The trend is clear: smartphones become cheaper over time and their uptake increases.

But we know that in Africa, especially, mobile usage is unevenly distributed and these stats are a good reminder that the non-smartphone user base is still huge. Many of us need to remain true to that reality if we want our ICT to be 4D.

The age old question – which mobile channel should we focus on? – has not gone away. And the answer still remains the same: it depends. What is your service? What devices do your users have? What are their usage preferences? Do they have data coverage and, if yes, can they afford data?

Low tech, like IVR and radio, can be beautiful and extremely effective. In a meta-study of education initiatives in Africa, the Brookings Institute found that most technology-based innovations utilize existing tools in new ways. They give Eneza Education as an example, which built its service on SMS (even though there is now an Android app available).

At the same time, apps are certainly rising in the development sector. While not in Africa, the Inventory of Digital Technologies for Resilience in Asia-Pacific found apps to be the dominant channel. From my own experience I’m seeing more apps, often as one part of a mix of delivery channels.

A forthcoming case study in the UNESCO-Pearson initiative is MOPA, a platform for participatory monitoring of waste management services in Maputo, Mozambique. Citizens report issues via USSD, website and, most recently, via Android app.

Usage patterns show that 96% of reports are still sent through USSD, 3% via mobile app, and only 1% through the website. Given that specific user base, and the quick-and-dirty nature of the transaction, it’s not surprising that USSD is a clear winner.

Another example of a channel mix is Fundza, the South African mobile novel library. It started life as a mobisite and now also has an app, which largely provides a window into the same content just in a nice Android skin.

The app is used by less than 1% of users, with the mobisite taking the lion’s share of traffic (via feature phone and smartphone). Fundza is also on Free Basics, where the breakdown is quite different: 65% mobisite, 45% app (perhaps pointing to the benefits of being bundled into someone else’s very well-marketed app).

There are many reasons why individual apps may or may not succeed, and these examples are not meant to downplay their utility. Overall, the world is going to smartphones.

However, the bottom line is that you should not write off the humble feature phone in Africa just yet. It does old tech very well, internet messaging and the mobile web, which for many ICT4D projects is still their bread and butter access channel.

In ICT4D We’re Principled, But Are We Practiced Enough? – Your Weekend Long Reads

Last month CO.DESIGN published the 10 New Principles of Good Design (thanks Air-bel Center for the link). The article, which is based on a set of industrial design principles from the 1970s, makes for important reading.

According to the author, and concerning commercial digital solutions, 2017 was “a year of reckoning for the design community. UX became a weapon, AI posed new challenges, and debate erupted over once rock-solid design paradigms.” What is most interesting — and wonderful to boot — is that many of the “new” principles we, the ICT4D community, have endorsed for years.

Good Design is Transparent

For example, the article calls for transparency in design. Apparently today, “amid a string of high-profile data breaches and opaque algorithms that threaten the very bedrock of democracy, consumers have grown wary of slick interfaces that hide their inner workings.”

We know that user-centered design is participatory and that we should expose the important parts of digital solutions to our users. We believe in telling our users what we’ll do with their data.

Good Design Considers Broad Consequences and is Mindful of Systems

The article warns that in focusing on the immediate needs of users, user-friendly design often fails to consider long-term consequences. “Take Facebook’s echo chamber, Airbnb’s deleterious impact on affordable housing,” as examples. Not for us: we understand the existing ecosystem, are conscious of long-term consequences and design for sustainability.

A Little History Lesson

Today we have principles for sectors — such as refugees, health and government (US or UK version?); for cross-cutting themes — such as identity, gender and mobile money; for research; and the grand daddy of them all, for digital development.

These principles have been developed over a long time. Fifteen years go I wrote a literature survey on the best practices of ICT4D projects. It was based on the work of then research pioneer,, drawing on a range of projects from the early 2000s.

In my paper, put forward seven habits of highly effective ICT-enabled development initiatives. By 2007 the list had grown to 12 habits — many of which didn’t look that different from today’s principles.

Do We Practice What We Preach?

But if these principles are not new to us, are we practicing them enough? Don’t get me wrong, the ICT4D community has come a long way in enlisting tech for social good, and the lessons — many learned the hard way — have matured our various guidelines and recommendations. But should we be further down the line by now?

The principles mostly outline what we should do, and some work has been done on the how side, to help us move from principles to practice. But I think that we need to do more to unpack the why don’t we aspect.

Consider this data point from a recent Brookings Institute report Can We Leapfrog: The Potential of Education Innovations to Rapidly Accelerate Progress (more on this report in a future post). Brookings analysed almost 3,000 education innovations around the world (not all tech-based, just so you know) and found that:

… only 16 percent of cataloged interventions regularly use data to drive learning and program outcomes. In fact, most innovations share no information about their data practices.

We know that we should be data-driven and share our practices. So what is going on here? Do the project managers behind these interventions not know that they should do these things? Do they not have the capacity in their teams? Do they not want to because they believe it exposes their non-compliance with such principles? Or perhaps they feel data is their competitive edge and they should hide their practices?

Time for ‘Fess Faires?

Fail faires are an excellent way to share what we tried and what didn’t work. But what about ‘Fess Faires, where we confess why we can’t or — shock horror — won’t follow certain principles. Maybe it’s not our fault, like funding cycles that ICT4D startups can’t survive. But maybe we should be honest and say we won’t collaborate because the funding pie is too small.

If fail faires are more concerned with operational issues, then ‘fess faires look at structural barriers. We need to ask these big questions in safe spaces. Many ICT4D interventions are concerned with behavior change. If we’re to change our own behavior we need to be open about why we do or don’t do things.

Good Design is Honest

So, on the one hand we really can pat ourselves on the back. We’ve had good design principles for almost twenty years. The level of adherence to them has increased, and they have matured over time.

On the other hand, there is still much work to be done. We need to deeply interrogate why we don’t always practice our principles, honestly and openly. Only in this way will we really pursue a key new principle: good design is honest.

Why Digital Skills Really Matter for ICT4D – Your Weekend Long Reads

In an increasingly online world, people need digital skills to work and live productively. One of the major barriers to digital uptake is a lack of these skills.

Across Africa, seven in ten people who don’t use the Internet say they just don’t know how to use it. This is not only a developing country problem: 44% of the European Union population has low or no (19%) digital skills!

It is no surprise, therefore, that the theme for this year’s UNESCO Mobile Learning Week is “Skills for a connected world”. (It runs from 26-30 March in Paris — don’t miss it!)

Global Target for Digital Skills

At Davos last month, the UN Broadband Commission set global broadband targets to bring online the 3.8 billion people not yet connected to the Internet. Goal 4 is that by 2025: 60% of youth and adults should have achieved at least a minimum level of proficiency in sustainable digital skills.

(I’m not quite sure what the difference is between digital skills and sustainable digital skills.) Having a target such as this is good for focusing global efforts towards skilling up.

The Spectrum of Digital Skills

Digital skills is a broad term. While definitions vary, the Broadband Commission report proposes seeing digital skills and competences on a spectrum, including: 

  • Basic functional digital skills, which allow users to access and conduct basic operations on digital technologies;
  • Generic digital skills, which include using digital technologies in meaningful and beneficial ways, such as content creation and online collaboration; and 
  • Higher-level skills, which mean using digital technology in empowering and transformative ways, for example for software development. These skills include 21st century skills and critical digital literacies.

Beyond skills, digital competences include awareness and attitudes concerning technology use. Most of the people served in ICT4D projects fall into the first and second categories. Understanding where your users are and need to be is important, and a spectrum lens helps in that exercise.

Why Skills Really Matter

Beyond the global stats, goals and definitions, why should you really care about the digital skills of your users, other than that they know enough to navigate your IVR menu or your app?

The answers come from the GSMA’s recent pilot evaluation of its Mobile Internet Skills Training Toolkit (MISTT), implemented last year in Rwanda.

Over 300 sales agents from Tigo, the mobile network operator, were trained on MISTT, and they in turn trained over 83,000 customers. The evaluation found that MISTT training:

  • Gives users confidence and helps them overcome the belief that “the Internet is not for me”;
  • Has the potential to help customers move beyond application “islands” — and get them using more applications/services;
  • Has a ripple effect, as customers are training other people on what they have learned (a study in Cape Town also found this); and
  • Increased data usage among trained customers, which led to increased data revenues for Tigo.

In short, more digital skills (beyond just what you need from your users) presents the opportunity for increased engagement, higher numbers of users and, if services are paid-for or data drives revenue, greater earnings. Now those are compelling ICT4D motivators.

Skills as Strategy

Therefore, we need to see skills development as one of the core components of our:

  • Product development strategy (leveraging users who can interact more deeply with features);
  • Growth strategy (leveraging users who train and recruit other users);
  • Revenue strategy (leveraging users who click, share, and maybe even buy).

But what about the cost, you might wonder? As Alex Smith of the GSMA points out, with the data revenues, for Tigo the MISTT pilot returned the investment within a month and saw an ROI of 240% within a quarter. That’s for a mobile operator — it would be fascinating to measure ROI for non-profits.

To get training, the Mobile Information Literacy Curriculum from TASCHA is also work checking out, as is the older GSMA Mobile Literacy Toolkit.

Image: CC by Lau Rey


Creating Killer ICT4D Content – Your Weekend Long Reads

Creating killer content is critical to ICT4D success. One of the major barriers to digital uptake is a lack of incentives to go online because of a lack of relevant or attractive content.

This weekend we look at resources for creating great content, drawing on lessons from the mhealth and mAgri sectors. If you are not an mhealth or mAgri practitioner, don’t stop reading now. While professions and sectors like to silo, in reality the ICT4D fields overlap enormously. For example, does a programme that educates nurses for improved obstetrics practices fall under mhealth or meducation? The details may differ, but the approaches, lessons and tech might as well be the same. From each sector there is much to learn and transfer to other m-sectors.

Let’s Get Practical and Make Some Content

Not long ago Dr. Peris Kagotho left medical practice to focus on mhealth. Since then she has successfully categorized, edited and contextualized over 10,000 health tips Kenyans. In a four-part blog series, she highlights techniques and learnings for effective and impactful content development. Read about the prerequisites for quality mhealth content; the principles of behaviour change messaging; creating content that is fit for purpose; and scheduling content for impactful delivery.

Making Content Meaningful Without Re-inventing the Wheel

While there is apparently an abundance of openly accessible health content, this alone is insufficient to make the world healthy and happy. The Knowledge for Health (K4Health) project knows the importance of providing the content in the appropriate context and the language of the people who will use it.

K4Health and USAID have therefore created a guide to adapting existing global health content for different audiences with the goal of expanding the reach, usefulness, and use of evidence-based global health content. Fantastic.

+ The Nutrition Knowledge Bank is an open access library of free to use nutrituion content.

Lessons on Content Placement, Format, Data and More

The Talking Book, a ruggedized audio player and recorder by Literacy Bridge, offers agricultural, health and livelihoods education to deep rural communities in four African countries. The UNESCO-Pearson case study on the project highlights key content development approaches and lessons, drawn from over ten years of experience. For example, it’s important not to overload users with too much content; the fist few messages in a content category get played the most, so those are the best slots for the most important messages; and these rural audiences prefer content as songs and dramas over lectures. The content strategy is highly data-driven.

Content Isn’t Delivered in a Vacuum

In 2016, the Government of India launched a nation-wide mobile health programme called ‘Kilkari’ to benefit 10 million new and expecting mothers by providing audio-based maternal and child health messages on a weekly basis. The service was designed by BBC Media Action and the GSMA case study describes its evolution, learnings and best practices, covering content and more. It is useful to zoom out and see the bigger picture of an mhealth initiative, and how content forms one part of the whole.

Image: CC by TTCMobile