Facebook’s Working On A Tool To Help The Blind “See” Images

Facebook connects 1.5 billion people all over the world, but for the blind and visually impaired, it can be difficult to gain access to what has become a vast platform for connectivity. That’s where Facebook’s accessibility team, led by Jeff Wieland, comes in. The sole purpose of the accessibility team is to help people with disabilities have a seamless experience on Facebook, and ultimately help the social network achieve its mission of connecting the world.

Right now, blind and visually impaired people who have access to screen readers — tools used to identify what’s displayed on a screen — can listen to what people are writing on Facebook, but there’s currently no way to figure out what’s going on in the millions of photos shared on Facebook every day.

“You just think about how much of your news feed is visual — and is probably most of it — and so often people will make a comment about a photo or they’ll say something about it when they post it, but they won’t really tell you what is in the photo,” Matt King, Facebook’s first blind engineer, told TechCrunch. “So for somebody like myself, it can be really like, ‘Ok, what’s going on here? What’s the discussion all about?’”

That’s why Facebook is currently working on an artificial intelligence-based object recognition tool to help blind users get an idea of what’s in all of the photos people share on Facebook. King, who started at the company just three months ago, recently showed me how he uses a screen reader to navigate Facebook.

“My view of the page is totally sequential,” King explained to me. “I can’t see the whole thing at one time. I see a little piece.”

As he scrolled down the page, the screen reader would tell King that he’s at a list of six items, which referred to the number of notifications he had at the time. It also told him when he reached a “convo box,” which signaled to him that he could interact with that element and leave a comment.

King eventually scrolled to a friend’s post that featured text and a photo. His friend, Anne, wrote, “Ready for picture day of first grade” accompanied with a photo. Thanks to the object recognition technology Facebook is prototyping, King heard: “This image may contain, colon, one or more people. Child.” Without it, all King would’ve known was that Anne wrote, “Ready for picture day of first grade,” and that she posted a photo — but nothing about what was in the photo. For another photo, the tool told him: “This image may contain colon nature, outdoor, cloud, foliage, grass, tree.” 

In the photo gallery below, you’ll see what a blind person hears read aloud when they’re using a screen reader to browse photos on Facebook.