NOTHING to see here: just an image of an empty street. But the investigator thinks there is more to this than meets the eye. With a few clicks of his mouse, he enhances a featureless shadow cast on the floor, apparently defying the laws of optics to extract a blurry image of two people lurking around the corner.
Technical wizardry like this seems far-fetched. But this isn’t CSI. The investigator is a computer scientist not a detective, and those characters are graduate students not suspects. More importantly, this technology is real, and it is being developed in labs right now.
Advertisement
The science of seeing around corners is new, fast-moving and breathtaking. We are discovering that the shadows are full of visual information that our eyes can’t see. Now, as people develop clever ways to make the invisible visible, they are exposing all manner of potential applications besides forensics. Autonomous cars that spot hidden hazards. Cameras that direct fire crews to people trapped in burning buildings. Endoscopes that guide surgery in unreachable parts of the body.
“It could be extremely powerful,” says , a computer vision researcher at the University of California, Berkeley. “Any information outside the frame could be interpretable.”
You don’t need novel science to see around a corner. You could just use a periscope, or any mirror for that matter. A mirror works because light rays bounce off the surface in a clean and predictable way – namely, at the same angle at which they hit it. As a result, all the visual information collectively contained within the light rays is preserved, so that you always see a clear image of whatever is out of view.
The problem is that most surfaces we encounter aren’t reflective, at least not in the sense that a mirror is. When you look at a painted wall, for example, you are observing light rays that have bounced, or scattered, from all sorts of random angles, preventing you from seeing an image of yourself. In fact, your image is there, but it is made up of only the tiny minority of light rays that happen to take the direct path from your face, into the wall and back into your eyes. The majority of light rays, which scatter through alternative paths, wash these out and thus render any image invisible.
To the human eye, at least. In 2012, computer scientist at the Massachusetts Institute of Technology (MIT) and his team hid an artists’ manikin behind a screen and then fired laser pulses onto an adjacent wall. They knew that some of the photons fired by the laser would scatter off the wall, rebound off the manikin and then scatter off the wall again, before finally being picked up by their photon detector. They also knew that this portion of photons would be tiny compared with the zillions taking different routes. The trick was in the precision of their detection system.
By timing the return of a photon to within a few trillionths of a second, they could calculate how far that photon had travelled after it had bounced off the wall. The haphazard nature of scattering made things difficult, because it was only possible to get a sphere of possibilities as to what point on what object’s surface each photon had come from. But by timing lots of photons returning from many different positions on the wall, the researchers ended up with numerous spheres of possibilities. Ultimately, the points where these spheres overlapped in their calculations formed .
Accidental cameras
Specialist laser systems like Raskar’s don’t come cheap, which could limit their application. Last year, some of his former group members, now at Stanford University in California, developed that could be run in conjunction with more widely available detection equipment. As the technology shrinks, they hope it could be integrated into surgical endoscopes. This might allow surgeons to see parts of an unhealthy intestine that are otherwise too tight to probe. It could also find use in autonomous vehicles, letting them spot other road users about to hurtle out of side streets. Exploiting it in CSI-style forensics will be trickier, because the technology would have to be incorporated into every CCTV camera at the manufacturing stage.
And yet, even everyday technology can be trained to see things outside the frame. The underlying concept here is different, relying on the existence of what are now being called “accidental cameras”, but the results are equally jaw-dropping.
We normally think of cameras as devices with glass or plastic lenses, but a camera can be anything that controls the light falling on a surface. Take the humble camera obscura, for example: by allowing light to enter a darkened room solely through a tiny hole, only light rays travelling directly from different points outside can get in. Unadulterated by any scattered light, these direct rays form a perfect, if inverted, image of the exterior scene on the wall opposite the hole.
Such a camera is almost always deliberately constructed. and at MIT started looking, they found unintended cameras almost everywhere – not just holes, but edges of any sort. A corner in a corridor, for instance. To understand how this vertical edge acts as a camera, and how, ironically, it can allow you to see the very scene it is obscuring, you first have to notice that the floor by the corner of the wall is in shadow. Known technically as a penumbra, this dark patch is easy to miss. Most of the floor is at the same brightness due to light scattering from everywhere in the corridor. At the penumbra, however, it is slightly dimmer because light from around the corner can’t quite reach it.
Given a photo of the floor near to the corner, a computer could subtract the contribution made by light that stays the same brightness everywhere to leave only the diminished light in the shadow region – that is, the contribution from around the corner. This would tell you the average brightness and colour of the hidden scene, which is pretty useless on its own. But the existence of the corner tells you something else about the light striking the shadow region.
To understand why, imagine standing with your shoulder to the wall, next to a corner, but so you can’t see round it. This is where the shadow is deepest. As you sidestep away from the wall, your view around the corner steadily improves. In the same way, the portion of the hidden scene exposed at any one point within the shadow depends on how far away that point is from the wall.
It is this constraint that makes the maths for converting the shadow to an image solvable, as Torralba and Freeman, together with Ye and others at MIT, discovered in 2017. Armed with nothing more than the basic geometry of a corner and video footage of the ground beside it, taken by an ordinary digital camera, .
There was a big snag with this work. The “images” making up the reconstructed video were only one-dimensional, like thin strips of normal photographs. That was enough to disclose movement, but not to recognise anyone. The reason was that the accidental camera itself, a vertical edge, was one-dimensional. As a result, moving away from the wall improved the view around the corner, but shifting up or down did not.
What the yucca sees
In January this year, however, a group led by at Boston University managed to of what was being shown on an LCD monitor. The feat required a slightly different accidental camera: a credit card-sized occluder set back from the corner, casting a shadow not onto the floor, but onto a wall even further back (see “Diagram”). “We’re getting two-dimensional reconstruction because the occluder itself is two-dimensional,” says Goyal.
Goyal hasn’t stopped there. Determined to make round-the-corner imaging more applicable to everyday situations, his group recently demonstrated improved algorithms that work with textured, rather than just featureless, surfaces. He even has preliminary results for two-dimensional images from one-dimensional corners without extra occluders. Position in the second dimension can be gauged from the relative brightness of the scattered light, he says. “It’s less robust, but we’ve had success.”
Meanwhile, Torralba, Freeman and their colleagues have shown that a . This works in the same way as the corner-imaging technique, in that different points of the plant’s shadow reveal information about different portions of the room. The distinction is that the computation required is far more complicated here, because the shadow is cast by leaves and stems sprouting in all directions. In fact, the image they created was only made possible thanks to a painstaking calibration procedure, which involved shining light onto the plant from every point within the room beforehand, to work out the geometrical relationship between shadow and hidden scene.
With the science moving so fast, it is tempting to speculate what these algorithms could do in the future. Ye points out that artificial intelligence is increasingly able to work out geometry from still photographs, without any calibration. Combined with the ability to interpret shadows, this potentially means that any photo could betray something of the scene outside the original frame. “Any camera has a limited field of view,” says Ye. “Even just for security or forensics, it would be incredible if you could increase that.”
“A 3-D image of a room can be constructed from the shadow cast by a houseplant”
Because the recording equipment need be nothing special, the images don’t have to be new, either. Imagine being able to shed light, retrospectively, on the context of rare historical photos or video footage, or contemporary photos presented as fake news. “It’s definitely plausible,” says Goyal. “It’s all just post-processing. You just need high resolution.”
Equally plausible, of course, are more nefarious uses, such as spying on people who believe they are out of sight. “I’ve thought a lot about this and I don’t think people should be too concerned,” says Ye. “These techniques are currently super-sensitive to things like camera motion, which is why we’ve mostly used fixed cameras. Even just for very slowly moving cameras, things become very hard, very quickly.” Although the technology is progressing fast by scientific standards, she adds, societies will still have plenty of time to get used to it, and push back against any uses they consider inappropriate.
In any case, it is hard to stem the flow of ideas, especially given a $28 million well of funding from the US Defense Advanced Research Projects Agency shared by many of the research groups. Another concept Goyal is working on is a combination of the penumbra and laser approaches, which he expects will make round-the-corner imaging more flexible and reduce acquisition time. Others, meanwhile, are using sound or Wi-Fi signals rather than light to see into hidden spaces (see “Hearing the sights”).
Then there are ways to employ the technology to see objects hidden not outside, but inside the frame. In 2016, Raskar and his colleagues exposed a printed manuscript to a laser operating in the terahertz range, which is midway between infrared light and microwaves. Terahertz light can penetrate materials and, unlike X-rays, it can also distinguish between white and black tones. From the precise arrival time of photons that are reflected, the . The capability could be a boon for historians looking to investigate delicate cultural artefacts.
Or, returning to vehicle safety, how about being able to see through fog? Last year, Guy Satat, then a PhD student in Raskar’s group, noticed that the wavelength of photons that have scattered off fog particles is skewed in a distinct way. The skewed photons can be discarded, leaving only those scattered from the object. Add some sort of photon-return timing system to judge depth and the .
This sort of technology is still in its infancy, but it is already clear it will save lives – and who knows what other applications are hiding around the corner?
Hearing the sights
Listening to the inaudible might sound like a paradox, but not according to a group including one of the pioneers of round-the-corner imaging. In 2014, William Freeman at the Massachusetts Institute of Technology and his colleagues captured high-speed video footage – without audio – of various objects from glasses of water to empty crisp bags while an instrumental version of Mary Had a Little Lamb played in the background. The .
Sound can see through walls, too. In June this year, of Stanford University in California and his colleagues hid an H-shaped object behind a wall. They then used speakers to bounce sound off another wall so it would go behind the first wall. Deploying microphones to pick up the returning sound waves, they could . They believe the set-up is a faster and cheaper way to see round corners than light-based systems.
If all this hidden imaging sounds a little cloak-and-dagger, be warned that people can be tracked through the walls of a home or office using ambient Wi-Fi and a smartphone. You need an app developed by at the University of California, Santa Barbara, and his colleagues, which can so long as you walk up and down a few times first to map the Wi-Fi environment. The researchers, who created the app to expose the privacy risk, are now developing defensive systems for Wi-Fi transmitters.
