[Toward the integration of real and virtual worlds] New experiences delivered through sensing technology

Natural bokeh effect using depth sensing

What experiences become available through the use of a smartphone camera with ToF image sensors? Here are some use cases.

One of them is the blurred background in photography.

“Many smartphone cameras come with the bokeh effect feature today. The ToF image sensor realizes this effect by delineating the foreground image from the background. While a simple RGB image captured by a standard camera can be processed to achieve the blurring effect to an extent, this method does not work well if, for example, the target person is wearing a piece of clothing whose pattern resembles the background. Whereas, the ToF image sensor can recognize the target clearly by the differences in distances. This is one of the advantages in using ToF image sensor technology that enables more intelligent bokeh effect processing,” says Baba.

Another imagery experience concerns the auto-focus feature.

“The ToF image sensor is also used for camera assistant features. Conventional image sensors are not efficient to use the autofocus in dark places. The ToF image sensor can help with the auto-focus no matter how dark the scene may be because it emits infrared light and identifies the distance to the target.” (Baba)

What is more, the depth data obtained through the ToF image sensor can enrich contents for smartphone apps and enhance the user experience.

“ToF-enabled apps can realize various services in a virtual space. You can virtually try clothes on, coordinate the interior design of your room, and more. For example, the sensor can recognize the shape and size of your foot, then you can have an app with which you can try on some shoes to see if they fit in terms of colors, designs and the size before purchasing. You can also use the camera to scan your rooms, including the shapes of the existing furniture and room layouts. The data is then used with an app to visualize new decorations by adding images of, say, a new rug, piece of furniture or some pot plants.”

The depth data are also useful to further improve the AI processing technology used in the ToF AR.

“Conventional deep learning mainly uses RGB images. Adding depth maps and other depth data to the learning process will accelerate the development of new AI applications and enhance the accuracy of recognition.”

These are a few examples of the advantages ToF image sensor technology can offer to users of smartphone cameras. Ideas for these and other new features are often proposed by SSS, but equally from the manufacturer side. In either case, our young engineers add proudly, they pursue feature extensions and enhancements for every release of new models of smartphones in order to respond to user feedback and meet their expectations.

Technological competence and comprehensive capacity enabling SSS’s flexibility in the face of client requirements

Baba started working on the development of ToF image sensors already in his second year at SSS. He is convinced that SSS’s strengths come from its technological competence and comprehensive capacity.

“In addition to the ability to meet the diverse client needs, SSS has the technological competence to develop not only individual sensors but also camera systems that utilize them. What is more, we can use specification data of laser, lens, etc. to develop high performance systems, which enables us to make comprehensive proposals.”

Eigen strongly agrees with him on this point.

“Our clients, that is, smartphone manufacturers and app developers, have different needs and requirements, such as capturing images using with large pixels and minimized power consumption. We can propose and deliver not simply the customization they require, but products that surpass their expectations, for which SSS’s technological competence is crucial.”

Desire to make sensing technology more widely available

The challenge they face is the fact that sensing technology is not yet a mainstream technology in the mobile domain. Technology is worth having only if many people can benefit from it. Baba emphasizes the importance of creating various apps and solutions that allow people to enjoy the benefit of ToF image sensor through day-to-day usage.

“It is a chicken-and-egg situation. We need to implement the ToF image sensors in as many smartphones as possible while there need to be more and more apps that use the sensor. Above all, it is important to provide many stakeholders, including manufacturers, app developers, and end users, with opportunities to experience the technology.”

To reach out farther and wider than ever, Eigen and the development team explore every possibility. Their efforts are driven by the lessons learned in the past, that AR/VR gaming apps alone can only appeal to limited user groups and not others.

“People would find it useful if they could control their smartphone apps by making a tapping or swiping gesture in front of the phone, like starting and stopping the movies they watch or browsing web pages. The technology could also be effective to operate a terminal in a convenience store or other places without touching its screen. To make AR and VR ubiquitous in everyday life, apps designed for social contexts such as these are indispensable.”

Another challenge is to achieve efficiency in power consumption. The ToF image sensor enhances the sensing accuracy by emitting ample laser beams, but this increases the power consumption proportionately. In addition, the development cost is also high, because it is necessary to design to reduce the power consumption for highly accurate sensing.

“In a case where camera assistant features are used, the camera must run the ToF image sensor on top of other, standard camera apps, and more power is consumed. Stronger laser beams can increase the sensing range, but it equally increases the power consumption and can cause the app to shut down. We are therefore trying to strike the best balance between the cost of reducing power consumption and performance day in day out.” (Baba)

Users already invest in their smartphones quite significantly, and pushing up the cost is in no way welcome. Smartphone manufacturers have, therefore, very stringent requirements on the cost performance front.

What SSS must deliver is the sensor and camera systems that satisfy the contradictory market needs—offering low power consumption and high performance, such as high frame rates and low noise, at the same time. This is a game-changing challenge, they say, that could rewrite the entire chart of the market.

“Some smartphone manufactures are trying to copy light-detecting and depth-sensing scanners that is incorporated in competitor’s model. SSS is not like them. We have rich ToF sensing technology, including the dToF and iToF, together with the technological capabilities to deliver the customization our clients need in terms of sensing ranges, resolutions, and power consumption levels. This is where our strengths are, that we are able to develop diverse apps. I would like us very much to lead the industry by leveraging our sensing technology and proactively proposing new ideas,” Eigen says passionately.

Baba acknowledges the progress that has been made with their clients, who developed in-depth understanding of ToF image sensor technology through the enduring communication, and with whom he can now have productive discussions. These young engineers consider that there is a very good chance of creating new markets.

“As I mentioned earlier, SSS has its strengths in the capability to offer camera systems as a total solution, pivoted on our sensing technology, to meet diverse client needs. In this sense, we are exceptional as we can offer a comprehensive package from software to hardware, covering the technology evaluation, manufacturing, and implementation.”

There is no doubt that SSS’s sensing technology will be playing ever greater roles in future smartphone cameras, delivering more enhanced convenience to users.