Speech delivered at the SEMICON Japan 2023
: Imaging and Sensing Technology playing a part in igniting emotions

The brand chosen by creators, driven by the power of technology

The Sony’s groupwide pursuit is to create and deliver Kando in people’s minds. The emotions are felt by people, but it is also people who inspire them. This is why we support the creators who aim to produce emotionally moving contents and aspire to be the brand of their choice across the world. To this end, it is crucial to develop the technology that empowers their creativity.

Contributing to creators’ contents creation through a combination of 2D and other dimensional information

To start with, let’s take a look at the 2D imaging as a base. This is like the technology of the conventional RGB imaging. Image sensors have evolved over the years, aiming to go “beyond the human eye” in order to deliver beautiful images to human. As far as still images are concerned, we have achieved this to an extent.

One of the technologies that drive the advancement in image quality is the global shutter. In November 2023, Sony Corporation launched “α9 III,” the world’s first^*1 mirrorless interchangeable-lens camera that features a full-frame global shutter image sensor. And it is us SSS who developed this image sensor. Global shutter is characterized by the simultaneous exposure and readout across all pixels, which enables to “frame the crucial moment” without a distortion even if the target object is in fast motion. A new technology such as this enables creators to broaden the potential of their expressions.

Further, and ultimate, advancement of 2D imaging is represented by the photon count imaging. The SPAD sensor adopts a pixel structure designed to amplify the electrons from a single incident photon, known as “avalanche multiplication,” which enables the sensor to capture the object even in faint light. Applying this technology to imaging can realize the high dynamic range capable of producing low-noise images in a dark location, and we are confident that it will facilitate imaging experiences that have never been known before. We are pursuing the development of this technology focusing on still images to start with.

Moving on, I will talk about the value of the information to be added to two-dimensional RGB imaging from other dimensions. We start off with the 5D imaging, a combination with various spectrum. The images shown represent a combination of an RGB image with an image of invisible light, that is, short wavelength Infra-Red (SWIR) light. One of the characteristics of SWIR images is the high penetrability through haze, fog and other weather conditions that obscure the visibility. As you can see, the technology offers the potential to create a dramatic image that captures even the ambiance of the real view, which has been difficult to achieve so far with the RGB imaging alone.

Another possible dimension to add is the polarization information, which is about the light wave orientation. These images (moving images below) show a slice of cake, captured and rendered as a CG image. You can see that adding the polarization information can enhance the expression of textures of the materials, which appear so real that defies the conventional CG enhancement.

Videos are becoming the mainstream medium of creation and communication today. We recognize this field to have a significant potential for improving the technology for movie imagery. Combining the RGB with the four dimensional Event-based Vision Sensor (EVS) enables the capturing of a fast-moving object in a video at high frame rates with reduced power consumption as the event data produced by the EVS compensates for the images of RGB frames. We have succeeded in mounting the RGB and EVS pixels side by side to develop a single chip sensor.

Fundamentally, EVS is a sensor to output data of the target object only where it changes, combined with coordinate and time information, contrary to the conventional frame-based RGB image sensors. The video illustrates that the EVS image on the right captures the image only by extracting the moving target unlike the RGB on the left.

What happens if we merge the images captured using the EVS and RGB sensors? Take a look at this video. By compensating for the gaps between the RGB frames with the event data from the EVS, a truly smooth moving image at 120 fps can be realized without blur even for high-resolution images at 4K. We are anticipating that the visual expressions will be further liberated for creators across the world as this technology becomes readily accessible with everyday mobile devices.

There is another possible combination. This is the RGB imaging and 3D, that is, depth information. These images are also a three-dimensional CG rendering of the image of an actual space. As shown in the image on the left, the RGB alone cannot accurately capture depths, and as a result, the image is partially compromised. By contrast, we can see that the combined RGB can render with high accuracy.

As these illustrate, we believe that the 2D imaging with added information from other dimensions can broaden the scope of content creation, thereby helping creators make contents that move their audiences emotionally.

Contribution to safety and security in society through technology that supports future of mobility

Advancing the autonomous driving technology is one of the major themes for the automotive industry. To achieve this, there are high expectations for the image sensors that accurately detect the surrounding conditions external to a vehicle. A autonomous driving car needs to recognize objects with high accuracy and feed into its vehicle control even if they are far away. We have announced a commercial launch of IMX735 this year. This is a sensor which boasts the industry’s highest^*2 pixel count of 17.42 effective megapixels, designed with the application for long-range front sensing in mind.
*2 Among CMOS image sensors for automotive cameras. As of announcement on September 12, 2023. Sony research

Take a look at these images, which demonstrate the long-distance visibility realized by IMX735.These (below) are taken on an expressway road. We can see that, in comparison with the right image by using an existing sensor, IMX735 can facilitate a clearer resolution of the road sign that is more than 100 m ahead.

Another theme we work on is the Sensor Fusion. We at Sony pursue the technology of Early Fusion. This is a concept of extracting optimal characteristics values from the raw data, before the signal processing, and merging them to obtain higher accuracy of object recognition. Applied to the front sensing, it will be able to improve the visibility in the hard-to-see conditions as shown at the top right here (below). Meanwhile, it is more readily applied to the surround sensing.

For this purpose, it is expected to be leveraged in a parking assist solution. We are currently developing software to recognize the external environment with high accuracy and automatically map the surrounding area, to realize smooth parking assistance.We believe that we can offer new value through a combination of hardware and software. We can see this in a video, too.

Contribution to sustainability through technology for sensing all over the globe

We believe that our sensing technology has great potential for making contributions in terms of the environmental issues. I present to you Sony’s Earth MIMAMORI platform. What we are trying to do with this is to develop a system with which to monitor every corner of the earth and detect early signs of environmental issues, natural disasters and other abnormalities, helping to take preventive actions.

There are three technologies that comprise this platform. These are diverse sensing devices for identifying various changes, LPWA network for conveying the change data over a wide area, and analytics and simulation technology for comprehending the changes in order to enable specific actions to be taken. By combining technological capabilities from across the Sony Group, we are pursuing proof of concept with our various partners. We will build momentum in our endeavors to make our contributions to the sustainability of our global environment.

There is also so-called multi-spectrum technology, which enables the sensing of a wide range of wavelengths at higher resolutions than those of the RGB imaging. SSS’s multi-spectrum sensor IMX454, combined with its dedicated software, can acquire information at different wavelengths in a single shot, and the data can be processed on the edge side for the segmentation of the target.

This technology is expected to have useful applications in agriculture. Here, you will see a scene in which the multi-spectrum sensor is used to capture the wave patterns unique to plants. In this image, the real plants are detected and rendered on the screen in pink. The hand is holding some artificial flowers, and the sensor does not pick it up even though it is also green.

As advancement is made in software development, we believe that multi-spectrum technology will enable nondestructive real-time monitoring of plants for their biometrics. The video shows stress responses of two pots of plants after they are watered at the same time. The plant at the top appears red at the beginning, indicating high levels of stress because it was not watered until the last minute and the soil was dry, and we can see that the color gradually changes to blue, meaning the stress levels are lowering as the plant absorbs the water. The RGB image on the left does not tell any difference, while the multi-spectrum image clearly shows the changes as they take place. The technology is still in the research and development stage, but when it is ready for large-scale rollout, it will be possible to observe crops through their growth processes scientifically and in real time, leading to the productivity enhancement in agriculture.

Delivering new experiences through technology for expanding the field of emotion

We consider VR and AR to be the technology that offers people the experience they have never known before, expanding the space to have the emotional engagement, which we call the field of emotion.For this technology, you see the illustration of potential applications in head-mounted displays and AR glasses, together with the possible devices for realizing them. We think that this technology represents the domain that presents great potential for the future.

We present an actual VR image viewed with an OLED Microdisplay with 4K resolution. While the screen is divided into two zones, one for the left eye and the other for the right eye, the images are identical. The subtle parallax provides for more three-dimensional perception when they are viewed on the actual VR head-mounted display, facilitating a very realistic visual experience.

To sum up, I mentioned the support for creators, safety and security in society, sustainability of the global environment, and expansion of the field of emotion. We believe that, in each domain of this “value chain that creates Kando,” we can offer our technologies as our contributions. The starting point for all this is our passion for distributing across the world this particular “Kando,” the keyword in our Purpose statement. We will continue to grow our business by pursuing this “Kando.”

The market for image sensors and semiconductors is expected to reach 1 trillion dollars in 2030, and this is a growing industry.
SSS is determined to be the leading company in the imaging and sensing technology domain and lead the market growth as we contribute to the further development of the semiconductor industry.