Memories Reborn: AI-Based Photo Colorization

A Fateful Meeting

Converting color photographs to monochrome or sepia can give them a nostalgic feeling, but what of the reverse, colorizing monochrome photographs? It can give them a new feeling of life and immediacy. The first time Niwata Anju noticed that mysterious power was three years ago, when she was a tenth grader.

As a high school student, Niwata was a member of a committee recording testimonies for the Hiroshima Archive, a project that University of Tokyo Professor Watanave Hidenori launched in 2011 as a way to preserve for posterity materials related to the atomic bombing through technology and art. Niwata met Hamai Tokusō to record his statement as an atomic bomb survivor. Hamai’s childhood home was in the old Nakajima area of Hiroshima, where his father also ran a barber shop before the bombing. He was rescued from the rubble of his home, but everyone else in his family perished.

Niwata listened to him speak about his family while they looked through his old photographs together, and had an idea. “I knew it would make him happy if I could give him colorized versions of those photographs,” she says. She had seen pictures of Okinawa from before World War II that Watanave had colorized in his workshop, and felt firsthand their power to bring memories of those distant years closer to today.

Watanave first encountered AI-based automatic monochrome photograph colorization technology in 2016. He used open-source automatic colorization software developed by Waseda University Professor Ishikawa Hiroshi to add color to photographs of Hiroshima taken just after the war, and the results shocked him. It was like the people in those photographs had come alive. He realized then why few visitors to the Hiroshima Archive seemed moved by the monochrome pictures on display, although reactions to recorded testimony from bombing survivors was strong. They did not do a good enough job of conveying their importance.

“Everyone is so used to color photographs now that monochrome photographs seem inorganic, static, almost frozen. It creates a distance from the war, depriving viewers of a way to relate it to themselves,” he says.

He colorized photographs of the bomb’s mushroom cloud and of the Japanese Special Attack Units, then posted them to social media. The photos were retweeted thousands of times, and people began offering advice and more information. After that, he began searching out valuable photographs taken before and after the war, colorizing them, and then posting them under the title “XX Years Ago Today.”

Members of a Special Attack Unit with puppy. Photo taken May 26, 1945, at the Kagoshima Bansei Aviation Ground two hours before a scheduled attack flight.

The colorized version adds liveliness and immediacy to the image. (Courtesy of the Rebooting Memories Project)

Colorized Photographs Bring Memories Sweeping Back

Niwata learned the colorization technique from Watanave and brought the retouched photographs back to Hamai. They helped bring back memories that the monochrome versions had not.

“It’s like my family was still alive,” he says. Looking at one photo, he remarked “Hey, I used to play with a toy gun here!”

A photo taken under the cherry blossoms at Chōjuen park in spring 1935. Hamai Tokusō is the boy in the white cap being held in his mother’s arms, fourth from the left. (Courtesy of Hamai Tokusō)

The colorized version. (Courtesy of the Rebooting Memories Project)

Next, Niwata visited Takahashi Hisashi. Takahashi was also born in Nakajima, where his family had run a photography studio. Like Hamai, he was the family’s only survivor of the atomic bombing. Takahashi was suffering from dementia and had trouble talking, but when he saw the colorized photos, he began to speak excitedly of his family. He says the reason he covered his face with the watermelon in this photograph was that he was frightened of the camera flash.

The picture, taken in summer 1936 at the Takahashi photo studio, shows the family enjoying summer watermelon with relatives. The boy covering his face with a watermelon rind is Hisashi. (Courtesy of Takahashi Hisashi)

The colorized version. (Courtesy of the Rebooting Memories Project)

“When I was little, I hated peace studies,” Niwata says. This course is part of the curriculum at Hiroshima elementary schools, and is taught in the runup to the August 6 anniversary of the atomic bombing. However, she says that she could never really feel the impact of the horror of war, and found herself ignoring the pictures displayed in these classes.

“I think it’s because we never felt any connection to our lives today, and it was like something from another world. I feel that colorizing these old black and white pictures can help show people who don’t know war that this is a personal thing. Before the war, people lived their lives just like we do today, but the atomic bomb took all that away in an instant. War isn’t something from the ancient past; it could happen again, to us, right here.”

Watanave thought of the words “rebooting memories” to describe the way these colorized photographs, and Niwata’s thoughtful approach, helped interviewees remember things they had forgotten. It was like time in the black and white world of the photographs was frozen, and color helped it begin flowing again. Suddenly, the war of the distant past feels more connected to modern life. These photographs help create a richer dialog as interviewees talk about the world, culture, and daily life captured on film. That realization is what started the two on their joint project.

Revealing Darkness Amid Happiness

Rebooting Memories is a collaboration between AI and human hands. First, the team uses an AI system trained on around 2.3 million monochrome and color photo pairs to automatically colorize the photographs. The AI is particularly good with natural colors like human skin tones, the sky, the sea, and mountains. On the other hand, it is not as adept with artificial things like clothing or vehicles, so unnatural color elements often remain. The automatic colorization is essentially just a start, and the finishing work is all done by hand based on feedback from the war survivors, comments from social media, and documentary records. The work takes time and effort, and some pieces can take months to complete.

In 2020, this work culminated in the publication of AI to karāka-shita shashin de yomigaeru senzen, sensō (Prewar and Wartime Japan Brought to Life Through AI and Colorized Photographs), a collection of 355 colorized shots depicting the tumultuous era. Watanave and his editor at Kōbunsha, Takahashi Kōsei, decided together to not include any grotesque or particularly tragic photographs. Much like director Katabuchi Sunao’s film Kono sekai no katasumi ni (In This Corner of the World), they decided the goal was to focus on daily life in wartime, thereby emphasizing the darkness that can haunt seemingly happy lives.

The project includes pictures taken by US military photographers in Japan, as well as shots of Japanese-American detainees in camps in the United States. Niwata placed special emphasis on gathering pictures showing the perspectives of children. She wanted to explore how war worked itself into the lives of average citizens, even the youngest ones, as Hamai’s and Takahashi’s tales of their childhood experiences demonstrated.

“Ms. Niwata’s resolve, her ingenuousness, and her unflinching, natural way of listening helped to bring out the memories and true feelings of the survivors,” noted Kōbunsha’s Takahashi.

A Never-Ending Journey

Their book, which was published on July 16, seemed to resonate with audiences. It became a hit, and went into its fifth printing in only a month. Watanave warns, though: “The colors of these photographs aren’t the true colors. We’ve tried to be as authentic as possible, but we aren’t yet at 100 percent. The book is simply the results of our work at the time of publishing.”

One particular event illustrates the truth of his comments. The book opens and closes with a picture of a young couple, a man and woman, looking out over the blasted remains of the city.

The caption reads: “August 5, 1946. The burnt ruins of Hiroshima are still visible a year after the bombing. This couple is looking to the southeast from the Fukuya Department store on Hatchōbori street.”

Watanave discovered the picture at the Wilmington College Peace Resource Center while he was visiting the United States in 2016. He colorized it and posted it to Twitter on August 6, 2018, the anniversary of the bombing of Hiroshima. The reaction was immense. It was even picked up by newspapers, but after the articles came out, a letter with details of the photo arrived from the Kyōdō News agency. According to the letter, the photograph had been taken by Kyōdō prior to August 5, 1946, and was part of a collection of pictures distributed to areas across Japan to commemorate the first anniversary of the atomic bombing and the war’s end.

It also explained that the Fukuya Department store, which had been destroyed in the bombing, had begun operating a dance hall well before the first anniversary of the bombing in hopes of helping revitalize the local area.

The identities of the couple remained a mystery, so Niwata and Watanave presented the image as a symbol of the hopes citizens had for Hiroshima’s rebuilding, and for a bright future ahead.

But the story did not end there.

Soon after the book was published, 90-year-old Kawaue Kiyoshi, a former oyster farmer from Minami Ward in Hiroshima, came forward to say that the couple pictured is him and the woman he later married. Niwata went to visit Kawaue after that. Unfortunately, his wife Yuriko had passed away in January 2020, at age 90.

Niwata immediately went to work, interviewing Kawaue and getting his help in fixing the colors of their clothes and the landscape. He noted that a field visible in the shot was still smoking, and was blacker than the picture showed. And he also explained that they were not “looking out with hope,” as Niwata and Watanave had imagined, but staring at where their homes had been, and grieving over the friends they had lost to the bombing. Even while the emotional wounds of the war remained open, Yuriko supported him through the painful time, and the two married five years later.

A couple gazing out at the burned out city, August 1946. (© Kyōdō News)

The uncorrected colorized version. (Courtesy of the Rebooting Memories Project)

The corrected colorized version, based on input from Kawaue Kiyoshi, pictured at right; Yuriko, who would go on to be his wife, is at left. (Courtesy of the Rebooting Memories Project)

“Experiencing these reawakened memories of my wife has been deeply moving,” says Kawaue. (© Niwata Anju)

“I imagine new testimonies will come to us, and when they do we will listen and keep improving the pictures,” says Watanave. He and Niwata are on a journey to rediscover the color of the past, likely one that will never end.

(Niwata Anju, at right, and Watanave Hidenori show the published outcome of their Rebooting Memories project. “We hope this book can be released in English and other languages,” they say. © Amano Hisaki.)