As a small amateur project, I experimented with the AI Gigapixel neural networks for an upscale of one of my favorite science fiction TV shows - Star Trek: Deep Space Nine (DS9), in the Russian translation of Star Trek: Deep Space 9.

Deep Space Nine Perspectives

Just like Final Fantasy 7, where I make upscale backgrounds, textures and videos in Remako mode , DS9 is deprived of plans to release an HD version. While the popular seasons of the Original Series and The Next Generation (TNG) were mostly filmed, DS9 was filmed in the mid-90s with visual effects superimposed (space battles, etc.).

Analog film can be re-scanned at a higher resolution, but this will not work with digital video. The remastering of the series becomes much more expensive: this is one of the reasons why it was not released in HD.
')

Machine Learned DS9 Remastering

I thought that neural networks could be used here. Using tools like AI Gigapixel, DS9 low-resolution frames can be scaled to high definition: 1080p or 4K. They never compare to good remastering, but this is a step in the right direction.

So I tried on a couple of shots what happens. The results are great. AI Gigapixel uses neural networks trained on real photos. They are well suited for the upscale video game renderers of Final Fantasy, but they also scaled shots and DS9 special effects in amazingly high quality.

Here are some examples:


Original 480p resolution (click to enlarge)


Improved 1080p resolution (click to enlarge)


Original 480p resolution (click to enlarge)


Improved 1080p resolution (click to enlarge)

These single shots looked promising. In the first set of images, the station’s personnel in spacesuits are only a few spots. After apskeyl, much clearer figures appeared.

A close-up of the hand also improved. Wrinkles and wrinkles on the skin look much more detailed, and the baseball has a better brilliance and intricate firmware.

Video

However, the real test is how neural networks cope with the sequence of frames, that is, the video. Will artifacts and other problems appear? In the end, AI Gigapixel is created for the upscale of individual images and does not take into account the relationship between the individual video frames.

To check, I decided to process part of the episode "Victim of the Angels" of the 6th season. This is a great series about the Dominion War, where there are epic space battles and more personal moments face to face.

I will tell you more about this process in future articles, but briefly: extracting and scaling frames, and then generating a new video from them took about two days. This is the processing of the first five minutes of the episode (summary, opening scene and introduction). It is a normal time for medium and high-end PCs with open source software.

The result led me to reverent awe. He looked better than I hoped. No problems and artifacts. Since (moving) images cost more than a thousand words, here are two videos to compare.

The first demonstrates the “before and after” situation on still frames.


The second compares two videos. Notice how much the improved version looks.


I highly recommend watching these videos through the YouTube app on TV, if possible. This gives a better feeling of how the enhanced DS9 will look directly on TV.

Comparisons are good, but what does a real video look like? Here are the first five minutes of the episode in full 1080p resolution :

What about 4K?

Honestly, I do not know. Although I can zoom to 4K, I do not have a TV or monitor with such a resolution to check the improvements. However, I recorded this video. It is interesting to hear the opinion of people with 4K equipment, whether it looks better compared to the 1080p version.

The rendering almost melted down my computer, since it is much more intense than 1080p, so we’ll confine ourselves to this 4K example.

What's next?

Since I do not own the rights to DS9, I can’t do what I want with the show. I would have released the full episodes, but this is just legally impossible. These videos are more likely to prove the concept, so that the copyright holders from CBS take a look at machine learning and neural networks for possible remastering of DS9, which will bring our favorite movie a little closer to the HD era.

Imagine what a professional team can do with powerful equipment, specially trained neural networks (perhaps having trained it on the TNG season — on a set of data from the original frames and frames after remastering) and accessing the SD source code instead of DVDRip, like mine.

Well, for my part, I can tell you more about the upscale process, how I conducted it, which will be the subject of a future article.

Let me know what you think about it.