How Did I Upscale Star Trek: DS9 Footage to 4K?

Note: I have made a better version of this tutorial, which you can see here.

Hello everyone, sorry for the long wait. I had promised a tutorial on how I upscaled SD video from Star Trek: Deep Space Nine. I also promised to get you some more upscaled DS9 footage. You'll get both if you just keep on reading.

But before I begin I want to just thank everyone for the great response to my 'DS9 Remastering Using Machine Learning' proof of concept. I read a lot of enthusiastic comments and this mini-project was even featured on TrekMovie.com and io9.com. Whenever my stuff gets mentioned on the big news sites that I read myself, I get a big kick from it.

More Upscaled DS9 Footage

Over the past few weeks, I have gotten a lot of great feedback from a lot of people. Not everyone could see the improvements my machine learning method was able to produce for example. Looking back, I could see why:

• YouTube did a number on the compression of some videos. On the 4K intro the 1080p resolution was much better than on the other videos, which had a max resolution of 1080p.
• The frame-by-frame and the side-by-side video didn't really show off the changes as well as other formats could have. They showed that the image was crisper, but didn't convey the improvement in details as well as it could have.
• The full 5 min video 1080p, that I had to host somewhere else, didn't come out as 1080p either. I hadn't noticed that in my rush to get news of this project out the door.

With these lessons learned and some improvements in my upscaling program (AI Gigapixel), I think I've made a much better trailer this time around. Hopefully, this can show everyone the promise machine learning has for upscaling old TV shows like DS9. So without further ado:

Note: I handled YouTube's compression algorithm much better this time around, but it still downgrades it slightly. You can download the source file here if you want to see it in its original glory.

EDIT: Here's an image slider with which you can compare a before-after scene yourself.

How To Upscale Video Yourself

Now on to the tutorial part of this post. To extract, convert and upscale footage I used the following tools:

• AI Gigapixel (paid, but it has a trial and you can get 15% off by using code captrobau15 at checkout)
• FFmpeg (free)
• VirtualDub (free)
• Audacity (free)

So go ahead and download those programs. Keep in mind that extracting an entire episode requires quite a bit of free space on your hard drive. The episode I exported needs 29 GB for the original frames. If you upscale to 1080p this will add another 72 GB. If you upscale to 4K you will need around 180 GB for the full episode.

This is all for Windows, by the way, I don't know how it would work on Mac or Linux.

Step 1. Export The Frames Of The Source Video

You will use FFmpeg to extract the individual frames of the source video. This is the tutorial I used,  which tells you about more options.

Click to enlarge

I will tell you how to export the entire episode below:

1. Get your source video and put it in a folder you can easily work from.
2. Put ffmpeg.exe in the folder you also put your source video into.
3. Open up command prompt (cmd)
4. In cmd, navigate to the work folder. Do this by typing in: cd C:\Users\\Desktop\Work. The part after cd depends on where your work folder is of course.

Now you will give FFmpeg the command to export the frames. Type in:

ffmpeg -i sacrificeofangels.mkv soa%04d.png -hide_banner

 What does this all mean?

1. sacrificeofangels.mkv is the source file in my case. Change it to the name and extension of your source video file.
2. soa%04d tells FFmpeg how to name the frames (soa0001.png, soa0002, etc.). Soa is an abbreviation of the episode title, which helped me keep multiple video's worths of extracted frames organized.
3. .png is the extension of the exported file. PNG is bigger than JPG, but JPG creates extra artifacts that the uspcaling program can't handle very well. So the lossless PNG format is preferred.
4. -hide_banner hides FFmpeg compilation information. Personal preference.

When you press enter on the filled in command, FFmpeg will get to work. This will take some time. Do something else in the meantime. Watch some Star Trek on Netflix, for example :)

Step 3. Upscale The Frames To 1080p/4K

You will use AI Gigapixel to upscale the frames from the SD original. There's a lot of things you can with Gigapixel, but I will focus on the settings that I used to upscale my DS9 footage. The program is very simple to use, so I suggest you just try out all the different settings on some photos to see what the effects of everything are. The program has been refined a lot over the past months, so all official tutorials are out-of-date. Trial-and-error is the quickest way to get a hang of this software.

Once you've installed the program, open it up. Drag one of the exported frames to middle of the program's window. You will then be shown something that looks like the screenshot below.

Click to enlarge

Below I will explain what all settings mean:

1. You can base the upscale's resolution by scale, width or height. I chose height, because...
2. I want to make the upscaled image have the height of 4K footage, so I input 2160 pixels. Full HD has a height of 1080 pixels.
3. Suppress Noise and Remove Blur can improve the quality of the end result. This really depends on the project. Just try out everything to see what gives the best result for your project. I chose Low and Low, as it gave just a bit of smoothing/noise removal without going overboard.
4. In my workflow I saved to a Custom Folder. This allows you to export your upscaled images to a different folder (I have one for original frames and upscaled frames). Ignore Prefix and Suffix.
5. For video it's best to output in JPG format. It is smaller and results in a fast upscale process. Since the original is PNG, you will need to choose Convert File Format: Yes. The few artifacts created by the JPG format at Maximum Quality won't be noticed when you later convert the frames to a video. Keep Color Profile to Yes, as it's faster.

Click Start to upscale the image. Upscale the original image until you are happy with the results.

Once you have a working setup, you can start upscaling footage. Select a few hundred frames when first getting your footing with this whole workflow and drag and drop them in the window. Upscaling an entire episode can take a long time, so start small.

How many frames you can upscale in one go depends on your PC's specs. It can be 1000s, 10000s, etc. You'll just have to find out for yourself.

Step 4. Exporting The Audio Of The Video

With Audacity you will rip the audio from the source file.

• Open up the video file in Audacity.
• Once it's loaded, go to the File > Export Audio.
• The file type you save it as depends on the program you use to create a video from all the individual frames. For my project, I used VirtualDub, which can't take all audio types. In this case, I use WAV 16-bit PCM.
• Save it and you're done.

Note: the source file for the DS9 video that I used had 5.1 surround sound. Basically, there are multiple audio layers. I have not yet figured out how to get the sound as surround sound into a format that VirtualDub can work with. Saving it as WAV16-bit PCM will collapse this into a mono format. It's okay for test purposes, but not for a final product. Let me know if you know how to get 5.1 surround into VirtualDub.

Step 5. Combining The Frames And Audio Into A Video

Everything will be brought together in VirtualDub.

Click to enlarge

1. Go to the Video menu. You will find Frame Rate and Compression there. Open up Compression.
2. Get the x264 encoder. If you don't have it, download it here. Click on Configure.
3. Here you can determine the quality. I found the settings as displayed the most appropriate.
4. Go to the Frame Rate menu. Set the frame rate in the two fields to the frame rate of the source video. For DS9 this was 29.970628 (a common frame rate). Click OK
5. Go to the Audio menu. We need to make sure Interleaving is set up correctly. Click OK.
6. In the Interleaving menu, set everything as displayed above. Click OK.
7. Make sure Sync to Audio is on in the Options menu.

Now everything is set up, you can go to File > Save as AVI. Save the file and let it render. You can uncheck Show Input Video, Show Input Video and set the processing thread priority to a higher priority to make the video export faster (at the cost of having fewer resources available for other programs).

This is how I did it. No doubt there are alternatives/improvements to the above workflow. If you have knowledge about these kinds of programs, please leave your suggestions in the comments below.

The Future Of Upscaling Old Footage

This blog post marks the end of this small side-project. While I would love to release full episodes, this is just not legally possible. The videos I made were a proof of concept for CBS, but also other companies, to look into machine learning and neural networks to help remaster old shows where traditional remasters are either too expensive or impossible due to lost originals.

While entire episodes might be out of the question, now that I've shared my workflow with the rest of the world it should be easier for others to try these methods out on clips and segments themselves. Go out and try it out yourself on DS9s battles of the Dominion War. Or go and see what this could do with Babylon 5 or other great shows that are still stuck in the SD era.