Remastering Star Trek: Deep Space Nine With Machine Learning

Note: You can find a higher quality trailer in this article.

As a little side-project, I have been working on putting the artificial neural networks of AI Gigapixel to the test and having them upscale another favorite thing of mine... Star Trek: Deep Space Nine (DS9).

The State of Deep Space Nine

Just like Final Fantasy 7, of which I am upscaling the backgrounds, textures, and videos in Remako mod, DS9 was also relegated to a non-HD future. While the popular Original Series and The Next Generation were mostly shot on film, the mid 90s DS9 had its visual effects shots (space battles and such) shot on video.

While you can rescan analog film at a higher resolution, video is digital and can't be rescanned. This makes it much costlier to remaster this TV show, which is one of the reasons why it hasn't happened.

Remastering Star Trek: Deep Space Nine With Machine Learning

This is where neural networks could come in, I thought. With tools like AI Gigapixel, I knew it might be possible the low definition frames of DS9 can be scaled up to a higher definition such as 1080p or 4K. It would never be the same as proper remastering, but it would a step in the good direction.

So I tried my hand at frame or two, to see what it could do. The results were great. AI Gigapixel uses neural networks trained on real photos. So while it did okay with upscaling the video game renders of Final Fantasy, it did amazing upscaling real-life footage and the bigger budget CGI effects of DS9.

Here are some examples below:


Original 480p definition (click to enlarge)


DS9 Enhanced 1080p definition (click to enlarge)


Original 480p definition (click to enlarge)


DS9 Enhanced 1080p definition (click to enlarge)

These still frames showed promise. In the first set of images, the maintenance crewmen in their spacesuits were nothing more than a few pixely blobs. The upscaling process turned the blobs into much more defined figures

The close-up of the hand also improved. The creases and folds of the fingers and hand look much more detailed, and the baseball really shows off its sheen and the intricate stitching.

Moving Images

The real test, however, was going to be if the upscaling process held up with a sequence of frames aka as a video. Would there be artifacts or other unsightly issues? AI Gigapixel was after all made for upscaling single images so it wouldn't take into account the relation between the individual frames of a moving image.

So I set about upscaling a portion of an episode. I settled on the season six episode: "Sacrifice of Angels". A great Dominion War episode that had both epic space battles and more personal face-to-face moments. And so I set out to work.

I will go into greater detail about my process in a future blog post, but it took me about two days to get everything extracted, upscaled and put it back together in a way that was pleasing. This resulted only in the first five minutes of the episode being done (the episode recap, the opening scene, and the intro). Still pretty good time for a mid-to-high end PC with software that isn't just available to professionals.

The result left me pretty awestruck. It looked better than I had hoped. No weird issues or anything. It looked pretty much like an HD version of DS9. Since (moving) pictures are worth more than a thousand words, here are two comparison videos that show off the improvement I was able to get with this machine learning based upscaling technique.

The first shows off the before-and-after situation with still frames.



The second puts the two videos side by side. Take note how much clearer and sharper the Enhanced version looks.



I highly recommend watching all these videos through your YouTube app on your TV if possible. It gives you more of a sense of how it would feel watching an enhanced DS9 on your TV.

Comparisons are all well and good, but what does it look like if you were to watch it normally? Below is a video of the first five minutes of the episode in full 1080p:




NOTE: I hadn't realized this hoster reduced the quality of the video so much. If you really want to see the quality of the above video, download the video itself here (139 MB).

What About 4K?

Honestly, I don't know. While I can upscale the image to a 4K resolution, I don't have a TV or monitor with a 4K native resolution to see if it looks better. I have nonetheless made this video and I am interested to hear from people with people with 4K equipment if it looks better over the 1080p version of the intro.

This nearly melted my computer, as it is a lot more intense to upscale than 1080p so I'll stick to this single video for 4K examples of DS9 Enhanced.



UPDATE: A month or so after I made the above videos, I released a much improved 4K comparison trailer. This shows off both space scenes and conventional scenes from DS9:



What's Next?

Since I do not own DS9, I can not just do what I want with it. While I would love to release full episodes, this is just not legally possible. These videos serve more as a proof of concept for CBS to look into machine learning and neural networks to help remaster DS9 and move it a bit closer to the HD era.

Imagine what a real team could do, with more powerful equipment, custom trained neural networks (perhaps training the network on TNG vs. TNG Remastered images) and access to the original SD files instead of a DVDRip like me.

What I will do is go into further detail about my process, which will be the subject of a future blog.

Let me know what you think.


Comments

Byronotron said…
This is incredible. Fantastic work! It's truly shocking how much detail the AI can recreate!
AlfredQack said…
I think its fantastic. Of course not real HD like next generation, but much, much better than I thougt it would be possible.

Looking forward your next blog post. I would love to try it on my own.
CaptRobau said…
Thanks! Even though I've been working with the tech for a few months it still surprises me with its good-looking results some times.
CaptRobau said…
I'm glad you think it's fantastic. Getting it to look as good TNG Remastered was never on the table, but even a Remastered Lite is still better than the old SD footage.
AlfredQack said…
This comment has been removed by the author.
Unknown said…
I'd think you would be in the same position as "Star Wars: Revisited" - fan edits like this are legally pretty solid.
Chris said…
These look absolutely fantastic! I've been playing with AI Gigapixel for a while and the results have been quite astonishing, so I'm happy to hear that creating videos with it seems to work as well. Though I wonder how much hard drive space those stills take up, let alone the time it takes to render all that!

Hopefully in the future, we might see some of these machine learning upscalers become plug-ins for programs like Premiere, which could make these kinds of enhancements much easier.
Unknown said…
From the producer’s own account it seems the most difficult part is to remaster the VFX and CG part of the show. Since many space scene were not shot using models on film and were rendered digitally. Perhaps they have los the original 3D assets or it is in formats that is hard to use with modern production software.
Unknown said…
Can you try it on some footage that has heavy motion blur?
CheerBeer said…
What are you using for source material? I actually had this exact same idea and just purchased the box set. I can tell you 4k does look better from my own proof of concept. However, it has quickly become evident that using Ai gigapixel, at least in it's current form, will not really be feasible. While it can do batch processing, adding hundreds of thousands of frames will often result in the program crashing. So I set it up overnight and wake up with only a fraction of the work done. While it can be done, the pita factor is pretty much making a non starter.
CheerBeer said…
You can eliminate motion blur with the high setting in AI gigapixel but it makes certain features look weird. I have only tried it on individual frames but to me it would not be faithful to the source material.
CheerBeer said…
Upscaling from a 480p image (500kb png) to 4k is about a 10x increase in file size. So to answer your question, a rediculous amount. Luckily I just bought a 10tb hard drive.
CheerBeer said…
Also, ideally it would be beneficial to use adjacent frames to upscale the footage, and then use AI to sharpen the upscaled footage. Luckily this software exists and if it works like I hope I'll post a proof of concept.
Unknown said…
I have a large screen 4k TV and I love DS9. The problem with the TV is lack of high quality video. Another problem is the real differences between 1990s TV and 2019 TV although DS9 stories hold up the video does not CBS would be wise to use this process and editing to create versions of Star Trek it could make CBS all Access worth having.
NAJMUL ISLAM said…
This comment has been removed by a blog administrator.
CaptRobau said…
A DVDRip. Those are somewhat optimized for downloading, so it's not going to be the format. Still I got a pretty good one, otherwise I wouldn't have gotten nearly the level of quality. A direct screengrab upscaled from you DVD boxset should be the best result. Although I imagine there's something with slightly better quality in the CBS archives. I assume they did a bit of reduction to fit everything nicely on a disc.

Gigapixel is indeed not suitable for large projects. A thousand or so frames it can load nicely, but go beyond that and it has to fit too much into memory at once. Some batch system that loads images one at a time would be the best for this.

Still it works well enough to get a few nice clips that can allow us to dream.
CaptRobau said…
I second that. Just leave a comment here or contact me via the forum. I'd be glad to showcase it or link to your own site/upload of this thing.

Good luck!
CaptRobau said…
I imagine they lost the assets or can otherwise not access them. I can't imagine that if they access them that they could not simply export at a higher resolution. So they must be gone.
CaptRobau said…
The first five minutes was:

3 GB for the exported PNG frames (you need that format so that the upscaler does not upscale compression artifacts)

5.5 GB for the upscaled frames in JPG with Maximum Quality. For a final, commercial, product you'd use PNG as well, since you wouldn't want to get any quality reduction. For my tests it was fine.

Multiply both by 9 (45 min episode) and you get 76 GB per episode.
Ed Lennon III said…
Would love to be able to donate some CPU cycles towards rerendering the whole thing.
Unknown said…
This would be awesome if it could be done to DS9 and Voyager! Those shows deserve it!
Gavin Greenwalt said…
Even if you had access to the assets they wouldn't be HD worthy. And if you're going to remaster you might as well go 4k. Would be cheaper and better to just buy new models off of the stock-model market. There are plenty of very faithful CG models that are superior to the original show assets.

And honestly you could set up one person for a year and they might be able to recreate all of the shots considering how expensive it was back then and how slow.
CheerBeer said…
It didn't work, I think I put too much stock in the super resolution technique. It may work better with a cleaner source, but the DVD source has way too much artifacts which Gigapixel seems to clean up really well.
Unknown said…
Like you, I'm skeptical this would be legal to provide episodes. But, I wonder if you could release the ML models, and we could feed it our own legal DVD-Rips on some GCP storage we pay for, if we could each make our own set. I don't think that'd run afoul of copywrite laws? I think it'd be pretty simple to build some scripts to make that happen. Plus, you get $300 of GCP credit to start out learning GCP. Sounds like a good idea to me!!
Aaron T. said…
According to this article http://trekcore.com/blog/2013/05/deep-space-nine-in-high-definition-one-step-closer/ they actually do have a lot of the original files, and they were designed with enough detail to hold up in HD. They're even an example frame re-rendered in HD for comparison.
Unknown said…
Please release this CBS! I would buy. & rewatch on my 4k TV.
Unknown said…
Perhaps you can use scene or cut detection software to break down the scenes as preprocess step. Then fed that scene as a single batch.
Unknown said…
The Worm hole will be one of hard ones to redo
Unknown said…
You may need some pro format like D1
Unknown said…
The 1080P one is meh, but the 4k sample is pretty good. I would love to see a clip with some actors in it to see what that looks like.
prajwal said…
This comment has been removed by a blog administrator.
Jared said…
Awesome! Do you think you would be willing to upscale the Battle of Wolf 359 from Emissary? I was going to try and I got all the original frames but my computer wasn't having any of it and AI Gigapixel crashed within the first 5 images.
Unknown said…
Very interested in your process... I own the DS9 DVDs and enjoy the show but was pretty disappointed with the quality of the video... I also am one of the people that purchased the TNG remastered DVD - they are spectacular BTW... I'm unfamiliar with the googlepixel AI neural networks but would be willing donate time in training if that'
s a possible thing?
Ms20 said…
Is it a memory issue? I got a machine with 256 gigs of RAM laying around if you wanna try.
Anonymous said…
This comment has been removed by the author.
Unknown said…
This is wonderful. This needs to happen. This also should happen to Babylon 5. They shot that show in HD, but the effects were SD.
Pajik said…
This comment has been removed by the author.
Pajik said…
I think you could get even better results if the ML algorithm was trained on video, (possibly even on the formats to be converted) rather than still images, and was allowed to use adjacent frames.
d34dl0ck said…
This is an awesome idea! The results are very promising, from trekkie to trekkie keep it going this is really cool! And from developer to developer that is a very creative approach for DL, nice idea!

All the best
Sven
CaptRobau said…
That is a good idea. I'll try it.
CaptRobau said…
I'll see what I can do
Parvaz Bappy said…
This comment has been removed by a blog administrator.
CaptRobau said…
It looks nice. A 2x jump in resolution isn't too obvious, but I can definitely see an improvement in the resolution when looking at the signage.
Jan Bert Stoel said…
The opening credits was always going to be problematic. Even the studio would probably have to actually redo the CGI of the sequence, to deal with the muggy comet's tail and polish up the lettering. That said, on my ± 2K screen, the 4K version did shine in regards to the rest, dealing effectively with some aliasing visible in the 1080p version. Perhaps upscaling beyond target resolution does intrinsically improve the result by providing more pixel data.

I'd love to know what the studio makes of this proof of concept. Well done.
Byronotron said…
Super Resolutions are very effective in creating a better looking image.
Star moon said…
This comment has been removed by a blog administrator.
Angelo said…
HI,
I own of AI Gigapixel and use it to enlarge jpegs and it does a far better job than Photoshop. I would like to try this on smaller trek videos (deleted scenes for props or costumes I own) but I have no idea how to extract the stills and reassemble them. Can you post how that is done or point me to a resource. Thanks.
Unknown said…
Actually it is possible to re-render the original VFX, Adam "Mojo" Lebowitz one of the guys that originally did the CGI for the show has re-rendered the original CGI models in HD and this was the result https://www.youtube.com/watch?v=bDWyfGjFsyU , I recomend that you download the original file from the description to avoid the heavy Youtube compression.
Unknown said…
Amazing job! Do you think similar results could be replicated with Waifu2x in Photo mode, It's basically a free open-source version of AI Gigapixel.
ValdikSS said…
I don't know. This doesn't look astonishing. You can perform almost the same upscale with avisynth and nnedi3_rpow2 neural network upscaler, which has been available since 2010.

http://forum.doom9.org/showthread.php?t=147695
taka bazar said…
Nice Blog ! Thank you for your very nice articles. I look forward to visiting your site in the future! hipatitshah
Maschinengeist said…
This comment has been removed by the author.
Maschinengeist said…
This comment has been removed by the author.
Maschinengeist said…
@ Unknown: I think you are doing something totally wrong or have a PC from the 80ies. I use the program myself and it works quite well. I can easily queue up 50.000 images - ok, it takes a few moments to load but there are no crashes or something. The process of 4000 to 5000 images takes ca. 1 hour ... and i have a normal mid-price PC.
So could it be you are extremly under the minimum requirement?
Maschinengeist said…
yes ... but it takes 10 times longer
Stu said…
God please, this! We run the risk of seeing low detail ships in HD, but the show deserves the chance.
Marcus said…
Yeah! This would be a good idea.
Video Nerd said…
I'm working with a similar approach, using nnedi3 as basis and a few other things. Look me up under "Video Nerd" on vimeo where i've got a few upscaled music videos. I'm trying to upscale DS9 too but it's damn low resolution and noisy...
Stefan said…
This comment has been removed by the author.
Stefan said…
This comment has been removed by the author.
Stefan said…
Thank you very much for showing this Proof of Concept. I have tried to dublicate your method and I have come up with the following:

RTX 2080 (non TI) @ 2000MHz + AI Gigapixel = 1 upscaled frame per second
--> 25fps x 60sek (in a minute) x 45minutes per episode = 67500 frames
67500 frames = 67500sek rendering time = 18.75 - ca. 19 hours of rendering per episode.

My PC consumed ca. 500Watts constantly while rendering - equals 9.5kWh per Episode (in Germany , where I live thats's ca. 2.6€ (@0.28€/kWh)

26 Episodes x 7 Seasons = 182 episodes (x 2.6€) = ca. 500€ of costs of electricity alone :(

This has not factured in the cost of buying AI Gigapixel (currently running as a trial).

I find the idea of an updated DS9 or VOY awesome - hopefully some time in the future the process will not be so tedious, time and energy consuming ;)

Best wishes from Germany and thank you for showing what is possible.

Update 1: I am currently trying to render parts of an episode in 4k - I can open 20.000 frames at a time in AI Gigapixel (i5 7500k, 16GB RAM, RTX2080 (8GB Video Memory)) without a problem. My taskmanager tells me that video memory is not used above 4.8GB at a time, while system memory is used up to 6.3gb (total). I am using AI Gigapixel v. 4.03

The total render time for a double episode (90min) would be 6 days :( and the resulting video file would be 2.2TB (uncompressed) - this is not something that you'd want to do on the whole library - hopefully somebody at CBS sees the potential and greenlights a remaster.
HalbarusEU said…
I just watched "What we left behind" Documentation in Cinema. It was epic and emotional. Maybe you could contact them, since they have Connections to CBS, about the Legal Stuff.
akhilapriya404 said…
The information which you have provided is very good. It is very useful who is looking for machine learning online course
Anonymous said…
This comment has been removed by the author.
BenRad1920 said…
Would anybody be interested in sharing specific setting or in this case number values used to upscale footage, specifically with the Topaz AI Software? I do own the entire software package but just can't get the results CaptRobau gets out of his material ... Any help would be greatly appreciated ...
dtj said…
How do you maintain audio sync with the newly assembled video?
CaptRobau said…
@dtj

This is based on a FFMPEG guide, so the audio parts of the FFMPEG guides just make that happen.
It’s hard to find experienced people about this topic, however, you sound like you know what you’re talking about! Thanks redmi note 8 pro display replacement bangalore You should take part in a contest for one of the finest sites on the web. I will recommend this website! realme x display replacement
Aw, this was an exceptionally nice post. Spending some time and actual effort to create a really good article… but what can I say… I hesitate a whole lot and never seem to get nearly anything done. samsung M30s display replacement I’m amazed, I must say. Rarely do I come across a blog that’s equally educative and entertaining, and without a doubt, you have hit the nail on the head. The issue is something that not enough folks are speaking intelligently about. Now i'm very happy I stumbled across this during my hunt for something concerning this. iphone 11 pro display replacement marathahalli

Oh my goodness! Awesome article dude! Thank you so much, However I am encountering problems with your RSS. I don’t understand the reason why I can't join it. Is there anybody else getting similar RSS problems? Anyone that knows the answer can you kindly respond? Thanks!! oneplus 7 pro display replacement bangalore An outstanding share! I've just forwarded this onto a colleague who had been conducting a little research on this. And he actually bought me breakfast simply because I found it for him... lol. So allow me to reword this.... Thanks for the meal!! But yeah, thanks for spending the time to talk about this subject here on your website. vivo V17 pro display replacement marathahalli
Markus said…
I really think CBS hates Star Trek by now. Stargate had its HD release. Even Baywatch can be watched in HD today. Why not one of the greatest shows of the 90s? How comes CBS can't make profit with Trek? They don't even try. Poor marketing decisions and their results provide lame excuses. It hurts.
Unknown said…
Hey there Capt! It looks great at 4k resolution! I have a very powerful computer (core i-9 5.0ghz and nvidia geforce 2080ti, 32gb RAM) and native 4k display equipment and would be happy to help with future attempts to upscale DS9. Respond to my comment and it'll be forwarded to my email.
Bobby Degnan said…
Bob Degnan says, Star Trek fans and U2 fans, I love 'em all. Great work. You made my day today.
Bobby Degnan said…
I wish I could add something technical but I'm unfamiliar with all this.. I would like to say, the end product, I'm hoping will be offered as a torrent, also, anything over 1.5 gigabytes per episode is unrealistic for me. Great work so far.
Mathew Marcum said…
This comment has been removed by the author.