Remastering Star Trek: Deep Space Nine With Machine Learning

As a little side-project, I have been working on putting the artificial neural networks of AI Gigapixel to the test and having them upscale another favorite thing of mine... Star Trek: Deep Space Nine (DS9).

The State of Deep Space Nine

Just like Final Fantasy 7, of which I am upscaling the backgrounds, textures, and videos in Remako mod, DS9 was also relegated to a non-HD future. While the popular Original Series and The Next Generation were mostly shot on film, the mid 90s DS9 had its visual effects shots (space battles and such) shot on video.

While you can rescan analog film at a higher resolution, video is digital and can't be rescanned. This makes it much costlier to remaster this TV show, which is one of the reasons why it hasn't happened.

Remastering Star Trek: Deep Space Nine With Machine Learning

This is where neural networks could come in, I thought. With tools like AI Gigapixel, I knew it might be possible the low definition frames of DS9 can be scaled up to a higher definition such as 1080p or 4K. It would never be the same as proper remastering, but it would a step in the good direction.

So I tried my hand at frame or two, to see what it could do. The results were great. AI Gigapixel uses neural networks trained on real photos. So while it did okay with upscaling the video game renders of Final Fantasy, it did amazing upscaling real-life footage and the bigger budget CGI effects of DS9.

Here are some examples below:


Original 480p definition (click to enlarge)


DS9 Enhanced 1080p definition (click to enlarge)


Original 480p definition (click to enlarge)


DS9 Enhanced 1080p definition (click to enlarge)

These still frames showed promise. In the first set of images, the maintenance crewmen in their spacesuits were nothing more than a few pixely blobs. The upscaling process turned the blobs into much more defined figures

The close-up of the hand also improved. The creases and folds of the fingers and hand look much more detailed, and the baseball really shows off its sheen and the intricate stitching.

Moving Images

The real test, however, was going to be if the upscaling process held up with a sequence of frames aka as a video. Would there be artifacts or other unsightly issues? AI Gigapixel was after all made for upscaling single images so it wouldn't take into account the relation between the individual frames of a moving image.

So I set about upscaling a portion of an episode. I settled on the season six episode: "Sacrifice of Angels". A great Dominion War episode that had both epic space battles and more personal face-to-face moments. And so I set out to work.

I will go into greater detail about my process in a future blog post, but it took me about two days to get everything extracted, upscaled and put it back together in a way that was pleasing. This resulted only in the first five minutes of the episode being done (the episode recap, the opening scene, and the intro). Still pretty good time for a mid-to-high end PC with software that isn't just available to professionals.

The result left me pretty awestruck. It looked better than I had hoped. No weird issues or anything. It looked pretty much like an HD version of DS9. Since (moving) pictures are worth more than a thousand words, here are two comparison videos that show off the improvement I was able to get with this machine learning based upscaling technique.

The first shows off the before-and-after situation with still frames.



The second puts the two videos side by side. Take note how much clearer and sharper the Enhanced version looks.



I highly recommend watching all these videos through your YouTube app on your TV if possible. It gives you more of a sense of how it would feel watching an enhanced DS9 on your TV.

Comparisons are all well and good, but what does it look like if you were to watch it normally? Below is a video of the first five minutes of the episode in full 1080p:




NOTE: I hadn't realized this hoster reduced the quality of the video so much. If you really want to see the quality of the above video, download the video itself here (139 MB).

What About 4K?

Honestly, I don't know. While I can upscale the image to a 4K resolution, I don't have a TV or monitor with a 4K native resolution to see if it looks better. I have nonetheless made this video and I am interested to hear from people with people with 4K equipment if it looks better over the 1080p version of the intro.

This nearly melted my computer, as it is a lot more intense to upscale than 1080p so I'll stick to this single video for 4K examples of DS9 Enhanced.


What's Next?

Since I do not own DS9, I can not just do what I want with it. While I would love to release full episodes, this is just not legally possible. These videos serve more as a proof of concept for CBS to look into machine learning and neural networks to help remaster DS9 and move it a bit closer to the HD era.

Imagine what a real team could do, with more powerful equipment, custom trained neural networks (perhaps training the network on TNG vs. TNG Remastered images) and access to the original SD files instead of a DVDRip like me.

What I will do is go into further detail about my process, which will be the subject of a future blog.

Let me know what you think.


Comments

  1. This is incredible. Fantastic work! It's truly shocking how much detail the AI can recreate!

    ReplyDelete
    Replies
    1. Thanks! Even though I've been working with the tech for a few months it still surprises me with its good-looking results some times.

      Delete
  2. I think its fantastic. Of course not real HD like next generation, but much, much better than I thougt it would be possible.

    Looking forward your next blog post. I would love to try it on my own.

    ReplyDelete
    Replies
    1. I'm glad you think it's fantastic. Getting it to look as good TNG Remastered was never on the table, but even a Remastered Lite is still better than the old SD footage.

      Delete
  3. This comment has been removed by the author.

    ReplyDelete
  4. I'd think you would be in the same position as "Star Wars: Revisited" - fan edits like this are legally pretty solid.

    ReplyDelete
  5. These look absolutely fantastic! I've been playing with AI Gigapixel for a while and the results have been quite astonishing, so I'm happy to hear that creating videos with it seems to work as well. Though I wonder how much hard drive space those stills take up, let alone the time it takes to render all that!

    Hopefully in the future, we might see some of these machine learning upscalers become plug-ins for programs like Premiere, which could make these kinds of enhancements much easier.

    ReplyDelete
    Replies
    1. Can you try it on some footage that has heavy motion blur?

      Delete
    2. You can eliminate motion blur with the high setting in AI gigapixel but it makes certain features look weird. I have only tried it on individual frames but to me it would not be faithful to the source material.

      Delete
    3. Upscaling from a 480p image (500kb png) to 4k is about a 10x increase in file size. So to answer your question, a rediculous amount. Luckily I just bought a 10tb hard drive.

      Delete
    4. The first five minutes was:

      3 GB for the exported PNG frames (you need that format so that the upscaler does not upscale compression artifacts)

      5.5 GB for the upscaled frames in JPG with Maximum Quality. For a final, commercial, product you'd use PNG as well, since you wouldn't want to get any quality reduction. For my tests it was fine.

      Multiply both by 9 (45 min episode) and you get 76 GB per episode.

      Delete
  6. From the producer’s own account it seems the most difficult part is to remaster the VFX and CG part of the show. Since many space scene were not shot using models on film and were rendered digitally. Perhaps they have los the original 3D assets or it is in formats that is hard to use with modern production software.

    ReplyDelete
    Replies
    1. I imagine they lost the assets or can otherwise not access them. I can't imagine that if they access them that they could not simply export at a higher resolution. So they must be gone.

      Delete
    2. Even if you had access to the assets they wouldn't be HD worthy. And if you're going to remaster you might as well go 4k. Would be cheaper and better to just buy new models off of the stock-model market. There are plenty of very faithful CG models that are superior to the original show assets.

      And honestly you could set up one person for a year and they might be able to recreate all of the shots considering how expensive it was back then and how slow.

      Delete
    3. According to this article http://trekcore.com/blog/2013/05/deep-space-nine-in-high-definition-one-step-closer/ they actually do have a lot of the original files, and they were designed with enough detail to hold up in HD. They're even an example frame re-rendered in HD for comparison.

      Delete
    4. Perhaps you can use scene or cut detection software to break down the scenes as preprocess step. Then fed that scene as a single batch.

      Delete
    5. The Worm hole will be one of hard ones to redo

      Delete
    6. Actually it is possible to re-render the original VFX, Adam "Mojo" Lebowitz one of the guys that originally did the CGI for the show has re-rendered the original CGI models in HD and this was the result https://www.youtube.com/watch?v=bDWyfGjFsyU , I recomend that you download the original file from the description to avoid the heavy Youtube compression.

      Delete
  7. What are you using for source material? I actually had this exact same idea and just purchased the box set. I can tell you 4k does look better from my own proof of concept. However, it has quickly become evident that using Ai gigapixel, at least in it's current form, will not really be feasible. While it can do batch processing, adding hundreds of thousands of frames will often result in the program crashing. So I set it up overnight and wake up with only a fraction of the work done. While it can be done, the pita factor is pretty much making a non starter.

    ReplyDelete
    Replies
    1. A DVDRip. Those are somewhat optimized for downloading, so it's not going to be the format. Still I got a pretty good one, otherwise I wouldn't have gotten nearly the level of quality. A direct screengrab upscaled from you DVD boxset should be the best result. Although I imagine there's something with slightly better quality in the CBS archives. I assume they did a bit of reduction to fit everything nicely on a disc.

      Gigapixel is indeed not suitable for large projects. A thousand or so frames it can load nicely, but go beyond that and it has to fit too much into memory at once. Some batch system that loads images one at a time would be the best for this.

      Still it works well enough to get a few nice clips that can allow us to dream.

      Delete
    2. This comment has been removed by the author.

      Delete
    3. @ Unknown: I think you are doing something totally wrong or have a PC from the 80ies. I use the program myself and it works quite well. I can easily queue up 50.000 images - ok, it takes a few moments to load but there are no crashes or something. The process of 4000 to 5000 images takes ca. 1 hour ... and i have a normal mid-price PC.
      So could it be you are extremly under the minimum requirement?

      Delete
  8. Also, ideally it would be beneficial to use adjacent frames to upscale the footage, and then use AI to sharpen the upscaled footage. Luckily this software exists and if it works like I hope I'll post a proof of concept.

    ReplyDelete
    Replies
    1. I second that. Just leave a comment here or contact me via the forum. I'd be glad to showcase it or link to your own site/upload of this thing.

      Good luck!

      Delete
    2. It didn't work, I think I put too much stock in the super resolution technique. It may work better with a cleaner source, but the DVD source has way too much artifacts which Gigapixel seems to clean up really well.

      Delete
    3. You may need some pro format like D1

      Delete
  9. I have a large screen 4k TV and I love DS9. The problem with the TV is lack of high quality video. Another problem is the real differences between 1990s TV and 2019 TV although DS9 stories hold up the video does not CBS would be wise to use this process and editing to create versions of Star Trek it could make CBS all Access worth having.

    ReplyDelete
  10. This comment has been removed by a blog administrator.

    ReplyDelete
  11. Would love to be able to donate some CPU cycles towards rerendering the whole thing.

    ReplyDelete
  12. This would be awesome if it could be done to DS9 and Voyager! Those shows deserve it!

    ReplyDelete
  13. Like you, I'm skeptical this would be legal to provide episodes. But, I wonder if you could release the ML models, and we could feed it our own legal DVD-Rips on some GCP storage we pay for, if we could each make our own set. I don't think that'd run afoul of copywrite laws? I think it'd be pretty simple to build some scripts to make that happen. Plus, you get $300 of GCP credit to start out learning GCP. Sounds like a good idea to me!!

    ReplyDelete
  14. Please release this CBS! I would buy. & rewatch on my 4k TV.

    ReplyDelete
  15. The 1080P one is meh, but the 4k sample is pretty good. I would love to see a clip with some actors in it to see what that looks like.

    ReplyDelete
  16. This comment has been removed by a blog administrator.

    ReplyDelete
  17. Awesome! Do you think you would be willing to upscale the Battle of Wolf 359 from Emissary? I was going to try and I got all the original frames but my computer wasn't having any of it and AI Gigapixel crashed within the first 5 images.

    ReplyDelete
  18. Very interested in your process... I own the DS9 DVDs and enjoy the show but was pretty disappointed with the quality of the video... I also am one of the people that purchased the TNG remastered DVD - they are spectacular BTW... I'm unfamiliar with the googlepixel AI neural networks but would be willing donate time in training if that'
    s a possible thing?

    ReplyDelete
  19. Is it a memory issue? I got a machine with 256 gigs of RAM laying around if you wanna try.

    ReplyDelete
  20. Hi,

    Can you tell me your thoughts about my efforts on Image Upscaling?

    Original
    http://www.mediafire.com/file/8ffo4a6i36d7gdn/Original.avi

    Result
    http://www.mediafire.com/file/coebq3y6c348139/Result%2810%29.avi

    Original2
    http://www.mediafire.com/file/acx2y95hnisn4ud/Original%282%29.avi

    Result2
    http://www.mediafire.com/file/mct2ddk37titqnv/Result10.avi

    Regards

    ReplyDelete
    Replies
    1. It looks nice. A 2x jump in resolution isn't too obvious, but I can definitely see an improvement in the resolution when looking at the signage.

      Delete
  21. This is wonderful. This needs to happen. This also should happen to Babylon 5. They shot that show in HD, but the effects were SD.

    ReplyDelete
    Replies
    1. God please, this! We run the risk of seeing low detail ships in HD, but the show deserves the chance.

      Delete
  22. This comment has been removed by the author.

    ReplyDelete
  23. I think you could get even better results if the ML algorithm was trained on video, (possibly even on the formats to be converted) rather than still images, and was allowed to use adjacent frames.

    ReplyDelete
  24. This is an awesome idea! The results are very promising, from trekkie to trekkie keep it going this is really cool! And from developer to developer that is a very creative approach for DL, nice idea!

    All the best
    Sven

    ReplyDelete
  25. This comment has been removed by a blog administrator.

    ReplyDelete
  26. The opening credits was always going to be problematic. Even the studio would probably have to actually redo the CGI of the sequence, to deal with the muggy comet's tail and polish up the lettering. That said, on my ± 2K screen, the 4K version did shine in regards to the rest, dealing effectively with some aliasing visible in the 1080p version. Perhaps upscaling beyond target resolution does intrinsically improve the result by providing more pixel data.

    I'd love to know what the studio makes of this proof of concept. Well done.

    ReplyDelete
    Replies
    1. Super Resolutions are very effective in creating a better looking image.

      Delete
  27. This comment has been removed by a blog administrator.

    ReplyDelete
  28. HI,
    I own of AI Gigapixel and use it to enlarge jpegs and it does a far better job than Photoshop. I would like to try this on smaller trek videos (deleted scenes for props or costumes I own) but I have no idea how to extract the stills and reassemble them. Can you post how that is done or point me to a resource. Thanks.

    ReplyDelete
  29. Amazing job! Do you think similar results could be replicated with Waifu2x in Photo mode, It's basically a free open-source version of AI Gigapixel.

    ReplyDelete
  30. This is a very amazing post for cheap web hosting services. in this post, you have provided all the basic information regarding.

    website builder for reseller

    ReplyDelete
  31. Comfortabl y, the post is really the freshest on that deserving topic. I harmonise with your conclusions and definitely will thirstily look forward to your next updates.

    private label website builder

    ReplyDelete
  32. Thank you for the link building list.I am going jot down this because it will help me a lot.Great blog! Please keep on posting such blog.

    white label website builder

    ReplyDelete
  33. I don't know. This doesn't look astonishing. You can perform almost the same upscale with avisynth and nnedi3_rpow2 neural network upscaler, which has been available since 2010.

    http://forum.doom9.org/showthread.php?t=147695

    ReplyDelete
  34. Nice Blog ! Thank you for your very nice articles. I look forward to visiting your site in the future! hipatitshah

    ReplyDelete
  35. This comment has been removed by the author.

    ReplyDelete
  36. Your post is very useful for learners. Thanks for providing such a great post...
    Blueprism online training

    ReplyDelete
  37. Everything looks fantastic.Amazing i m really impress with your content and very useful it.topJesseDiaz

    ReplyDelete

Post a Comment