jUpscaler - AI upscale any image/video to 4k [Real-ESRGAN GUI]

Hey, honestly I abandoned the project but I'm at home for two days, I'll see if I can do something.AV1 has a royalty-free licensing model so i think i could add it.

sonicboomecho1 year ago

Thank you so much for doing this! Your project will continue to be useful for many years to come.

Hello, I was just playing around with adding AV1 to the program, and I have some bad news. After changing the encoder from x264 to AV1, the video rendering takes several minutes instead of just a few seconds. I tried to figure it out for an hour, but I couldn't make it work. I know AV1 itself is slow in rendering, but this difference is quite big. However, if you'd like to give it a try, all you need to do is change "libx264" to "libaom-av1" in lines 246 and 622 of the GUI.pyw file. Keep in mind that you can't launch the program through GUI.exe then and you'll have to run it directly from GUI.pyw using Python.

sonicboomecho1 year ago

Thanks for trying. It is a real shame about aom-av1. Though have you tried libsvtav1 instead? The svt-av1 is much faster encoding than libaom-av1.

I read about SVT-AV1, and apparently, it's faster, but the quality is terrible. However, I checked available codecs in FFmpeg once again, and indeed, there are more to choose from (previously, I didn't expand the terminal window :P). I tried other encoders, and they are much faster (around few seconds). The whole idea of the program is "do it automatically," so I need to figure it out how to detect the most optimal encoder when the program starts. An update is coming soon.

Update added. Several changes + a new upscaling method for general use. It seems to me that it's not so much that you don't understand programming, but rather that my code was quite messy, which is why it took so long.

View more in thread

Flowgame1 year ago

Is there a minimum requirement to run this?

No there is not. I assume you're asking about the requirement for Nvidia graphics cards, but this version also works on AMD cards as well as integrated chipsets (cpu). Just a weaker computer specification will increase upscaling time.

Thanks for the great work! I do run into one problem, both in the previous version, as well as in v2 when using video as an input.

Using videos with integer FPS, such as 30fps, 25fps, etc, no problems with auto mode or auto FPS option.

However, videos with film-type FPS, e.g. 23.976, output at 23fps with desynchronized audio. In the previous version, when I tried manually inputting 23.976 fps in the box, the video would be extracted and upscaled, but the merge/mux step never occured. Entering an integer fps works as expected, though with the expected audio desync issue from lengthening or shortening the video track duration.

The current v2 appears to no longer have this FPS input option, and truncating the output fps to 23fps is still the behavior observed.

My untrained suspicion is somewhere along the way, when the source file's FPS is read to be automatically matched in the output, the decimal gets truncated, either when read or when sent to ffmpeg. I don't have any other non-integer framerate videos to try at the moment, though I suppose I could find/make one to test on, like a 29.97 or 59.94 fps file to see if it outputs 29 or 59 fps.

If passing the full decimal value is not possible, perhaps rounding to the nearest integer rather than truncating would be acceptable.

Example:

35484 frames source at 23.976fps -> 24.67 min original length

23.000 -> 25.71min (+1.04min or 62.4s) very desync by end, noticable early

24.000 -> 24.64min (-0.03min or 1.8s) much closer, only slightly noticable throughout

Of course, this would be worse for much longer source videos than half hour anime.

Sorry for the wall of text. Again, thanks for the work on the app, it is already proving useful beyond this edge case. The batch image file upscaling particularly is really great.

Thanks!

Can confirm: 29.97 video input results in output 29.00 fps video.

Also noticed that only first audio track and no subtitle tracks get muxed into the final output file when multi-audio is present. Possibly a future feature to consider :D

Probably just do a batch/shell script for the other audios/subs and fix the framerate after running the upscaler

Oh, thanks for feedback. I probably know what is problem with 23.976fps i'll try to fix that quickly.
"Also noticed that only first audio track and no subtitle tracks get muxed into the final output file when multi-audio is present." - this will be harder to fix i think. Maybe few days fixing.

Anyway, do you want to turn back manual fps setting? I thought no one is using this, but now i see i was wrong.

I don't think the manual fps is necessary if the auto mode can catch the non-integer framerate and pass them along the chain.

The only time I've intentionally altered fps has been to match audio/subs from other sources, like if I have good audio from one file, but good video from another, and one is 25fps and the other is 23.976. Other methods are more appropriate for dealing with that case though.

Thanks!

The fps issue should be fixed.
The audio/subtitles things, will be fixed soon i think.

Thanks. I've been trying it since the release and it works. I've only found one case in which it doesn't, which is videos which have soft telecine 3:2 pulldown that play in 29.97fps but are stored in original source framerate of 23.976fps. The playback device/software uses these tags that tell it which frames to duplicate to output 29.97 from the film source 23.976.

Here's an example of such an anime video's mediainfo output

The actual framerate is 23.976 or 24fps based on the # of frames, but the file contains those pulldown tags to specify which frames get duplicated by the player. This allows fewer actual frames to be encoded and stored, by not encoding the duplicates to save space.

When used as an input file, this produces a video stream that is 29.97 fps, but has a shorter duration than the original file and its audio tracks because the duplicate frames were not included and upscaled, and the soft pulldown tags are not retained. The resulting file runs out of video before the audio track ends.

When I use ffprobe -i on the file, 29.97 fps and 29.97 tbr are reported for the video stream.

I suspect this should be fixable, as if I use jUpscaler to extract and upscale the frame images, then run

.\ffmpeg.exe -y -r 23.976 -i ..\upscaled_frames\frame%08d.jpg -i '.\testin.mkv' -map 0:v:0 -map 1:a? -map 1:s? -c:a copy -c:s copy -c:v libx264 -r 29.97 -pix_fmt yuv420p testout.mkv

I get the desired result, with everything in sync, with the upscaled video stream, 23.976 stored fps, 29.97 playback fps, and all audio and subtitle streams copied over. Mediainfo says the file is vfr,

So keeping all audio and subtitle streams should be as easy as adjusting the ffmpeg calls jUpscaler makes, instead of -map 1:a:0?, use -map 1:a?; and same for subtitle, -map 1:s?

If an input file has no subtitle tracks, this should not throw any errors about it.

I don't know how to detect the 23.976 pulldown thing to do this automatically when needed, but for now I can just extract and upscale and batch run this ffmpeg argument set.

Thanks for all your hard work! Hope this info is useful.

View more in thread

BastianGVF2 years ago

guis good, just a tip for people using this but want to use more models then the ones included, here's a three step tutorial on how to get the right model files to swap out on the GUI Step 1: download cupscale, get the model that you want from https://upscale.wiki/wiki/Model_Database, select the realesrgan option on cupscale, drag an image into it, and then run it, that converted the model you wanted into bin and param files Step 2: go to .ncnn models in the cupscale folders and find the folder for the model you want, copy the bin and param files and then paste it into the models folder on jupscaler step 3: change the file name to the name of the model that you wanna swap with the other model, once it asks if you wanna replace the file you want to swap with the other model hit yes

Oh, thanks for information i didnt know that. Probably i include a few of those in future.

BastianGVF2 years ago

if you do that maybe include a 1x model option in the app for video clean up esrgan models

Well, i couldn't add the new methods because the are for ESRGAN not for Real-ESRGAN. Even if there is few of them, they don't work with converter. I will look closer for ESRGAN engine and if it works with non-nvidia cards i will try to implement

BastianGVF2 years ago (2 edits)

i made a mega link with some converted esrgan models, if you need that or something, here https://mega.nz/folder/CQggyLaB#Zz57uPUItv49njiyjmEkBw

Deleted account2 years ago

Deleted post