This is my second blog, and it is a small guide to my journey to try to implement new features to Nuke, sometimes it’s easy, and sometimes is super hard and complex. This one is in a middle I think. Nuke is just supporting software, heavy lifting is done in conda. Below you could find result before and after.
The Fields directed Duncan Guymer – footage from Hugo’s Desk course
STEP 1.:
Prep plate XMem2 support JPEGs and ViTMatte PNGs also for saving GPU memory XMem2 have to by downsize to 480p I will provide script there is Size calculator for saving time.
STEP 2.:
Finding best frames for mask reference which we will feed to XMem2, lets try use CV2 and calculate image weight. Its easy to decide when we have sequence of 100 frames, but it will be harder with sequence of 400 frames and we want this proces automated for maximum, because process itself is long, but worthed.
For this I used edited code from Gabrielle Garam You.
STEP 3.:
Roto out 4 frames and convert this roto to 1 object, in script I provide CV2 tool for it, because XMem2 can read only 1 object mask.
STEP 4.:
Duplicate first frame and copy files to right folder, where is stored XMem2, for this is in the script other automated button, you just need to adjust your path.
STEP 5.:
XMem2: Production-level Video Segmentation From Few Annotated Frames
1, conda activate X2
2, cd “C:\Users\WORKSTATION\Desktop\XMem2-main”
3, python interactive_demo.py –images example_videos/CHANGE FOLDER NAME
UI of XMem2 is super cool!
STEP 6.:
Upscale Alpha from XMem2 for Trimap creation. I simplified Trimap tool, shuffle alpha to RGBA and than use this tool.
STEP 7.:
Run ViTMatte and we have good Mask as a starting point for compositing, there is still lot of edge work, but we save huge amount of time. More about ViTMatte you could find in my first article. Only one note, ViTMatte using padding, for fitting mask in right place we have to adjust mask with transform in my case it was -9px at Y.
Here you could find the script.