Remove video background
In this notebook, we go through a few example methods to remove the background in a video.
Note, although these methods are relatively quick (examples are shown for running it on a single CPU core, multiple CPU cores, a couple of methods for running it on the GPU), you’d want to stick these methods in a loop to iterate over all the videos you probably have to get the most of it.
[1]:
from ipywidgets import Video
from IPython.display import HTML, Image
from simba.video_processors.video_processing import create_average_frm, video_bg_subtraction, video_bg_subtraction_mp
[2]:
# BEFORE WE START, LETS TAKE A LOOK AT THE ORIGINAL VIDEO WHICH WE WILL REMOVE THE BACKGROUND FROM
video_url = 'https://raw.githubusercontent.com/sgoldenlab/simba/master/docs/_static/img/open_field.webm'
HTML(f''' <video width="600" height="600" controls> <source src="{video_url}" type="video/webm"> </video>
''')
[2]:
[3]:
# PATH TO VIDEO WE WANT TO REMOVE BACKGROUND FROM
VIDEO_PATH = "/mnt/c/Users/sroni/Downloads/bg_remove_nb/open_field.mp4"
#LOCATION WHERE TO SAVE THE VIDEO WITH THE BACKGROUND REMOVED
SAVE_PATH = "/mnt/c/Users/sroni/Downloads/bg_remove_nb/bg_removed_ex_1.mp4"
# VALUE BETWEEN 0 AND 255. HIGHER VALUES AND MORE PIXELS WILL BE CONSIDERED BACKGROUND
THRESHOLD = 15
# SET VERBOSE TO TRUE TO GET PRINT OUTS TO FOLLOW THE PROGRESS.
VERBOSE=False
[4]:
# IN THIS FIRST EXAMPLE, WE PERFORM A VERY SIMPLE BACKGROUND REMOVAL.
# MEANING, NOTHING FANCY, WE JUST MAKE THE BACKGROUND WHITE WHILE THE FOREGROUND (THE ANIMAL) RETAINS THE ORIGINAL COLORS.
video_bg_subtraction(video_path=VIDEO_PATH, save_path=SAVE_PATH, verbose=VERBOSE, threshold=THRESHOLD, bg_color=(255, 255, 255)) # WE RUN THE BACKGROUND SUBTRACTION
[5]:
#EXPECTED RESULTS
video_url = 'https://raw.githubusercontent.com/sgoldenlab/simba/master/docs/_static/img/bg_removed_ex_1_clipped.webm'
HTML(f''' <video width="600" height="600" controls> <source src="{video_url}" type="video/webm"> </video>
''')
[5]:
[6]:
# AS SEEN AS IN THE EXAMPLE, IT'S NOT GREAT, AND WE MAY WANT TO CONSIDER MORE PIXELS TO BELOW TO THE BACKGROUND.
# WE UP THE THRESHOLD FOR WHAT SHOULD BE CONSIDERED BACKGROUND FROM 10 TO 40
#LOCATION WHERE TO SAVE THE BACKGROUND REMOVED VIDEO
SAVE_PATH = "/mnt/c/Users/sroni/Downloads/bg_remove_nb/bg_removed_ex_2.mp4"
# VALUE BETWEEN 0 AND 255. HIGHER VALUES AND MORE PIXELS WILL BE CONSIDERED BACKGROUND
THRESHOLD = 40
# WE RUN THE BACKGROUND SUBTRACTION
video_bg_subtraction(video_path=VIDEO_PATH, save_path=SAVE_PATH, verbose=False, threshold=THRESHOLD, bg_color=(255, 255, 255))
[7]:
#EXPECTED RESULTS
video_url = 'https://raw.githubusercontent.com/sgoldenlab/simba/master/docs/_static/img/bg_removed_ex_2_clipped.webm'
HTML(f''' <video width="600" height="600" controls> <source src="{video_url}" type="video/mp4"> </video>
''')
[7]:
[8]:
# ALTERNATIVELY, WE REMOVE THE BACKGROUND, MAKING THE BACKGROUND COLOR RED, WHILE THE FOREGROUND BLUE.
#LOCATION WHERE TO SAVE THE BACKGROUND REMOVED VIDEO
SAVE_PATH = "/mnt/c/Users/sroni/Downloads/bg_remove_nb/bg_removed_ex_3.mp4"
# WE RUN THE BACKGROUND SUBTRACTION
video_bg_subtraction(video_path=VIDEO_PATH, save_path=SAVE_PATH, verbose=False, threshold=THRESHOLD, bg_color=(0, 0, 255), fg_color=(255, 0, 0))
[9]:
#EXPECTED RESULTS
video_url = 'https://raw.githubusercontent.com/sgoldenlab/simba/master/docs/_static/img/bg_removed_ex_3_clipped.webm'
HTML(f''' <video width="600" height="600" controls> <source src="{video_url}" type="video/mp4"> </video>
''')
[9]:
[10]:
# ALTERNATIVELY, WE MAY HAVE A SECOND VIDEO OF THE BACKGROUND WHICH CAN BE USED TO COMPUTE THE AVERAGE BACKGROUND TO REMOVE THE BACKGROUND.
# THIS WORKS BEST IF WE PLAN TO REMOVE THE BACKGROUND FROM VIDEOS WHERE THE ANIMALS ARE LARGELY IMMOBILE.
#LOCATION WHERE TO SAVE THE BACKGROUND REMOVED VIDEO
SAVE_PATH = SAVE_PATH = "/mnt/c/Users/sroni/Downloads/bg_remove_nb/bg_removed_ex_4.mp4"
#PATH TO BACKGGROUND VIDEO
BG_VIDEO_PATH = "/mnt/c/Users/sroni/Downloads/bg_remove_nb/MY_BACKGROUND_VIDEO.mp4"
# WE RUN THE BACKGROUND SUBTRACTION
video_bg_subtraction(video_path=VIDEO_PATH, bg_video_path=BG_VIDEO_PATH, save_path=SAVE_PATH, verbose=False, threshold=THRESHOLD, bg_color=(0, 0, 255), fg_color=(255, 0, 0))
[11]:
# ALTERNATIVELY, WE MAY WANT TO COMPUTE THE AVERAGE FRAME FOR BACKGROUND SUBTRACTION FROM THE FIRST 20 SECONDS OF THE VIDEO
# THIS WORKS BEST IF THE RECORDING HAS AN EMPTY ARENA FOR THE THE FIRST 20 SECONDS. THEN WE CAN USE THOSE 20 SECONDS TO ACURATELY CAPTURE THE BACKGROUND.
#LOCATION WHERE TO SAVE THE BACKGROUND REMOVED VIDEO
SAVE_PATH = SAVE_PATH = "/mnt/c/Users/sroni/Downloads/bg_remove_nb/bg_removed_ex_5.mp4"
# WE RUN THE BACKGROUND SUBTRACTION
video_bg_subtraction(video_path=VIDEO_PATH, bg_start_time='00:00:00', bg_end_time='00:00:20', save_path=SAVE_PATH, verbose=False, threshold=THRESHOLD, bg_color=(0, 0, 255), fg_color=(255, 0, 0))
[12]:
# ALTERNATIVELY, WE CAN PERFORM BACKGROUND SUBTRACTION BUT USING MULTI-CORE PROCESSING (USING THE 31 CORES i HAVE AVAILABLE IN THE EXAMPLE BELOW).
# CAN BE QUICKER IF THE VIDEO(S) ARE LONG, YOU HAVE MANY OF VIDEOS, AND/OR YOU HAVE PLENTY OF CPU CORES
video_bg_subtraction_mp(video_path=VIDEO_PATH, save_path=SAVE_PATH, verbose=False, bg_color=(255, 255, 255), fg_color=(0, 0, 255), core_cnt=31)
Frame batch 1 completed...
Frame batch 2 completed...
Frame batch 3 completed...
Frame batch 4 completed...
Frame batch 5 completed...
Frame batch 6 completed...
Frame batch 7 completed...
Frame batch 8 completed...
Frame batch 9 completed...
Frame batch 10 completed...
Frame batch 11 completed...
Frame batch 12 completed...
Frame batch 13 completed...
Frame batch 14 completed...
Frame batch 15 completed...
Frame batch 16 completed...
Frame batch 17 completed...
Frame batch 18 completed...
Frame batch 19 completed...
Frame batch 20 completed...
Frame batch 21 completed...
Frame batch 22 completed...
Frame batch 23 completed...
Frame batch 24 completed...
Frame batch 25 completed...
Frame batch 26 completed...
Frame batch 27 completed...
Frame batch 28 completed...
Frame batch 29 completed...
Frame batch 30 completed...
Frame batch 31 completed...
Joining open_field multiprocessed video...
Input #0, concat, from '/mnt/c/Users/sroni/Downloads/bg_remove_nb/temp_open_field_20241202094237/files.txt':
Duration: N/A, start: 0.000000, bitrate: 805 kb/s
Stream #0:0(und): Video: mpeg4 (Simple Profile) (mp4v / 0x7634706D), yuv420p, 480x432 [SAR 1:1 DAR 10:9], 805 kb/s, 30 fps, 30 tbr, 15360 tbn
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
Stream mapping:
Stream #0:0 -> #0:0 (mpeg4 (native) -> h264 (libx264))
Press [q] to stop, [?] for help
[libx264 @ 0x562e1fabd600] using SAR=1/1
[libx264 @ 0x562e1fabd600] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 0x562e1fabd600] profile High, level 3.0, 4:2:0, 8-bit
[libx264 @ 0x562e1fabd600] 264 - core 164 r3108 31e19f9 - H.264/MPEG-4 AVC codec - Copyleft 2003-2023 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=13 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to '/mnt/c/Users/sroni/Downloads/bg_remove_nb/bg_removed_ex_5.mp4':
Metadata:
encoder : Lavf60.16.100
Stream #0:0(und): Video: h264 (avc1 / 0x31637661), yuv420p(progressive), 480x432 [SAR 1:1 DAR 10:9], q=2-31, 30 fps, 15360 tbn
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
encoder : Lavc60.31.102 libx264
Side data:
cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
frame= 4236 fps=2815 q=29.0 size= 1792kB time=00:02:21.13 bitrate= 104.0kbits/s speed=93.8x
SIMBA COMPLETE: Video concatenated (elapsed time: 2.2029s) complete
SIMBA COMPLETE: Video saved at /mnt/c/Users/sroni/Downloads/bg_remove_nb/bg_removed_ex_5.mp4 (elapsed time: 6.5732s) complete
[out#0/mp4 @ 0x562e1fa9fbc0] video:2162kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 2.688034%
frame= 4980 fps=2832 q=-1.0 Lsize= 2220kB time=00:02:45.90 bitrate= 109.6kbits/s speed=94.3x
[libx264 @ 0x562e1fabd600] frame I:20 Avg QP: 8.02 size: 4486
[libx264 @ 0x562e1fabd600] frame P:1419 Avg QP:20.67 size: 666
[libx264 @ 0x562e1fabd600] frame B:3541 Avg QP:24.82 size: 333
[libx264 @ 0x562e1fabd600] consecutive B-frames: 2.5% 5.3% 8.0% 84.2%
[libx264 @ 0x562e1fabd600] mb I I16..4: 92.1% 0.7% 7.1%
[libx264 @ 0x562e1fabd600] mb P I16..4: 0.2% 0.4% 0.3% P16..4: 2.0% 0.8% 0.6% 0.0% 0.0% skip:95.6%
[libx264 @ 0x562e1fabd600] mb B I16..4: 0.1% 0.1% 0.0% B16..8: 2.3% 0.7% 0.4% direct: 0.2% skip:96.3% L0:52.7% L1:42.5% BI: 4.9%
[libx264 @ 0x562e1fabd600] 8x8 transform intra:23.8% inter:1.6%
[libx264 @ 0x562e1fabd600] coded y,uvDC,uvAC intra: 8.8% 22.9% 20.6% inter: 0.7% 1.1% 1.0%
[libx264 @ 0x562e1fabd600] i16 v,h,dc,p: 89% 6% 5% 0%
[libx264 @ 0x562e1fabd600] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 7% 7% 85% 0% 0% 0% 0% 0% 0%
[libx264 @ 0x562e1fabd600] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 21% 14% 50% 3% 2% 3% 2% 3% 2%
[libx264 @ 0x562e1fabd600] i8c dc,h,v,p: 72% 12% 15% 1%
[libx264 @ 0x562e1fabd600] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x562e1fabd600] ref P L0: 69.4% 3.9% 16.5% 10.2%
[libx264 @ 0x562e1fabd600] ref B L0: 73.9% 21.3% 4.8%
[libx264 @ 0x562e1fabd600] ref B L1: 93.3% 6.7%
[libx264 @ 0x562e1fabd600] kb/s:106.66
[21]:
# INSTEAD OF A SINGLE CPU CORE, OR MULTIPLE CPU CORES, WE USE THE GPU TO PERFORM BACKGROUND SUBTRACTION.
# THIS CAN BE FASTER IF YOUR VIDEO(S) ARE LONG, YOU HAVE MANY OF VIDEOS,BUT NOT TOO MANY CPU CORES.
#TO GET THIS TO WORK, WE FIRST MANUALLY HAVE TO COMPUTE THE "AVERAGE" FRAME IN THE VIDEO.
#THIS AVERAGE FRAME CAN BE CREATED USING THE CPU...
from simba.data_processors.cuda.image import bg_subtraction_cuda, bg_subtraction_cupy
avg_frm = create_average_frm(video_path=VIDEO_PATH)
[14]:
# LETS TAKE A LOOK AT THE BACKGROUND TO SEE WHAT WAS COMPUTED
Image(url= 'https://raw.githubusercontent.com/sgoldenlab/simba/master/docs/_static/img/bg_avg_frm.webp')
[14]:
[15]:
#... OR SHOULD YOU PREFER, WE CAN US THE GPU TO COMPUTE THE AVERAGE FRAME
from simba.data_processors.cuda.image import create_average_frm_cuda, create_average_frm_cupy, create_average_frm_cupy
# THE AVERAGE FRAME CAN BE COMPUTED ON THE GPU USING NUMBA CUDA...
avg_frm = create_average_frm_cuda(video_path=VIDEO_PATH)
[16]:
# ... OR, ALTERNATIVELY, WE CAN COMPUTE THE AVERAGE BACKGROUND USING THE GPU AND CUPY
avg_frm = create_average_frm_cupy(video_path=VIDEO_PATH)
[17]:
# ONCE WE GOT THE AVERAGE FRAME, WE REMOVE THE BACKGROUND FROM THE VIDEO USING THE AVERAGE FRAME ON THE GPU AND NUMBA CUDA
SAVE_PATH = SAVE_PATH = "/mnt/c/Users/sroni/Downloads/bg_remove_nb/bg_removed_ex_6.mp4"
BATCH_SIZE = 500 # HOW MANY IMAGES WE SEND TO THE GPU SEQUENTIALLY. INCREASE THIS VALUE IF YOUR GPU ALLOWS IT.
#AGAIN, THIS OPERATION CAN BE PERFORMED USING NUMBA CUDA
bg_subtraction_cuda(video_path=VIDEO_PATH, avg_frm=avg_frm, fg_clr=(255, 0, 255), bg_clr=(186, 142, 35), save_path=SAVE_PATH, threshold=THRESHOLD, batch_size=BATCH_SIZE)
Processing frame batch 1 / 10 (complete: 0.0%)
/home/simon/miniconda3/envs/simba_310/lib/python3.10/site-packages/numba/cuda/cudadrv/devicearray.py:886: NumbaPerformanceWarning: Host array used in CUDA kernel will incur copy overhead to/from device.
warn(NumbaPerformanceWarning(msg))
Processing frame batch 2 / 10 (complete: 10.0%)
/home/simon/miniconda3/envs/simba_310/lib/python3.10/site-packages/numba/cuda/cudadrv/devicearray.py:886: NumbaPerformanceWarning: Host array used in CUDA kernel will incur copy overhead to/from device.
warn(NumbaPerformanceWarning(msg))
Processing frame batch 3 / 10 (complete: 20.0%)
/home/simon/miniconda3/envs/simba_310/lib/python3.10/site-packages/numba/cuda/cudadrv/devicearray.py:886: NumbaPerformanceWarning: Host array used in CUDA kernel will incur copy overhead to/from device.
warn(NumbaPerformanceWarning(msg))
Processing frame batch 4 / 10 (complete: 30.0%)
/home/simon/miniconda3/envs/simba_310/lib/python3.10/site-packages/numba/cuda/cudadrv/devicearray.py:886: NumbaPerformanceWarning: Host array used in CUDA kernel will incur copy overhead to/from device.
warn(NumbaPerformanceWarning(msg))
Processing frame batch 5 / 10 (complete: 40.0%)
/home/simon/miniconda3/envs/simba_310/lib/python3.10/site-packages/numba/cuda/cudadrv/devicearray.py:886: NumbaPerformanceWarning: Host array used in CUDA kernel will incur copy overhead to/from device.
warn(NumbaPerformanceWarning(msg))
Processing frame batch 6 / 10 (complete: 50.0%)
/home/simon/miniconda3/envs/simba_310/lib/python3.10/site-packages/numba/cuda/cudadrv/devicearray.py:886: NumbaPerformanceWarning: Host array used in CUDA kernel will incur copy overhead to/from device.
warn(NumbaPerformanceWarning(msg))
Processing frame batch 7 / 10 (complete: 60.0%)
/home/simon/miniconda3/envs/simba_310/lib/python3.10/site-packages/numba/cuda/cudadrv/devicearray.py:886: NumbaPerformanceWarning: Host array used in CUDA kernel will incur copy overhead to/from device.
warn(NumbaPerformanceWarning(msg))
Processing frame batch 8 / 10 (complete: 70.0%)
/home/simon/miniconda3/envs/simba_310/lib/python3.10/site-packages/numba/cuda/cudadrv/devicearray.py:886: NumbaPerformanceWarning: Host array used in CUDA kernel will incur copy overhead to/from device.
warn(NumbaPerformanceWarning(msg))
Processing frame batch 9 / 10 (complete: 80.0%)
/home/simon/miniconda3/envs/simba_310/lib/python3.10/site-packages/numba/cuda/cudadrv/devicearray.py:886: NumbaPerformanceWarning: Host array used in CUDA kernel will incur copy overhead to/from device.
warn(NumbaPerformanceWarning(msg))
Processing frame batch 10 / 10 (complete: 90.0%)
/home/simon/miniconda3/envs/simba_310/lib/python3.10/site-packages/numba/cuda/cudadrv/devicearray.py:886: NumbaPerformanceWarning: Host array used in CUDA kernel will incur copy overhead to/from device.
warn(NumbaPerformanceWarning(msg))
SIMBA COMPLETE: Video saved at /mnt/c/Users/sroni/Downloads/bg_remove_nb/bg_removed_ex_6.mp4 (elapsed time: 25.0268s) complete
[18]:
#EXPECTED RESULTS
video_url = 'https://raw.githubusercontent.com/sgoldenlab/simba/master/docs/_static/img/bg_removed_ex_6_clipped.webm'
HTML(f''' <video width="600" height="600" controls> <source src="{video_url}" type="video/mp4"> </video>
''')
[18]:
[23]:
# OR, ALTERNATIVELY, WE CREATE THE BACKGROUND IMAGE USING THE GPU AND THE CUPY LIBRARY.
SAVE_PATH = SAVE_PATH = "/mnt/c/Users/sroni/Downloads/bg_remove_nb/bg_removed_ex_7.mp4"
bg_subtraction_cupy(video_path=VIDEO_PATH, avg_frm=avg_frm, fg_clr=(0, 0, 255), bg_clr=(139, 128, 0), save_path=SAVE_PATH)
Processing frame batch 1 / 10 (complete: 0.0%)
Processing frame batch 2 / 10 (complete: 10.0%)
Processing frame batch 3 / 10 (complete: 20.0%)
Processing frame batch 4 / 10 (complete: 30.0%)
Processing frame batch 5 / 10 (complete: 40.0%)
Processing frame batch 6 / 10 (complete: 50.0%)
Processing frame batch 7 / 10 (complete: 60.0%)
Processing frame batch 8 / 10 (complete: 70.0%)
Processing frame batch 9 / 10 (complete: 80.0%)
Processing frame batch 10 / 10 (complete: 90.0%)
SIMBA COMPLETE: Video saved at /mnt/c/Users/sroni/Downloads/bg_remove_nb/bg_removed_ex_7.mp4 (elapsed time: 11.3978s) complete
[24]:
#EXPECTED RESULTS
video_url = 'https://raw.githubusercontent.com/sgoldenlab/simba/master/docs/_static/img/bg_removed_ex_7_clipped.webm'
HTML(f''' <video width="600" height="600" controls> <source src="{video_url}" type="video/mp4"> </video>
''')
[24]:
[ ]: