Egocentric data and video alignment
In this notebook, we will egocentrically align pose estimation and associated video data.
Egocentric alignment is a crucial preprocessing step with various applications.
One primary use case is in unsupervised deep learning scenarios, where images of animals are used as inputs to train algorithms. Standardizing the orientation of the subject (e.g., an animal) ensures that the model and subsequent analyses do not incorrectly interpret the same behavior in different orientations (e.g., grooming while facing north versus south) as distinct behaviors. By aligning the animal to a consistent frame of reference, we eliminate orientation as a confounding variable.
While egocentric alignment is an essential first step, it is often insufficient by itself for comprehensive analyses. Additional preprocessing steps are typically required, such as:
Background subtraction to isolate the animal from its surroundings (see other methods / notebooks).
Geometric segmentation to slice out and focus on the subject (again, see other methods / notebooks).
In this notebook, we will focus exclusively on performing egocentric alignment. Further preprocessing steps are outlined in related materials.
[1]:
from simba.data_processors.egocentric_aligner import EgocentricalAligner
from ipywidgets import Video
from IPython.display import HTML
[2]:
# WE DEFINE HOW WE SHOULD EGOCENTRICALLY ALIGN THE DATA AND VIDEOs
ANCHOR_POINT_1 = 'center' # Name of the body-part which is the "primary" anchor point around which the alignment centers. In rodents, this is often the center of the tail-base of the animal.
ANCHOR_POINT_2 = 'nose' # The name of the secondary anchor point defining the alignment direction. This is often the anterior body-part, in rodents it can be the nose or nape.
DIRECTION = 0 # The egocentric alignment angle, in degrees. For example, `0` and the animals `ANCHOR_POINT_2` is directly to the east (right) of `ANCHOR_POINT_1`. `180` and the animals `ANCHOR_POINT_2` is directly to the west (left) of `ANCHOR_POINT_1`.
ANCHOR_LOCATION = (250, 250) # The pixel location in the video where `ANCHOR_POINT_1` should be placed. For example, if the videos are 500x500, 250x250 will place the anchor right in the middle.
GPU = True # If we have an NVIDEA GPU availalable, we can use it to speed up processing. Otherwise set this to `False`.
FILL_COLOR = (0, 0, 0) # We are rotating videos, while at the same time retaining the original video size. Therefore, there will be some "new" areas exposed in the video (see below for more info). This is what color to color these new areas.
VERBOSE = False # If True, prints progress (like which frame and video is being processed etc). However, this information will flood this notebook is I am going to turn it off.
[3]:
# WE DEFINE THE PATHS TO THE DIRECTORIES HOLDING THE DATA AND VIDEOS, AND DIRECTORY WHERE WE SHOULD STORE THE RESULTS.
DATA_DIRECTORY = r'C:\Users\sroni\OneDrive\Desktop\rotate_ex\data' #DIRECTORY WHICH IS HOLDING POSE-ESTIMATION DATA
VIDEOS_DIRECTORY = r'C:\Users\sroni\OneDrive\Desktop\rotate_ex\videos' #DIRECTORY WHICH IS VIDEOS, ONE FOR EACH FILE IN THE DATA_DIRECTORY. NOTE: IF YOU SET THIS TO None, THEN THE ALIGNMENT WILL BE PERFORMED ON THE DATA ONLY.
SAVE_DIRECTORY = r"C:\Users\sroni\OneDrive\Desktop\rotate_ex\rotated" #DIRECTORY WHERE WE SHOULD SAVE THE ROTATED POSE-ESTIMATION AND ROTATED VIDEOS.
[4]:
# NOW WE ARE GOOD TO GO, USING THE INFORMATION ABOVE, WE DEFINE AN INSTANCE OF AN SimBA EGOCENTRIALIGNER AND RUN IT
# ON THE 6 VIDEOS AND VIDEO DATA INSIDE THE DATA AND VIDEO DIRECTORIES, RESPECTIVELY.
aligner = EgocentricalAligner(anchor_1=ANCHOR_POINT_1,
anchor_2=ANCHOR_POINT_2,
data_dir=DATA_DIRECTORY,
videos_dir=VIDEOS_DIRECTORY,
save_dir=SAVE_DIRECTORY,
direction=DIRECTION,
gpu=GPU,
anchor_location=ANCHOR_LOCATION,
fill_clr=FILL_COLOR,
verbose=VERBOSE)
aligner.run()
SIMBA COMPLETE: Video concatenated (elapsed time: 16.214s) complete
SIMBA COMPLETE: Egocentric rotation video C:\Users\sroni\OneDrive\Desktop\rotate_ex\rotated\501_MA142_Gi_Saline_0513.mp4 complete (elapsed time: 102.2651s) complete
SIMBA COMPLETE: Video concatenated (elapsed time: 15.0453s) complete
SIMBA COMPLETE: Egocentric rotation video C:\Users\sroni\OneDrive\Desktop\rotate_ex\rotated\501_MA142_Gi_Saline_0515.mp4 complete (elapsed time: 101.7265s) complete
SIMBA COMPLETE: Video concatenated (elapsed time: 14.7956s) complete
SIMBA COMPLETE: Egocentric rotation video C:\Users\sroni\OneDrive\Desktop\rotate_ex\rotated\501_MA142_Gi_Saline_0517.mp4 complete (elapsed time: 105.1418s) complete
SIMBA COMPLETE: Video concatenated (elapsed time: 16.5156s) complete
SIMBA COMPLETE: Egocentric rotation video C:\Users\sroni\OneDrive\Desktop\rotate_ex\rotated\502_MA141_Gi_Saline_0513.mp4 complete (elapsed time: 107.4481s) complete
SIMBA COMPLETE: Video concatenated (elapsed time: 14.7832s) complete
SIMBA COMPLETE: Egocentric rotation video C:\Users\sroni\OneDrive\Desktop\rotate_ex\rotated\502_MA141_Gi_Saline_0515.mp4 complete (elapsed time: 103.6864s) complete
SIMBA COMPLETE: Video concatenated (elapsed time: 14.9907s) complete
SIMBA COMPLETE: Egocentric rotation video C:\Users\sroni\OneDrive\Desktop\rotate_ex\rotated\502_MA141_Gi_Saline_0517.mp4 complete (elapsed time: 106.1234s) complete
SIMBA COMPLETE: Egocentrically aligned data for 6 files saved in C:\Users\sroni\OneDrive\Desktop\rotate_ex\rotated (elapsed time: 627.5204s) complete
[14]:
#EXAMPLE VIDEO EXPECTED RESULTS SNIPPET
video_url = 'https://raw.githubusercontent.com/sgoldenlab/simba/master/docs/_static/img/egocentric_nb_1.webm'
HTML(f''' <video width="600" height="600" controls> <source src="{video_url}" type="video/webm"> </video>
''')
[14]:
[7]:
#NOW, LET'S CHANGE A FEW SETTINGS, TO GET A FEELING FOR HOW IT BEHAVES …
ANCHOR_LOCATION = (500, 100)
FILL_COLOR = (255, 0, 0)
DIRECTION = 180
[8]:
# ... WE CREATE A NEW INSTANCE BASED ON THE UPDATED INFORMATION ABOVE, AND RUN IT.
aligner = EgocentricalAligner(anchor_1=ANCHOR_POINT_1,
anchor_2=ANCHOR_POINT_2,
data_dir=DATA_DIRECTORY,
videos_dir=VIDEOS_DIRECTORY,
save_dir=SAVE_DIRECTORY,
direction=DIRECTION,
gpu=GPU,
anchor_location=ANCHOR_LOCATION,
fill_clr=FILL_COLOR,
verbose=VERBOSE)
aligner.run()
SIMBA COMPLETE: Video concatenated (elapsed time: 16.2393s) complete
SIMBA COMPLETE: Egocentric rotation video C:\Users\sroni\OneDrive\Desktop\rotate_ex\rotated\501_MA142_Gi_Saline_0513.mp4 complete (elapsed time: 108.1662s) complete
SIMBA COMPLETE: Video concatenated (elapsed time: 15.0821s) complete
SIMBA COMPLETE: Egocentric rotation video C:\Users\sroni\OneDrive\Desktop\rotate_ex\rotated\501_MA142_Gi_Saline_0515.mp4 complete (elapsed time: 101.4825s) complete
SIMBA COMPLETE: Video concatenated (elapsed time: 14.7555s) complete
SIMBA COMPLETE: Egocentric rotation video C:\Users\sroni\OneDrive\Desktop\rotate_ex\rotated\501_MA142_Gi_Saline_0517.mp4 complete (elapsed time: 102.9438s) complete
SIMBA COMPLETE: Video concatenated (elapsed time: 16.4037s) complete
SIMBA COMPLETE: Egocentric rotation video C:\Users\sroni\OneDrive\Desktop\rotate_ex\rotated\502_MA141_Gi_Saline_0513.mp4 complete (elapsed time: 109.8081s) complete
SIMBA COMPLETE: Video concatenated (elapsed time: 14.8025s) complete
SIMBA COMPLETE: Egocentric rotation video C:\Users\sroni\OneDrive\Desktop\rotate_ex\rotated\502_MA141_Gi_Saline_0515.mp4 complete (elapsed time: 103.8362s) complete
SIMBA COMPLETE: Video concatenated (elapsed time: 15.0489s) complete
SIMBA COMPLETE: Egocentric rotation video C:\Users\sroni\OneDrive\Desktop\rotate_ex\rotated\502_MA141_Gi_Saline_0517.mp4 complete (elapsed time: 107.5951s) complete
SIMBA COMPLETE: Egocentrically aligned data for 6 files saved in C:\Users\sroni\OneDrive\Desktop\rotate_ex\rotated (elapsed time: 634.8647s) complete
[17]:
#EXAMPLE VIDEO EXPECTED RESULTS SNIPPET
video_url = 'https://raw.githubusercontent.com/sgoldenlab/simba/master/docs/_static/img/egocentric_nb_2.webm'
HTML(f''' <video width="600" height="600" controls> <source src="{video_url}" type="video/webm"> </video>
''')
[17]:
[11]:
# FINALLY, LET'S USE A DIFFERENT EXPERIMENT, WITH DIFFERENT BODY-PART NAMES TO SEE HOW IT BEHAVES
ANCHOR_POINT_1 = 'butt/proximal tail'
ANCHOR_POINT_2 = 'snout'
DIRECTION = 0
ANCHOR_LOCATION = (600, 300)
FILL_COLOR = (128, 120, 128) #SET BACKGROUND TO GRAY
DATA_DIRECTORY = r'C:\troubleshooting\open_field_below\project_folder\csv\outlier_corrected_movement_location'
VIDEOS_DIRECTORY = r'C:\troubleshooting\open_field_below\project_folder\videos'
SAVE_DIRECTORY = r"C:\troubleshooting\open_field_below\project_folder\videos\rotated"
[12]:
# ... AGAIN WE CREATE A NEW INSTANCE BASED ON THE UPDATED INFORMATION ABOVE, AND RUN IT.
aligner = EgocentricalAligner(anchor_1=ANCHOR_POINT_1,
anchor_2=ANCHOR_POINT_2,
data_dir=DATA_DIRECTORY,
videos_dir=VIDEOS_DIRECTORY,
save_dir=SAVE_DIRECTORY,
direction=DIRECTION,
gpu=GPU,
anchor_location=ANCHOR_LOCATION,
fill_clr=FILL_COLOR,
verbose=VERBOSE)
aligner.run()
SIMBA COMPLETE: Video concatenated (elapsed time: 6.5711s) complete
SIMBA COMPLETE: Egocentric rotation video C:\troubleshooting\open_field_below\project_folder\videos\rotated\raw_clip1.mp4 complete (elapsed time: 44.5313s) complete
SIMBA COMPLETE: Egocentrically aligned data for 1 files saved in C:\troubleshooting\open_field_below\project_folder\videos\rotated (elapsed time: 44.5873s) complete
[21]:
#EXAMPLE VIDEO EXPECTED RESULTS SNIPPET
video_url = 'https://raw.githubusercontent.com/sgoldenlab/simba/master/docs/_static/img/egocentric_nb_3.webm'
HTML(f''' <video width="600" height="600" controls> <source src="{video_url}" type="video/webm"> </video>
''')
[21]:
[ ]: