Geekly Articles each Day
📜 ⬆️ ⬇️

The Big Bang Theory and Python Practice

Introduction


Recently, actively learning the programming language Python. I was especially interested in using Python in recognizing and classifying faces. In the article, I will try to apply face recognition for the Big Bang Theory series.



A little about the series
The series tells the story of two young talented physicists (Sheldon Cooper and Leonard Hofstedter), their attractive neighbor on the landing, a waitress and aspiring actress Penny, and their friends astrophysics Radzhesh Kutrappali and engineer Howard Volovice. The action of the series takes place in Pasadena, California. At the moment, shot 265 episodes (12 seasons).
IMDB rating - 8.2.

I am a non-professional programmer. Programming is my hobby. All code is terrible, but I'm learning and trying to make it better.

As a tool for recognizing and classifying faces, I used the Face Recognition Project on MXNet .
Insightface
In this repository, we provide training for deep face recognition. The training data includes the MS1M and VGG2 datasets, which were already packed in the MxNet binary format. The network backbones include ResNet, InceptionResNet_v2, DenseNet, DPN and MobileNet. The loss functions include Softmax, SphereFace, CosineFace, ArcFace and Triplet (Euclidean / Angular) Loss.

Also it needs pre-trained models of Face Recognition models and Gender-Age model (for determining gender and age).
')
The rest of the tools:


Most additional Python packages are installed with the command:

pip install _

Stage 1. Preparatory


For classification, it is necessary to create a selection of images of persons (different in perspective, season) for each character we are interested in. For this, I took literally the first video from Youtube that I got at the request “funny moments of the big bang theory”. With the help of opencv, I extracted the images from the video at 1 second intervals, manually selected them and renamed them using the script for the characters.

Sample image selection


Further, with the help of a neural network and the use of a pre-trained model, I walked through all the selected images. From each detected face, a 512-dimensional vector representation is extracted that characterizes it. This view and character name is stored in pandas.dataframe.

Code
 for filename in glob.glob('knownimage/*.jpg'): img = cv2.imread(filename) ret = detector.detect_face(img, det_type=0) if ret is not None: bbbox, points = ret #draw = img.copy() for inum, bbox in enumerate(bbbox): bbox = bbox[0:4] #cv2.rectangle(draw, (int(bbox[0]), int(bbox[1])), (int(bbox[2]), int(bbox[3])), (255, 255, 255)) pp = points[inum] pp = pp.reshape((2, 5)).T # print(bbox) # print(points) nimg = preprocess(img, bbox, pp, image_size='112,112') nimg = cv2.cvtColor(nimg, cv2.COLOR_BGR2RGB) aligned = np.transpose(nimg, (2, 0, 1)) f1 = get_feature(mmodel, aligned) df = df.append({'name': filename.split('.')[0], 'np': f1}, ignore_index=True) df.to_pickle('known.pkl')


For the 24 main and minor characters of the series, 382 files were selected.

Character list
  • Howard Joel Wolowitz ( govard , here and hereinafter in brackets indicates the name of the character in the face recognition system) - aerospace systems engineer, husband Bernadette Rostenkovski-Wolowitz, father Halley Wolowitz, best friend Rajesh Kutrappali. He graduated from the Massachusetts Institute of Technology.
  • Dr. Rajesh Ramayan Kutrappali ( rajesh ) is an astrophysicist from India, the best friend of Howard Wolowitz.
  • Priya Kutrappali ( priya ) is the younger sister of Raja, the daughter of a doctor and Mrs. Kutrappali.
  • Penny Hofstedter ( penny ) - Leonard Hofstedter’s wife, a sales representative for a pharmaceutical company.
  • Dr. Bernadette Maryann Rostenkovsky-Wolowitz ( bernaded ) is a microbiologist, Howard's wife Wolowitz, mother of Halley Wolowitz and Michael Wolowitz.
  • Dr. Beverly Hofstedter ( mama_leonarda ) - Leonard Hofstedter's mother, psychiatrist and neurobiologist
  • Leonard Leakey Hofstedter ( leonard ) - experimental physicist, employee of the California Institute of Technology, holder of a doctoral degree.
  • Dr. Leslie Winkle ( lesly ) is a University of California employee who works with Leonard Hofstedter.
  • Dr. Sheldon Lee Cooper ( sheldon ) is a theoretical physicist working at the California Institute of Technology. He has several degrees, including a doctoral degree.
  • Melissa "Missy" Cooper ( sestra_sheldona ) is Sheldon Cooper's twin sister.
  • Mary Cooper ( mama_sheldona ) is Sheldon Cooper's mother.
  • Dr. Amy Farrah Fowler ( emmy ) is a neuroscientist, Sheldon Cooper's wife.
  • Wil Wheaton ( will ) - Star Trek: New Generation actor, whom Sheldon considered his worst enemy.
  • Stuart David Blum ( stuart ) - the owner of a comic book store, after a fire became unemployed and moved to live with Howard Volovitsa.
  • Dr. V. Kutrappali ( papa_rajesh ) - the father of Rajesh Kutrappali.
  • Barry Kripke ( barry ) - a colleague of the main characters in the university.
  • Ramona Nowitzki ( navitsky ) - graduate student of the Department of Physics.
  • Dr. Geiblhauser ( geibelhauser ) - the new head of the Faculty of Physics.
  • Zack ( bivshiy_penny ) - Penny's former boyfriend.
  • Mrs. Kutrappali ( mama_rajesh ) is the mother of Rajesh Kutrappali.
  • Dr. Stephanie Barnett ( medik_leonarda ) is one of Leonard's girls.


Stage 2. Video processing


Now we begin to process the entire series. With the help of opencv, we extract 1 frame per second from each video and find faces on it. For these persons, we obtain a 512-dimensional vector representation, which characterizes it and compares it with the base that we obtained in the previous step. We also determine the estimated age and gender.

Video processing








For each found and classified person in the csv file we save the name, gender, age, frame number, series, season. Detected but not classified persons are saved in a separate directory.

Processing of all series took ~ 30 hours, 263775 frames were extracted, 353031 characters were classified. Processing of one frame takes from 0.2 to 1 second depending on the number of faces. In the folder with unclassified persons - ~ 20,000 persons (ie, 6%). This is mostly extras, rare characters or bad angle of the main characters of the series.

Examples of unclassified face images




Piece of code
 home_video = '/run/media/home/DATA/torrent/The.Big.Bang.Theory/' all_series = (len(glob.glob(home_video+'*.avi'))) now_series = 0 for filename1 in glob.glob(home_video+'*.avi'): now_series = now_series + 1 print(filename1) cam = cv2.VideoCapture(filename1) fps = cam.get(cv2.CAP_PROP_FPS) # Gets the frames per second fps = fps #print(fps) total_frames = cam.get(7) #print(total_frames) rer = round(total_frames/fps) for kk in range(0,rer): #while True: start_time = time.time() cam.set(1, round(kk*fps)) ret_val, img = cam.read() scale_percent = 170 # percent of original size width = int(img.shape[1] * scale_percent / 100) height = int(img.shape[0] * scale_percent / 100) dim = (width, height) # resize image img = cv2.resize(img, dim, interpolation=cv2.INTER_AREA) #print(ret_val) #img, draw = get_input(detector,img) ret = detector.detect_face(img, det_type = 0) if ret is not None: bbbox, points = ret draw = img.copy() for inum,bbox in enumerate(bbbox): bbox = bbox[0:4] pp = points[inum] pp = pp.reshape((2,5)).T #print(bbox) #print(points) nimg = preprocess(img, bbox, pp, image_size='112,112') nimg = cv2.cvtColor(nimg, cv2.COLOR_BGR2RGB) aligned = np.transpose(nimg, (2,0,1)) f1 = get_feature(mmodel,aligned) for index, row in df.iterrows(): np1 = row['np'] name = row['name'] dist = np.sum(np.square(np1 - f1)) if dist<1.1: #print(name, dist) dd = dist nnn = name.split('/')[1] break else: nnn = 'None' dd = dist if (nnn=='None') and (int(bbox[3])-int(bbox[1]))>112: crop_img = draw[int(bbox[1]):int(bbox[3]), int(bbox[0]):int(bbox[2])] cv2.imwrite('unknown/' + str(int(time.time())) + '.jpg', crop_img) #cv2.rectangle(draw, (int(bbox[0]), int(bbox[1])), (int(bbox[2]), int(bbox[3])), (255, 255, 255)) if (nnn != 'None'): gender, age =3, 100 gender, age = get_ga(mga_model,aligned) font = cv2.FONT_HERSHEY_SIMPLEX bottomLeftCornerOfText = (10, 20) fontScale = 1 fontColor = (255, 255, 255) lineType = 2 cv2.putText(draw, nnn + ' ' + str(round(dd, 2)), (int(bbox[0]), int(bbox[1])), font, fontScale, fontColor, lineType) # with open('test_10_1.txt', "a") as myfile: # myfile.write(name + ';' + str(kk) + ';' + str(age)+';'+str(gender)+';'+ str(dist) + ';' + filename1 + '\n') else: gender, age = 3, 100 if cv2.waitKey(1) == 27: #df.to_pickle('test.pkl') print('exit') cv2.destroyAllWindows() #break # esc to quit quit() cv2.imshow("detection result", draw) tt =round(time.time() - start_time,2) print(' '+ str(kk) + '/'+str(rer) + ' > '+str(now_series)+'/'+str(all_series)+'; ' + '  : '+str(tt)+'; ~~'+str(round((rer*tt)*(all_series-now_series)/60))+'', end= "\r")


Stage 3. A little analysis


In the beginning I wanted to use Pandas to analyze the data obtained, but, unfortunately, I did not master it to the necessary extent. On the Internet, the Falcon SQL Client program was found, into which all the data was downloaded. Also note that the 12 season is not removed until the end.

Falcon SQL Client
Falcon is a free, open-source SQL editor with inline data visualization. It currently supports connecting to RedShift, MySQL, PostgreSQL, IBM DB2, Impala, MS SQL, Oracle, SQLite and more.


The figure below shows a graph of the number of frames with the main and secondary characters in the seasons. Here it is clearly seen that the share of Sheldon and Leonard is decreasing, but the role of Bernadette and Amy is growing.



Here, the main characters of the series are taken out separately.



The main characters of the series as a percentage. The clear leader is Sheldon Cooper.



I suggested that the appearance of a “pair” in the series should correlate with the joint appearance in the frame, i.e. the closer the relationship, the more often they appear together in the same frame. A schedule was built for the “couples”: Leonard - Penny, Leonard - Sheldon, Sheldon - Amy, Howard - Rajesh, Howard - Bernadette.



The graph clearly shows the dependencies. Let's consider separately the "couples" Leonard - Penny and Leonard - Sheldon.



We see that as Leonard and Penny evolve, Sheldon more and more remains in the “span”. It is also known that Penny and Leonard begin to meet in season 3, which can also be seen on the chart.

Consider separately Sheldon-Amy and Sheldon-Leonard.



The peak of the development of Sheldon-Amy relations falls on season 10, which we see on the graph.
The following pair of "pairs", which we consider separately - Howard-Bernadette and Howard-Rajesh.



Judging by this schedule, “strong male friendship” no one will destroy and Rajesh continues his communication with Howard. We also note that the peak of the Howard-Bernadette relationship falls on season 5 (this season they have preparation and the wedding itself).

Now let's check how the pre-trained model for determining age and sex works. Let's start with the age: I took for each main character the average age in the episode and put it in a separate table (the actual age of the hero at the time of the shooting of the season is shown in brackets).



It can be seen that the make-up, the right light and the post-processing does its job and the neural network does not do well with the determination of age. However, she discovers an increase in age with each subsequent season.

Let's look at the definition of gender for the main characters (closer to 1 - male, to 0 - female).



Here the neural network does much better and shows results close to real ones. Only with Amy, the result is a bit suspicious.

It seemed to me interesting to use a neural network to detect and classify individuals. It can be seen that it does not provide 100% accuracy, but this should not be. In the future, I think to simplify the stage of creating the initial sample, using the search by image on the Internet and using this project in video surveillance.
Thanks for attention.

List of materials:


  1. MTCNN face detection implementation for TensorFlow, as a PIP package.
  2. Face Recognition Project on MXNet
  3. The big bang theory
  4. List of characters of the series
  5. Falcon SQL Client
  6. Summary CSV file

Source: https://habr.com/ru/post/435888/

More articles:


All Articles