NJIT Students Come in 1st With Election-Inspired Hackathon Project
It was a timely inspiration that garnered two Honors College students a first-place hackathon win. With the upcoming election season expected to be fraught with doctored videos of famous figures aiming to influence voters — known as deepfakes — Aarati Srikumar and George Aboudiwan, along with two teammates from other schools, developed a web-based solution to detect such deception and provide “a level of protection and reality for users.” Dubbed DeFake, the web browser extension and application employs machine learning to analyze and determine if online videos are fake or have been manipulated using artificial intelligence.
Srikumar and Aboudiwan, both computer science juniors at the university’s Ying Wu College of Computing, presented DeFake at PennApps, a weekend-long college hackathon run by students, and hosted by and held at the University of Pennsylvania. Projects were rated on originality, usefulness, polish and technical difficulty by a panel of judges from the tech industry.
The DeFake team beat some 250 tech projects to emerge victorious and received prizes that included tech items, software subscriptions and gift cards. Aboudiwan and Srikumar participated once before in PennApps, so they knew the competition would be stiff and were duly surprised about their ranking.
“My honest reaction was disbelief. I was so ecstatic and it was surreal,” recalled Srikumar about hearing the announcement they had won. Aboudiwan shares her feeling.
Here is how DeFake works. When a user views a video, DeFake’s Chrome extension first scans it to identify the subject. It then applies its machine-learning model to uncover deepfakes targeting the subject, and displays its findings — whether the video is real or phony— in real time to the user.
“Our web app offers the same service,” noted the team in their submission. “Users are able to upload a link to any video they are concerned may be tampered with, and receive a result regarding the authenticity of the video.”
In training the machine-learning model for the web extension and app, the DeFake team began by breaking down sample videos into one-second frames, mining concavity values of facial features key in identifying speech patterns and behaviors of an individual. These values were then run through principal component analysis, a technique that searches for variation and reveals patterns in data — in this case, differences in movements and mannerisms.
“We looked at how much the top and bottom eyelids were bent … and how much the lips were bending, too. … These minute dilations of the eyes and lips are what we extracted from the videos as concavity values, which we fed into the model,” Aboudiwan explained.
Training also incorporated other videos of the famous figures, and involved a linear kernel support vector machine (a state-of-the-art learning algorithm). The model’s algorithm was created in Python.
“We understood the importance of the misinformation issue, and how it is possible to utilize a conjunction of a machine-learning model with an effective user interface to appeal to and attract internet users of all ages and demographics,” the team remarked.
Up next for the DeFake team is to train the model with more celebrities and adapt the Chrome extension to identify whether or not the videos are authentic as the user scrolls. Srikumar and Aboudiwan and their fellow developers hope to be able to access Facebook’s deepfake datasets when they are released this December as part of the company’s Deepfake Detection Challenge, which is open to the public.
In the meantime, Aboudiwan and Srikumar are relishing their hackathon success and lessons learned from the event.
“One of the best takeaways I had was working with my group. … We became really close to each other during the weekend and really worked as a unit to complete our project. Hackathons give students an opportunity to expand their skills, try new things and meet great people,” offered Aboudiwan.
“I took away that you should always ask questions and be curious. We had spoken to an [Amazon Web Services] representative who had come about the problem we were trying to solve, and he told us there was no good way to do it, and that it is a problem even big companies like Facebook are trying to solve,” Srikumar said “Yet, we were still able to make a breakthrough in the problem of fake news, and it was a gratifying feeling to be able to help solve such real issues.”