Vishaal811 - Machine Learning Pathway

Final Assessment

Name: Vishaal Ranjan
Team: ML Team 8 (Bertinator)

Overview of Things Learned:

Technical Area:

  • I learned how to create web crawlers from scratch using Scrapy and BeautifulSoup

  • I also learnt about the API calls that are made in the websites that use infinite scrolling and how to scrape the data for such pages

  • Performed data cleaning and analysis on the scraped data from the forums

  • Concatenated the data from all the forums and used regular expressions to remove HTML tags and some unnecessary characters

  • Used the BERT model in order to predict the forum for any given post

  • Achieved an accuracy of 97% while through the BERT model while using data from 5 different forums

  • Learned how to deploy the model on AWS through the use of Lambda and Serverless

Tools Used: Python, Pandas, Scrapy, BeautifulSoup, BERT, AWS, Git, Colab, Visual Studio Code

Soft Skills: Learned how to Collaborate with teammates from different time zones, solved technical issues by interacting with the team and task leads and coming up with a solution, communicated my learnings effectively to the whole team

Achievement Highlights

  1. Successfully scraped the airline forum through Scrapy (which included 36000+ entries and 13 features)
  2. Successfully joined all the dataframes together after cleaning the data and handling the null values
  3. Researched and got a better understanding of the BERT model through which I was able to create the custom model for our usecase
  4. Made use of Slack and Github in order to collaborate and share code
  5. Learned about model deployment on AWS through the use of its services like Lambda and Serverless

List of Meetings attended

  • Attended all the team meetings that have been organized so far (2 meetings/week)
  • Git Webinar
  • Scraping with BeautifulSoup and Scrapy Webinar
  • Recommender Systems Webinar
  • Weekly meetings with Technical Lead Maleeha