User Survey and Data Collection for AI Code Assistants
Foundational Track - Introductory AI Explorations
This project has been selected for expansion into 2024 virtual-internships. Interested? Apply here.
Objective
The primary focus of this project is to develop students’ skills in User Survey and Data Collection, essential for modern careers in data engineering. It also introduces UX research fundamentals, emphasizing user interviewing and persona building. In today’s AI-driven landscape, the boundaries between disciplines are blurring. This project underscores the growing importance of cross-disciplinary skills, equipping the next generation of data engineers with the tools to harness UX insights for enhanced data solutions.
Students will engage with Generative AI-powered code assistants like AWS CodeWhisperer, GitHub Copilot, and Google Duet AI. Through interviews, surveys, web scraping, and trend analysis, they will gain a deep understanding of user behaviors, motivations, preferences, and challenges when interacting with these AI tools.
By leveraging AI chatbots and conversational AI platforms such as ChatGPT, Google Dialogflow, or IBM Watson Assistant, as well as data analysis tools like Tableau’s Ask Data, students will transform collected data into actionable personas and design strategies. This experience will demonstrate how AI can enhance data processing, improve pattern recognition, and facilitate the derivation of meaningful conclusions.
Through this project, students will gain hands-on experience in applying AI technologies to real-world data challenges. They will develop a comprehensive understanding of how user insights can drive innovation in data engineering and AI-powered tools.
Learning Outcomes
Upon successful completion of this project, students will:
-
Proficiency in Data Collection Techniques: Master diverse data collection methods, including user interviews, surveys, and web scraping, to gather comprehensive data on user interactions, trends, experiences, and motivations related to AI coding assistants.
-
AI-Driven Data Transformation: Gain proficiency in converting raw data into structured, actionable insights. Employ AI chatbots and conversational AI platforms to assist in identifying patterns, performing sentiment analysis, and deriving meaningful insights.
-
Data Visualization and Reporting: Acquire skills in using data visualization tools and techniques to effectively communicate insights derived from AI-assisted data analysis. Learn how to create compelling visualizations, dashboards, and reports that convey key findings and recommendations to stakeholders.
-
Understanding AI Coding Assistants: Develop a deep understanding of AI-powered coding tools, their features, user interaction patterns, and the challenges and opportunities they present for developers and data engineers.
By achieving these learning outcomes, students will possess a comprehensive skill set that enables them to collect, transform, analyze, and visualize data effectively, while also considering the user experience aspects of AI coding assistants.
Steps and Tasks
1. Familiarization with AI Coding Assistants
Embark on an in-depth exploration of AI coding assistants like AWS CodeWhisperer, GitHub Copilot, and Google Duet AI. This initial step is crucial for gaining a deep understanding of the domain you will be researching. Interact with these tools to discover their features and capabilities, and understand how they assist developers in their coding tasks. Refer to the official documentation and comprehensive user guides to build a strong foundational knowledge.
As you familiarize yourself with these AI coding assistants, also consider how AI itself can be utilized as a powerful tool in the UX research process . AI can serve as a UX assistant by automating analytical tasks, synthesizing large volumes of user data, and providing insights that drive more informed design decisions.
2. Understanding UX Research and Data Collection
Begin by exploring the fundamentals of UX research, focusing on how it integrates into broader data collection and analysis efforts in product development. Learn about user experience principles and how they help gather actionable insights. Start with the What is UX Research? from the Interaction Design Foundation.
Set clear goals for your user interviews and surveys, emphasizing the collection of data that reveals user interactions with AI-powered tools. Develop a structured interview guide that aligns with these objectives, ensuring questions are targeted to uncover deep insights into user behaviors and needs as well as trends. For guidance on creating effective interview guides, check How to Write an Interview Guide.
Alternatively, you can also consider web scraping to gather data. To learn web scraping, refer to the following project: WebScrapper DataDive
3. Collecting Data through Interviews, Surveys, or Web Scraping
Engage with users through structured interviews and widely distributed surveys to collect diverse data on user experiences with Generative AI code assistants. Use platforms like online forums to find participants. Learn more about conducting effective interviews at How to Conduct User Interviews.
If you choose to collect data through web scraping, follow the techniques learned in the web scraping project to gather relevant information from online sources , such as:
- GitHub Issues and Pull Requests for AI coding assistant repositories
- Developer forums and communities (e.g., Stack Overflow, Reddit programming subreddits)
- Product review websites (e.g., G2, Capterra)
- Social media platforms (e.g., Twitter, LinkedIn)
- Blog posts and Medium Articles discussing experiences with AI coding assistants
Remember to respect the terms of service and legal requirements when web scraping.
Note: If you find that you haven’t collected adequate data, consider joining the Code-Along Discussions. This will allow you to combine your data with that of other participants, enhancing the depth of insights you can generate.
4. AI-Assisted Data Analysis, Visualization, and Reporting
After collecting the data, employ AI-assisted tools like ChatGPT to transcribe and analyze the information. Utilize AI capabilities for pattern recognition, sentiment analysis, generating summaries, and keyword extraction. Explore how AI can help identify recurring themes, user preferences, and pain points from the collected data. This AI-assisted analysis will help streamline the process of converting raw data into structured insights that can be used to make informed decisions about product development.
To further enhance your data analysis and visualization skills, consider using emerging AI tools like Tableau’s Ask Data or dive into coding with packages like Matplotlib, Seaborn, or Plotly to create informative and engaging visualizations.
5. Creating Actionable Personas and User Journey Maps
From the analyzed data, construct detailed user personas that represent various user segments. These personas should include demographic details, user goals, preferences, and pain points. Utilize these personas to guide the design and development of solutions that meet actual user needs. Refer to User persona examples for templates and inspiration.
Illustrate the complete user interaction with the AI-powered code assistants through journey maps. Each map should capture key touchpoints, emotional experiences, and potential areas for enhancing user satisfaction. This visual tool helps in understanding and improving the overall user experience. See Journey map examples for how to create effective maps.
6. Generating Actionable Insights and Comparative Analysis
Based on the AI-assisted data analysis, derive actionable insights that can inform the development and improvement of AI coding assistants. Identify the key features users find most beneficial, the challenges they face, and the areas where these tools can be enhanced. Use data visualization techniques to present these insights in a clear and compelling manner.
Conduct a comparative analysis of the different AI coding assistants based on the user feedback and insights generated. Identify the unique strengths, weaknesses, and user preferences for each tool. This analysis will provide a comprehensive understanding of the current state of AI coding assistants and the opportunities for improvement and differentiation.
Evaluation Process
For a comprehensive understanding of the evaluation process and STEM-Away tacks, please take a moment to review the general details provided here. Familiarizing yourself with this information will ensure a smoother experience throughout the assessment.
For the first part of the evaluation (MCQ), please click on the evaluation button located at the end of the post. Applicants achieving a passing score of 8 out of 10 will be invited to the second round of evaluation.
Advancing to the Second Round:
If you possess the required expertise for an advanced conversation with the AI Evaluator, you may opt to bypass the virtual internships and directly pursue skill certifications.
Evaluation for Virtual-Internships Admissions
-
Start with a Brief Project Overview: Begin by summarizing the project objectives and the key activities you engaged in (User Survey, Data Collection, AI Analysis, Persona Building). This sets the context for the discussion.
-
Discuss Data Collection Techniques:
- Methodology Selection: Reflect on your choice of data collection methods (interviews, surveys, web scraping) and how effectively these methods aligned with the project goals.
- Execution and Adaptability: Evaluate the execution of your data collection strategies. Discuss any challenges you faced, such as finding participants, ensuring diverse data, or web scraping from various sources, and how you adapted your methods in response to these challenges.
- Quality of Data Gathered: Consider the quality of the data collected. How did you ensure the reliability and validity of the data? Discuss any improvements you implemented to enhance data quality.
-
AI-Assisted Data Analysis Proficiency:
- Tool Selection: Discuss the AI-assisted tools you employed for data analysis, such as ChatGPT, Tableau’s Ask Data, or coding packages like Matplotlib, Seaborn, or Plotly. Explain your rationale for selecting these tools and how they aligned with your project objectives.
- Analysis Techniques: Detail the analytical techniques used to process and interpret the collected data. Highlight how you utilized AI capabilities for pattern recognition, sentiment analysis, generating summaries, and keyword extraction.
- Insight Generation: Reflect on your effectiveness in identifying recurring themes, user preferences, and pain points from the collected data. Share any specific instances where AI-assisted analysis led to unexpected or valuable insights.
-
Visualization and Reporting Skills:
- Visualization Techniques: Discuss the data visualization techniques you employed to present insights in a clear and compelling manner. Explain how you used tools like Tableau’s Ask Data or coding packages to create informative and engaging visualizations.
- Reporting Effectiveness: Evaluate the effectiveness of your data reporting and presentation skills. Consider how well you communicated key findings, actionable insights, and recommendations to stakeholders.
-
Persona Creation and User Journey Mapping:
- Persona Development: Describe the process of constructing detailed user personas based on the analyzed data. Discuss how you incorporated demographic details, user goals, preferences, and pain points into the personas.
- Journey Map Creation: Explain how you developed user journey maps to illustrate the complete user interaction with AI-powered code assistants. Highlight the key touchpoints, emotional experiences, and potential areas for improvement that you identified.
-
Comparative Analysis:
- AI Code Assistant Comparison: Discuss your comparative analysis of different AI coding assistants based on user feedback and insights. Explain how you identified the unique strengths, weaknesses, and user preferences for each tool.
- Insights and Recommendations: Share the key insights and recommendations derived from your comparative analysis. Discuss how these findings contribute to a comprehensive understanding of the current state and future opportunities for AI coding assistants.
-
Learning and Growth:
- Skill Development: Reflect on how your skills in data collection, AI-assisted analysis, visualization, and UX research have improved throughout the project.
- Impact on Understanding: Discuss how this project has deepened your understanding of user behavior, AI-powered tools, and their importance in product development.
- Future Applications: Consider how you might apply these skills in future projects or career endeavors. What additional areas would you like to explore or improve upon?
-
Ask Questions: Show curiosity by asking the AI mentor questions. For example:
- “How do you see the role of AI-powered code assistants evolving in the next few years? What are the biggest opportunities and challenges in this field?”
- “What are some of the key metrics used to evaluate the effectiveness of AI code assistants like GitHub Copilot or AWS CodeWhisperer in real-world applications?”
- “How do you balance the need for user-friendly interfaces with the complexity of AI algorithms in developing these tools?”
Evaluations for Skill Certifications on the Talent Discovery Platform
-
Expertise in Data Collection Techniques:
- Methodology Selection: Discuss the methodologies you chose for data collection (interviews, surveys, web scraping) and their alignment with project goals. Explain why these methods were suitable for understanding user interactions with AI-powered tools.
- Execution and Adaptability: Evaluate the execution of your data collection strategies. Share any challenges you faced and how you adapted your approach to overcome them, such as refining survey questions, employing new tools for better data accuracy, or handling complexities in web scraping.
- Quality of Data Gathered: Assess the quality and comprehensiveness of the data collected. Discuss measures taken to ensure data reliability and validity, such as piloting your surveys, using diverse sampling techniques, or implementing data cleaning processes.
-
Proficiency in AI-Assisted Data Analysis:
- Tool Selection and Integration: Demonstrate your expertise in selecting and integrating AI-assisted tools like ChatGPT, Tableau’s Ask Data, or coding packages for data analysis. Discuss how you leveraged these tools to streamline the process of converting raw data into structured insights.
- Analysis Techniques: Showcase your proficiency in applying advanced analytical techniques using AI-assisted tools. Highlight specific examples where you utilized capabilities like pattern recognition, sentiment analysis, summarization, and keyword extraction to derive meaningful insights.
- Insight Generation and Interpretation: Demonstrate your ability to generate and interpret actionable insights from the AI-assisted data analysis. Share case studies or examples where your insights led to significant discoveries, informed design decisions, or drove product improvements.
-
Data Visualization and Reporting Excellence:
- Visualization Design: Showcase your expertise in designing effective and visually compelling data visualizations using tools like Tableau’s Ask Data or coding packages. Discuss your process for selecting appropriate visualization techniques, considering factors such as data type, audience, and key messages.
- Storytelling with Data: Demonstrate your ability to create impactful data stories through your visualizations and reports. Explain how you structured your narratives, highlighted key insights, and provided actionable recommendations based on the data.
- Stakeholder Communication: Discuss your experience in effectively communicating data insights to various stakeholders, including technical and non-technical audiences. Highlight your ability to tailor your presentations and reports to different levels of understanding and decision-making needs.
-
Persona Creation and User Journey Mapping Expertise:
- Persona Development: Showcase your proficiency in constructing comprehensive user personas based on the analyzed data. Discuss your process for synthesizing demographic details, user goals, preferences, and pain points into representative personas.
- Journey Map Creation: Demonstrate your expertise in developing user journey maps that effectively illustrate the end-to-end user interaction with AI-powered code assistants. Explain how you identified and visualized key touchpoints, emotional experiences, and opportunities for improvement.
- Insights and Recommendations: Discuss how your personas and journey maps informed actionable insights and recommendations for enhancing the user experience with AI coding assistants. Share specific examples of how your work contributed to design improvements or feature prioritization.
-
Comparative Analysis Proficiency:
- AI Code Assistant Evaluation: Demonstrate your ability to conduct a comprehensive comparative analysis of different AI coding assistants based on user feedback and insights. Discuss your methodology for evaluating and contrasting the strengths, weaknesses, and user preferences of each tool.
- Insights and Recommendations: Showcase your expertise in deriving meaningful insights and recommendations from the comparative analysis. Explain how your findings contribute to a holistic understanding of the AI coding assistant landscape and inform strategic decisions for product development and differentiation.
-
Continuous Learning and Growth:
- Skill Advancement: Reflect on how your skills in data collection, AI-assisted analysis, visualization, and UX research have advanced throughout the project. Discuss any new techniques, tools, or best practices you have learned and applied.
- Domain Expertise: Demonstrate your deep understanding of user behavior, AI-powered tools, and their significance in product development. Discuss how this project has enhanced your domain knowledge and your ability to contribute to the field.
- Future Growth: Share your plans for continuous learning and skill development in the areas of data analysis, AI, and UX research. Discuss any specific areas you have identified for further exploration or improvement based on your project experience.
-
Thought Leadership: Show your thought leadership by sharing your insights and opinions on the future of AI-powered code assistants and their impact on the software development industry. Discuss potential advancements, challenges, and ethical considerations in this field.
-
Ask Questions: Engage in a meaningful dialogue with the AI mentor by asking thought-provoking questions. For example:
- “How can AI code assistants be designed to foster collaboration and knowledge sharing among developers with different levels of expertise?”
- “What are the potential long-term impacts of AI code assistants on the skills and job roles of software developers?”
- “How can we ensure the responsible development and deployment of AI-powered tools in the coding ecosystem, considering factors such as transparency, fairness, and accountability?”
Resources and Learning Materials
-
AI as a UX Assistant - This article from the Nielsen Norman Group discusses the various roles AI can play in enhancing UX design, focusing on how AI can assist in automating tasks, providing insights, and creating more personalized user experiences.
-
Harnessing the Power of AI in UX Research and Design - This blog post explores the integration of AI in UX research and design, discussing how AI tools can help streamline processes, enhance user understanding, and drive innovation in design practices.
-
AI-Driven User Experience Design: Exploring Innovations and Challenges in Delivering Tailored User Experiences - This publication on ResearchGate examines the impact of AI on user experience design, highlighting both the innovative possibilities and the challenges faced in creating AI-driven, tailored experiences for users.
-
Planning Research with Generative AI - This NN/g article provides insights on how generative AI, including AI chatbots, can be effectively used to plan and execute successful user research, emphasizing the importance of context, prompts, and careful analysis.
-
UX Research Methods - A comprehensive guide from the Nielsen Norman Group that outlines various UX research methods, helping designers and researchers choose the most effective approach for their specific needs.
-
Secondary Research - An insightful article from dscout’s People Nerds blog that discusses the importance of secondary research in UX, providing a foundation for understanding user behaviors and needs before conducting primary research.
-
How to Design a Survey - A detailed guide from the Pew Research Center that offers best practices and methodologies for designing effective surveys that can yield reliable and meaningful data.
-
How to Conduct User Interviews - This NN/g article shares strategies and techniques for conducting effective user interviews, which are crucial for gathering deep, qualitative insights into user behaviors and preferences.
-
Affinity Mapping - This resource explains the process of affinity mapping, a technique used in UX research to organize and analyze complex data, helping teams identify patterns and insights from research findings.
This project aims to set the groundwork for students’ final internship by equipping them with a solid foundation in data collection and analysis, and their crucial role in the design and enhancement of a user-centered AI-powered code assistant.
STEM-Away® Recordings
