Method


  • In - depth Interviews

  • Online surveys

Team


  • 2 UX Researchers


My role


  • UX Researcher


Timeline


  • Overall: 16+ weeks

  • Preliminary Research: 4+ weeks

  • Interviews an surveys : 6 + weeks

  • Analysis : 7 + weeks


What is this project about?

The Research process….

Central Research Question

-How do programming students make use of ChatGPT to understand concepts covered in class or to aid them with their coursework?

-What are user perceptions and motives when using such tools?

Background and Motivation

With generative machine learning models becoming more advanced each year, interactive text-based models like ChatGPT are being actively used by people to generate ideas and solve problems.

  • Though this might seem like a detrimental problem to student learning and academic integrity, we wanted to find out if such text-based models can actually supplement learning and lead to a more rounded learning experience for students.

  • In today’s times, findings from this research problem can potentially benefit professors and students in assessing the impact of AI-based learning on learning styles and knowledge decay levels. This will allow them to structure their lectures based on optimal learning and give us an idea of potential pros and cons of such models.

  • Narrowing down on our research problem, we conducted qualitative research on how programming students use Chat GPT to aid them with academics by taking their first-hand accounts.

  • We wanted to know more about how the use of ChatGPT affects student learning in modern times and determine if such tools are beneficial or detrimental.

Additional Research Questions

- What are user expectations when turning to generative machine learning models?

-What perceived advantages do users believe they have when using such models over other academic resources?

- What are some drawbacks of AI-based learning tools that still make them imperfect?

Current Research and knowledge

Research Methods

Since there is little to no research published on student use of ChatGPT, we looked at adjacent research domains to aid us in gathering insights on the current gap in research.

  • Current quantitative research suggests that chatbots supplement student learning resulting in no difference in student scores between groups that study using ChatGPT and face-to-face delivery.

  • However, the study also suggests that a perceived usefulness contributes heavily to student performance

  • Another study by GitHub reflects on potential benefits programmers experienced in one year of the launch of GitHub Copilot.

  • Researchers collected data on first-hand experiences to chart out how GitHub Copilot had impacted their approach to problem solving.

  • However, this research was focused on the end result rather than programmer learning.

Analysis


We used two methods to gather data - coding interviews and analyzing diary studies.

To code the interviews, we used a bottom-up approach to analyze the data, which means that we allowed themes to emerge from the data itself. This was done to prevent any internal biases from tampering with our results.

We analyzed all entries thematically for diary studies, looking for patterns and trends in the students' experiences with ChatGPT. We also compared the themes that emerged from the diary study to those that emerged from the interviews, to ensure consistency and validity of our findings.

  • As we worked in pairs, we collaborated on the analysis by discussing and comparing our findings, resolving any disagreements, and ensuring that our interpretations were consistent with the data. We also engaged in reflexivity throughout the research process, which involved acknowledging and reflecting on our own biases, assumptions, and expectations. We kept a reflective journal to document our own thoughts and experiences during the research process, sought feedback from each other and other researchers, and were open to unexpected findings that challenged our initial assumptions.

  • Overall, our data analysis process was rigorous and unbiased, which allowed us to generate meaningful insights into how programming students use ChatGPT for learning.



Key Finding 1

ChatGPT is being used by programming students as a tool for learning new programming languages. Participants reported that they often use ChatGPT to clarify concepts and syntax in new programming languages that they are learning, which helps them to build a solid foundation of knowledge in the language.

"I mostly use Chat GPT to help me understand programming concepts better and to generate examples when I'm learning new programming languages. It helps me visualize how the code works, and it's been pretty useful for my engineering projects too."

-Student A.

Key Finding 2


  • Our study revealed that participants are actively using Chat GPT to generate ideas and sample code for their programming projects. By posing specific prompts or questions to Chat GPT, students are able to obtain tailored responses that assist them in brainstorming and formulating their own solutions. 

  • This not only aids in the development of their programming skills but also fosters creativity and independent problem-solving abilities. By leveraging ChatGPT's capabilities, students can explore various approaches and enhance their understanding of programming concepts, ultimately leading to more innovative and well-rounded project outcomes.

Key Finding 3

While ChatGPT is a valuable resource for learning, participants also emphasized the importance of verifying information from other resources. They reported that they often use ChatGPT in combination with other resources, such as textbooks, online forums, and video tutorials, to gain a more comprehensive understanding of the subject matter.

"I use ChatGPT for brainstorming ideas or getting unstuck when I'm working on a project. It's also been useful for learning new concepts and understanding the pros and cons of different approaches."

-Student B.

"I use ChatGPT for brainstorming ideas or getting unstuck when I'm working on a project. It's also been useful for learning new concepts and understanding the pros and cons of different approaches."

-Student B.

Methods

First-hand User Interviews: Asked participants about their experiences using AI-based learning and if they think they achieved their goals effectively? Asked them why they started using generative learning models over traditional academic resources.

Diary Entries: Asked participants to maintain a diary for a short period of time and answering a series of questions every time they make use of AI-based learning to complement their learning ability.

Participants

Undergraduate or graduate college students taking programming classes who make use of AI-based learning models on a regular basis to seek help with coursework.

Background and Reasearch

What is already known

Research Methods

Key findings

Conclusion

Next Steps

The research aimed to evaluate the effects of AI-based learning on learning styles and knowledge decay levels in programming students.

  • The findings of this research have the potential to provide valuable insights for professors and students.

  • By assessing the impact of AI-based learning, professors can better understand how this teaching method affects different learning styles.

  • Professors can use the findings to tailor their teaching strategies to accommodate various learning styles, improving the overall learning experience for students.

  • Students can also benefit from the research findings by gaining insights into how AI-based learning can impact their learning styles.

  • This knowledge can help students adapt their study habits and approach to learning, optimizing their educational outcomes.

  • Understanding the impact of AI-based learning on knowledge decay levels is crucial for both professors and students.

  • By evaluating knowledge decay levels, professors can determine the effectiveness of AI-based learning in retaining information over time.

First hand interviews

Conducted five in-depth interviews with our target audience.

  • Each interview lasted between 30-40 minutes.

  • The interviews were conducted remotely, over Zoom.

  • The interviews were recorded with the participants' permission, and then transcribed and analyzed for common themes and patterns.

Diary Study

The 5 interviewees recruited previously were asked to maintain a diary for a week and make an entry each time they make use of ChatGPT for academic purposes.

  • Participants were briefed with the diary entry procedure and given a rough format they could use to enter data during the given time period.

  • We gathered information in a simultaneous manner i.e. users were asked to maintain a diary entry for a week between which they scheduled an interview at a time they were comfortable with.

How did we recruit our participants?

The 5 interviewees recruited previously were asked to maintain a diary for a week and make an entry each time they make use of ChatGPT for academic purposes.

  • Participants were briefed with the diary entry procedure and given a rough format they could use to enter data during the given time period.

  • We gathered information in a simultaneous manner i.e. users were asked to maintain a diary entry for a week between which they scheduled an interview at a time they were comfortable with.

Discussion: -

ChatGPT can be a valuable tool for programming students to supplement their learning.

- Students reported that ChatGPT helped them generate new ideas and perspectives. - However, ChatGPT should not be relied upon as the sole source of information, as it is not always accurate or reliable.

- The limitations of AI-based learning tools should be acknowledged, as they are not perfect and may have drawbacks.

Implications:

- Professors and students can consider incorporating ChatGPT and similar AI-based learning tools into their educational practices to enhance learning experiences.

- The findings can help educators structure their lectures based on optimal learning, taking into account the potential benefits and limitations of AI-based learning tools.

- Further research can explore ways to improve the accuracy and reliability of AI-based learning models like ChatGPT. - The insights gained from this research can inform discussions on academic integrity and the appropriate use of AI-based learning tools in educational settings.

- The small sample size of the study may not fully represent the experiences and perspectives of all programming students.

- The convenience sample used in the study could result in skewed representation, potentially favoring participants with more technology interest or those with positive experiences with ChatGPT.

- The diary study method employed in the research may have limitations due to inaccurate reporting or difficulties in recalling specific details, despite efforts to mitigate this through prompts and reminders.

- Further research with larger and more diverse samples is needed to validate and generalize the findings.

- The study focused specifically on programming students, and the findings may not be applicable to students in other disciplines or educational contexts.

Our study highlights several potential avenues for further research on programming students' use of ChatGPT:

  • Investigate experiences of students who haven't used ChatGPT or had negative experiences to better understand its benefits and limitations.

  • Examine the extent of reliance on Chat GPT for specific tasks, like generating sample code or concept clarification, using qualitative and quantitative methods like surveys and interviews.

  • Explore Chat GPT's impact on programming education outcomes, such as coding proficiency and student engagement, through longitudinal studies.

    These directions offer a foundation for future investigations into Chat GPT's role in programming education.

Discussions and Implications

Limitations

Next Steps

Recruited Participants

Participant Profile

College students taking programming classes who make use of ChatGPT on a regular basis as an alternate academic resource.

  • Students typically between the ages of 18-23 pursuing an undergraduate or graduate degree.

  • Students with some prior coding ability and experience learning programming languages.

To recruit participants, we sent out a recruitment screening questionnaire to our personal contacts in programming courses as well as interested classmates.

  • We then screened for participants that closely matched our target audience and then invited them to participate in a brief group call where we briefed them with the diary entry procedure.

  • Participants were then asked to schedule a remote interview at a time comfortable for them. This occured before participants started maintaining their diaries for the study.

👈🏻 Back

Thankyou :) Also, Remember to stay hydrated 🍉

Designed by Aditya :)