A computer would deserve to be called intelligent if it could deceive a human into believing that it was human.
-Alan Turing
Since OpenAI introduced ChatGPT 3, a chatbot designed based on the GPT-3 language model, in November 2022, developments in AI technology have not only gained popularity among scientific and engineering communities but have also entered the daily lives of many from all walks of life. OpenAI's decision to make this language model available as a chatbot for public use had several objectives. These objectives encompassed delving deeper into the chatbot's strengths and weaknesses, enhancing its functions, and addressing security concerns. Therefore, for over a year now, OpenAI has been leveraging user experiences to refine ChatGPT, improve its accuracy and reliability, and, perhaps most importantly, detect misuse and make it safer and more useful for society. As a result, the company has been releasing updates following a feedback-driven, iterative development policy.
For instance, following a thorough evaluation and fine-tuning of ChatGPT 3's responses, OpenAI introduced ChatGPT 3.5 in March 2023.[1] Since then, this fine-tuned and polished version has been made freely available to the public. Right after this release, ChatGPT 4, developed based on the GPT-4 language model and boasting approximately 580 times more parameters than its predecessor, became accessible for trial use.[2] Currently, the free version of ChatGPT only grants access to ChatGPT 3.5, while subscribers to ChatGPT Plus, priced at $20 per month, gain access to ChatGPT 4 and thus enjoy a more enriched artificial intelligence experience. ChatGPT Plus offers significant benefits, such as the ability to integrate files in various formats into the chatbot and analyze the data contained in those files using the Advanced Data Analysis Mode (formerly known as Code Interpreter Mode). In addition, ChatGPT Plus users can seamlessly utilize a wide range of third-party applications through an extensive selection of hundreds of plugins. (Figures 1 and 2)
Figure 1: ChatGPT4 Interface Figure 2: Plugin Store available to ChatGPT Plus users
With the widespread use of advanced language models, the increased accessibility of information and the ease of transforming it into an essay format have become significant concerns, particularly in the academic world. Educators from a variety of professions and disciplines are expressing concerns that chatbots like ChatGPT, Google Bard, and Microsoft Bing may reduce students' motivation and hinder their ability to generate original text and engage in critical thinking. These debates surrounding AI are being amplified by the continued innovations brought forth by leading AI developers such as OpenAI. For instance, this July, they released the Code Interpreter Mode, equipped with the capabilities of the Python programming language and its more than 300 libraries. This update transformed ChatGPT 4 from a chatbot into an AI agent capable of generating code in response to specific prompts, executing that code within its interface, and sharing the results in textual or visual formats, depending on the user's preference and requirements. What is more, in the last days of August, OpenAI updated the Code Interpreter Mode with the addition of additional Python libraries and packages and renamed it Advanced Data Analysis Mode to highlight its enhanced abilities for data analysis.
Python experts will undoubtedly unlock the advanced mode's full potential. Nevertheless, even individuals lacking prior programming experience can utilize it as a valuable assistant for code writing and data analysis. ChatGPT's unique ability lies not just in generating code based on user inputs and running it in a safe environment but also in its proficiency to test the code internally, rectify errors, and ensure smooth execution and optimal performance. This distinctive capability makes ChatGPT a valuable tool, broadening its accessibility to individuals without any prior coding experience. So, how does the Advanced Data Analysis Mode enhance the research process for those dealing with extensive datasets?"
Literature Review and Brainstorming on Research Questions
ChatGPT provides valuable support for researchers at different stages of their project, ranging from the initial step of conducting literature reviews and crafting research questions to analyzing textual or numerical data, creating visualizations, and the final step of writing the final report. However, when utilizing the free version of ChatGPT to search for articles written on a specific topic, there's always a risk of receiving inaccurate or even made-up results. This obstacle can be resolved by upgrading to the paid version, which grants access to the plugins designed for literature reviews. In particular, Scholar AI, Litmaps, and Scholarly are useful plugins for discovering studies related to a research topic and downloading them if they're open source. However, given that these tools are predominantly web-based, it does not make sense to subscribe to ChatGPT Plus just for literature research.[3]
Researchers can use ChatGPT 3.5, which is the fastest version of released GPT chatbots, to brainstorm about their research questions. The language model can provide researchers with topic-specific suggestions, encourage critical thinking, and inspire them with new perspectives. For example, you can ask which analyses would be appropriate for a particular study or ask for ideas for the title of your article. It is important to ask the right questions in a dialogue with the chatbot and to critically evaluate its answers. The researcher's knowledge of the research subject and how they formulate their questions for ChatGPT directly influence the quality of the results. For example, relying on the chatbot's insights about less familiar topics without subjecting its responses to critical examination can misguide the researcher. Furthermore, using the texts produced by any AI agent without proper citation is considered plagiarism, leading to academic integrity issues.[4] The crucial aspect to consider at this point is ChatGPT's ability to spark creative thinking. While the language model can introduce new perspectives, it ultimately falls upon researchers to develop these novel ideas.
Quantitative Analysis and Visualization
Before starting to analyze numerical data, it’d be helpful for a researcher to clearly define their research methodology and approach. Subsequently, they can upload their data files into the Advanced Data Analysis Mode, prompting ChatGPT to load and examine the provided data. Upon accessing the file, the chatbot scans roughly 500 characters to get a feel for the content and subsequently asks for the desired method of analysis. At this point, users have the flexibility to either seek guidance from the chatbot regarding statistical methods (such as regression, classification, clustering, time series, etc.) or specify their method directly. It's worth noting that the Advanced Data Analysis Mode comes equipped with a variety of visualization tools, allowing the representation of the analysis results as graphs or maps. (Figure 3)
Figure 3: In response to a query about the types of graphics it can produce, ChatGPT 4 generated this image.
Consider that you are working with a rich dataset that includes a wide array of details about airports from all over the world, including the airport's name, exact location, type, size, and altitude.[5] You might be curious about how an airport's altitude relates to its size classification (large, medium, or small). To address this, ChatGPT generates and processes a code to calculate the correlation coefficient, shedding light on the nature of this relationship. After interpreting the obtained value, you can prompt the chatbot to provide a visual depiction of the results. (Figure 4) In the next query, you can ask ChatGPT to compute the density of airports worldwide by calculating the number of airports per square kilometer in each country. Later, you can write a prompt to see the top ten countries with the highest airport density on a chart that aligns with your specific choice or one the chatbot deems suitable. (Figure 5)
Figure 4: The diagram produced by ChatGPT. It depicts the association between the altitude of airports and their respective size categories.
Figure 5: The bar diagram produced by ChatGPT. It illustrates the top ten countries based on the number of airports per square kilometer.
Given that our dataset includes the geographical coordinates of each airport, translating the spatial data into a map becomes a swift process. By simply prompting ChatGPT to map the "large airports,” a distinct category in the dataset, ChatGPT can generate an interactive map by executing the relevant code in a few minutes. (Figure 6) For a successful mapping process, the dataset must contain complete and accurate geographic coordinates for each location. If not, ChatGPT 4 fills in the gaps using its own data, which was last updated in September 2021. However, one should approach such automated assignments with caution, as they might not always be precise, raising concerns about the reliability of the generated maps.
Figure 6: The interactive map produced by ChatGPT. Geographical Distribution of Large Airports.
ChatGPT 4 enables its users to reach Python’s most widely used libraries for geographic data visualization, Folium and Geopandas, and their associated packages, allowing map customization. This enables users to design the maps according to their specific needs and preferences by changing the location markers, using a different map template, or adding legend tables. Another important aspect to bear in mind is that, although ChatGPT 4 can display simple visualizations like graphs within its interface, it cannot display the interactive maps it creates. Therefore, it generates download links that allow users to save these maps as HTML files, facilitating easy sharing and access.
Görsel 7: The interactive map produced by ChatGPT. Geographical Distribution of Large Airports. The customized version.
Qualitative Analysis and Visualization The present version of the Advanced Data Analysis Mode does not yield satisfactory results for qualitative analysis due to its lack of essential libraries and packages for text analysis. Therefore, researchers might find it more beneficial to turn to other tools tailored for qualitative analysis. This includes relevant software solutions or programming languages like R and Python, equipped with relevant libraries like NLP (Natural Language Processing) and NER (Named Entity Recognition). Moreover, web-based applications like Voyant Tools, which do not necessitate programming expertise, can also be an alternative choice for performing such tasks. [6] This does not mean that the Advanced Data Analysis Mode is unable to conduct text analysis but that it relies on simpler methods that can lead to errors. For example, it lacks the relevant modules of Python’s NLTK library for tokenization, a foundational step in frequency analysis. Instead, it resorts to the split() method, which divides text based on spaces, which is a rudimentary form of tokenization. This method can be used for simple analysis, but it can lead to errors in more complex text analysis tasks because it only tokenizes over a single separator (space, period, comma, etc.) and does not recognize the nuances of the word structures of languages. To overcome this shortcoming of the Advanced Data Analysis Mode, the relevant libraries must be uploaded to the system. Installing the appropriate libraries and modules for each analysis is a time-consuming process and does not always deliver the desired performance. Briefly, the Advanced Data Analysis Mode, with its current capabilities, can perform frequency analysis, text coding, thematic analysis, and sentiment analysis at a basic level. It's crucial to emphasize, though, that the accuracy of its results is closely tied to the text being in English and the prompts being written in the correct way and order. To test the abilities of the Advanced Data Analysis Mode in qualitative analysis, I first asked ChatGPT to produce fictional interviews with five university students about their experiences during the early stages of the COVID-19 pandemic. Once the content was generated, I prompted ChatGPT 4 to conduct a frequency analysis and visualize the preliminary results. In the first phase of the analysis, it generated the following word cloud.
Figure 8: The Word Cloud generated with the Advanced Data Analysis Mode.
In the next query, I requested ChatGPT to encode the text and conduct a thematic analysis. Based on the results of the frequency analysis, it offered five themes. (Figure 9) Once it has identified the themes based on the most frequently used words, one can also ask for a list of themes, associated codes, and representative sentences for each code.
ChatGPT also achieves a certain degree of success in performing sentiment analysis on the interview text. The process begins with tokenization via the split() method to break down the text into individual words. In the next step it uses Python's TextBlob library to calculate the emotional valence of interviews. The emotional valence scores fall within a range from -1 to 1, with a score closer to -1 indicating a predominantly negative affective response, a score closer to 1 suggesting a positive sentiment, and a score closer to 0 indicating neutrality or ambivalence in the emotional response. (Figure 10) As a final step of the analysis, ChatGPT can generate a report that introduces the scope, methodology, and purpose of the analysis. This report will also include a detailed description and interpretation of the analysis's results.
Figure 9: The bar chart showing the main themes and the number of associated sentences.
Figure 10: The bar chart illustrating the results of sentiment analysis
Sentiment Polarity: 0.15, Sentiment Subjectivity: 0.46
Limitations of ChatGPT 4’s Advanced Data Analysis Mode
Based on my personal experience, I can outline the limitations and drawbacks of ChatGPT 4 in the realm of data analysis as follows:
● Given that ChatGPT is not designed for tasks such as geographic data analysis, text analysis, or statistical analysis, its analysis and visualization capabilities are dependent on the available Python libraries and the active plugins.
● ChatGPT Plus users are allowed to send fifty prompts in three hours. This restriction can be limiting, especially in projects that require detailed analysis or long interactive sessions.
● Since the GPT language models are predominantly trained on English texts, conducting analyses in different languages tends to produce less consistent and detailed results.
● Since ChatGPT does not have direct internet access, it does not have the possibility to instantly download the libraries required during analysis. Therefore, the analysis may get stuck at a certain point.
● The Advanced Data Analysis Mode can perform basic analyses on the uploaded data sets, but to fully harness its capabilities, it should be used together with Python or any other programming language. This collaborative approach not only provides access to relevant libraries but also enhances the code-writing process with ChatGPT's assistance, ensuring a more successful analysis process.
As a concluding remark, ChatGPT's data analysis capabilities are currently bound by the Python libraries it relies on. This means that it can only perform basic analysis on uploaded datasets. While this may seem like a limitation, it's a promising sign of ChatGPT's potential to expand its data processing capabilities with future updates. In the near future, ChatGPT could become a versatile tool for performing in-depth data analysis, opening up exciting possibilities for social scientists, researchers, analysts, and data enthusiasts alike. This possibility makes it a potential game-changer in data science.
Note: Special thanks to Arif Yasin Kavdır for sharing his valuable insights regarding data analysis with ChatGPT.
Emine Öztaner
Koç University, ARHA, Ph.D. Cand.
Suggested Readings:
https://www.unite.ai/exploring-openais-chatgpt-code-interpreter-a-deep-dive-into-its-capabilities/
https://dev.to/ppiova/advanced-data-analysis-in-chatgpt-replaces-code-interpreter-3lil
https://www.ipl.org/div/chatgpt/data-science-and-chatbots-chatgpts-advanced-data-analysis-plugin
https://medium.com/@Keenious/chat-gpt-in-the-academic-world-c21ecc6888fe
[1] https://help.openai.com/en/articles/6825453-chatgpt-release-notes [2] https://openai.com/gpt-4 [3] Some of the web-based AI tools developed for conducting literature reviews are www.elicit.org, www.concensus.app, and www.litmaps.com. [4] https://www.nature.com/articles/d41586-023-00107-z [5] This analysis was conducted using open-source datasets downloaded from https://ourairports.com/airports/ (airports.csv) [6] https://voyant-tools.org/