User Experiences: Early Career Members’ Feedback About AI Chatbots

Using an AI Chatbot to Brainstorm

Brainstorming allows the user to get some ideas from the bot and then implement those ideas themselves. A chatbot is built on a large language model that has learned statistical relationships between text in documents. This means that with your prompt, a chatbot can provide a lot of relevant material quickly and with little effort on your part. This can take some of the hard work out of idea generation and help the user break out of a slump in ideas or writer's block.

Using an AI Chatbot to Optimize a Color Palette for Graphing

One early career member was very pleased with the ability of a chatbot to optimize a color palette for graphing. Given that these bots are text‐based, it was impressive that they were able to correctly identify the hex color shades that were in between the supplied values and provide warnings about color combinations that might be hard for humans to distinguish.

Using an AI Chatbot to Edit Text

We've all been there—it's your third read of that document, and you just don't think you can catch any more spelling or grammar errors. Do you send it to a colleague who can check it with fresh eyes? This can be a good opportunity to ask a chatbot to edit your text. However, this typically only works for short text sections, and the bot may make unwanted changes. One user asked a bot to edit only for spelling and grammar but found that it inserted some unnecessary words. Another early career member used chatbots to edit blog posts and was pleased with many of the suggestions though it did mix up some job titles for people mentioned in the text. These chatbots also seem to favor bulleted lists, so if this is not your style, you might have a good bit of checking and training to do.

Using an AI Chatbot to Draft a Document

Several members spoke about using chatbots to help draft documents. When the user doesn't provide the document, the bot will return a template that the user can fill out (for example, a resume template). If the user has some text to base the draft on, then the bot will adapt the content to the format it thinks best represents the requested document. This can work well when the goal is to start with technical text and convert it to explain science or statistical concepts in plain language. One early career member was pleased with the draft of a press release that a chatbot created using a project report. However, there was one big issue: the bot recognized that the team had made progress in economics, education, and fundraising but failed to perceive that the main project focus, soil science, was important enough to mention in the press release. Whoops!

Using an AI Chatbot to Answer Questions

The early career members we spoke with had success asking chatbots simple questions with lots of information available on the internet. Questions such as “What is soil?” generally had trustworthy answers though they often lacked the nuance academics might hope for. More complex questions yielded unreliable responses. For example, one member provided a draft manuscript that described how to map spatial units that would have similar soil health. When they asked the chatbot what a soil health sampling group was, it asserted that this was a group of people who go out and sample soils—not at all what the writer intended.

Using an AI Chatbot to Write or Troubleshoot Code for Data Analysis or Visualization

Several members used a chatbot to write or edit code for data analysis or visualization, mostly in the coding language R. This worked well for simple tasks and for users who themselves were fairly proficient at coding, but the users did report that the provided codes often didn't run. Two experienced R users said that it created more headaches than it solved for their R coding, noting that each iteration seemed to result in increasingly complex errors. Users who wish to supplement their data analysis with code from a chatbot should make sure they understand the analysis taking place so that their interpretations match the analysis conducted.

Using an AI Chatbot to Search for Databases and Repositories

One member hoped that AI could help them find relevant data sources but found that this was mostly a dead end. The chatbot sent them to Google Scholar, which might work in a few cases for which papers did a good job of documenting the dataset, but in most situations, it was unhelpful.

In summary, chatbots are becoming part of everyday life for many early career members. They can benefit users by helping with tedious or draining tasks and injecting novelty when ideas run stale. But as disclaimers note, users always should fact‐check the output of chatbots, and using them may take mental calibration because they are equally emphatic about subjects for which they have both poor and excellent “understanding.” Job seekers and employees should be cautious of relying on these tools as a substitute for skills or knowledge, and scientists and authors should make sure to understand their responsibilities for material they present and publish. With no uncertainty, no citations, and limited grasp of complex topics, it's wise to use these tools for what they are best at—helping the user generate ideas, drafting material based on provided content, and editing communications for general or interdisciplinary audiences.