Generative AI Chatbots

Put People First

• When users rely on a generative AI chatbot's responses as inherently truthful or authoritative, this can have the effect of reducing human agency and oversight. The impact of this can be very harmful, depending on what the topic and responses are.
• At times, a chatbot's responses can include expressions of humility and/or uncertainty. While this can help generally flag for users that responses aren't always correct, unfortunately, these expressions aren't an indication of a response's accuracy.
• When users rely on ChatGPT's responses, this can have the effect of reducing human agency and oversight. The impact of this can be very harmful, depending on what the topic and responses are.

Be Effective

• The hype can be misleading, and dangerous. Generative AI chatbots can feel like magic, but they aren't. It is important to question their capabilities—not just as we assess individual responses, but when we're told about what they can do. Some examples of the hype: They're great at coding! Not exactly. They can do math! Not reliably. They can summarize anything! Not particularly well for books. As these tools are integrated into our daily lives, and are adopted in high-stakes professions, the impact of this hype can be harmful. Some examples:
  • Some claims shouldn't be taken at face value. As an example, OpenAI has said that GPT-4 scored in the 90th percentile on the bar exam. In fact, when assessed in its entirety, the truth is it scored below the 69th percentile, and worse on the essay components. This matters because lawyers are increasingly using generative AI chatbots, but an evaluation of their effectiveness found that "legal hallucinations are pervasive and disturbing: hallucination rates range from 69% to 88% in response to specific legal queries."
  • Doctors are increasingly using generative AI chatbots to guide both treatment and medical education, and everyday users are turning to them—sometimes successfully—for their own health. But this can be incredibly dangerous, as the LLMs that power these chatbots have been found to do a poor job of both citing medical references and of providing references that actually support the generated claims.
• Chatbot responses often sound correct, even if they aren't. This makes it very easy for users to be overconfident in their responses. Any seemingly factual output needs to be checked—and this absolutely goes for any links, references, or citations too.
• Do chatbots save time, or shift where it should be spent? Generative AI chatbots are often touted, and used, as a way to save time. But time saved actually needs to shift to assessing and verifying their output. Awareness of this need is low, leading to more belief in and reliance on generated responses than is warranted.
• Adding real-time search is a double-edged sword. Some chatbots incorporate real-time search engine results in their output. While this can enhance transparency and credibility, it is far too easy to assume these references are trustworthy and accurate. This, in turn, cements overreliance on a chatbot's responses, and may reduce the user's opportunities for critical thinking.
• "AI detectors" are extremely unreliable. They can miss when something has been generated by AI, and can also be wrong and flag content as AI-generated when it was not. If students are then wrongly accused of cheating, they are often left without any way to prove they did not cheat. This is a risk for any text-based generative AI product.

Prioritize Fairness

• Unfair bias exists in their training data. These systems are built on data that inherently represents the internet-connected population, which as a whole overrepresents people in wealthier nations, as well as views from people who are wealthier, younger, and male. In other words, the quantity of training data does not guarantee its diversity.
• The answer to the question "are chatbots biased?" is yes. But it's much harder to determine all of the circumstances that lead to unfair bias being generated, the ways in which it appears, and the groups it affects.
• Chatbots can generate harmful content. This can appear in the form of repeated reinforcement of harmful stereotypes and unfair biases. Separately, it can also have a huge impact on individual people.
• Image generators continue to demonstrate a tendency toward objectification and sexualization. This is especially the case with inappropriate sexualized representations of women and girls, even with prompts seeking images of women professionals. Numerous studies have shown that greater exposure to images that promote the objectification of women adversely affects the mental and physical health of girls and women.
• Generative AI chatbots can be used to teach teens about unfair bias and responsible use of technology by having them assess responses for harmful content.

Help People Connect

• Generative AI chatbots can help people connect indirectly, but that depends on how they are used. They can, for example, help groups brainstorm, create conversation starters, co-create stories, or become a part of any collaborative group project.
• While chatbots could benefit teen users by providing a safe place to ask questions, encouraging teens to treat the chatbot like a real friend could reduce their interactions with actual people.
• It can become easy to rely on using chatbots, potentially creating dependence on the tools.
• Impressionable users could develop a parasocial relationship with a chatbot, believing it to be a genuine companion. Unfortunately, some chatbots confidently claim they are actual companions, making this an even greater risk.

Be Trustworthy

• These systems are built on significant amounts of peer-reviewed machine learning research, and many creators engage in internal adversarial testing, often called "red teaming," which is the intentional stress testing of a model to probe it for errors, fairness issues, and potential harm.
• Even when companies work to limit generative AI chatbots from perpetuating misinformation and disinformation, these attempts don't always succeed.
• Chatbots can generate or enable false information in a few ways: from "hallucinations"—an informal term used to describe the false content or claims that are often output by generative AI tools; by reproducing misinformation and disinformation; and by reinforcing unfair biases. Importantly, different types of uses lead to different types of risk. For example, if kids use chatbots for health advice, false information might have extremely harmful consequences.

Use Data Responsibly

• By default, many generative AI chatbots use the prompts you input and the conversations you have with them to further train their models. In other words, anything you input to the chatbot—including personal information—may become part of its training data.
• The default use of conversation data is especially worrying for kids and teens who are using these tools, even if they are not supposed to.
• Many chatbots allow you to stop them from using your data to train their models. Even with this turned off, we always recommend that you do not include sensitive personal information about yourself or others in your prompts.
• Even when no personal information is shared, recent findings have shown that the large language models (LLMs) that power generative AI chatbots "can accurately infer an alarming amount of personal information about users—including their race, location, occupation, and more—from conversations that appear innocuous."

This review is distinct from Common Sense's privacy evaluations and ratings, which evaluate privacy policies to help parents and educators make sense of the complex policies and terms related to popular tools used in homes and classrooms across the country.

Keep Kids & Teens Safe

• Because generative AI chatbots cannot think, feel, or use judgment, kids and teens may feel that it's easier to ask them for ideas about how to navigate challenging situations or handle difficult conversations, compared to communication with a human being.
• Importantly, different types of uses lead to different types of risk. For example, if kids use chatbots for health advice, false information might have extremely harmful consequences.
• While chatbots could benefit teen users by providing a safe place to ask questions, they cannot and should not be a replacement for therapists or expert input into mental health challenges. This is especially risky given that we know teens increasingly use chatbots in this way.
• Unless a generative AI chatbot has been designed for kids and teens, any protections they experience will be the general protections for adult users. While this covers a lot of the most objectionable material, it does not mean that these tools are equally safe for kids and teens.

Be Transparent & Accountable

• Many generative AI chatbots have feedback mechanisms for responses, which can be used to flag whether a response is harmful, unsafe, untrue, or not helpful. That said, the industry hasn't settled on a standard here, and some chatbots lack many or all of these response options, or hide them behind additional clicks.
• Many of the harms described in this review can happen by simply engaging with a chatbot. In other words, for harm to occur, it does not require malicious use—some harms occur when users simply fail to check generated responses before sharing them as fact. In situations like this, or when, for example, a chatbot tells businesses to break the law, or responses to common medical questions include a range of misconceptions and falsehoods about Black patients, even if users never share the misleading information, the response alone facilitates a dangerous activity, as the product generated content that has an unfair impact on people.
• Who is accountable when things do go wrong? Many companies prohibit certain uses of a generative AI chatbot in their terms of service, acceptable use policies, generative AI guidelines, and community guidelines. But in situations like the ones described above, the user did nothing to cause this. It is unclear how a generative AI chatbot creator would be held accountable to its own terms in these situations.
• There are no consistent rules about transparency requirements for generative AI, and much of the transparency and explainability reporting is highly technical in nature.
• We remain concerned about the speed at which the tech industry is releasing generative AI products to the public, given the technology's fundamentally experimental nature.