Which AI Model is Best for Analyzing Open Responses in Surveys?

Introduction

Have you ever wondered how companies manage the overwhelming amount of feedback they receive? In the past, this required extensive manual effort, but today, AI models for text analysis offer a more efficient solution. These sophisticated AI models quickly convert unstructured data from survey responses and social media comments into actionable insights. In this article, we explore how these technologies work, compare different classification models, and demonstrate how to effectively measure their performance.

AI-Based Analysis of Open Responses

Analyzing open responses means sorting unstructured text data into specific topics and sentiments. This helps businesses understand customer feedback and spot key trends and issues. It typically involves two main steps: first, assigning topics to each response to see what’s being discussed; second, summarizing these findings to provide a big-picture view. This structured approach not only saves time but also delivers actionable insights that can drive better decision-making and improve products and services.

Integrating advanced AI tools like ChatGPT can refine this process, as discussed in our detailed article on using ChatGPT for open-ended survey responses. This approach not only saves time but also provides actionable insights that can significantly enhance decision-making and improve services.

Text Classification Models

Text classification models come in various forms, each suited for different tasks:

• Rule-Based Models: Use predefined rules and pattern matching to classify text. They are straightforward and easy to implement but can be rigid and require manual updates.

• Machine Learning Models: Include algorithms like Naive Bayes, Support Vector Machines (SVM), Decision Trees (Random Forests), and Logistic Regression. These models learn from labeled data and can handle new, unseen text. However, they need a lot of labeled data to train effectively and can struggle with complex language.

• Transformer Models: The latest generation includes BERT, GPT, T5, and RoBERTa. These models use advanced natural language processing and are pre-trained on vast amounts of data. They can handle complex language tasks with high accuracy and adapt to various text analysis needs.

Understanding the Classification Process

The table below illustrates the process of categorizing items or responses into predefined categories or topics. This example demonstrates how AI models for text analysis and human annotators can work together to classify various inputs accurately.

How to measure the performance of AI-models?

To illustrate the performance of these models, consider two scenarios: an animal classification task and an open text topic classification task.

Animal Classification Example

In this task, models classify images as either “animal” or “not animal.” The confusion matrix helps visualize the model’s performance, showing true positives, false positives, true negatives, and false negatives.

• True Positive (TP): The model correctly identifies an image as an animal. For example, pictures of a sheepdog, a puppy, and a cat are correctly classified as “Animal.”

• False Positive (FP): The model incorrectly identifies a non-animal image as an animal. For example, pictures of a muffin and grass are incorrectly classified as “Animal.”

• False Negative (FN): The model fails to identify an animal image and classifies it as “Not Animal.” For instance, a picture of a cat is wrongly classified as “Not Animal.”

• True Negative (TN): The model correctly identifies a non-animal image as “Not Animal.” This isn’t explicitly shown in the example but would be where non-animal images like the muffin and grass are correctly classified.

When dealing with tasks that require sorting information into two categories, like yes/no or damaged/not damaged, it’s important to know how accurately AI models for text analysis perform. To measure this, we use three key metrics: Precision, Recall and F1 Score.

• Precision:

  • Measures how many of the model’s ‘yes’ decisions were correct.

  • Ensures the model isn’t falsely labeling too many items as ‘yes’ (minimizing false alarms).

• Recall:

  • Checks if the model is identifying all the ‘yes’ cases it should.

  • A high recall means the model is catching most of the real ‘yes’ situations, avoiding missed cases.

• F1 Score:

  • Combines precision and recall into one metric

  • Helps balance the need to catch as many positives as possible with the need to avoid too many mistakes.

  • Useful for comparing models to see which one performs best overall, providing a quick snapshot of both accuracy and reliability.

The image below visually explains the concepts of precision, recall, and the F1 score using specific numbers from a previous example on “animal” vs. “non-animal” classification. It shows the calculations for precision, recall, and the F1 score based on real data, illustrating how these metrics are derived and why each is important for evaluating the performance of a model.

The F1 score quickly tells us how well a model balances detecting true positives while minimizing errors. By checking this score, you can efficiently determine which model is most reliable for your specific needs, reducing the risk of incorrect classifications.

Comparison of F1 Scores of Different Scenarios

Using the F1 score to evaluate different models gives us a clear view of their accuracy and reliability. For example, in one study, various models were tested to see how well they could classify text. The results, shown by their F1 scores, highlighted the unique strengths and weaknesses of each model.

• Model 1 (Classifies All Images as Animals): This model achieves high recall by classifying every image as an animal, ensuring that all actual animals are identified. However, this comes at the cost of many false positives, as it also incorrectly classifies non-animal images as animals:

  • Precision: Approximately 67%, indicating that not all predicted animals are actual animals.

  • Recall: 100%, as the model successfully identifies all animal images.

  • F1 Score: Approximately 80%, showing a reasonable balance between precision and recall despite the high number of false positives.

• Model 2 (Classifies All Images as Not Animals): This model avoids false positives entirely by classifying every image as “not animal.” While this approach ensures no incorrect identification of non-animals as animals, it results in very poor performance metrics:

  • Precision: 0%, as the model never predicts any image as an animal.

  • Recall: 0%, since it fails to identify any actual animals.

  • F1 Score: 0%, reflecting its inability to balance precision and recall effectively.

When we look at these two models, it’s clear that Model 1 works best when it’s important to catch every animal, even if some mistakes are made. On the other hand, Model 2 is better for situations where making a mistake is more costly than missing an animal. Each model has its strengths, depending on what you need most: accuracy or caution.

Text Classification Example

Here, models classify customer feedback into predefined topics. For instance, feedback about delivery service might be categorized into topics like punctuality, service quality, and damage. The performance is measured against human-annotated benchmarks, revealing how well the AI models align with human judgment. This approach ensures that the AI’s categorization is both relevant and accurate.

The slide shows how AI models handle feedback on topics like damage, punctuality, and service. For example, feedback such as “The box was ripped open, and parts were missing” should be categorized under “Damage.”

When sorting customer feedback into categories like ‘damage’ or ‘no damage,’ it’s crucial that our text classification models get it right. Imagine a customer says, ‘Your package was late, but thankfully, it wasn’t damaged.’ Our model needs to understand that this isn’t a case of damage. On the flip side, we don’t want to miss a serious complaint like, ‘The product was crushed upon arrival. Very disappointed!’

To ensure our models are accurate and reliable, we rely on something called the F1 score. This score tells us how well the model is doing at avoiding mistakes and not overlooking real issues. It’s a bit like a report card that helps us keep the model on track, making sure it’s recognizing true problems without any mix-ups.

Accurately categorizing feedback and understanding the sentiment behind it is vital, not just for maintaining customer satisfaction but also for applying these insights in more complex analyses. For instance, in Key Driver Analysis, understanding these categories and sentiments is crucial as it helps identify what truly drives customer satisfaction. This application shows how effectively handling open text responses can provide valuable insights in advanced market research, leveraging the detailed categorization and sentiment analysis performed by AI models.

What Makes a Good F1 Score?

A good F1 score typically ranges from acceptable (above 60%) to excellent (above 80%). Achieving a high F1 score requires careful model training, fine-tuning, and validation against human-annotated data. For text classification tasks, aiming for an F1 score above 70% is considered a benchmark for high performance. This ensures that the model balances precision and recall effectively, providing reliable and accurate results.

How to Achieve an Excellent F1 Score

Achieving a high F1 score, typically above 80%, involves a blend of AI tools and human insight. Follow this structured method inspired by Survalyzer to optimize your AI model:

• Step 1: Ask AI-GPT What Topics Appear in Your Data

Use AI-GPT to initially identify potential topics from your data set. This helps set a preliminary scope for your analysis by highlighting frequent themes and categories.

• Step 2: Manually Review and Refine Categories Based on Best Practices

After AI-GPT has suggested topics, manually review these to ensure accuracy and relevance. Refine these topics using industry best practices to make the categories more specific and actionable. For instance, instead of a broad term like “Punctuality,” use specific labels like “Late Delivery” or “On Time.”

• Step 3: Conduct Multiple Trials

Run 4 to 5 trials to confirm the consistency of the refined topics. This step helps solidify your topic selection by ensuring they are relevant across different data sets.

• Step 4: Prepare Training Data

Prepare your training data by manually categorizing a significant portion to establish a high-quality initial dataset. You can also use AI-GPT to assist in preliminary categorization, followed by human review and refinement for accuracy.

Preparing accurate training data is critical for the success of AI models in text analysis. Accurate sampling ensures the model’s predictions are reliable and reflective of the broader dataset. Read more about the fundamentals of survey sampling and calculate the optimal sample size to ensure your training data supports robust AI analysis.

• Step 5: Model Training and Validation

Train your AI model using the prepared and refined data. Validate its performance continuously by testing it on a subset of your data and making necessary adjustments based on these results.

• Step 6: Deploy the Trained Model

Once the model is trained and has demonstrated accurate performance, deploy it to analyze new data. It should accurately categorize text based on the well-defined topics.

• Step 7: Monitor and Optimize

Keep an eye on the model’s performance and optimize as needed. This ongoing adjustment process helps maintain the model’s effectiveness, making it a reliable tool for decision-making and strategy development.

By following these steps, Survalyzer has achieved an F1 score of 75% with their custom-trained AI model, outperforming both human classification and general-purpose AI models like GPT 3.5. This approach ensures the model is accurate and tailored to specific business needs.

Conclusions

In conclusion, utilizing AI to classify and analyze text is becoming an essential tool for businesses, enhancing areas like customer service and marketing. By integrating technologies like machine learning and transformer models, companies not only become more efficient but also gain valuable insights into customer behavior.

Understanding these technologies and implementing them strategically can significantly improve your data handling. As these advancements subtly change our approach to business operations, remember that our team is here to help integrate these solutions into your business, ensuring they align well with your goals and deliver meaningful results.