Building Visual Questioning Answering System Using Hugging Face Open-Source Models

A Step by Step Guide to build Visual Questioning Answering System

Visual Question Answering (VQA) is a complex task that combines computer vision and natural language processing to enable systems to answer questions about images. 

In this technical blog, we explore the creation of a VQA system using Hugging Face’s open-source models. The article begins with an introduction to multimodal models and the VQA task, providing foundational knowledge for understanding how these systems operate. 

We then guide you through setting up the working environment and loading the necessary models and processors. By preparing both image and text inputs, we illustrate how to perform visual question answering. 

This step-by-step tutorial demonstrates how to leverage Hugging Face’s powerful tools to build sophisticated VQA systems, enhancing readers’ understanding of multimodal AI applications.

Table of Contents:

1. Introduction to Multimodal Models

2. Introduction to Visual Questioning Answering Task

3. Setting Up Working Environment

4. Loading the Model and Processor

5. Preparing the Image and Text

6. Performing Visual Questioning-Answering

---

---

1. Introduction to Multimodal Models

When a task requires a model to take more than one type of data, such as an image and a sentence, we call it multimodal. Multimodal models are designed to handle and integrate different forms of input, like text, images, audio, and even video, to perform a variety of tasks.

These models are increasingly important in applications that require a deep understanding of complex data, such as image captioning, visual question answering (VQA), and multimodal content creation.

One prominent example of a multimodal model is ChatGPT with GPT-4. This model allows users to send text, images, and even audio, making it a versatile tool for a wide range of applications.

GPT-4 can understand and generate human-like text, and when enhanced with multimodal capabilities, it can also interpret images and audio, offering responses that are contextually relevant across different types of data.

Multimodal models have numerous applications across various fields:

1. Image Captioning: Generating descriptive captions for images by understanding the content within them.

2. Visual Question Answering (VQA): Answering questions about the contents of an image by combining natural language processing with computer vision.

3. Text-to-Image Generation: Creating images based on textual descriptions, useful in creative industries and design.

4. Speech Recognition and Synthesis: Converting speech to text and vice versa, enhancing communication tools and accessibility.

5. Augmented Reality (AR) and Virtual Reality (VR): Integrating multiple data types to create immersive and interactive experiences.

In this article, we will explore one of these tasks which is image-text retrieval or matching. In the coming articles of this series, we will cover the rest of these topics.

2. Introduction to Visual Questioning Answering Task

Visual Question Answering (VQA) is a computer vision task involving answering questions about an image. The goal of VQA is to teach machines to understand the contents of images and provide answers in natural language. Questions are typically open-ended and may require an understanding of vision, language, and commonsense knowledge to answer.

VQA has gained attention in the AI community due to its challenge in enabling computers to comprehend image contents similar to humans. It has been suggested that the problem is AI-complete, confronting the Artificial General Intelligence problem. Applications of VQA include aids for visually impaired individuals, education, customer service, and image retrieval.

3. Setting Up Working Environment

Let’s start by setting up the working environments. First, we will download the packages we will use in this article. We will download the Transformers package and the torch package to use Pytorch.

!pip install transformers
!pip install torch