As the differences in models are diminishing, data quality has a greater impact on results. Mini GPT-4, on the other hand, lacks quantitative results.Last, it should be clarified the focus of this line of work is data-centric, not model-centric. LLaVA has rigorous quantitative results, including the level of similarity with Visual Chat and GPT-4, the SoTA accuracy on Science QA, and ablation studies on data iteration and model design. When some users compared LLaMA to MiniGPT-4, Li pointed out that LLaMA could reproduce image-based results from the GPT-4 paper, which MiniGPT-4 could not. LLaVA co-author Chunyuan Li answered several questions about the work on Twitter. A pre-training process first trains the project matrix, and then a fine-tuning process updates both the projection layer and the LLaMA decoder weights the CLIP weights are frozen. The image and word tokens are then passed to a LLaMA decoder which produces output. The LLaVA architecture consists of a CLIP foundation model followed by a projection matrix layer to convert images into a word embedding space textual input is also transformed into the same space. Overall, the generated dataset contains 158K samples. Because the images are annotated with captions and object bounding boxes, the team fed this data into a text-only GPT-4 along with prompts asking GPT-4 to output instruction-following data, including: imagined conversations between a person and an assistant, questions about the details of the image content, and questions requiring reasoning about the image content. The LLaVA team's goal was to train a model end-to-end with visual instruction tuning. To do this, the researchers started with images drawn from the COCO dataset. The next step in development of AI assistants has been the addition of the ability to handle image data, as shown with the release of GPT-4 and Visual ChatGPT. InfoQ recently reported on LLaMA, which has only 7B parameters compared to GPT-3's 175B, but can outperform GPT-3 on many tasks. The technique of fine-tuning large language models (LLMs) with instruction-following datasets has led to gains in performance, as demonstrated by ChatGPT, and has prompted researchers to explore this technique with smaller LLMs.
It achieves excellent visual chat experience when fine-tuned on multimodal chat data. We have presented an automatic pipeline to create language-image instruction-following data, based on which we train LLaVA, a multimodal model to follow human intent to complete visual tasks.
This paper demonstrates the effectiveness of visual instruction tuning using language-only GPT-4. When further fine-tuned on the ScienceQA training dataset, LLaVA achieved an accuracy of 92.53%, a new record for the benchmark. The team also used GPT-4 to evaluate LLaVA's responses in experiments, by asking it to rate LLaVA's output on a scale of 1 to 10. This dataset was used to fine-tune the LLaVA model, which consists of two foundation models: CLIP for vision and LLaMA for language, with an additional network layer to tie the two together. The researchers used GPT-4 to generate the instruction-following dataset, which contains virtual conversations between a human user and an AI assistant about the content of images. LLaVA is based on a CLIP image encoder and a LLaMA language decoder, is fine-tuned on a synthetic instruction-following dataset, and achieved state-of-the-art accuracy on the ScienceQA benchmark. Researchers from Microsoft, the University of Wisconsin–Madison, and Columbia University have open-sourced Large Language and Vision Assistant (LLaVA).
0 kommentar(er)
