Anthropic releases Claude 3.5 Sonnet latest model is comparable to GPT 4o and can run code in the chat window

5 min readJun 24, 2024

Anthropic released its latest model, Claude 3.5 Sonnet, which surpasses previous versions and its competitor GPT 4o model in reasoning, knowledge and coding ability evaluation. It runs twice as fast as Claude 3 Opus and costs one-fifth of it.

The model is available for free on Claude.ai and the Claude iOS app.

Claude 3.5 Sonnet is priced at $3 per million input tokens and $15 per million output tokens, with a context window of 200K tokens. It is cost-effective and suitable for mid- to high-end application scenarios.

The model also outperforms previous versions on visual reasoning tasks, such as interpreting diagrams and transcribing text from imperfect images. Visual capabilities also surpass GPT 4o…

Claude 3.5 Sonnet performs well in multiple evaluation dimensions, including reasoning ability, knowledge reserve, encoding ability and visual performance.
Overall results : Compared with previous versions and competitor models, Claude 3.5 Sonnet has significantly improved overall performance, providing users with a more intelligent and efficient solution.

The Sonnet outperformed rival models on key assessments and was twice as fast as the Claude 3 Opus at one-fifth the cost.
Sonnet excels in graduate-level reasoning, coding, multilingual mathematics, and textual reasoning.
The preview version of Artifacts is now available, which can generate documents, codes, charts, vector graphics, etc.
Sonnet excels on visual tasks, especially those that require visual reasoning.
Sonnet sets new benchmarks in graduate-level reasoning, undergraduate knowledge, and coding proficiency.
Sonnet has shown significant improvement in understanding nuance, humor, and complex instructions.
Claude 3.5 Sonnet is free to use, and has higher usage limits for Claude Pro and Team subscribers.
The rest of the Claude 3.5 series (Haiku and Opus) will be released later this year and will develop new capabilities and features.

https://interserver.awesome-vps.com

Performance and speed

Claude 3.5 Sonnet performed well in several performance evaluations, including:

Graduate Level Reasoning (GPQA) : Excels in complex reasoning tasks and is able to handle advanced logic and analytical tasks.
Undergraduate Level Knowledge (MMLU) : A high score on the knowledge test demonstrates a broad range of knowledge and understanding.
Coding Skills (HumanEval) : Outstanding performance on the coding test, able to solve complex programming problems, demonstrating a high level of programming and debugging skills.

speed

Claude 3.5 Sonnet runs twice as fast as Claude 3 Opus. This performance boost makes it ideal for handling complex tasks such as:

Context-sensitive customer support : Ability to quickly understand and respond to customers’ complex questions.
Coordination of multi-step workflows : Efficiently manage and execute across multiple tasks.

Cost-effectiveness

Encoding performance

In an internal programming evaluation, Claude 3.5 Sonnet solved 64% of the problems, while Claude 3 Opus solved 38%. This shows that Claude 3.5 Sonnet has significant improvements in the following aspects:

Bug fixing : Ability to independently identify and fix bugs in code based on natural language descriptions.
Feature addition : Ability to add new features to existing open source code base.
Code Translation : Easily handle code conversions, especially useful for updating legacy applications and migrating code bases.

Visual performance

Claude 3.5 Sonnet excels on standard vision benchmarks, especially in tasks that require visual reasoning, such as:

Chart and Graph Interpretation : Ability to accurately understand and analyze chart and graphical information.
Image-to-text transcription : The ability to accurately transcribe text from imperfect images is particularly important in areas such as retail, logistics, and financial services.

Artifacts

Artifacts is a new feature introduced in Claude 3.5 Sonnet that allows users to interact with AI-generated content more easily. Simply put, it is like an intelligent assistant that can help you create and edit various content, and let you view and modify these contents in real time in a dynamic workspace.

The preview version of Artifacts is now available, which can generate documents, codes, charts, vector graphics, etc.

When you use Claude to generate content (such as code snippets, text documents, or website designs), these generated contents appear in a dedicated window called the “Artifacts window.” You can view, edit, and further improve these contents in this window.

for example:

Suppose you are developing a website and you need Claude to generate some HTML code for you. After you request Claude to generate the code, the code will appear in the Artifacts window. You can modify the code directly in this window, see the real-time effect , and integrate the modified code into your website project.

The following are its main features and application scenarios:

Generate and edit in real time :

When you ask Claude to generate some content, such as a code snippet, a document, or a website design, Claude displays that content in a dedicated window.
You can view, modify and improve the content directly in this window, just like creating it on a real-time workbench.

Easy integration into projects :

Whether you are writing code, writing documentation or designing a website, the Artifacts feature can help you seamlessly integrate the content generated by Claude into your project.
You no longer need to copy and paste, you can work directly in the dynamic workspace provided by Claude.

Increase work efficiency :

With the Artifacts feature, you can use Claude’s intelligent creation capabilities more efficiently without having to frequently switch between different tools and windows.
This centralized way of working greatly improves work efficiency and is particularly suitable for tasks that require frequent modification and adjustment of content.

Teamwork :

Although it is currently mainly used by individuals, Claude also plans to expand the functionality of Artifacts to support team collaboration in the future.
Team members can share and edit content in the same workspace, improving team collaboration efficiency.