Claude 3 versus GPT-4: comparing Anthropic’s new AI with the current leader on political prompts

Archive note: this article records tests from March 2024. The two prompts and all four model responses below are translated in full from the Spanish source and quoted as historical AI output. Names, testimonials and factual examples within those responses have not been independently verified. Availability, credits and benchmark claims describe that period.

Last week, Anthropic launched Claude 3, its latest language model, which outperforms GPT-4 on several benchmarks, making it the most advanced publicly accessible AI model at present. Claude 3 comes in three versions: Opus, Sonnet and Haiku, with Sonnet available free on Anthropic’s website. The model promises “near-human” understanding and stands out for its advanced computer vision capabilities, even surpassing GPT-4 Vision. One of its major advantages is its enormous context window—the ability to work directly with large amounts of external information—initially 200,000 tokens, with the possibility of expanding beyond 1 million for special cases. Claude 3 demonstrates almost perfect recall of that information in complex evaluations.

Here are the benchmarks provided by the company itself, comparing the model with GPT-4 and the Google Gemini models currently available:

Accessible transcription of the historical image. Values and evaluation settings are reproduced as shown; a dash means no value is supplied.

Scroll across the table to read all columns.

AI journalist Maxim Lott has also conducted an intriguing experiment, giving different AI systems an intelligence test from Mensa, with the following results:

Accessible transcription of the image titled “AIs ranked by IQ”, attributed to MaximumTruth.org:

Scroll across the table to read all columns.

Given the model’s apparent potential, and Reddit users’ particular praise for its writing and text generation—and the fact that, unlike Gemini Ultra, it works perfectly in Spanish—we decided to test it directly against GPT-4. After trying around ten prompts, our impression is that the two are at a very similar level: one produces better responses in some cases, the other in others.

So that you can judge for yourselves, beyond general benchmarks or our personal impressions, we present two examples—no more, to avoid making the newsletter excessively long—with responses generated by Claude 3 Opus and GPT-4 using Chatbot Arena, a tool specifically designed to compare large language models. We gave it two prompts previously shared in this newsletter’s prompt section. In principle, then, these were better adapted to GPT-4, as they had been created for that model. Here are the responses from each:

Screenshot note: the selected models are claude-3-opus-20240229 and gpt-4-1106-preview. The rules allow side-by-side conversations over multiple rounds and voting for A, B, a tie or “Both are bad”. “Clear history” starts a new round. The substantive Spanish prompt and responses shown in the image are translated in full below.

First example: urban mobility¶

Historical prompt¶

Act as a political marketing expert specialising in urban election campaigns, applying the PASTOR framework (Problem, Amplify, Story, Testimonials, Offer, Response) to develop a specific strategy for a candidate for mayor of a large city. Identify the central urban mobility problem affecting citizens, amplify the urgency and challenges associated with this issue, use real stories from affected residents to illustrate its impact, and incorporate testimonials from urban planning and transport experts and from citizens satisfied with solutions proposed in other cities. Present a clear offer of innovative policies and measures that the candidate will implement to improve urban mobility, and end with a call to action aimed at urban voters, encouraging them to support the candidate to achieve tangible, positive change in their city.

Claude 3 Opus — historical response¶

Problem:

Urban mobility has become a pressing challenge for our city’s residents. Endless traffic jams, inefficient and overcrowded public transport, and a lack of adequate infrastructure for cyclists and pedestrians are negatively affecting citizens’ quality of life. This problem not only robs people of valuable time but also contributes to air pollution and daily stress.

Amplify:

The situation has reached a critical point. Workers lose precious hours of their day stuck in traffic, arriving late for work and losing valuable time with their families. Students struggle to reach classes on time, while older people and people with disabilities find public transport inaccessible and inconvenient. Congestion is also choking the city’s economy, delaying deliveries and reducing productivity. We must address this problem now, before it gets worse.

Story:

Ana, a nurse and single mother, leaves home an hour earlier every morning simply to make sure she reaches the hospital where she works on time. Despite living only a few kilometres away, she gets stuck in traffic every day, struggling with the anxiety of being late and the stress of navigating congested streets. She longs for a city where she can get to work efficiently and reliably, and have more time for her son.

Testimonials:

Urban planning and transport experts have praised innovative solutions implemented in cities such as Curitiba and Copenhagen. “By prioritising efficient public transport, creating dedicated bus lanes and encouraging cycling, these cities have transformed urban mobility,” says Dr Silva, a renowned urban planner. Residents also attest to the positive effects. “Since the new transport systems were introduced, I get to work in half the time and without stress,” says María, an office worker in Curitiba.

Offer:

As your mayor, I pledge to implement a comprehensive urban mobility plan. I will invest in modernising and expanding our public transport system, creating dedicated bus lanes and ensuring reliable service frequencies. I will build a network of safe cycle lanes and promote bike-sharing schemes. I will implement “complete streets” policies that prioritise pedestrians and cyclists. I will also work with technology companies to develop smart traffic-management solutions. These measures will transform how we move around our city.

Response:

Citizens, the future of our urban mobility is in your hands. By supporting my candidacy, you are voting for tangible, positive change. You are choosing less time stuck in traffic and more time with your loved ones. You are choosing cleaner air and safer streets. Together, we can create a city where getting around is a pleasure, not a burden. Vote for me, and let us set off towards better urban mobility for everyone! Your vote will be the first step on this exciting journey.

GPT-4 — historical response¶

Problem:

The central urban mobility problem in our large city is chronic traffic congestion, which causes excessive journey times, increased air pollution and reduced quality of life for residents. The current transport infrastructure is not equipped to handle the growing volume of vehicles or meet the needs of a dynamic, expanding urban population.

Amplify:

To amplify the urgency of the problem, we highlight the negative impact of inefficient urban mobility on the local economy: losses running into millions because of working hours wasted in traffic, the emotional strain on citizens forced to spend extra hours away from their families, and public health risks caused by environmental pollution. We emphasise that without immediate, decisive action, these problems will intensify, jeopardising our city’s sustainable future and competitiveness.

Story:

We use real stories from affected residents to illustrate the problem’s impact. For example, a nurse who arrives late at hospital because of traffic, affecting patient care, or a father who misses important events in his children’s lives because he is stuck in his car for hours. These personal stories create an emotional connection with voters and highlight the need for effective solutions.

Testimonials:

We include testimonials from urban planning and transport experts who support the candidate’s proposals, as well as from citizens satisfied with similar solutions implemented in other cities. This provides credibility and demonstrates a track record of success in improving urban mobility.

Offer:

The candidate presents a clear offer of innovative policies and measures, including:

• Expanding and modernising the public transport system to offer faster, more reliable travel alternatives.
• Creating more cycle lanes and pedestrian paths to promote active travel.
• Implementing a smart mobility plan that uses technology to optimise traffic flow and reduce congestion.
• Initiatives to encourage car-sharing and reduce the number of vehicles on the roads.
• Incentives for electric vehicle use and the installation of more charging stations throughout the city.

Response:

We conclude with a call to action aimed at urban voters, encouraging them to support the candidate to achieve tangible, positive change in their city:

“It is time to take the wheel of our future. Together, we can unlock our city’s potential with urban mobility that moves us forward instead of holding us back. This is our moment to choose a greener, faster, more efficient route. A vote for [Candidate’s Name] is a vote for less traffic, cleaner air and more time with our loved ones. Let us join forces on [election date] to move our city in the right direction!”

Additional strategic notes:

• The strategy should be personalised to reflect the city’s particular characteristics and unique challenges.
• Campaign events should be held at transport trouble spots to illustrate the problems directly and discuss the proposed solutions.
• The campaign should use a range of media, both digital and traditional, to reach a wide range of voters.
• There should be a focus on inclusive mobility, ensuring that the proposals consider all communities and socioeconomic groups.

Second example: AI and blockchain in public policy¶

Historical prompt¶

Act as a forward-thinking political consultant specialising in integrating emerging technologies into public policy design. Develop detailed strategies to incorporate artificial intelligence and blockchain effectively and ethically into public policy. What best practices and approaches are recommended to ensure that these technologies are used in ways that maximise benefits for society while respecting ethical and legal principles? Include considerations of data privacy, transparency, accessibility and inclusion in your advice, as well as examples of successful use cases and recommendations for collaboration between the public sector, technology experts and key stakeholders.

Claude 3 Opus — historical response¶

As a forward-thinking political consultant specialising in integrating emerging technologies into public policy design, I recommend the following approaches and best practices for incorporating artificial intelligence (AI) and blockchain effectively and ethically:

1. Establish a robust ethical and legal framework. Before implementing any technology, it is crucial to develop clear ethical and legal guidelines that ensure responsible, transparent use of AI and blockchain. This includes principles such as non-discrimination, data privacy, security and accountability. Government bodies, ethics and legal experts, and key stakeholders should collaborate to create this framework.
2. Encourage transparency and explainability. AI systems used in public policy must be transparent and explainable. Citizens have a right to understand how decisions affecting them are made. I recommend mechanisms that allow the reasoning behind AI-based decisions to be audited and explained, fostering public trust.
3. Ensure data privacy and security. The collection and use of personal data must comply with strict privacy and security standards. Techniques such as anonymisation, encryption and controlled access should be applied. Blockchain can be used to create immutable, secure records, particularly in sensitive areas such as health or digital identity.
4. Promote accessibility and inclusion. Implemented technologies must be accessible to all citizens, regardless of socioeconomic status, technical skills or disabilities. Intuitive interfaces, multiple access channels and training programmes can help overcome digital divides. AI can even be used to personalise services according to individual needs.
5. Cross-sector collaboration. Governments, the technology industry, academia and civil society should work together on the design, implementation and monitoring of these technologies. I recommend establishing multidisciplinary working groups and citizen feedback mechanisms to co-create people-centred solutions.
6. Pilots and continuous evaluation. Before large-scale implementation, I suggest pilot testing in controlled settings to assess impacts and risks. Once implemented, initiatives should undergo regular evaluations to measure effectiveness, detect biases and make necessary adjustments.

Some promising use cases include:

• Blockchain for secure, transparent voting systems, such as the pilot conducted in Sierra Leone in 2018.
• AI to optimise resource allocation and personalise public services, such as the AI-based triage system used in hospitals in Denmark.
• Blockchain-based digital identity for secure, interoperable access to government services, such as the e-Estonia programme.
• AI to combat corruption and fraud in public procurement, such as predictive models used in South Korea.

In summary, the ethical and effective integration of AI and blockchain into public policy requires a multidisciplinary, people-centred approach grounded in sound principles. With appropriate safeguards and ongoing collaboration, these technologies have the potential to transform governance, improve services and empower citizens. As consultants, our role is to provide expert guidance to harness these benefits while carefully managing the associated risks and challenges.

GPT-4 — historical response¶

As a political consultant specialising in incorporating emerging technologies into public policy design, my approach focuses on ethical, effective and equitable implementation. Below are strategies and recommendations for integrating artificial intelligence (AI) and blockchain into public policy:

1. Establish an ethical and legal framework:

• Develop clear regulations defining the ethical and legal boundaries within which both AI and blockchain must operate, with a focus on data protection, informed consent and non-discrimination.
• Implement privacy standards such as the GDPR and promote the adoption of “Privacy by Design” principles in technology development.
• Ensure transparency in AI algorithms, allowing independent audits and reviews to prevent bias and unfair decisions.

2. Encourage transparency and accountability:

• Use blockchain to create transparent, tamper-proof record-management systems, improving trust in public processes.
• Establish accountability mechanisms for AI developers and users, ensuring avenues for redress in the event of errors or abuse.

3. Improve accessibility and inclusion:

• Ensure user interfaces are accessible to people with disabilities, removing barriers to digital public services.
• Implement policies that promote digital inclusion, providing training and resources so that all citizens can benefit from new technologies.
• Use AI to personalise public services and improve their efficiency, adapting them to individual needs.

4. Promote collaboration:

• Establish partnerships between the public sector, technology companies, academic institutions and civil society organisations to share knowledge and best practices.
• Create AI working groups or ethics committees that include technology experts, philosophers, legal scholars and other stakeholders to assess technology’s social impact.

Examples of successful use cases:

• AI in public health: AI-assisted diagnostic systems that improve the accuracy and speed of disease identification, enabling a faster response to health emergencies.
• Blockchain for civil registries: Systems for recording births, marriages and deaths that ensure data integrity and immutability.
• AI in the administration of justice: AI tools that help review cases and legislation to assist judges and lawyers, improving the judicial system’s efficiency.
• Blockchain in the management of public funds: Platforms that track government spending in real time, increasing transparency and reducing corruption.

Recommendations for collaboration:

• Organise hackathons and innovation competitions that encourage the technology community to develop solutions to public problems.
• Create public policy laboratories that act as incubators for AI and blockchain projects, facilitating experimentation and phased implementation.
• Promote international collaboration, sharing experiences and establishing common standards to address AI and blockchain’s ethical and technical challenges globally.

In conclusion, AI and blockchain should be integrated into public policy through a human-centred approach, ensuring that these technologies serve the public interest, respect privacy and fundamental rights, and promote a fairer, more inclusive society.

Trying the model¶

If you would like to use Claude 3 Opus, the most powerful version, it is not available free on the website, unlike the intermediate version. You can, however, try it through Chatbot Arena, as we did, or use the most powerful model through Anthropic’s API, for which new users receive $5 in credit.

Finally, we would also like to recommend two very useful tools created by Anthropic to help users get the most out of its new models: a prompt library, with many interesting examples, and a specialist prompting guide.

Fernando Nieto Lobato

Director of Digital Innovation at Institución Educativa ALEPH