Using TRL And OpenEnv To Develop An AI That Paints Watercolours

🔍 Read the full analysis: Using TRL And OpenEnv To Develop An AI That Paints Watercolours on ThorstenMeyerAI.com

An independent engineer has released a complete, open-source reproduction of Surya Narreddi’s viral watercolour-painting language model, built with TRL and OpenEnv on Hugging Face infrastructure. This project includes all datasets, training scripts, reward functions, and trained models, enabling the community to explore AI that learns aesthetic preferences through reinforcement learning.

The original model, created by Surya Narreddi and showcased in August, used a language model to generate JavaScript code that paints watercolour-style images via the p5.js library. The model’s output, being code, is human-readable and editable, allowing inspection of each brushstroke decision. The reproduction follows Narreddi’s approach but is fully open, with all artifacts released under an open license.

The project trains a Qwen model with reinforcement learning, optimizing against a weighted reward mix that includes a style judge, code length, compilation check, and a human preference model called HPSv3. For more details on training techniques, see the original analysis. The style judge is a vision model called Qwen3-VL-30B-A3B-Instruct, which compares generated images against four reference images. The reward function emphasizes style, naturalness, and adherence to artistic taste, with the entire training process orchestrated via Hugging Face’s infrastructure, involving 110 training steps, 240 episodes per step, and multiple generations per episode.

All datasets, environments, and scripts are openly available in a Hugging Face collection, making it possible for others to reproduce, modify, or extend the work. This approach aligns with open research practices discussed in the original analysis. The project aims to investigate whether reinforcement learning can be effectively applied to optimize models based on aesthetic taste rather than traditional, verifiable correctness, such as passing tests or solving problems. Insights into this innovative approach are detailed in the original analysis.

Implications for AI Art and Aesthetic Learning

This open reproduction marks a significant step in AI art research by providing transparency and accessibility to models that learn aesthetic preferences rather than objective correctness. The approach challenges conventional reinforcement learning paradigms, which typically rely on quantifiable rewards, by instead focusing on subjective taste, potentially broadening AI’s creative capabilities. The release lowers barriers for researchers and artists to experiment with aesthetic RL, fostering innovation in AI-generated art and design.

Moreover, since the model produces readable code that can be inspected and edited, it offers a level of interpretability uncommon in pixel-based image generators. This transparency allows users to understand how artistic decisions are made, which could influence future developments in explainable AI for creative tasks.

Background and Evolution of AI Art Generation

The project builds on a lineage of early generative art experiments, such as DeepDream (2015), neural-network portraits by Mario Klingemann, and datasets curated by artists like Anna Ridler. These works explored the medium itself, often emphasizing human-AI collaboration and the exploration of AI’s creative potential. Narreddi’s earlier work involved prompting text-to-image models, but his subsequent focus shifted toward training models directly on artistic datasets with reinforcement learning to optimize aesthetic qualities.

The viral success of Narreddi’s original watercolour video in August, which garnered over 1.5 million views, highlighted the public interest in AI-generated art that appears imperfect and handmade. His approach to training a language model to generate code that produces watercolour effects was seen as a novel way to combine interpretability with artistic expression. The open release of the reproduction now enables the community to build on this foundation, fostering further experimentation and validation.

“By openly sharing all datasets, code, and trained models, we aim to democratize research into aesthetic reinforcement learning and enable others to explore AI that learns taste.”

— Thorsten Meyer, project lead

Unresolved Questions About Model Performance

It remains unclear how the reproduction’s outputs compare quantitatively to the original viral videos in artistic quality or style fidelity. The project presents visual median outputs for different reward mixes but does not provide a definitive evaluation or user study results. Additionally, the full technical report from Narreddi, which might clarify these aspects, has not yet been published. The degree to which reinforcement learning improves aesthetic appeal over baseline models is still under investigation.

Future Research Directions and Community Access

The next steps involve waiting for Narreddi’s full technical report to better understand the technical nuances and performance metrics. Meanwhile, the open artifacts allow researchers to experiment with different reward configurations, datasets, and models. The project sets a precedent for open, taste-based reinforcement learning in creative AI, encouraging further exploration into how models can learn subjective aesthetic preferences and produce more human-like, interpretably artistic outputs.

Key Questions

What is the main innovation of this reproduction?

The reproduction is fully open, providing all datasets, code, and trained models, and demonstrates how reinforcement learning can optimize AI for aesthetic taste, not just factual accuracy or technical correctness.

How does the model generate watercolour paintings?

The model writes JavaScript code that uses the p5.js library to produce watercolour-style images, with style enforced through a reward system that favors natural, translucent, and soft-edged effects.

Can others modify or extend this project?

Yes, all artifacts are openly available on Hugging Face, allowing researchers and artists to experiment with different reward functions, datasets, or model architectures.

What are the limitations of this approach?

The qualitative and quantitative comparison to the original viral outputs is still pending, and the impact of different reward mixes on aesthetic quality is not yet fully understood.

Why focus on aesthetic rewards instead of traditional metrics?

This approach explores whether models can learn subjective taste, potentially broadening AI’s creative capabilities beyond objective correctness, and fostering more human-like artistic expression.

Primary source: Hugging Face · via ThorstenMeyerAI.com