Google Attaches a Small Controller Network to Image AI to Improve Instruction-Following
The technique attaches a small controller network to an existing image generation model to adjust how strongly it follows an instruction. Google says it improved both image quality and instruction-following, even where a model's internals are hard to modify.
Google Research unveiled Diffusion Controller on September 29 local time. It is a technique for steering the image generation process. The approach attaches a small controller network to an existing model so that the elements a user asks for actually show up in the picture. Google says the method makes models follow instructions more faithfully while preserving the stability of the base model, even in settings where the model’s internals are hard to touch.
Why a lizard in sunglasses is hard
Image AI often draws the requested subject but misses specific details in the instruction. Google’s example is a lizard wearing sunglasses. The model renders the lizard realistically but leaves out the sunglasses, or it forces the sunglasses in and warps the face in the process. The new technique targets exactly this trade-off: keeping the requested elements while keeping the image natural.
Previous approaches either adjusted how strongly the written instruction influenced the picture midway through generation, or fine-tuned the model with extra training. Google explains that the two methods operated separately, making it difficult to strike a balance between quality and user preference. Diffusion Controller treats the entire process, from random noise to a finished image, as a single control problem. The idea is to compare different steering methods within the same mathematical framework.
The base model stays put; only the direction is corrected
The attached controller network looks at the intermediate states as the image forms and nudges the generation direction step by step. It intervenes to move the output closer to the desired style or content without altering the base model itself. If the output drifts too far from the base behavior, the controller applies the brakes. Google says this mechanism reduces the problem of mangling a subject’s shape or texture while inserting a requested object.
Image: research.google
Users can dial the control strength up or down with a single setting at generation time. It is effectively a knob for how strongly the written instruction is applied. Google claims the base image stays stable even so. For developers who cannot readily modify a model’s internals, the pitch is that a separate controller network alone is enough to customize the results.
How did it perform in testing?
Google ran performance tests on top of Stable Diffusion v1.4 across three fine-tuning setups. Evaluation used HPS-v2, a score that measures how well a generated image matches human preferences. According to Google, a controller network with restricted access to the model’s internals posted a higher win rate than LoRA in two of the training setups. LoRA is a method that additionally trains only part of a model. Google says the controller achieved this result while touching fewer internal layers than LoRA.
Image: research.google
The version that freely modified the model’s internals recorded a 90% win rate against the base model, Google said. In human side-by-side evaluations, it also scored best on image quality and instruction alignment for complex prompts that demand several conditions at once.
Google points to personalization, safeguards that reduce the generation of harmful images, and control of video models as directions for follow-up research. The thing to watch, in the end, is whether the gains seen in image testing carry over to those use cases.