Generate and Organize: structured output from LLMs

Ever hit a wall trying to get LLMs to generate structured text?

We’ve all been there. In our case we faced challenges in Named Entity Recognition (NER) and Relation Extraction (RE); getting these models to output structured information effectively has always been a tough nut to crack.

But my team has been exploring a simple yet powerful approach that’s been yielding exciting results: "Generate and Organize" (G&O).

Here’s how it works:

Generate: Let the model freely produce content in natural language.

Organize: Then, guide the model to restructure that content into the desired format.

It might sound straightforward, but the impact has been remarkable. For example, in a zero-shot NER task, instead of training the model to recognize entities directly, we had it generate a narrative and then organized the content into structured data. We did see a significant improvement in accuracy without complex tweaks.

This approach reminded me that sometimes, the most effective solutions are elegantly simple. Interestingly, OpenAI seems to agree when they recently launched “Structured outputs”, which aligns closely with this concept. It’s incredible to see LLM giants adapting swiftly, making tasks like ours even more efficient.

As we look ahead, I’m confident we’ll see native solutions to many other wrappers. What wrappers are you working on or hoping to see native solutions for?

PS - 1st comment has the links for those interested.