ChatBuild
Back to Projects

ChatBuild

ChatBuild was a small idea to turn text prompts into real structures in Minecraft: describe a build, and Gemini places it block by block in front of you.

GamesJavaSpigotGoogle GeminiMinecraft

ChatBuild: A fun demo of AI use cases

I built this way back in 2024 and got pretty good results, but want to maybe try get it up and running again to see whats possible!

Type what you want, look at a spot on the ground, and watch it appear. ChatBuild is a Spigot plugin for Minecraft: Java Edition that sends your prompt to Google Gemini, gets back a list of blocks, and places them in the world.

Live demo: a prompt goes in through chat and the build appears in front of the player.

How It Works

  1. A player runs /build fast <prompt> or /build pro <prompt> while looking at a block.
  2. The plugin sends the prompt to Gemini along with a long system prompt that explains the job: answer only in JSON, put the centre of the build at 0,0,0, and use only blocks from a given list.
  3. Gemini returns something like {"blocks": [{"type": "minecraft:oak_planks", "x": 0, "y": 0, "z": 0, "fill": false}]}.
  4. The plugin parses that and places each block relative to the spot the player was looking at. Entries marked fill: true carry endX/endY/endZ and fill the whole box, which keeps responses short. Filling with minecraft:air hollows out interiors.
  5. When the build is done, an oak sign goes up next to it showing the plugin version and a GitHub credit.

The API call runs off the main thread so the server doesn't freeze, and the blocks are placed back on the main thread once the response arrives. While it waits, the player sees a boss bar counting down.

Fast vs. Pro

The first argument chooses the model: fast uses Gemini 1.5 Flash and pro uses Gemini 1.5 Pro. Flash comes back quickly with simpler builds. Pro takes longer but adds more detail. Tab-complete suggests both options.

Long prompts are awkward to type in chat, so /build pro -book reads the prompt from the book and quill in your hand instead.

What I Learned

Most of the work was making the model's output reliable, not writing the Minecraft side. The system prompt asks for JSON only three separate ways because Gemini kept wrapping its answer in markdown fences or adding a friendly sentence first, and either one breaks the parser. Big builds also hit the 8,192-token output limit and got cut off mid-JSON, so the plugin checks for MAX_TOKENS and tells the player instead of placing half a castle.

Details

  • Java 21 Spigot plugin for Minecraft 1.20, built with Maven
  • Google Gemini 1.5 (Flash and Pro) through the REST API with OkHttp
  • Retries each request up to three times before giving up
  • Prompt-to-JSON-to-blocks pipeline with box fills to keep responses small