Factorio Learning Environment

The Factorio Learning Environment, or FLE, is an open source framework built on top of the game Factorio. It exists to evaluate large language models on long term planning, program synthesis, and resource optimization. As models begin to saturate older benchmarks, FLE offers open ended challenges that rise in difficulty from basic automation to factories handling millions of resource units per second. It provides two evaluation protocols: lab play, which includes 24 structured tasks, and open play, where the model builds the largest factory it can from scratch. The project was developed by Jack Hopkins and colleagues and is free to use.

- Test a model's planning and automation skills through structured lab tasks

- Let a model attempt open ended factory building in open play

- Interact with the environment through a Python API and submit programs

- Measure performance using production scores, milestones, and error feedback

- AI researchers who need a harder, open ended benchmark

- Developers checking how well a model can synthesize working programs

- Enthusiasts comparing different models on the same set of tasks

Setting up FLE requires Python and some familiarity with running code, so it is not a one click tool. The game environment needs reasonable hardware, especially for large open play runs. Results depend heavily on how well the interaction program is written, so weak programs can make a capable model look bad. Because it is a research project, expect rough edges and changing interfaces. Treat the scores as one data point about model ability, not a complete ranking.