OffWorld is developing a new generation of autonomous industrial robots to do the heavy lifting first on Earth, then on Moon, Mars and asteroids. We see reinforcement learning as one of major candidate technologies that could allow us to reach a high level of autonomy. While RL has achieved remarkable results in games and simulators, its adoption for real physical robots has been slow. In this talk we will go over a few projects we did at OffWorld that relate to applying RL on real robots, we then make the case that there is an apparent gap between RL community's aspirations to apply RL on real physical agents and its reluctance to move beyond simulators. To bridge this gap we introduce OffWorld Gym —a free access real physical environment and an open-source library that allows anyone to deploy their algorithms on a real robot using the familiar OpenAI gym ecosystem and without the burden of managing a real hardware system nor any knowledge of robotics.