Interesting, I saw something like this by looking at the https://github.com/mozilla/pdf.js/ project. Take a look at the closed pull requests. They have some bots listening for commands in comments and do stuff like unit testing and previewing. The result gets posted as comment from the responsible bot. Another one is checking master branch for changes and automatically builds and pushes at gh-pages. Seems to work very well, but don't know how they build/did it.