Cleanup repo for inference

LLaMA worker is now removed and several pieces of cleanup on plugins/worker have been done. It would be good to do some further cleanup and documentation.

This goes hand in hand with #1473 which I think needs to be prioritised now that the inference pipeline is fairly stable and not changing regularly. We have many users asking for details on how to run their own instances.