> most models I've seen are 300GB+ and require significant computational resources to operate (think several $15k NVIDIA A100 compute nodes).
What? Where have you been the last 3 months?
> the quality of the responses from the model are correlated with how large (and therefore how much compute) the model has
There's a lot more to this including the model structure, training methods, number of training tokens, quality of training data, etc.
I'm not at all saying that Vicuna/Alpaca/SuperCOT/Other llama based models are as good as GPT3.5 - but they should be capable of this, they still create coherent answers.
You need preferably 24GB of vram, but you can get away with less, or you can use system memory (although that'll be slow).
There is a openai api proxy that might let this work without too much work actually
EDIT: It actually says in the readme they plan to support StableLM which is interesting because at least at the moment that's not a well performing model
Even if you're a more lightweight model, it's still not very practical to require a dedicated 24GB GPU for every active gamer, whether local or cloud hosted.
For all intents and purposes, it's as much of a non-starter in a production game as the multiple A100 scenario.
Of course that isn't going to remain the case for long as the recent advancements in optimization make their way into live systems, but still.
> it's still not very practical to require a dedicated 24GB GPU
totally agreed, you could get away with 12GB too which is in the midrange.
That said yeah it's still not something you could make a game with yet, I'm just pointing out 300GB+ of VRAM isn't the bar for entry here, it is reachable for medium-high end consumers but that's not really including the games resources either, and most gamers aren't medium-high end so...
> EDIT: It actually says in the readme they plan to support StableLM which is interesting because at least at the moment that's not a well performing model
I chose StableLM because that's the only other model I knew of besides ChatGPT. I'm open to adding support for other models after I fix some bugs first.
What? Where have you been the last 3 months?
> the quality of the responses from the model are correlated with how large (and therefore how much compute) the model has
There's a lot more to this including the model structure, training methods, number of training tokens, quality of training data, etc.
I'm not at all saying that Vicuna/Alpaca/SuperCOT/Other llama based models are as good as GPT3.5 - but they should be capable of this, they still create coherent answers.
You need preferably 24GB of vram, but you can get away with less, or you can use system memory (although that'll be slow).
There is a openai api proxy that might let this work without too much work actually
EDIT: It actually says in the readme they plan to support StableLM which is interesting because at least at the moment that's not a well performing model
EDIT 2: You should try the replit2.8B model - This is surprisingly good at programming - https://huggingface.co/spaces/replit/replit-code-v1-3b-demo