After reading the reddit comments, it looks like a primary problem is that the author doesn't (didn't?) understand how to benchmark it correctly. Like comparing the time to mmap() a file with the time to actually read the same file. Not at all the same thing.
I mean, its open source so people can create benchmark and independently verify if the AI was wrong and then have the claims be passed to the author.
I haven't read the reddit thread or anything but If the author coded it by hand or is passionate about this project, he will probably understand what we are talking about.
But I don't believe its such a big deal to have a benchmark be written by AI though? no?
> I mean, its open source so people can create benchmark and independently verify if the AI was wrong and then have the claims be passed to the author.
Thank you for volunteering. I look forward to your results.
> Thank you for volunteering. I look forward to your results.
Sure can you wait a few weeks tho? I know nothing about benchmarking so gonna learn it first and I have a few tests to prepare for irl.
I do feel like someone else more passionate about the project should try to pick the benchmarking though.
I don't mind benchmarking it but I only know tools like hyper for benchmarks & I have played with my fair share of zip archives and their random access retrieval but I feel like even that would depend from source to source.
There are some experienced people in here who are really cool at what they do, I just wanted to say that if someone's interested and already has the Domain Specific knowledge to benchmark & they enjoy it in the first place, this having AI benchmark shouldn't be much of a problem in comparison.
Why would someone spend their time checking someone else's AI slop when that person couldn't even be bothered to write the basic checks that prove their project was worthwhile?
