I am building this for two reasons. One is that I went out with some colleagues who work in DevOps and they wouldn't stop complaining about vs code crashing every time they tried to open one of our system log files. The other as a challenge to see if I understand the problem area well enough to produce something better.
Where grep is the usual thing people use, or glogg or even notepad++, what I aim to build is something that is as fast as grep but gives a UI that is modern and capable of handling these 50GB files. There are other advantages of having a UI - being able to see the context in which the error appeared for example without having to run multiple queries.
So I am building Strata. A native desktop application (cross-platform) that uses memory mapping to handle massive files without fully loading them into RAM, it takes 8 seconds to open a 50GB file and once you do scrolling is buttery smooth with 60fps.
I have baked some additional features into it already: Live SSH Tail to allow people to view server logs instantly without transferring files around, Smart JSON analysis to detect and format JSON logs that have been collapsed into a single line making it impossible to read, currently working on improved search beyond just text to allow for regex and being able to search by via JSON properties.
I'd love to hear from other DevOps and SRE people, what features would make your life easier? Does anyone else need to deal with logs of this size and what are your current workflows and pain points?
That's a tough one! I've considered creating a non-gamified learning app based on spaced repetition and active recall - think Anki flash cards but on steroids :D
The reason why I haven't done it yet is because I never have the time, but if you do then I for one would love an app like that.
There's a lot of talk about today's market, but I think doing something and sticking at it will attract the right attention
I'm working on a Log analyzer, something that doesn't choke on huge files. I ended up building something that can open a 50GB log file in 8 seconds and not eat your RAM, while maintaining 60fps when scrolling.
All this with a native C++ engine (able to run on Windows, Linux and MacOS) with a Flutter UI.
With additional features that DevOps teams and SRE can use like Live SSH tailing, JSON analysis and formatting, context aware search with regex, text, and search via JSON fields. All those things also performing incredibly fast - analysing a 5GB file for JSON logs takes about 2s on my laptop.
Still in beta currently, with about 70% of the functionality done! You can find more here https://getstrata.co.uk/ :)
Where grep is the usual thing people use, or glogg or even notepad++, what I aim to build is something that is as fast as grep but gives a UI that is modern and capable of handling these 50GB files. There are other advantages of having a UI - being able to see the context in which the error appeared for example without having to run multiple queries.
So I am building Strata. A native desktop application (cross-platform) that uses memory mapping to handle massive files without fully loading them into RAM, it takes 8 seconds to open a 50GB file and once you do scrolling is buttery smooth with 60fps.
I have baked some additional features into it already: Live SSH Tail to allow people to view server logs instantly without transferring files around, Smart JSON analysis to detect and format JSON logs that have been collapsed into a single line making it impossible to read, currently working on improved search beyond just text to allow for regex and being able to search by via JSON properties.
I'd love to hear from other DevOps and SRE people, what features would make your life easier? Does anyone else need to deal with logs of this size and what are your current workflows and pain points?
Thanks everyone!