It is absolutely open source. AGPL for source, MIT for the binaries. Both of those are free software.
Their enterprise version has a different license; the source for that one is not available and the binaries are released under a much more restrictive license, but that's nothing to do with the normal one.
Disclosure: I run Zulip, a Mattermost competitor that has intentionally avoided taking venture funding in part because we don't want to be forced by investors to be Open Core.
Venture-funded Open Core software like Mattermost is an awkward place for the terminology. The free version is distributed under an open source license, as so there is a thing that is "open source", but the clear intent of Open Core licensing policies is primarily to achieve the goals of:
(1) Being able to market the software as open source.
(2) Maximizing the portion of users who buy the paid/proprietary version.
While in theory open core can be run responsibly, the incentive structure is to intentionally not include features important for typical use cases in the "open source" version of the software that any similar community-driven open source project would have considered an essential, early feature.
FOSS means different things to different people, but my personal perspective is that venture-funded open core software like Mattermost feels like FOSS to me about as much as proprietary software with a free plan like Slack or GitHub does.
For Mattermost in particular, this thread is a good reference: https://news.ycombinator.com/item?id=21820583. They made changes to the specific items mentioned there in response to the community pressure, but that doesn't change their fundamental business strategy.
Their enterprise version has a different license; the source for that one is not available and the binaries are released under a much more restrictive license, but that's nothing to do with the normal one.