Your own personal Google: how Mem plans to reinvent note-taking apps with AI
In the summer of 2019, Kevin Moody and Dennis Xu started meeting with investors to pitch their new app. They had this big idea about reshaping the way users personal information moves around the internet, coalescing all their data into a single tool in a way that could actually work for them. But they quickly ran into a problem: all of their mock-ups and descriptions made it seem like they were building a note-taking app. And even in those hazy early days of product development before they had a prototype, a design, even a name they were crystal clear that this would not be a note-taking app.
Instead, the founders wanted to create something much bigger. It would encompass all of your notes but also your interests, your viewing history, your works-in-progress. Imagine if you had a Google search bar but for all nonpublic information, Xu says. For every piece of information that was uniquely relevant to you.
Thats what Moody and Xu were actually trying to build. So they kept tweaking the approach until it made sense. At one point, their app was going to be called NSFW, a half-joke that stood for Notes and Search for Work, and for a while, it was called Supernote. But after a few meetings and months, they eventually landed on the name Mem. Like Memex, a long-imagined device that humans could use to store their entire memory. Or like, well, memory.
Either way, its not a note-taking app. Its more like a protocol for private information, a way to pipe in everything that matters to you your email, your calendar events, your airline confirmations, your meeting notes, that idea you had on the train this morning and then automatically organize and make sense of it all. More importantly, its meant to use cutting-edge AI to give all that information back to you at exactly the right time and in exactly the right place.
The most ambitious version of the product will require a huge amount of work and a lot of technology that either doesnt exist or only barely works. It will require partnerships with huge tech firms that may be reticent to share their data, trust from users who have to compile it all together, and a user experience that is infinitely flexible without being too complicated or overwhelming. But if Mem can accomplish its goals or even come remotely close, it could become the most powerful productivity tool on the planet. It could maybe even help shift the way the internet works.
Theres all this data in the world about us, Xu says, and none of it we can actually use, right? Its all trapped inside of Googles servers or Facebooks servers, Netflix, YouTube, all that. What we want to do is actually put that in your hands.
Theres all this data in the world about us, and none of it we can actually use, right?
So far, Mem is mostly a note-taking app. Its blisteringly fast and deliberately sparse mostly just a timeline of every mem (the companys parlance for an individual note) youve ever created or viewed, with a few simple ways to categorize and organize them. It does tasks and tags, but a full-featured project manager or Second Brain system this is not.
But if you look carefully, the app already contains a few signs of where Mem is headed: a tool called Writer that can actually generate information for you, based on both its knowledge of the public internet and your personal information; AI features that summarize tweet threads for you; a sidebar that automatically displays mems related to what youre working on.
All this still barely scratches the surface of what Mem wants to do and will need to do to be more than a note-taking app. The competition will be steep, too: Dropbox recently acquired Command E to power cross-platform document search, Neeva is building personal search right next to web search, while Evernotes Chrome extension lets you search your notes through Google. Googles even tried to build this itself, though it gave up on Google Desktop when it became clear cloud storage was the future.As generative AI gets more powerful and less expensive, every app you use will find a way to incorporate it.
The point is, a search engine for your private stuff is not a new or even particularly novel idea. But thanks to huge recent advancements in AI and a generational shift in how we manage our digital lives, the time is ripe for someone to build it.
Theres no lightbulb moment at the founding of Mem, more a series of cobbled-together ideas over the years. Moody and Xu were Stanford classmates and had talked about working together while Moody was at Google and Xu at Yelp. But Moody does remember one specific aha moment from 2018 when he was at a restaurant in Oregon. I was like, Shit, I think Ive been here before, and I forget what I ordered, he says. He couldnt remember if he should reorder what he got last time or was potentially about to make the same menu mistake again. This immediately felt like a solvable problem. I wanted to be able to come back to the same restaurant, he says, and a note would just pop up and say, The seared ahi, 10/10; the meatball sub, send it back before it even gets to your table. He wrote that idea down in Apple Notes and never quite forgot about it.
From the very beginning, Mems founders knew they needed a huge amount of data to get their vision off the ground. (Google doesnt work if theres no internet to crawl, right?) So the Mem app was built with easy capture in mind, which meant making the app fast to open and fast to type into. Before they had a mobile app, they built a way for users to text notes to Mem through SMS or WhatsApp. That proved to be one of Mems most popular features. Even in the early days, when Mem didnt do much, it was still one of the easiest tools for just quickly jotting something down.
But the dream is not quick notes. The dream is for all your info and everything about you to just appear in Mem, where it can be sorted and searched and made useful on your behalf. If a tool could access your Netflix history and your Spotify playlists, just to name one example, it could learn a lot about your taste. So the team began early on to search for even bigger sources of personal information, ones that dont require users to take their own notes.
It started with the biggest player of all: Google. After all, if you wanted a single source of a persons important online information, their inbox is a good place to start. Mem has been working with Google for months to let users connect to their Gmail and allow Mem to index all of their emails since time immemorial. Theyre still working on the implementation, but the goal, Xu says, is to essentially turn all of that unstructured information into something that is really easy to use for you. After that, Mem will need to do something similar for, well, every other platform on the internet.
Theres a good chance its impossible to bring the entire internets knowledge of you into a single place even Xu and Moody acknowledge that most companies have no incentive to give Mem or anyone else access to the data they have. And even if it can pull it off, Mem would instantly become a security nightmare. Xu and Moody say theyve invested in platform security since the beginning and that they take their security responsibility seriously. If we succeed even close to the degree we want to, were going to have the most valuable data in the world, Xu says, and thats going to put a huge target on our back. In the long run, theyre interested in using decentralized platforms to avoid having a single point of failure. Both co-founders say theyll never sell or otherwise abuse user data. Still, to do what they plan will require an unprecedented amount of trust from users at a time when trust in digital platforms seems awfully hard to come by.
Mems best maybe only chance is to make the tradeoff worth it.
A useful question to ask about any internet service is, what are they doing with all your data? Often, its targeted ads, which are of specious value. Other times, its to make better recommendations for stuff to watch or buy or listen to. In Mems case, it thinks that by collecting and combining all your information, it can both help you find all that data and make it much more useful.
In our very first conversation in 2020, Moody off-handedly asks a question that hinted at things to come: why doesnt Google Docs know whats in your docs? Docs can search through the words, sure, but it doesnt understand what your doc is about. It cant generate a table of contents or a one-paragraph summary; it cant tell you which of your other docs might be relevant to the topic at hand. (Ironically, in the months since, Google has actually begun to roll out some of these features. Moody was onto something.) Its not just Google, either, he says. Too many notes apps, document holders, and collaboration tools are just dumb containers of information. Mem aspires to be more than that.
Mems real theory of the future starts with transformers. Not the robots-in-disguise kind the machine-learning kind
Mems real theory of the future starts with transformers. Not the robots-in-disguise kind the machine-learning kind. It starts with a whitepaper from 2017 called Attention is All You Need, written by several Google researchers, that changed the way people think about AI.
The short version of a long paper goes something like this: Using transformers, AI engines can be trained to understand full sentences, paragraphs, even entire documents at a time. That gives those engines much more context with which to infer meaning transformers allow computers to figure out whether you wrote bank meaning the house of money or the side of a river or whether the tone of a story is happy or sad or somewhere in between. Transformers are changing the way the whole tech industry thinks about language processing. This has been an incredible breakthrough, says Zoubin Ghahramani, a distinguished researcher at Google, that I feel like has enabled a lot of these other interesting language models to flourish over the last five years.
The way Mem sees it, transformers flip the whole model of information management on its head. If the computer can deeply understand and organize your stuff, it means you dont have to. For as long as digital note-taking and information storage tools have existed, theyve required users to carefully tend to them. It takes a lot of time and effort to organize everything, Xu says, and keep everything in the right place and make sure you have the right tags. Sometimes its just impossible because you dont realize what connections are going to form and the ways youre going to want to use a certain piece of information until you get new information.
As he explains this to me, Xu shares his Mem screen over Google Meet and opens up a mem with a Twitter thread hes writing about the death of folders and digital organization systems. As he types, an automated list of related mems appears in the right sidebar: a few tweets hes collected on the subject of information storage, an article from The Verge about how young internet users dont see the internet in terms of files and folders anymore, a few mems that looked like other drafts of the same thread, an early manifesto Moody wrote about Mems plans. If a human were to read this, Xu says as he scrolls through the list, youd understand what its about, right? And whats surfacing here is not only things Ive written in the past; its things that Kevin has shared with the team, mission statements you can create this kind of hive mind.
This is Mem X, Mems most advanced auto-organizational tool. Mem X is how Mem becomes a self-organizing workspace, the term Xu and Moody have come to use to describe the platform. (Its also how Mem plans to make money: the basic app will be free, the company has said, but advanced features like Mem X and Writer will come at a price.) Through a partnership with OpenAI, Mem has begun ingesting huge amounts of user data and finding connections without any user input.
Mem It for Twitter is another example: Mem can grab a Twitter thread and generate a summary of its contents while also recommending similar threads from other users. Going forward, Mem wants to do the same for every URL you save to the app not just understand the URL or even the words in the title but everything on the page and what its about. Pair that with knowledge about your history, habits, and taste, and Mem can also become your own personal guide to the internet.
The tech is far from perfect: Xu, at one point, searches for transformers to prove that Mem X understands the difference between language models and Optimus Prime, and indeed Optimus Prime crept into the search results. When he searched for my name which should bring up all the meetings wed had together, all the emails Id sent, and more it only brought up a few results because I have multiple email aliases. But Mem X drew enough unexpectedly correct connections that the value already outweighs the imperfections. Mems AI isnt perfect, and neither are the fundamentals its based on. But they both already work.
