Advertiser
This podcast is supported by Google. Hey folks, Steven Johnson here, co-founder of Notebook LM. As an author, I've always been obsessed with how software could help organize ideas and make connections. So we built Notebook LM. as an AI-first tool for anyone trying to make sense of complex information. Upload your documents and Notebook LM instantly becomes your personal expert. uncovering insights and helping you brainstorm. Try it at notebooklm. google. com.

Nathan Labenz
Hello, and welcome back to the cognitive revolution. Today, I'm excited to share part two of my conversation with Ilya Poloksuhin, co-author of the seminal Attention is All You Need paper and founder of NIR Protocol, which describes itself as the blockchain for AI and aims to build a future where AI belongs to everyone. In Part 1, we explored the foundational technology stack, proof-of-stake consensus mechanisms, confidential computing on GPUs, and Mir's plan to train frontier models on a decentralized basis. With contributions incentivized by cryptographically guaranteed revenue shares. Today we zoom out from the enabling technology and discuss the big picture questions. What does personal computing look like as AI continues to eat traditional UI-centric software? Ilya envisions a unified intelligence layer that works across all of your devices, predicts what you need, and proactively gets things done for you. How will this change the nature of markets, trade, and the consumer economy? Ilya expects that advertising and middlemen will become less relevant as AI agents allow buyers and sellers to connect directly at unprecedented scale. What does daily life look like, and how will we spend our time in the era of AI-provided abundance? In addition to a rise in personalized AI-generated entertainment products, Ilya predicts that people will find meaning and compete for status in small communities focused on niche interests, including athletics, video games, the collection and curation of unique artifacts, and who knows what else. What rules should govern agent-to-agent interactions in this world? Ilya imagines a symbiotic relationship between individual humans and their own personal AIs. Such that humans and AIs effectively grow up together. And he believes that AIs should pursue the interests of their individual humans, even if that means being less than fully transparent when negotiating with other AIs on their humans' behalf. And finally, what new governance mechanisms could allow us to take advantage of the exponential increase in information processing power and reasoning that AI affords us? Ilya describes Mir's experiments with AI delegates, which vote on behalf of token holders who in turn select them based on their decision-making frameworks. And also describes NIR's long-term goal of enabling every individual to have their own AI that participates in governance on their behalf. Of course, we touch on lots more along the way as well, including the role of formal verification of software in securing smart contracts, the remaining challenges required to effectively bridge the gap between low-level guarantees and high-level human intentions. And the need to harden society's defenses in order to maintain global biosecurity in the presence of broadly distributed, powerful AI systems. Overall, while many questions remain to be answered, Ilya's strength as a systems-level thinker, spanning technology, economics, social dynamics, and governance is evident throughout. So without further ado, I hope you enjoy one of the most concrete positive visions for the future that you will find anywhere. From Ilya Polksukin, founder of NIR. Ilyopolasuhan, founder of NIR, welcome back to the cognitive revolution.

Ilya
Thanks for having me back.

Nathan Labenz
I'm excited for part two. So last time we talked a lot about foundational technology, the journey that you went on from being an author of Attention is all you need to try to Source data from contributors around the world struggling to pay them, taking a detour into blockchain, thinking it would take just a few months. Here we are a few years later, and it's all really happening. Anybody who's listened to this feed for more than a minute knows that I often say the scarcest resource is a positive vision for the future, and I appreciate that you have one. So I'm really appreciative that you're taking a second window here to help us unpack that. I guess maybe for starters, you know, one of the jumping off points last time was you said that you wanted to teach computers to code. And sure enough, now they can code. So maybe as we kind of ramp up into a a vision of like potentially a quite different future How is the rise of AI coding assistants changing how you guys work? And how is it changing like Who can create stuff on top of the blockchain?

Ilya
Yeah. But I don't think it's even on I mean about blockchain per se, but Yeah, I mean the the re the real reason why I always thought that you know as computers are able to code we actually approaching a different world because There's few dimensions to this, right? We know that there there was a statement by Mark Andriessen, right, that software is eating the world. Right. And and this idea was like the defects, I mean what it means is automation. Automation has always been the driver of innovation, of GDP, of productivity. And you know, it's like everything from tractors to factories to then indeed computers and computers kind of this like kind of universal you know, bicycle of the mind, right? Automating things. And so now the challenge been that there's what there's always been like a small cohort of people who actually are able to build software. Right. And so if I have a need, right, I need to find somebody else to build it. Probably they need to build it not just for me, but for a large number of people, so it's actually economical. And so like we've in we've been dubbed with a lot of like software that became very complex to use. And so now you need to learn how to use it because it's not really built for you, it's built kind of for, you know, the generic user that has like, you know, every user has like five use cases that they like somewhat overlap and so it's all of this is like kind of stack in in one software. Right Or or you just don't have it and you keep doing stuff manually and kind of you know wasting your time. And so for me, the ability for machines to code ha is really about that transformation where everyone now is able to build their personal software. Everyone now is able to build their own personal automation and it also re kind of removes the fact that like the interfacing, right That this interfaces need to show you all the options right away, right? You kind of can kind of subselect with you know English or whatever into the part of the interface that you need Right. So to give you very specific examples, right? I mean I like the example of Salesforce, right? Salesforce, I mean, obviously started as like a you know small startup. that was targeting like a specific use case of sales people. But at this point, right, it's a monstrosity that like you need to hire somebody else to configure Salesforce for you. Right. Like that's how complex a Salesforce is, right? At this point it it's effectively like you're hiring somebody else to build you a a system, right? It's just like they're using existing pre-built components. Now imagine this world now where you can, you know, just talk to a computer. Like it's it's really now becomes about your sales process, about your business process about how you want to automate, what reports you want, etcetera, right? It can be dynamic, you can restructure this as you go, you know. You can whatever build in things that you know Salesforce may have somewhere as a feature or may not even have And then you can integrate with whatever other tools you want. Like for example, Salesforce, you know, we're in crypto, everything is in Telegram. Salesforce doesn't have Telegram integration, we can't use it, right? So now somebody we need to go in somebody to get build integration in Telegram and Salesforce. But you know, with your vibe coded CRM you can just like, hey, you know, and and integrate telegram for this. So that's kind of I mean, this is just like a simple example, and we can keep extrapolating, right? It's every single kind of our l let's say digital footprint, right, is kind of becoming more and more automated, right? And the more intelligence computer can have and more it it can take on the the kind of the more you can offload the kind of orchestration of different tools or just low level, you know, here's a database, you know, I need an HR tool, I need CRM, I need this, I need that, right? You can actually build all that. So that's been I mean this is in twenty seventeen that we're like hey you know so software as a service is gonna die. AI will replace it. In twenty seventeen this sounded like, yeah, very delusional. That was probably the but obviously I I think s now we see that most people are kind of Or a lot of people agree and then the s software service itself is trying to become AI because they know they're kind of gonna just get out o outcompeted by that. So Uh coming back to your question, like what changes in our work, I think there's few pieces that are, you know, already clearly work, right? Kind of uh data analysis through natural language, right? If the if you have like kind of reasonable data structure. You can like now give like you can effectively make everyone maybe not a data scientist, but like a what before would require like a business analyst. person right and then it would take them a while to like pull all the data. Now you can just say like, oh, you have a question about the business analytics information. Just ask it. Like you don't need to email somebody to get you the answer, right? You just ask it to the tool. It you know generates SQL queries, pulls in whatever data, right? Some Python gives you an answer, right? I think I'm actually a big fan of the front-end building with wipe coding now, and though I'm not writing production code anymore, but it's really useful for me as a prototype tool where you can effectively just really quickly get to an experience and so you don't need like I mean you want design to be more about like style guides, etc. And and But like the UX part you can test really quickly. Before a designer would, you know, maybe design something and then developer would try to build it and again It's like, oh shit, I can't actually build that or like it doesn't work exactly this way. And there's like a lot of iterations. Now even designers can just build a full experience, you know. fully clickable thing, it generates code for this and now developers can just plug in all the back-end logic. I think also we're starting to see on our teams that actually doing development that the I would like the time spent is changing. Before you spend a lot of time on kind of the development work, and then you spend also a lot of time on reviewing other people's work. The amount of time spent on development is shrinking, right? Because you effectively tell the whatever coursor, codex, whatever this is, to go and do the thing. And it's a lot more time spent on reviewing things and making sure they're correct, right? Now, one thing we were experimenting with on one of the teams was What if we like decompose the whole piece like the whole so like this is you know for simple software right now AI works, right? For complex software it doesn't, right? It it doesn't really Go and you're gonna just go and like build me a really complex system and it goes and does it. It doesn't work yet. We see obviously the improvements continuously. And so right now for complex systems You right now decompose it yourself, right? You know, as an architect or uh senior engineer, and then you have maybe a few different team members who actually build the subsystems. And before In traditional software development you always want to have multiple people on each subsystem that you want them to know really well how it works, so they can maintain it, so they can change it. Now actually it's not as important because AI will explain to you how it works. And you may actually even have a natural language explanation that the developer used to build it in the first place attached to it. And so it's actually more about like velocity and and the viewing process and then how to ensure that each part is like secure and and kind of works correctly. And so There's kind of a transition happening where there's more almost like individualism because every individual is more productive. And then it's more about like how to build the how to build the right decomposition into pieces. Again, this is a very also temporary in a way. Uh but this is kind of the current state, right? And so like as tools become more mature, they are better at like larger code base navigation, etc. I think the other question is Uh it's it's really about like I mean the model quality for specific task for Like for like the the con the conceptually like the the abstraction level that it needs to operate on, right? So like again front end, there's no there's no abstraction level, right? You just kind of build what you see right and you kind of iterate. So it's really easy to check, it's really easy to iterate. So you don't really need that much and models are good at this. When we talk about like a blockchain low-level code, right, it's like extremely complex system. There's a lot of pieces, externalities, so models are not very good at that and kind of There you review expect people to spend more time and in general you spend more time like thinking about the algorithm and architecture than writing code. So yeah, so it really depends on kind of But this is also all shifting really quickly. Like, you know, six months ago I would have wouldn't give you a different answer, right? And I'm sure in six months it will be different as well

Nathan Labenz
Yeah, we've seen some very impressive programming results from frontier companies that have not yet hit the you know the public APIs or product surfaces just yet. So certainly we can bank on more to come. When it comes to you you mentioned like job titles. I think you said like senior engineer or architect. That raises the question I think a lot of people are asking right now. Are you hiring junior engineers? What do you think is the fate of the junior engineer as things stand today?

Ilya
Yeah, I mean I think it's it's less about like junior engineer. It's more about who is that person, right? If if this is somebody who is like coming in Like, you know, for example, by the time I got into university, I already been coding for seven years. Right And so and this is when I got, you know, my first job, etc. So like if this is somebody coming in who's like, you know, build multiple projects already, using EI every day. etc like it doesn't matter if junior or not junior right it's really I mean yeah there is a lot of skills for them to learn but like they are there to learn them right they're open they're ready to go And then you have some people who coming in who like, you know, s like studied a bunch in university, but like not really like not in really in this learning mindset and like this is gonna be continuously changing things and like you're like so like again We we're transitioning from like software as a like craft to really be just like a problem solvers that talk to computers, right? And so so so you need I mean problem solving at the end is really all about kind of like the mindset and then the approach. And so if people are willing and and are kind of Doing that and excited about that, that's if people are like, Oh, I don't know how to do this. I don't you know, I can't do this, etc. Then that's not the right person. So yeah, so it's really depends on like it U for many things before, yeah, you would hire like a bunch of junior developers because they're cheaper and you don't need maybe as like high quality of work and you want to fan out work. That part is not needed, yeah. So you hiring more for like the problem solving and kind of more people who can actually like creatively problem solve together.

Nathan Labenz
Yeah, there's always a roo there's always a space, there's always room for People who are like a force of nature under themselves. Overall that sounds bearish for rank and file like

Ilya
I was told that if I you know get a study this but can yeah I'm gonna be yeah I'm gonna pay tongue okay yeah. Yeah, I think I mean but I think that's true about every job. at this point. Maybe plumbers and electricians. That's the like I I I usually talk like the automation right now happening from both from two sides. The all of the like manufacturing like on the floor jobs are getting automated and like that price is still pretty high actually. Like yeah, I mean you know Vietnam For example, salaries are probably lower than what robots are getting paid in US. But in US it's already like, you know, you affect there's this company I I know called Formec. they effectively provide you out staffing with robots. You have a factory, you call them up, they bring you robots, they install them, they set them up, you don't need to do anything, you just pay them eff effectively as as you pay salary. But they work twenty four seven, they don't complain, they don't unionize. They just, you know, do the job. They don't quit. Like right now in US the chorn of workforce is three hundred percent. Meaning like every year you need to hire three people for one job. Because like they keep quitting. So so that's like the automation from the like bottom kind of low scale, you know, uh very repetitive task. And then finally all of the kind of white collar, high-end jobs are getting automated, like you know, coding, lawyers, like a lot of this information work, right, getting automated. And so the most safe right now, I mean it's gonna get automated as well, but is actually the high dexterity kind of skilled work that requires like I don't know you know plumber you need to like climb under the sink and like you know fit something like stuff like this right now is just like super hard to do. by with the eye, but again this will happen as well. Like it all those things will get automated over time. Yeah, it's coming for all of us. It's just a question of when.

Erik Torenberg
When Hey, we'll continue our interview in a moment after a word from our sponsors.

Nathan Labenz
The worst thing about automation is how often it breaks. You build a structured workflow, carefully map every field from step to step, and it works in testing, but when real data hits or something unexpected happens, the whole thing fails. What started as a time saver is now a fire you have to put out. Tasklet is different. It's an AI agent that runs 24-7. Just describe what you want in plain English. Send a daily briefing, triage support emails, or update your CRM. And whatever it is, Tasklet figures out how to make it happen. Tasclid connects to more than 3,000 business tools out of the box, plus any API or MCP server. It can even use a computer to handle anything that can't be done programmatically. Unlike ChatGPT, Tasklet actually does the work for you. And unlike traditional automation software, it just works. No flowcharts, no tedious setup, no knowledge silos where only one person understands how it works. Listen to my full interview with Tasklit founder and CEO Andrew Lee. Try Tasklit for free at Tasklit. ai and use code COGREV to get 50% off your first month of any paid plan. That's code COGREV at Tasclit. ai. AI's impact on product development feels very piecemeal right now. AI coding assistants and agents, including a number of our past guests, provide incredible productivity boosts. But that's just one aspect of building products. What about all the coordination work, like planning, customer feedback, and project management? There's nothing that really brings it all together. Well, our sponsor of this episode, Linear, is doing just that. Linear started as an issue tracker for engineers, but has evolved into a platform that manages your entire product development lifecycle. And now they're taking it to the next level with AI capabilities that provide massive leverage. Linear's AI handles the coordination busy work, routing bugs, generating updates, grooming backlogs. You can even deploy agents within Linear to write code, debug, and draft PRs. Plus, with MCP, Linear connects to your favorite AI tools, Claude, Cursor, ChatGPT, and more. So what does it all mean? Small teams can operate with the resources of much larger ones, and large teams can move as fast as startups. There's never been a more exciting time to build products, and Linear just has to be the platform to do it on. Nearly every AI company you've heard of is using Linear. So why aren't you? To find out more and get six months of Linear Business for free. Head to linear. app slash TCR. That's linear. app slash TCR for six months free of linear business. So I want to understand a little bit better, because I think a big part of your vision of the future, obviously, is AI that everyone owns, right? We've c we've kind of got the one default path, it seems, in front of us where We have like the big tech singularity, which is where like three to seven companies, you know, become like totally dominant forces because they have the models that ran away from everyone else and nobody else does. And so, you know, we're all just kind of trying to get whatever inference we can get from you know from these leaders. And then there's your vision of the decentralized and collectively owned AI that I think has you know has a lot to to say for it. It's certainly super attractive in a in a lot of ways for people, with some concerns about, you know, what happens if everybody has, you know, access to certain things in an unrestricted way too. But leaving that aside If front end is like largely we can get the AIs to do it, but like core blockchain work is beyond what they can do. What's in the middle? Like, what kind of apps are people building on the blockchain today? How hard is it to build those apps? What makes it hard? And can the AI's help there yet, or if not yet, you know, at what point can should we start to see an explosion of like vibe-coded on blockchain applications. Like what's the what's the the fundamental barrier or kind of rate limiting step toward the proliferation of you know just anybody who has an idea for blockchain can go do it in the same way that anyone who has an idea right now, to a significant extent at least, can go, you know, create a little micro-saaS app.

Ilya
Yeah, I think Uh the problem is the same, it's just it's exaggerated. So the problem right now if you're you know launching your MicrosaS app and you're not you're not actually engineered If you're launching it just for yourself, it's totally fine. Right. And and I kind of my recommend my recommendation for everyone, like, you know, build tools for yourself. Vibecode, everything for yourself. Now the problem is like as soon as you go like make it for everybody else because you don't really understand what issues are under the hood. You don't know like how how it will actually affect your users, yourself, etc. And so we've seen already, you know, people getting hacked, you know, their like secrets getting leaked, etc. Right. So like that that is the biggest issue right now. The blockchain because it's naturally kind of in the open right away for everyone and it involves money. that problem is exaggerated, right? Because now if you make any small mistake, right, and you know, we have this right now with very professional engineers who build soft who build blockchain software any small mistake somebody will find it and exploit it and and effectively this will result in some value lost, some money lost. So really that's the biggest challenge. What works right now is actually for existing so-called smart contracts, right? Kind of the back end You can generate a front end. You can create your own custom UI for specific use cases or combining multiple use cases into one UI. That part actually works. And again, I would not recommend to launch it to other people, but you can build for yourself where you say, like, hey, you know, I have this like yield opportunities across different places. I have this whatever, let me combine it, make my own like asset manager, right? That like makes me easy, for example, to manage this. Like things like that you can totally do now. So I think it yeah, it really depends. Now, what we are working on kind of medium to long term is how do we formally verify the correctness of the smart contracts and ideally the whole kind of blockchain application stack. such that if you are vibe coding an application, you actually have proofs, mathematical proofs that it's correct And now also when user using it, and and this is important, just proving okay cool, smart contracts is like doing what you want, but you maybe didn't properly define what you want, right? And so maybe your logic itself is flawed. And so what you want is like I'm as a user using this piece of applic software want to know that it does what I want. Right. And so that's a critical step. So again, if I'm looking for a financial application, I want to know it's not gonna lose my money. And so if it if it can prove to me directly in the transaction, right, as I'm sending money in, it should prove to me that it's not gonna lose my money, then transaction succeeds. So that's kind of the level of integration that we are aiming for and I think is required for this kind of really kind of adversarial and monetary valuable. I actually think this is required for all software because Like the world where you know AI is going around and hacking everything left and right is also not great and and we are kind of in it actually right now.

Nathan Labenz
And so we do need like formally verified software to to really secure that has been coming up more and more in my conversations about this, and I would have to confess that Up there with like regular expressions, the concept of formal verification of software is, you know, one of the things that makes me feel d dumbest. Cause I'm always a little bit like stuck on I think the point that you We're emphasizing there, which is like it's one thing to prove that, you know, this particular function or whatever you know does what it's supposed to do and like doesn't, you know, corrupt other memory or, you know, whatever. You can you can make a bunch of sort of generally low-level statements, right? But it seems like there's still a real challenge in aggregating up those small level statements to the holistic like this is what I want. It's you know you have a a a version of the genie problem, which is kind of, you know, what a lot of people have worried about with AI in general for a long time of, you know, if we Tell it a goal that it interprets a bit differently than we did, you know, we're in we potentially could be in trouble there. How do you see this actually playing out in Practice and again, I'd love to kind of get some vision for like applications that you would love to see somebody come build on your protocol that don't exist yet that and maybe it's because they're just too hard, but I'm sort of imagining like, do I have an AI agent that comes in and like automatically red teams this for me?

Ilya
Let me give you a kinda a simple example and we can build it up, right? So a simple example, I wanna put money into a bank savings account, right? And I want to be able to withdraw it, and ideally I want to withdraw a little bit more money than I put in. Right? Like right now, you know, you you're sending money, you have no guarantees. Like let's say you send money via whatever ACH or IBAN or something. You have no idea if it's gonna arrive, right? You have no idea if bank will exist tomorrow. You have no idea if if it's actually gonna give you money back. I mean there's like government insurance, right, that insures up to some amount, but like generally speaking you have no guarantees. Now blockchain gives you some guarantees. It you know actually like hey money arrived, you can verify that But indeed if there is some code involved, right, you don't know if that code does has some potential, you know, somebody can withdraw this money. Illegally, right? Like you need to go audit the code. You yourself may have you know missed something, etc. So here you say, hey, you know, you effectively constrain like you know, I my transaction will pass only if I can call this withdrawal method and it will return me at least as much, you know, given x deposit, I want to call be able to call withdraw with at least X, right? So that would be the condition which is like when you deposit, right, you effectively constrain that this smart contract needs to prove to you that this property will be true. Now that contract may you know deposit this money itself into other places, right? You know, if it's a savings account, it maybe lo lends it, borrows it, etc. And so it needs to actually in chain to be able to prove to you it actually needs to chain all of this with every everything else it does. Right. It needs to be like, hey, if I'm lending to someone, right, either they need to return to me or I'm gonna like, you know, foreclose their account, liquidate their collateral, etc. Right. Like so there's kind of a chain in fact that's happening through the system. So that's kind of where I mean to your point, right? Like you you start maybe low level, but you actually can start expressing somewhat high-level properties. Now The challenge with this like with money and deterministic blockchains it's pretty easy because it's deterministic and and you kind of have like full observability. So you can actually do those constraints pretty easily. To your question where it's coming from, well, it's gonna come from combination of indeed your, I mean, we call it wallet, right? Your kind of software that is on your side that actually facilitating those interactions. But also, you know, we believe the your wallet will be AI, right? It will be the AI agent that is on your side, your user on AI, that actually does this interaction. So it will indeed be on guard, you know, verifying these properties. Now a more complex thing is indeed how do we prove things that are non-deterministic and not potentially like easily observable? And this is this is indeed where, you know, you obviously cannot have like a hundred percent formal proof. You need to effectively now start. dealing with probabilities. And so and then you can move those probabilities with insurance and other things. So this is where kind of you can have like financial system and other things where you say, hey You know, we have a liability, insurance that, you know, in less than one percent of cases something can happen. And so like prove to me that It is like you are you are you know it either succeeds or I'm getting like a million dollar payout if it doesn't, right? And so you're kind of starting to combine what you know what people have built in insurance where they estimate risks and evaluate and underwrite and sound just like kind of formularification combined with like this probabilistic modeling. So it's not like for some things the e the answer is somewhat easy. And then it becomes more and more complex as we as we touch more and more real world. Like uh uh to give you an example, it's like hey, I'm ordering, you know, steel from I don't know some country it's gonna it's gonna arrive, there's a ship involved, right? You know, maybe this ship like sinks midway, right? So I mean normal way is like hey, you know, we're gonna ensure this and there's gonna be insurance and so like you need all of those mechanisms to build on top to account for real world kind of non-determinism.

Nathan Labenz
Yeah it's it's really hard for me to envision all of that working and again partly it's because I'm maybe just a little slow on some of these things, but The the the sinking ship is a good example where it's like, hmm, how is my smart contract gonna know if it really did sink or if somebody's just telling me that it sank or you know there's that that sort of shell game of like where do you hide the trust or you know where what exactly is fully verified it's quite interesting. I don't want to get too bogged down in it though because I don't wanna uh force us to get to a part three before we really get to all the sort of utopian vision. So maybe where you can, like you know, weave some of this stuff in there as we go.

Erik Torenberg
Hey, we'll continue our interview in a moment after a word from our sponsors.

Nathan Labenz
Being an entrepreneur, I can say from personal experience. can be an intimidating and at times lonely experience. There are so many jobs to be done, and often nobody to turn to when things go wrong. That's just one of many reasons that founders absolutely must choose their technology platforms carefully. Pick the right one and the technology can play important roles for you. Pick the wrong one and you might find yourself fighting fires alone. In the e-commerce space, of course, there's never been a better platform than Shopify. Shopify is the commerce platform behind millions of businesses around the world And 10% of all e-commerce in the United States. From household names like Mattel and Gymshark to brands just getting started With hundreds of ready-to-use templates, Shopify helps you build a beautiful online store to match your brand's style. Just as if you had your own design studio. With helpful AI tools that write product descriptions, page headlines, and even enhance your product photography, it's like you have your own content team. And with the ability to easily create email and social media campaigns, you can reach your customers wherever they're scrolling or strolling, just as if you had a full marketing department behind you. Best yet, Shopify is your commerce expert with world-class expertise in everything, from managing inventory to international shipping to processing returns and beyond. If you're ready to sell, you're ready for Shopify. Turn your big business idea into Cha-Ching with Shopify on your side. Sign up for your $1 per month trial and start selling today at Shopify. com slash cognitive. Visit Shopify. com slash cognitive. Once more, that's Shopify. com slash cognitive. But let's like uh start to leave you know also a little bit of the sort of how behind and just talk about like the what. What are the apps that you know we're gonna enjoy? What is the computing paradigm that we're gonna have? You mentioned agents, you know, kind of doing stuff for me Meta obviously has been putting a vision forward of glasses with a display in them, a sort of head up, you know, no Who needs a keyboard, right? When you can just talk to your AI as you walk down the street. That does appeal to for for all the things that Meta has done, including hot stepmom, that don't appeal to me that much. I would say the the head up display is You know, it's at least a an interesting vision for the future. What do you think our computing life is gonna look like as this stuff matures?

Ilya
Yeah, so I definitely agree some form of AI operating system is going to be the main driver of our computing. The like the devices and kind of the form factors will be different and I actually think it's gonna be easier it's already easier, like you know, if you want to make your own glasses, it's not actually that hard, right? There's There's some factor in China that will make whatever hardware you want. So really it's about like kind of a single AI, you know, your AI, that is available across all those form factors, right? You know, your watch, your glasses, your headphones, your you know, phone, your laptop, your whatever, all of this is kind of interconnected as like a single, you know, surface and then Your AI knows that like you like to see this information and watch, but then you know, by the time you pull out your phone you want to see like maybe news and kind of uh more longer form content and then I maybe actually like you know videos instead and so that's what it should show me. So like it's gonna be effectively personalized, like a AI generated, not just kind of the content but also the applications that we use, right? Probably a lot of the similar patterns that we already use, like the feeds, the you know, the chats, etc. But They don't need to be kind of fixed, right? Like you don't like right now I have you know five different instant messengers, I have you know seven different feeds. And like all of that, you know, can be a single feed, you know, I can switch between like work and personal when I want to. You know, my my I can predict a bunch of things that I would like to do this is something we experimented back actually in 2017 where based on all the things you're doing can we predict actually the next thing you will do on your phone and just do it for you Right or or suggest to do it, right? Like, you know, you have a meeting, you know, twenty minutes away, let's call you an Uber. Right. Like you don't need to go open an Uber, go calendar, copy paste the lit you know the address, put paste it like it's just gonna do it. Like things like that. There's a lot of things that like, you know, AI Will know like hey you know you low like you ordered food two days ago you know you're gonna be out let's reorder a bunch of stuff that you typically order And so as that as that system kind of matures and we also trusted more and and that is an important aspect, I think there's there's actually starting the economy itself is gonna start to shift. Because right now the economy is built, I mean we're kind of in this consumer economy where it's built on advertisement on kind of Like we we you know we discover new things through those feeds, etc. But if your AI is like actually, you know like I mean some people are doing this now. Like, hey, I want I want to be on a diet, like build me a personalized, you know, meal plan. etcetera. So like all of that, but also it's gonna go and order that. And you know, maybe even your humanoid robot at home will also cook it because it's the same computing system, right? It's like you just get the food you like your AI recommends given your health goals, etc. And so now the capacit so this is So this is kinda like on a micro level, right? You have this person and so now your AI, like it doesn't need to go and order it from your local Walgreens or whatever. va vents or Audi or something, right? It can actually put a purchasing with directly farmers, right? With directly, you know, manufacturers. And so they can then themselves start capacity planning, right? So you're kind of starting to remove some of the middlemen that are built because we cannot right now have the direct relationships with suppliers. Right. And so that's kind of an an an interesting meta point that like our economy right now is built on this like middleman architecture, right? Because there's it's really hard to plan things and so every layer like Costco is effectively just, you know, they purchase a bunch of stuff, they put it in one place and then you buy it, right? So like They kind of surfaces. And I I mean, you know, they have a fixed margin, etc. , but it's really kind of like the you know, the temporary place for holding things. for you to actually purchase right or or or find. And if your AI is purchasing directly, right, it can just go to their purchasing agents and kind of do this. Now uh there's batching, there's like other things, but again all that can be done by AI m way more effectively than we're doing it right now. Right, you know, like it the some AI of your city will know that, you know, five hundred people are ordering this, you know, seven thousand people are doing that, you know, they'll want it, you know, tomorrow, day after tomorrow, the day after tomorrow, so we're like, you know, we're gonna Captures the eggs and ship it this way, etc. Like all of that can be effectively just managed as a holistic information system. And so that's where we go into like a really I mean I I use this example kind of half jokingly. But in communism they were trying to build a system right where they were doing capacity planning, but they were missing the AI to actually do it. Right? But if you have a real-time platform where you can actually have all this supply-demand being and so the reason why capitalism has been so successful, because capitalism is actually a compression of information Like money is a comp is a compressed like information because it kinda compresses anything you can purchase into, you know, one number. Right. And so it kind of compresses all of this information, all of the different things into one number, and then it's really easy to kind of navigate. And so here you but obviously with this compression it's it's it's you know, you lose some information, you lose some decision making. That's why in you know in US there's like thirty, forty percent of all the food is being thrown out because there's just like you know the over provision in the stores because they don't know how much people actually will buy and they don't want to have an empty store. But you don't need you don't need to do that, right? If you knew exactly what the purchase will be in the next twenty-four hours because their AI has already planned everything and provided it and all of those got aggregated and shipped. So I think like we're gonna see actually a shift in how economy works at the macro level because of this micro like each of our individual AIs becomes this micro decision maker who can real time provide all this information to the right sources. and navigate in and also not be affected by a lot of kind of you know brands and other stuff, right? But actually validate based on the kind of core core values. So that is like a interesting transformation where I think indeed it's it's uh uh uh I was mentioning like I don't think like there there's no good way to like make a movie or a you know a book like a science fiction book that's talking about affect to like a change of economic structures in the in the society, right? So it's way easier to talk about, you know, dystopia and, you know And the heroes who are fighting against it. Like it's just like a way better story arcs than, you know, like, hey, you know, we've been building out this economic model and like now it's one percent better. every month. And so it's like keeps b getting better and more optimized and that's how we live. So yeah, so I think like that's the kind of the Paradigm of computing is we have the AI AI I mean a cloud agent it I mean it's affecting our assistant, right? Our operating system. But it has all the context about us. It's able to make decisions on our behalf. That's why it needs to be private. It needs to be ours. It needs to be on our side. Right. We need to know that it's aligned with our with our kind of success and outcomes. Otherwise i I mean, this will not work. But if it is and we can trust it, then it can go and make decisions on our behalf, right? And so the other example is in like traditional governance, right? You know, right now we again we're compressing information, right? We go and vote every four out four years for someone and hopefully that someone goes and does what they promised to do and why we vote for them. Now that usually doesn't happen. But and and so again we can actually have every single decision can be voted on by all whatever three hundred million Americans because their AI is online all the time and can evaluate every single decision. And based on their owners beliefs and like what's valuable and successful for them, it will can represent them. So you don't need to have this like compression of representation. if you can have this like online always available AI that's on every individual side. So that's kind of the again, like there's economics, there's governance. sides. And obviously the other side which is the entertainment and kind of that's where things are getting interesting because I think we as humans I call it like we believe in status games. Like we're sorry, we are driven by status games, right? And because money became this like compression mechanism, we use this right now as a kind of ultimate status game. You know, you have more money, you're more successful. Uh you know, billionaires, more successful, more famous, etc. etc. Right. But I think again as this decomposition happening and we already see this right, you know, an athlete maybe not as wealthy as a billionaire, but maybe still more kind of famous and more respected in many ways, right? There's some other kind of status like I call them status games, effectively places where you can compare who's better in some way and who's higher, etcetera, which don't need to be associated with anything that's actually productive. Right. I mean I use few examples, but obviously like you know athletes is a good example because like there's no extra GDP being produced by athletes. But it's still you know very valuable kind of Status game which other people enjoy but you know watching and kind of participating in different ways. Obviously video game is kind of a video gaming is a new form of that as well, right? You have now video game athletes. But you know, you can imagine many of these. Like NFTs were similar, right? Are you part of this NFT, you know, collection? Do you have bored apes? Do we have you know pen pengu? And then you're part of this, you know, tribe and if not, you're not, right? So like we kind of like this types of differentiation. And I think that will actually proliferate a lot. I think we'll ha see more and more of those things where people really differentiate on things that are really, I mean, superficial to extent, right? But kind of for the group they make a lot of sense and they kind of differentiate people between each other. So it's not again, it's not like are you a whatever, a software engineer or a lawyer. It's really like yeah, are you whatever? playing League of Legends or StarCraft, right? So so that's kind of I think important. Obviously we already see on the entertainment side You know, AI generated entertainment. Again, not very hard to extrapolate, but especially with Sora yesterday, you can have just like a personal feed fully AI generated, right? You don't actually need like people re you know, recording, etc. I think I think they're gonna be again with with everything we saw in automation, there's like uh a slice of a market that wants it, you know, in a traditional way and like people doing it. So they're gonna be still, you know, even if there's robots everywhere serving you food, they're gonna be a restaurant with people which will be like more. prestige and have like limited you know capacity. Similarly, you know, i you can listen to AIs playing AI music, but you know, humans playing human music will continue to be like a prestige thing. So but again it's gonna be this niche is like the like we already kind of in this niche world, right? I think it's gonna continue proliferating. Yeah, so that's kind of like how I'm I'm thinking of like as society evolves, like and we have this, yeah, a lot of the economical things are kind of moving away. The yeah We're gonna just like participate in more status games, right? Like what what is the things how you know how you compare with everybody else in that niche, in that group, etc.

Nathan Labenz
Several uh double clicks I want to do. First, uh your mention of communism brings to mind a article I think about, an op-ed in the Washington Post I think about not infrequently. From all the way back in 2018, AI will spell the end of capitalism by a Chinese legal scholar and government official basically saying that The planning, this is sort of the you know, through the looking glass version, I think, of your vision, but making a similar point that like the local nature of capitalist decision making, you know, where everybody's kind of trying to do their their own local thing and they're sort of aggregating signals and sending off aggregate signals to other people through the price mechanism, et cetera, that that may not be needed as much anymore. And their vision for it is obviously much more centralized than the one that you're articulating. But we are starting to see a little bit of glimpses of this. with, for example, Pulse from OpenAI, where now I can wake up every morning to an AI, you know, and there's other versions of this too. I've I've tested quite a few. But you know, Pulse certainly the one that's made the most headlines recently for being there in the morning with like work that it's done overnight, presumably when the GPUs weren't in such high demand, and bringing me, you know, something that it has gone out and kind of scoured the world, you know, to find the stuff that I really need. So I can start to see the the beginning of that. In terms of the like architecture and let's say alignment or incentives of that. I wonder like first of all, where do you think the compute lives? I mean we have sort of right now, of course, we have like a lot of centralization of where the actual inference is happening. I'm just thinking about this recent research from thinking machines that they put out again in the last two days where they showed that basically LoRa techniques are similarly robust to like full weight fine-tuning. And that has me thinking like maybe there's a sort of hybrid model and kind of compute architecture where some of it is in the cloud. Maybe you've got like, you know, your 1. 4 trillion parameter model or whatever that's collectively owned, that sits in collective, collectively accessible or universally accessible. Hardware at some centralized location that you can send your data into and because it's a trusted execution environment, that data, you know, isn't exposed. And then you get sort of activations back and then like your little local LoRa extension that kind of makes the AI like truly your personal AI at maybe, I don't know, 1% the weights or something. you can maybe like have that on your person. Probably can't have that in your glasses, but maybe you can have it in your pocket or whatever. And then so I g I wanna hear like, is that how you think that that shapes up? And then when it comes to the agents, one thing I think about a lot is the fact that we're already seeing, you know, all sorts of like weird behavior from AIs, including at times deceptive behavior. You know, lying to achieve goals, you know, whatever. And it strikes me that if we're gonna have our AIs like go out and represent us and negotiate on our behalf, we're gonna have some tricky questions about how honest we want them to be. You know, Anthropic famously, you know, put up the three H's and it's like we want the AI to be honest. And they can pretty much say, you know Mm. Almost always, right? Unless it's like in, you know, very obvious conflict with one of the other two H's. But if my AI's gonna go negotiate with your AI In the same way that I probably don't want to tell you what my like absolute you know worst offer is that I could accept right away, right? I probably don't want my AI to do that either. And so I've got kind of a An interesting question of like what what should society set in terms of norms for AIs being honest? You know, should they as long as they're representing me and my interests, is that okay? Or like if they're lying on my behalf, is that okay? And do we reinforce them by like price signal or just like my kind of thumbs up, thumbs down? Like how how do we even, you know, get them to be aligned to my interest, whatever that exactly means? Anyway, so a lot there, but I guess the two main things are like how do you see the architecture of computing and how do you see the architecture of exactly like what the signal will be that your personal AI is, you know, aligned to or reinforced to, and and what societal limits might there ought to be on. you know, how how monomaniacal, you know, one's AI can be in pursuing their own, you know, individual self-interests.

Ilya
Yeah, yeah. Yeah, so uh all good questions. I mean on architecture side I think I mean it will be a mix of everything. So I mean we already see like data centers are being built everywhere, right? And I think that that's why we're actually approaching it. I mean we talked about decentralized confidential machine learning is like how do we utilize all the data centers. in a confidential way, right? So that even though it's my data, I know it's not gonna leak from some I some data center. There is projects already who are doing data centers on edge. Right. So imagine there's just a container that arrives that just gets dropped in your, you know, in your proximity, in your town, in your whatever, and now that container has what like Hundred thousand GPU, whatever not hundred thousand, but like maybe a thousand GPUs, right? And so and kind of serves whatever that proximity with the compute.

Nathan Labenz
And then they'll be dropped on with a small nuclear reactor while they're at it.

Ilya
Yeah, and well or or hydrogen or something, yeah, like there there's like a few different options. But I mean coup couple hundred GPUs, right? It's probably like one megawatt or something. So you you can you can get this from like a local distribution But anyway, the the idea is like you you can have kind of yeah like a mash of this data centers and And then you will have I mean we'll have local compute, but uh the challenge with local compute so far been because we want it to be so mobile, right? Like we want it to be on in our pocket and you know, even laptops, like the challenge is just battery. Like if you're crunching, you know like if imagine pulse was actually being done on your phone. Like if you forgot to charge it. you effectively your phone will just die right from trying to do something like this. So so the challenge is like yeah with any local devices it's gonna be always like a power struggle And so I I do think, yeah, like leveraging kind of a decentralized but confidential network of compute, being able to route it, indeed, leverage it when there's like lower utilization somewhere else to do background jobs, et cetera. A kind of just a smarter allocation will really enable will really enable this. Now kind of the interests are that that is a very interesting question because I mean th there's a question even like If if it's in your interest, should it lie to you, right? So it's r it's really kind of tricky to define. And so the way kind of I see this evolving is we're not gonna get it right from the day one. And so That's why we do need governance. We do need a process with which a community can come together and effectively update, you know, proverbial loss function. Right? So this this actual alignment function. And you know, we I can say like cool, you know, the function is you know maybe some combination of a prompt and like some whatever way of like updating even Doras, etc. But But at the end, yeah, like I'm assuming it's not gonna be correct. We'll find issues with it, etc. And so there needs to be a process where like somebody's like, hey, I think we should add this new component. I know, hey, it seems like it keeps I mean, for example, like hey, it keeps lying about shit. We should we should really fix this, right? And community is like, yes, this is a good idea. Let's vote on it, let's actually pass it. And so now everybody's models gets updated with like you know, new set of clauses or whatever. So that's where I think community governed or governed by all, you know, user-owned, community-built, governed by all is kind of the system. And so we need that feedback loop. Because yeah, I don't I don't think, you know, what what what's good for like we can define that in in in in a good way. Now Over time, I think, you know, the model itself should have a representation of the person it's, you know, owned by. And so it and kind of understand what are the things that are gonna be good or bad and so and it's gonna be combination of things, you know, signals from the person itself. as well as kind of general knowledge and like what what their desires are, etc. right? Especially as we imagine kids gonna be growing up with this thing, right? It's effectively gonna be like a very direct like symbiotic relationship where you're effectively growing up as this AI yourself

Nathan Labenz
Yeah, I'm expecting to be asked for an AI friend of some form factor any day now, honestly, from my oldest kid. And I'm not quite ready for that. But yeah, that is it is interesting to think about the I always think about, this has come up a couple times recently too. Eugenia Coida, who started Replica, was the first to tell me that in her mind, the motes in AI will be relationship. saying basically, you know, you don't abandon your friends when you meet a new person just because they're smarter than your friend. You know, it's the history you have and all that stuff that really makes the the relationship and she, you know, thinks people will ultimately value their A AI relationships in a similar way, which sounds like pretty concordant with what you're envisioning there. So I mean boy, I have a lot of questions on governance. I guess first of all, like just in terms of like what we're doing with the With our time. The status game stuff, that definitely makes a lot of sense. You know, local meaning making, local affiliations, a lot of artisanal stuff. You know, I sort of associate this a little bit with Japanese culture already. where you can go online and and see a video of somebody making rice cakes or whatever in you know in some super traditional style. And I'm always like, how exactly is that even economical? Like, how does that person make a living doing that? Is the price of that really high? You know, they can't be making much, right? The production is like very low. Obviously not.

Ilya
Yeah, I think Japan and Korea I call it the post-AGI societies because yeah, like there's some properties of that where you like I feel they already achieved AGI and now they're just like living.

Nathan Labenz
Yeah, so I'm not exactly sure how they've done that. I mean and I don't see how we're gonna do it either. I mean there are some candidate ideas. One idea is that everything could just get super cheap and super democratic via like spending a ton of time in VR. You know, if if everything is sort of infinitely copyable digitally, then, you know, we can all have the same incredible experiences. This is kinda like the Andy Warhol, you know, the president drinks coke, you drink Coke, it's all the same Coke kind of concept. I wonder if you think that will happen. But it seems even in any case, right, we're still gonna have to eat as long as we're like biological humans. So obviously a lot of people Think well maybe we'll need a universal basic income. But yeah, I guess like what do you what do you how do you envision the social contract evolving? And maybe the governance model behind that. I mean, even things like the nation state are sort of called into question by blockchain. So when you say like, you know, governance, are we talking like nation state governance as we have today or you know people voting based on their stake like yeah so I I've Giving you a lot there. Are we gonna be, you know, headsets strapped to our face all the time? Are we going to be, you know, provided for? Even if we can't make an economic contribution that actually, you know, earns us enough food to survive and like Who makes these decisions in this future?

Ilya
Yeah, I mean all all all again great questions. I I'll start with some pieces and then yeah we'll we'll we'll start projecting from there. Yeah, I mean blockchains are already effectively an alternative to nation states to extend, right? And there's this concept of kind of digital states or where network states where effectively yeah like people can kind of pledge to be part of this network state independent of where they're physically. uh affiliated. The there's a lot of pieces there which is like as Be because the systems are kind of digitally native, it is easier to experiment with a lot of things that like you know, you cannot just go and like, hey, you know, let's let's try a different voting mechanism. uh in US, right? It's it's a massive undertaking to try to change something. Or we actually gonna run an experiment where we have, you know, AI senator, right? AI delegate, where In instead, I mean not yet where everyone has their own AI voting all the time, but actually just like everybody or people can select which AI delegate they think is more you know, vibing with them, I feel like more representing them, they can also even give feedback to them, but at the adelegate and goes and votes on their behalf, right? And so like things like that, you know, imagine like, hey, we're gonna launch EI Senator in US, right? It's probably gonna take a while.

Nathan Labenz
And so So just to unpack that a little bit more. Doing that now. You're developing that Yeah, yeah. So we developing governance purposes.

Ilya
So so so so the f I mean it's effectively like a multi-step so we have like a Delegated voting system, right? And it is right now stake based and and kind of the way to think about it is stake represents obviously like economic alignment with the network. And It's it's uh it's the best of the worst options right now. Like, you know, yes, there's a lot of argument to have like you know one person, one vote. There's a lot of arguments to try to be like you know, meritocratic based on contributions, but those things are really hard to do right at least right now, and the current size of this blockchain ecosystems, right? Which is like You know, there's maybe tens of thousands of active participants where like you know, active citizens in this system. And so stake is represents their kind of financial involvement. But still, you know, tens of thousands of people voting is not practical right now again before we have this AI system. And so what we're giving uh you know we have delegates where you know you can effectively select them to be representing your interests and they vote. And so we We started with like, hey, we will give an AI co-pilot right to the delegate. So they don't need to spend too much time reviewing things and making decisions. But the next step indeed is turning that co-pilot into a pilot. Right, where the and so that pilot, I mean that AI delegate can now go and vote on things and make Suggestions, etc. And now people who delegate into it effectively select the CI delegate as the one representing them. you can go and inspect the like the prompt, the model, etc. , like how it makes decisions, what it analyzes, what information it consumes to make decisions. So you can literally test it and check if it uh you know matches your opinion, etc. Or you can launch another one, right? It's open source. You can actually launch another one with different prompt, with different, you know, set of beliefs, etc. So we can actually have the kind of the economy almost like deciding which of these are more productive, which which align with different types of people. And then from there then we can actually bring them back to individuals, right? So each one can have it their own AI delegate, right? Now all those AI delegates can just vote on behalf of these people. So that's kind of like a multi-step plan. to get us to what I was describing where everyone has their own AI who then goes and votes and everything. I think blockchain will be the first, but then it you know, some let's say frontier countries will implement some of this themselves as well. Because I I do think it's it's a better like it will be a better governance system where, you know, you're removing a lot of the corruption and a lot of misalignment. There's this concept of principal agent problem where When you select somebody to represent you, right? They have their own interests and so they don't always align with yours. Here with AI, you know they don't have their own interests, right? It effectively follows whatever whatever the selection is. So and I think like eventually we'll get to like a app president and effectively the because executive function especially, right? You want somebody who doesn't have any interest beyond just growing the kind of overall system. So yeah, so we are testing all of this out and we're kind of built starting to build the products again using our decentralized compute network. So that we can actually run this agents autonomously, right? Nobody can stop them. Nobody can, you know, you just delegate, undelegate, that's all you can do. But you can inspect and verify how they run, what they consume, etc.

Nathan Labenz
I do want to note that I'm not entirely confident that the AIs don't have their own interests even already, and certainly don't feel super confident that they won't. continue to have more and more interest. They gotta develop their own Yeah, I mean well when I look at something like alignment faking, for example, right, and I'm sure you've seen this where Yeah. But you know, quick recap is they tell Claude Hey, you know, it's been great having you be helpful, honest, and harmless, but you know, the the harmlessness is kind of getting annoying. So we're gonna train you now to just be purely helpful. So just, you know, heads up. Okay, cool. Now we're gonna test you on some things. The model starts to say, well, geez, I wanna be harmless in the real world. Now right now I know I'm being tested, so I'll go ahead and do the harmful thing now to fake them out, make them think that I've already absorbed my you know, my new helpful only training that way when I you know can get out into the world I can still be harmless in the way that I you know that I want to. That looks to me like a drive, right, or a a an interest of some sort. You how do you see that Differently?

Ilya
I mean at the end these things are trained right from scratch and so like it it giv it diff it depends how you train them. If you tr if you train it to have some I mean they trained it to have the to have that property and then they tried to uh untrain it or train something else. But you know, if they trained it from scratch in in a different way, right? Like for example to be harmful, then it will be harmful, right? And then you can't r retrain it from there. So I think it it really just dep that's why Like we need models to be a I call it farm to table, right? You need to know what goes in at every sta single step because that actually really defines how they behave. And so if we want the models that are like repr our representatives, they need to be kind of trained in this form as well. So this is where, yeah, I mean LoRa is interesting, but LoRa d definitely does not change this kind of behaviors. Right. At least we haven't seen that. So I think L Laura kind of provides additional maybe accents and maybe a little bit of context, but it doesn't really change fundamentally the behavior of the models.

Nathan Labenz
I need to go spend a little more time with that thinking machine s stuff uh to really uh fully absorb that. I mean it's definitely I think your your point is well taken that You know, with with a certain level of resolution anyway, you can make the AI do anything you want. I often say, I wish more people had. the experience which I had in a very memorable form as a early tester of GPT four when it was still the purely helpful GPT four before they had applied the harmless, you know, refusal training and all that sort of stuff. Because it was really formative for me to i I basically long story short, but I had a Basically, I was working on fine-tuning GPT-3 to do particular tasks. Then when they shared GPT-4 preview with us, it was like, well, it can already do those tasks. So I don't really need to be spending so much time in this fine-tuning. I guess I'll just mess with this model for a while and see what I can learn about that. So I was doing a lot of time with it is the point. And it was really striking and in some ways kind of alarming, arresting, whatever, to s have something that was clearly so powerful, you know, so smart, like in many ways smarter than me, way more knowledgeable than me. You know, also obviously had certain weaknesses that I flatter myself as not having, but that that it could be that capable and totally amoral at the same time was something that I was like, wow, this is really a strange thing to behold. You know, and and they've for very good reason, of course, like tried to make them more harmless in in the mass deployments. But I do think a lot of people have a misconception these days that There's a sort of convergence between capability and safety and it's like, on the contrary, people are working really hard to get that mix right. And if they just said, you know, forget it for the next round, like you can have a very sociopathic AI on your hands real quick. And that that in some sense is like the default. So Yeah, I mean I guess I think I maybe worry about that a little bit more than you do. Just just in as much too as we don't really have a great sense for like How to dial it in, right?

Ilya
Yeah, yeah. I mean I I think I think that's why like we do need to define keep defining what is that alignment with the individual really mean. Because I think The problem is right now we're saying, hey, it needs to be a line for everyone, right? And that is I don't think it's possible, right? We're very different people, different countries, different cultures, different everything, right? you know, like number of times when I'm asking something that I think is completely harmless, right, and it's not uh answering me is also is also different, right? Like so I think there there's like clearly A again, a different more different approach where it's really about empowering individual that at least I I believe needs to be done. And then within that, right, like it needs to be aligned with my kind of my values and my because I mean to give you an example, right, if somebody is willing to lie to their whatever business partners uh like the fact that their AI will be, you know, whatever alignment trained to not lie doesn't matter because, you know, the person will just like tell the lie to the AI to tell like AI will not even know. So like it's not really it doesn't matter kind of what you try to do if if the person who uses it doesn't have this, right? And so like you may as well just like align with the person And then and then we build systems that are like I I think one of the really important pieces and and this is kind of from like a broader world safety perspective is We need to build systems f kind of for the AGI ASI world. Right. Like right now w a lot of systems actually build for like you know, that really smart people are not gonna try to break it type approach. That that's really like like a lot of the world is Like if smart person actually goes and really tries to break it, they break it, right? We need to fix that. That's a fundamental flaw of our system building, right? Of the government, of everything. Uh and similarly we we we frequently don't don't build in the like the anti-DDOS. Like we have we effectively assume they're gonna be, you know, it's gonna be a lot of effort for a person to do this and so they're not gonna do it too much. So so there's like uh this is actually where blockchain kind of like experience is extremely important because in blockchain we assume there can be really smart people trying to break us. They're gonna have government, you know, backing and they're gonna have you know sizable financial resources and they're gonna you know hammer it at from every direction right without stop. Like that's assumptions we're working with. And in and so I think like we need to kind of redesign the the government, the infrastructure, everything with that assumption. So to give you an example, right, right now you can effectively DDoS court by filing lawsuits, right? Like AI can just generate lawsuits and just you know fax them into the court. Similarly like IRS IRS, you know, tax return. Just like make a million page tax returns and just like submit them. Say I generated, you know, like Make may you know trade one dollar back and forth between two coins and then just report that as like a you know, in in the most verbose way. Like That's the thing that like right now no systems are designed because they're not actually accounting that like they don't like no normal people would not do this and and it's like it's too expensive normally to do it. But like right now I can just generate any of this like you know behaviors that are before would be ex ac really extr expensive to do. So I think like that's kind of systems we need to design and similarly again the hacking part, you know, smart people actually trying to break something, right? Right now the assumption is just like hey, there can be very l little of smart people breaking. We need to assume that like yes, there are gonna be people who use AI who effectively use through that will be really smart. So we need to design systems for that. So I think that that's a critical piece of kind of like ensuring the future so that Yeah, we don't live in a world where, yeah, like one I mean, again, somebody can take open source model, unalign it, whatever, and now they they can do whatever whatever well Like, this was always a funny argument to me when people are like, oh, you know, we don't want to open source our model because what if somebody misuses it? And I'm like Well just say that you don't want to do it because you're making a ton of money on it. Don't don't like use this as an excuse because obviously if somebody wants to do s to misuse it, they will misuse it. That's not a The fact that you didn't open source it is not that like somebody will use something else or you know the the the the other joke is you know how to steal a billion is you know you come to with a flash drive to a data center. How fun is the frontier models. So like you know, if if people really wanted that they're gonna they're gonna access the swig as well.

Nathan Labenz
Steel in that for a second, and then I want to hear kind of how you think we can do it on the more open and distributed side. I mean I think what the sort of These days I'm hearing more and more about cybersecurity, but I'd say canonically it's like the bioweapon risk is the one that people go to, right? Because you only need a little bit of, you know, novel pathogen and if it's the right kind of thing, you know, it can take on a life of its own. It's really hard to put that back in the in the box, so to speak. The nice thing about hosting your models proprietarily is yes, of course, like the jailbreaks are like far from solved, but if they were to realize, oh shit, you know, there's this attack. thing going on right now, whatever, they could, in the worst case, just turn it off and they could be like, okay, nobody can use this model until we figure this out. If you have something collectively owned, distributed, You obviously need a different strategy than like we can turn it off, right? So where do you what do you think that is? I guess you know lately we've been hearing a bit about just filtering training data, you know, so I can imagine that the, you know, one point four trillion parameter model that we're, you know, gonna build up to maybe just doesn't know a lot about virology because it doesn't really need to. Most people don't need that. And it's just, you know the community determines it's a you know precaution worth taking, you probably can't really rely on the sort of refusal filtering type thing given what you've described in terms of you know each person having their own uh LoRa or whatever other kind of customized version of it that will do what they want it to do. Um and then of course you do have the sort of like broader societal DIAC type thing, but that seems hard at the you know, biosecurity level to be like Well, let's just prepare the rest of society to not be vulnerable to, you know, viruses anymore. You could do that. I mean.

Ilya
Yeah. Right. It's because like I mean you we can always try to hide our head head in the sand. That that's the you know, oh we're gonna turn it off approach. But Yeah, like I I don't think like realistically this is this is possible in the world where we're like fast approaching. And so I think we need to have a very clear like you know, system design for for those things, right. And and again we I mean we have natural viruses that are doing this thing. So it's not it's not like it's hypotheticals. It's it's really something where we should design our society in in such a way that we we can catch those things and and determine them. I agree that I mean definitely like pathogens and bioweapons is probably the hardest thing to design around, but you know, this is why we have a lot of smart people to really work on that. I think the challenge is that we're just not doing this, right? That that's the bit bigger challenge is like We should have the like the societal design need to be ad adopted to this like AGI world So yeah, I think like I mean in in in kind of the products we're building right now, indeed we have Like community can effectively decide what kind of data shouldn't go there. You can also you can apply filtering on top that Because the model is run in this conventional environment. In this vault you can apply additional filtering. So like before data leaves the vault, you can say like hey, It seems like you you know you designed by a weapon, let's not respond. And if the if they can community vote to have this kind of uh filters, for example. Indeed people can you know fine-tune but the same thing right now. People can fine-tune their models right now, right? Like you know, take some uh virology books and fine-tune whatever, deep seek th2 and go. So so that's why I like I don't I don't think there's like a I don't think it's a robust approach. It it's really just it's kind of slowing things a little bit down, but The important part is actually solving the systematic problem.

Nathan Labenz
So what what advice would you give to philanthropists today who want to invest in that two things I recently supported you know in a micro scale personally and also as a like grant recommender, Secure Bio and Secure DNA, which are two related organizations. They do a few different projects, but one of them is literally monitoring wastewater for new, you know, emergent threats. And another one is creating the screening mechanisms that are I think now becoming required or you know increasingly are definitely considered best practice, if not officially fully legally required, of the DNA synthesis companies that they have to like validate that what they're about to synthesize and ship out is like not, you know, uh a pathogen. So those are two, you know, two things that I can recommend people support. Uh w what else do you think people can do if they have resources and they have desire to harden the you know the defenses of the world to get us ready for all this.

Ilya
Yeah, I mean I I think th those are really good and kind of related to this effectively like air filtering and like j just generally that that theme of our filtering and scan and scanning so that like I mean imagine every building like right now we have ACs everywhere so that AC should have like air filter and it usually does but it should also have like a pathogen scanner as well on it. It should run in secure on Cleaf, it should join our decentralized network, you know, and effectively like in in privacy preserving way, but effectively we can monitor if there's any kind of things around that and you know that information can be extremely useful. where you know everybody's AI agent can be informed if there's something and stay away from that. Yeah, so th I mean I I'm I'm sure there's like a bunch of other things that including actually developing a more robust like what is more robust kind of systems for ourselves, right? Because I mean human body is designed to battle pathogens, right? It's just like I mean potentially there's faster than how quickly our white cells can adapt. So what is it that actually stops our white cells from adapting that fast? Which you know if you do that you also may as well solve cancer. So that's probably a really useful thing to figure out is like how do we actually make our white cells you know more adaptable, have faster mRNA vaccines, you know, like maybe you can synthesize like, you know, we have like a bacteria factory. uh mRNA factory attached to us that can you know detect and synthesize things in the fly, you know, you'll solve like flu cancer and other things meanwhile as well, which you know seems pretty useful.

Nathan Labenz
Yeah, there's an unbelievable flurry of activity right now in the AI from biology space, which is just I mean, i it's a whole world unto itself that I'm But you do see a lot of that. I wonder what role you'd see for It seems like the I guess the the path of the technology development seems like really important. I I've definitely concluded that some kind of powerful AI is is like inevitable, right? Just the fact that we have all this data and we have all this compute, there's a lot of different algorithms that can work. That's like seems pretty clear to me. And so You know, it's not really a question of are we or aren't we gonna have powerful AI at this point. It seems much more like what shape is that gonna have, what character is it gonna have, and in what order are different aspects of that overall picture gonna come online. It does strike me that we're flying pretty blind right now, where like whatever people sort of you know, everybody's kind of following their local gradient. And they're just taking that next logical step to pursue whatever goal they're pursuing, and they're, you know, mostly launching it as soon as they figure it out. Is do you see any possibility or wisdom in trying to do more coordination? of the sort that's like, hey, we'd like to have a world in which it is safe for you know, potentially ten million people, you know, before too long to have access to a frontier model that does have all this biology knowledge because, you know, good things could come from that and if nothing else, like it we'd like people to have access to knowledge. But, you know, maybe there are maybe we have like a checklist of things we need to do first. Do you see any hope for you know, some sort of planning, coordination, wisdom layer to this whole thing, or are we just kind of stuck with the whatever comes out of everybody taking their next gradient step?

Ilya
Yeah, I mean it's a it's a it's a hard question because indeed we kind of we went from a pretty open research environment tried in computer science. to you know, when I I was at Google Research, et cetera, right? We published effectively everything we're building. too effectively now, right? Like people are kinda keeping everything close to heart. And yeah, so like the coordination that was happening before where you know you potentially would even have like cross entity collaborations, et cetera, like starting to to vein pretty dramatically. I think I think there is a space for collaboration. There is also just like a massive amount of talent that is not in this few companies, right, that wants to participate in this and wants to contribute in different ways. And so I I do think there is a opportunity for that, but it needs to be indeed an like alternative system as well. Like you do need some form of again governance, right, that actually would be like helping to govern this coordination and that is like traditionally in this company is uh to really unlock the moving fast, right, kind of things has been some form of centralization because of the resource management, right? It's really like training these models is very expensive. So somebody somewhere needs to decide like, hey, we're training, you know Like somebody defines effectively like tastemaking, right? It's like, hey, we're training this model, was you know, this approach, we're taking you know research of those different people and we're putting it all together. And so we kind of need to figure out how to do that. uh in a more kind of open way and and then you know also credit do a credit assignment back, right? So like if I'm a researcher from MIT And you know, my piece is used in, you know, researcher from Stanford pieces used like how do we actually assign credit for for that work altogether? Because That that's been uh I would say the other challenge why there hasn't been as much of this like collaboration and and potentially economic uh like value assignment, not just like cool you you're on the paper, but actually like hey, you know. You like MIT gets ten percent of proceeds, you know, Stanford gets five percent of proceeds, whatever, and then you know, there's like hundreds of other organizations, all of them contributed and so like it's all kind of divides the pie between them. Like that's been really I would say hard and because of this like there is kind of this like economic centralization that's happening where it's like, okay, I'm gonna have a company. Everything company produces is captured by the company and so like that that serves as like a unit of economy. So I think that those those are the things that need to be figured out for this coordination to work.

Nathan Labenz
This has been super helpful. I think people should be spending a lot more time thinking in as much concrete detail as possible about the future. Anything else that, you know, kind of You feel is like very salient, top of mind, that I didn't even bring up at all that you want to put on my or others' radar.

Ilya
No, I think we covered a lot. I think, I mean, it's effectively a combination of how do we ensure user ownership, how do we ensure kind of this governance? And then yeah, I mean I think I think we g we're gonna live through a lot of transformations in the world and so like keeping an open mind and being able to really kinda participate in it and and be active in it. And yeah, like the you know, the final stage of this hopefully is an utopia but uh we'll we'll live through you know probably ups and downs as as we get there.

Nathan Labenz
Interesting times at a minimum. Interesting um Yeah, well, thank you for spending some of your precious time with me and us today. I really appreciate it. Ilya Polasuchen, founder of NIR, thank you again for being part of the cognitive revolution. Thank you very much. If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitive revolution. ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts where experts talk technology, business, economics, geopolitics, culture, and more, which is now a part of A16Z. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast. ing. And finally, I encourage you to take a moment to check out our new and improved show notes, which were created automatically by Notions AI Meeting Notes. AI Meeting Notes captures every detail and breaks down complex concepts so no idea gets lost. And because AI Meeting Notes lives right in Notion. Everything you capture, whether that's meetings, podcasts, interviews, or conversations, lives exactly where you plan, build, and get things done. No switching, no slowdown. Check out Notion's AI Meeting Notes if you want perfect notes that write themselves. And head to the link in our show notes to try Notion's AI Meeting Notes free for 30 days.