Coding is a solved problem, says AWS’s Jonathan Allen. Judgment is the new bottleneck, and it has to be built. Adam Blue and Jonathan trace three ways to build it: training judgment like pilots train for a crisis, using the pager as a forcing function for ownership, and practicing skills deliberately instead of letting AI do all the thinking.
"The Mythical Man-Month," by Fred Brooks
Join Jonathan Allen and I on our latest Cut to Context, where we talk about the future and nature of work, escalating expectations about what AI can achieve, ideas about how we’re going to bring along the next series of software engineers and participants in the product, and ideas like cognitive surrender get examined on this episode of Cut to Context, which you are not going to want to miss.
Adam Blue
Hi, everyone. I’m Adam Blue and welcome to Cut to Context. I’m joined today by Jonathan Allen, executive in residence at Amazon Web Services. I’m joining you today from the most liminal possible space that I could find in North America. This is a hotel in Palo Alto, so I am literally at the white-hot center of AI today for a very special Cut to Context.
So thanks for joining today, Jonathan. You spend a lot of time, I think you said you just came in from Japan, which is amazing. You spend a lot of time out on the road talking with folks that are implementing AI, and Amazon has, I think, 26,000 or more agents in production now. So maybe let’s start off with, just give us a sense from your travels in the field, what’s working and what’s not working in agentic AI today?
Jonathan Allen
Hey, Adam, thanks for inviting me. Good to see everyone.
What’s working and not working today? Interesting. Most of my executive conversations right now, and I have the privilege of speaking to between 150 and 200 leadership teams a year around the world, every CXO is laser focused on it that I’m speaking to, and they’re now well beyond the experimented stage. We’re into the lessons learned stage, and the lessons learned as you look at it are kind of like threefold. One is everyone knows out the loop is pretty dumb. It’s where we started and where we were vibe coding. Then immediately we are trying in the loop and then we’re also now learning some of the painful failures of being in the loop. And then we’re moving to a better place of being on the loop with AI. And before I go into any of those, of course, the big thing that’s happening is there’s kind of two paradoxes going on right now on the technology front.
Obviously in the last year we’ve seen the development and emergence, the last 18 months I guess, of Claude Code and then Amazon Kiro and Spectrum Development and then the agentic harness coming in for development. And that’s just in a record time had a radical change in what’s happening to the world.
And then of course for our business colleagues, they’re now realizing leveraging agents in production is and continues to have and will continue to have for a foreseeable future a dramatic impact on actually their value, their output. And then that isn’t even getting into remotely, obviously the lessons learned of us as individuals, as professionals of using these tools. And they’re in the huge spec of conversations around the impact on us as people, which is just changing absolutely daily.
So across all of those things, massive privilege to have those conversations. And I’ll end that question by saying I think one of the really interesting aspects of my role is I kind of have this really privileged seat to work with customers that are 10 years in the future and customers that are probably 10 years behind where the current practices are. And I find myself continually digging into my tool bag to help them and then learning from those in the cusp of that 10-year period, as well.
Adam Blue
Interesting. Interesting. Well, it’s a great survey of where folks are. I know one of the things I think about and we’re starting to talk about more now that we’re, I don’t know, I’d call it realistically 24 months into usable AI, is there’s this notion around cognitive surrender. And I think it’s maybe a little less traumatic than the term implies from the two words put together. But the idea being that when human beings are using AI, their performance is often lower over time in performing tasks without AI once they start to use it. And I think that the human brain, and we’ve talked about this before on the podcast, it’s kind of a recurring theme. The human brain is radically efficient around the conversion of calories into neurology and it’s just the way biology works. And so our brains are rife with shortcuts and heuristics instead of real analysis and snap decisions.
And it’s all in the name of efficiency effectively. Your brain is lazy. You have to force your brain to work hard and your brain is like, I don’t know, 5 to 7% of your body weight and consumes 20% of the calories you consume, which is just completely outsized. So you are a meat robot operated by a human consciousness. So when you start offloading some parts of that capability, and we went through this with Google and cell phones and people said, “Well, nobody’s ever going to remember anything anymore.” And I don’t know my wife’s phone number. I have to ask her all the time what’s your phone number? Because I don’t dial it. I don’t look at those digits.
So I can feel that we’re on the cusp of this cognitive surrender mechanism. And you’re talking to a lot of folks and I’m wondering what is the feedback you’re getting from them and how are they thinking about it? And is there a movement to try and resist it or mitigate it or do we just leap in with both feet and start doing a completely different set of things and give up the set of skills that agents can perform?
Jonathan Allen
I think I’m going to reject that premise, Adam, cognitive surrender. It’s a really interesting phrase of words. And as you were kind of telling me that, I was looking at it and I was like, look, the way I’m using both my agentic harness and the way I’m leveraging multiple lanes, that doesn’t reflect what I’m seeing within leaders play out. What I’m seeing is for me personally and those leaders, and I had this really interesting conversation with a really senior CIO in APAC last week, and they’re saying there’s a few things happening to individually how they work and I work, which kind of resonated with me, which probably touches on a little bit on what you’re saying.
The first thing is that we’ve removed ourselves from using UXs in an extraordinary rapid period of time. It’s like, please don’t give me a UX. Just give me an MCP or an API that I can talk to via my agents. And that was really interesting insight.
The second thing is, though, that the idea that we can cognitively surrender what we’re doing, I think I outright reject that because when you look at where the long pole in the tent has been previously with anything we’ve been trying to accomplish in business, primarily in the binary world, it’s been the writing of software. And that has now shrunk to fait accompli. That’s done. I haven’t written a line of code since March and I don’t think I’ll ever write a line of code again. Yet the ability for judgment to come into play is now extraordinarily demanding in what we’re doing. So all the bottleneck is now in the judgment. So I wouldn’t call that cognitive surrender and judgment. If anything, I’d say the bar has risen on that.
And I find myself now staring at the research that’s been done, reading it thoroughly, and actually having to use a lot of brain power to figure out, OK, is what the AI just told me, is it biased? Is it real? What’s the proof? Have we checked the proof? Where does this lead us? What’s going to be around the corner? What are the tangential angles? So I would say in a way, the boring stuff has gone. The bar has risen on the judgment for you as a leader. The bar has also risen on how you do your job and the expectation in you is higher than it has ever been along with the quality bar.
So unless you are really good at offloading the parts of your job that you need to, to whatever system and setup you are doing and are focusing on that judgment, to me that judgment is now harder. And not only is it harder, I can now go across multiple different fields at any one time. And I find that in both my personal projects and my work projects. So that’s causing a really interesting behavior though, Adam.
So one of the things when we talk to leaders, and both we’ve seen this inside Amazon as well, is there’s been a propensity to hire more senior individuals, not just in leadership roles, but in development roles. Who wants to take on the graduates anymore when I’m hiring now for judgment, not your ability to code and have Python syntax expertise or even understand data structures. I now want your ability to drive an agentic harness or to make a judgment decision or to expand from your role as an SDE to be also, how should we be doing the UX? What should we be doing in testing? What should we be doing? What product should we be developing? Where should our partnership be next? And that has created some really weird things in the last year where people have almost stopped hiring juniors. I think there was a figure from, was it Ravio, 77% less technical hiring in Europe for junior roles and that is just not good.
And I’m pretty proud inside of Amazon that we’re deliberately hiring 11,000 grads and interns into those roles so we keep developing that next wave of talent. But that is forcing us to focus on judgment skills. We’ve got to focus on judgment. So I guess that’s a long way of me saying I kind of reject the thesis of cognitive surrender. I do accept the fact the bar has risen.
Adam Blue
Yeah, I think it’s an interesting perspective. So let me give you an example, and this is from my own personal exercise. So I built a feature with Claude. Me and Claude hang out a lot. We make features together. So Claude and I make a feature, a feature doesn’t work. And I asked Claude to figure out why the feature doesn’t work. And Claude comes up with a pretty convoluted explanation that involves the Nomad infrastructure blocking particular requests that have a particular shape. And then I thought that feels unlikely. And it’s a great answer. It’s well thought out. It’s very specific. It’s so reasonable. It’s so reasonable.
And I look down at the command line and I had not pushed Enter on the CLI to deploy the change. So I had tested the old version of the code that did not actually functionally contain the change. And then I told Claude it didn’t work. And so Claude has constructed for me this elaborate fiction of why it doesn’t work. And the reason it didn’t work is because I didn’t press Enter on the command line that was sitting there ...
Jonathan Allen
How good do you feel?
Adam Blue
... staring me in the face. And I just thought, I’ve had 30 years of being bad at software engineering to develop enough skepticism to arrive at that conclusion pretty quickly. And to your point about juniors, I don’t worry about them not learning Python syntax. I think I worry about them not developing the intellectual skepticism and intellectual curiosity that leads them to say, well, how does this actually work? What do those packets look like on the wire? If I read that protocol trace and I find out that Windows boxes send out WINS packets in reverse direction to incoming socket connections in that way, that’s why it takes three seconds to get this socket open across s boundary that doesn’t pass those packets. They don’t learn those things experientially.
Jonathan Allen
I completely agree with you. I’m totally aligned on that one. But interesting if I may interrupt you. One of the really interesting things that I’ve learned, and I’ve had the privilege of engaging with a number of CEOs from airlines, and they’re like, “We’ve had this problem before in humanity.” And I’m like, “Really? Tell me, we’ve had this problem before?” And they’re like, “Yeah, if you think back over the 80, 90 years of aviation, if you’re a pilot coming in to an airline, the first time you will fly maybe an Airbus A320, the first time you’re going to fly that as a pilot, there’s passengers on there for real. You’re not going up in a trial A320. You’re flying it for real with passengers. You’re in the right seat with the pilot in command in the left seat. And how do we do that? Well, not only do we put you through 13 exams, the ATP exams, you’ve got to go through a heck of a long time in a simulator while we throw every single scenario at you that allows you to build the muscle memory of judgment. We’re going to give you those aviation lessons learned over those 18, 90 years. You’re going to experience them for real.”
And how do we now duplicate that experience that a junior pilot goes through in our engineers, in our business development professionals, in our marketeers? Does that mean we have to get them with something like Amazon Web Services fault injection simulator? Do you need to see what a SQL injection attack looks like? Do you want to see what a network storm looks like? And I’m increasingly thinking the answer is yes. We actually need a completely different way of onboarding our engineers. Either the universities can do this with us or we as professional entities need to also look to develop that with them so they can go through those same experiences you and I have had as I have had when I was a network engineer dealing with a network call failure, dealing with a DDoS attack.
These things, we’ve got to feel it to know what that feels like, to build the calibrated paranoia to deal with these situations rather than just relying on what the consoles told us.
Adam Blue
I think we’re collectively as an industry unprepared for how to move people from entry level to the first stage of experience in a post-AI world.
Jonathan Allen
Agreed. I think that’s something we’ve got to solve. We’ve got to solve it. I was speaking to a whole bunch of universities recently and they are waking up to this fact that the way that students are learning is changing and has got to change. And one of them was said, “Well, how do we teach organic chemistry when they can just cheat with AI?” And I’m like, “You’re looking at it the wrong way around. Your job should be how can they use organic chemistry with AI? How do you use organic chemistry with AI safely? What is your responsibility? How should you be getting the most out of a language model? How should you be using a literature store to assist you maybe with a RAG database to assist you in the development of whatever that theoretical topic is?” And the moment I sort of changed that paradigm around with the deans, they were like, “Huh, can our professors develop that course curriculum to do that?”
That’s now the new normal. And I’m saying, yes, that is the new normal now. Just saying you can’t use AI because you’re going to cheat with it is completely the wrong paradigm.
Adam Blue
I agree with you and I like that approach. The only other thing I’d say is I think it still has value to ask people to do things without the assistant some fraction of the time. I think there’s still value in that. Human beings are physical in a way that artificial intelligences are not, and there is a certain amount of muscle memory and experience that leads to learning that is outside the intellectual realm. I think it’s in “The Mythical Man-Month.” It’s like in the first chapter, I think he talks about if you want to have a team, so get them together and tell them they’re going to make spaghetti and then don’t give them any instructions and then watch and see what they do. And it’s an exercise in organizational dynamics and it’s an exercise in understanding the team that’s completely divorced from software engineering or building things or whatever.
And I wonder if we’re going to have to lean harder on things in the physical space, things that actually co-integrate that kind of notion of humanity some fraction of the time. Because again, I don’t think we’re ready for how to progress people from level zero to level one in their careers anymore.
Jonathan Allen
No, no. And without a doubt, I think rightly or wrongly being out of the office hasn’t helped. Certainly when I was 20 years old, straight out of college, for me, being able to go into the workplace and see how others behaved, how others acted in situations, learning from those was incredibly helpful to me. The human interaction is so critically important. I actually think it’s been said many times, but team sports, team activities as you’re growing up is fundamentally important to being a well-rounded human in that regard, learning how to get on in a team, how to deliver things. Although I think the ability for people with agency to execute with AI is still extraordinary and going to continue to be extraordinary. I think to achieve outcomes in life, your ability to engage with other humans to collaborate, negotiate, form thesis, opinions, prove them and prove things out is still essential. None of us want to work with the individuals that can’t do that.
Adam Blue
Yeah. So let’s talk about another dynamic I think that regrettably kind of runs counter to that. So Amazon is very famous for the two-pizza team, and at earlier points in my career that would’ve probably been two people, me and Eric Smith, we probably could eat two pizzas handily, but normally that means what, five, six people, a two-pizza team?
Jonathan Allen
Eight to 10. We find actually,
Adam Blue
Eight to 10, wow. Big pizzas.
Jonathan Allen
Big pizzas. Big pizzas. We find, actually, Adam, holding the pager becomes the thing that takes the size. So typically a two-pizza team is providing a service, and that would be a combination of multiple different skill sets depending on the business outcome of the challenge that they’re solving. So it could be SDEs, it could be UX, it could be data, it could be product management. It’s the on-call bit, by the way, that comes to be the ultimate determinator of the size we find.
Adam Blue
OK, interesting. So we’re starting to hear people throw around terms like a two-slice team or, “Hey, we’re just going to have two builders build this product and not a designer, a product manager, a product engineer, a quality engineer, etc.” And so as the team gets smaller, I think that’s exciting because there’s less communication overhead, there’s less friction, you can move faster. But I also wonder, do we lose something as we end up with these smaller, almost more atavistic teams in a sense?
Jonathan Allen
Well, we’ve been on a journey in the last year and a half, I’d say in Amazon in this regard. So the two-pizza teams is globally well known and Werner Vogels at re:Invent last year talked about the two-slice team as you just mentioned. But the journey that we’ve been on from when we’ve gone from two-pizza teams where you’ve got 10 people going down to three people, and then very rapidly, the thing that breaks that is the pager.
Adam Blue
Interesting.
Jonathan Allen
So having been in my previous professional life on a one-in-four rotation, it sucks. It sucks. That’s not tenable. Yeah, we’ll try that and quickly roll back out of that. That’s not working very well because we have an intense responsibility to our customers for security, for operational stability, for the features, and pretty much in that order, by the way. So actually, although we have these smaller squads, what we found in the last months is that those squads then bump into each other, both from their dependencies from an API point of view, but also understanding of what each of those squads is doing on their own roadmap. So that’s the first thing that happened.
And then the second thing, which I can pick out from you that you’ve said is the job role families have blurred, right? If you give somebody the SDE title, what do they do? They act like an SDE. You give somebody the UX title, what do they do? They act like a GUI graphical user interface expert. And we’re seeing obviously the models completely blur the lines of that. And where we’re moving to, and I’m going to talk more about this in re:Invent, by the way, I have a talk called “A Leader’s Guide to Advanced Team Models in an Agentic World.” Where we’re sort of moving to is bigger teams with these small units working within them and then the pager responsibility being shared. So you’re going to co-locate those smaller teams together. And so actually co-locating these smaller teams together, but into a larger unit, and then the pager responsibility is shared. And of course you’re co-locating those teams that have the most dependencies, the most interactions together. They need to understand each other’s roadmaps, they need to understand each other’s AIs.
That ability to hold the pager is a forcing function to not only learn, but to instill the responsibility that obviously as you’re pushing things out, the best forcing function I’ve ever seen in my entire professional career as a developer and an engineering leader is you’ve got to hold the pager. If you’re woken up at 3 a.m. because one of your teammates has done something dumb, that’s going to be fed back.
Adam Blue
Yeah. Yeah, I agree. And that sentiment is … that’s not just in the software engineering industry or the computing industry or the tech industry. It’s kind of everywhere. Yeah, I saw an interesting study. They took people and they had them perform lab tasks, like pouring things in and out of graduated cylinders and lighting Bunsen burners—organic chemistry stuff. And they had some people do it in lab coats and some people do it in street clothes. And the people in lab coats, they perform better, which makes no sense logically. But it turns out that the lab coat is like plus-four science. It’s like the world really is modelable as a role-play game.
So I tried this out at home. So I got a chef jacket because I do all the cooking at home. So I got a proper chef’s jacket, like a really good one, and it looks good and it’s crisp and tight. And then I didn’t tell my wife and I put on the chef jacket, I make dinner. I came out, I put dinner down with a chef jacket. She’s like, “This is a fantastic dinner.” And literally it was better. It was a better dinner. I can’t explain it.
And this role blurring, to your point, you tell someone they’re a UX designer and they say, “Well, I’m a UX designer. I’m going to do UX designer things. I’m going to inhabit this role. I’m going inhabit this personality.” And then when we start telling people you’re part of the meaningless gray goo that collectively produces these outcomes for customers, I think we’re going to have to think about the way people regard themselves because the psychology of work, the way people produce differential effort and the way they care about the work and the way they relate to each other in knowledge work I think is even more fundamental, even more critical than it is in other kinds of labor.
And AI is accelerating things and producing extraordinary value. But it feels to me, and I’d love to hear your perspective because you’re out talking to so many people, that the psychology of work and the workplace, it’s already changed and that change is accelerating.
Jonathan Allen
Absolutely.
Adam Blue
Is that something you’d agree with or am I just full of shit?
Jonathan Allen
No, completely agree. It is and it’s going to continue to do so. And I think being aware of what that’s doing and has done to your role and will continue to do to your role and that self-awareness is absolutely essential. And your story of the chef is just so, it’s exactly what it is. I think one of the screw-ups I made earlier in my career was confusing my personal persona with that of my job title and conflating the two. I remember at different stages of my career, I was a network engineer, I was a client-server engineer, I was even a desktop engineer when I started out and did different things. And it’s kind of like you inherit the persona and much like you putting on your chef outfit or your lab coat. And I think that has now changed.
I find it really interesting that, for example, Anthropic have gone with a member of technical staff for the majority of their employees, and of course they’re preserving levels based on your judgment and impact, but they’ve gone to that member of technical staff. So that’s the lab jacket they want their folks to wear. Now, whether Amazon will change that in the future, that’s beyond my pay grade and that’s somebody else’s decision. But we’re doing a lot of experiments. We’re looking at different things, but I think it does have a deterministic impact on how you position yourselves.
I think the agency that it gives people is extraordinary though. My daughter’s 23 and she’s now looking at things going, “Hey, should I start my own business?” She’s talking to Claude the same as you do. It’s like, “How do I do that? Where would I host this thing? How do I make it scalable? I can host this on cloud. That’s pretty straightforward. I don’t need to buy any infrastructure. How do I ramp it up?” It’s like she’s having a conversation and educating herself about things we would have to previously do with a book. And really interestingly, she presented me a business plan the other day and I’m like, “Well, that’s OK. The level of research you’ve been able to get to in a few days is astonishing.” So yeah, it’s a new world.
Adam Blue
Yeah. Interesting. So this is probably our last topic, but I think it’s an important one. A lot of people talk about managing up and the expectations at the CXO level for what AI will do for their companies are … they vary from wildly aggrandized to completely insane, in my opinion. And I suspect you’re also talking to people a little further down the org chart that are feeling that pinch between individual employees that are being asked to do things and then the expectations of ... And sometimes it’s even the board. I hear this from bankers all the time. It’s like the board wants to know what are we doing about stablecoin? What are we doing about cyber? What are we doing about AI? And that … it can feel, as a senior manager, you can feel very pinched between those two things.
And so the people that are handling this the best, aside from drinking heavily, what are they doing? How are they thinking about that? How are they approaching that squeeze from insane expectations from both sides?
Jonathan Allen
Yeah, I haven’t got all the answers on that one yet, Adam. I think, look, the bar has risen. My advice to all individuals, whether they’re the CEO, board members, or graduate intern-level engineers is you have to take an interest in this. And I also tell people at the junior level, pay attention to what you pay attention to because your future career is probably dependent on your interest in that and your passion. But then what are you doing in your personal life to actually develop that in your own personal space, to use that, to push yourself, to broaden it, to challenge it, to build those legs? What projects have you done?
I find it interesting that GitHub has been dealing with extraordinary demand as people have been doing their own projects and companies using GitHub have pushed it out. That’s a sign of the times. So what are you doing to learn in that space? Because the more you can see what is possible, the more you’re going to be in control of your own career and know where you want to develop your own expertise. So the pressure is there. I’m very privileged to spend a lot of time with people that have a lot of agency, are very determined, that are very hungry to learn and push forward, and keep doing that.
And also, if that’s not for you, acknowledge what you want to still be doing hands-on for, right? I mean, as much as we’re pushing forward with electric vehicles and electric planes and rockets, the aviation world is kind of interesting. Things move really slowly in the aviation world because quite rightly, everything is safety critical first and pilots is still very hands-on. So where do you want to do your career? Where do you want to take it? Think about that really closely and always be looking, trying to look around the corner of where things are going to go.
And you’re in California right now. The AI right now is as bad as it’s ever going to be. And I often say to people, I’m using some pretty ... I have a multiharness setup. I found out the work between multiple agents, but even when I’m doing a complex query, the work takes time to come back. What happens when the inference bottleneck is gone? What happens when that return with your information back is instant? What happens when there’s no delay? What happens when that inference to establish your critical infrastructure is complete? What are you now doing as a human to add the value in the judgment, in the reasoning, in the business development? Because it’s as bad as it’s going to be right now.
Adam Blue
Yeah, that’s an interesting notion. Well, thanks very much for joining us today, Jonathan. This was fantastic.
Jonathan Allen
Thank you, Adam. It’s great to be here.
Adam Blue
Yeah. At the end, we like to talk a little bit about something from modern culture and fit it in. And have you seen “The Bear”? It’s a television show.
Jonathan Allen
I haven’t.
Adam Blue
It’s American. Yeah, it’s fantastic. There’s multiple seasons. I think the first season is probably my personal favorite. It’s about a bunch of people with some pretty substantial emotional issues who work together in a restaurant. And it is a fantastic slice of the impact of technology and modernity among a series of people trying to figure out who they are in a high-pressure environment. And what stands out to me is something that becomes more and more important is notions around judgment, discretion, taste. It’s the thing that you can’t get from the LLM.
And when you watch this show, which is a loving tribute to people that make food for other people to eat and take money for it, you can see the characters evolving their perspective and developing taste and judgment. And you can see how expensive it is for them to do that and how brutal it is for them to do that. And the show is, I mean, it’s brilliantly shot and has fantastic editing and cinematography. The acting is brilliant. The needle drops alone are worth watching the show for.
But I think there’s a lesson in there around what it means to do work and what it means to approach work that is both independent of and deeply coupled to the changes we go through today. Because to your point, what happens when the inference cost is zero and all that’s left is taste and thoughtfulness and judgment?
Jonathan Allen
And human interaction.
Adam Blue
Yeah. How will you get there? What kind of person will you choose to be when you don’t have the excuse of, “Well, that was hard or it took a long time.” That is a little terrifying. It’s a little terrifying in the best possible way maybe.
Jonathan Allen
It changes what we think work is, like you said. That’s the interesting bit to me. So much of the stuff that used to frustrate me about work, I’ve automated, but I can’t escape ... It’s what I can’t escape which is interesting to me. It’s what I can’t automate, which is interesting to me. And the expectation on that has risen as well.
Adam Blue
Yeah, totally agree. All right. Well, thanks everybody.
Jonathan Allen
Thank you, Adam.
Adam Blue
Thanks for joining us for Cut to Context. Thank you, Jonathan.