Live from Google NYC: AI, Now What?

Chapters

0:00 - Intro
4:33 - Should Colleges Forge More Coherent AI Policies?
9:29 - What Are the Cost and Benefits of the Patchwork Approach to AI?
12:06 - A Case for a Coherent AI Strategy at Colleges
15:33 - A Case Against Standardization in AI
18:08 - How Can College Leaders Measure AI Progress?
20:46 - The Importance of Stopping Some AI Pilots
20:24 - How to Make Decisions When AI Changes So Rapidly
23:59 - Sponsor Break
24:29 - Should Colleges Build, Buy, or Partner on AI
27:54 - Should Undergrads Learn AI From Day 1?
31:41 - What Research Shows About AI and Learning
34:12 - Convincing Students to Build Human Skills
38:49 - Do Selective Colleges Need to Change?
41:24 - How Can Colleges Show They Teach Durable Skills?
43:50 - Is Higher Ed Ready to Fully Assess Learning?
46:26 - How Will Colleges Deal With the Costs of AI?
48:26 - AI Could Help Some Colleges Leapfrog Others
50:50 - Colleges That Get AI Right Will …
52:15 - Closing Thoughts on AI at Colleges

Relevant Links

“Why College Could Matter More than Ever in the AI Era,” by Michael Horn.

Transcript

Andrea Goldsmith

In any organization, top-down leadership without a shared vision and shared commitment to the goals usually fails.

That is especially true in universities. I mean, we talk about getting faculty or students to do something through a top-down approach as herding cats.

Michael Horn

Jeff, we took Future U. to New York.

Jeff Selingo

It was Google's Higher Education Leader Summit, and we went in with one question, Michael. Everybody says they want an AI strategy, but is a coherent one even possible?

Michael Horn

Or is the mess actually the point?

Klara Jelinkova

You know, one of the things that's interesting and I think has been a long term strength of American higher education is that we are not uniform. Right? I'm sort of somewhat opposed to uniformity in higher education, and I think differentiation has really served as well.

Michael Horn

That's Klara Yolenkova, CIO at Harvard, and Andrew Goldsmith, president of Stony Brook.

Jeff Selingo

And Sandra Laughlin is a learning scientist who now hires the graduates. So she sees who actually arrives.

Sandra Loughlin

People do not learn when there is not friction. That is just the nature of how the mind has evolved. And so the challenge with AI is that it's so good at removing some of the friction from learning because if we are evaluating students on the things that AI is really good at and we're not evaluating the students on the things they need to be good at, we have removed the friction.

Michael Horn

We get into what to build, what to buy, who to partner with.

Jeff Selingo

And what it actually costs.

Michael Horn

And the question I can't stop thinking about.

Klara Jelinkova

I think it is not about what pilots do you start, but what pilots do you stop.

Jeff Selingo

Plus whether AI is about to let a scrappy institution leapfrog everybody with a bigger budget.

Andrea Goldsmith

This is a time when institutions that are agile and creative and innovative and not afraid to take risks and fail and learn from those failures are gonna disrupt higher education.

Michael Horn

Live from New York, it's Future U.

Let's go.

Sponsor

Bring the best of Google's secure, research-backed AI tools, like Gemini for Education and Gemini Notebook, to your institution. Elevate leaders, faculty, and students with purposeful technology, transform learning, and drive innovation.

Learn more at edu.google.com.

Subscribe to Future U wherever you get your podcasts. And if you enjoy the show, share it with your friends so others can discover the conversations we're having about higher education.

Michael Horn

Welcome, everyone. So good to be in New York City, and a huge thank you to Google for hosting us at the Higher Education Leader Summit today.

Jeff Selingo

So our goal for today's discussion and we're going to introduce our panel in a minute is to really move past the idea of should we use AI, which has really been the debate on higher education campuses for the last four years, and really start to think about what a coherent institution-wide AI strategy looks like.

What does it require?

What might it cost?

Who owns it?

And is this word, 'coherence,' which we're going be talking around today, even achievable? Or is it desirable when we think about strategies for AI on our campuses, especially when the technology, as we know, resets every few months and the labor market is already rewriting the jobs graduates train for.

That's our goal today in our discussion.

And with that, I'll have Michael introduce our panel.

Michael Horn

Well, you already heard we have an incredible group of panelists to talk about this from a variety of angles and I'm really excited about that.

To my immediate left, Andrea Goldsmith, the President of Stony Brook University. That's good New York applause, I would say. Yeah. Okay.

And then all the way at the end, have Klara Jelinkova, who's the VP and CIO at Harvard University. Welcome.

And we have Sandra Loughlin in the middle there, the chief learning scientist at EPAM Systems and the former director of the office of transformational learning at the University of Maryland.

Alright we're going to start off with a lightning round for all three of you.

So, quick answers to this but a little bit of explanation.

Four years now into the era of the AI Large Language Models that really have pervaded campuses, The dominant approach that we've seen in a lot of places has been around policies, sometimes restrictions, isolated pilots for sure, and I'm curious for each of you, has this been the appropriate posture in your opinion for higher education or would something more uniform, coherent, dare we say, and faster have been a better approach and why?

I suspect you all have slightly different takes here.

So Andrea, let's start with you and run it down.

Andrea Goldsmith

Sure. So thank you for hosting this. It's a pleasure to be here.

AI is the most transformative new tool. It's not four years that it's been introduced. ChatGPT 3.5 in November '22 was really what ignited the imagination of the world because you could talk to it and it would talk back and it seemed human.

It has launched the biggest paradigm shift in how educators teach and how students learn in generations.

And it's also a basic workforce requirement for most jobs, even though it's still evolving and we're in the very early days.

So I'm an engineer, whenever there is a new technology, there are the early adopters who move fast and break things. There's the kind of medium-term adopters who are more cautious and thoughtful and prescriptive of how they use things. And then there's the laggards who are resisting change.

And I think the phase that we are ending now is the one of the early adopters. The ones who took this tool even when it was in the early stages where it couldn't do elementary school arithmetic and started using the tool and understanding how to use it and how to integrate it as educators, as teachers in their classes and students starting to use it to solve homework problems or write essays.

And we've learned a lot from that early stage, those early adopters who did break things. We saw uneven standards and policies and practices about the use of AI, was it cheating, was it not cheating? We saw faculty who adopted it in their syllabi and teaching and failed and actually slowed down learning or degraded the learning process because they didn't know how to use the tool or the tool was not ready to be used.

And we also saw uncertainty by students. Is it cheating if I use ChatGPT? By faculty, can I use it in my pedagogy?

So we're coming out of that learning phase and I think now is the phase where generally with technology transformation, a much broader set of faculty and students and universities and other organizations can take the lessons learned from the early adoption and apply them systematically.

So I think we actually did it right as universities in terms of letting the early adopters move fast and taking the lessons learned from them and using that for this next phase. Super interesting. Sandra?

Sandra Loughlin

I would have to agree. Mean, it's a ‘Monday morning quarterback’ question as well. In theory, what should we have .... Well, in theory, we probably could have done it better. But in practice, the whole world was hit with something that is truly transformative, in ways that we are still grappling with.

And so I don't know that there was a better option than to let people kind of mess around and mess up.

But to just start to help us as a society both at the enterprise level and the university level, start to figure out what are the parameters of change. What do we need to do better in phase two And what do we definitely need to not repeat from phase one?

Klara Jelinkova

One of the things that's interesting and I think has been a long-term strength of American higher education is that we are not uniform. Right?

We have different institutions, different sizes of institutions and just kind of different approaches. And so one of the things that technology sometimes does, it kind of homogenizes things. And I personally hope that that is not the answer. Right? And that we will continue to have different types of institutions that provide consumer choice or provide choice to students and provide choice to families.

And I really hope that people are going to continue to try different things, and institutions will continue to try to find their unique path within this. Right? And I think that's kind of the call for leadership that every institution is now trying to address.

So I just ... I'm sort of somewhat opposed to uniformity in higher education, and I think differentiation has really served as well.

Michael Horn

Yeah. Let me flip the question a little bit and go to Sandra with this one, which is because students even within the institutions, because there's been sort of this early adopter, they're the ones moving, others are not yet.

Approach: Students are the ones living in this patchwork at the moment, and they have different rules and different courses, mixed signals about, as you said, whether AI is cheating or the future.

From where you sit in industry watching how graduates actually show up in the workforce, help us understand the cost or the benefit of the patchwork approach to this point.

What shows up in how students use AI or fail to because institutions and even individual faculty frankly, right, within the same discipline or even course proceeded with a range of approaches?

Sandra Loughlin

I think that regardless of the rules — the unwritten rules, the structure – students are using it. They have used it. They've been using it through university.

And they are showing up in the workplace with the burden of that experience.

And I mean that both positively and negatively. In that, a lot of students — again, everyone kind of responds to incentives and students are incentivized to just get the job done, not to necessarily do the job themselves.

And so we are seeing in our kind of new hires a variety of levels of anxiety, a variety of levels of confidence, but absolutely a variety of levels of skills that we would otherwise have expected to see from a new hire.

And I think that the patchwork approach has created this confusion for students and it has created this confusion for employers about what should we be expecting, how should we be looking for skills?

Is the degree still a strong signal? Like, now it's becoming even less clear.

I don't know that a strong, consistent, coherent policy would necessarily solve that, but it is forcing us in enterprise and in higher education to reckon with a technology that does what we have always prioritized people to do.

And as a society at every level, we're gonna have to reconcile what that means going forward.

Jeff Selingo

So Andrea and Klara, this word coherence keeps coming up. The word strategy keeps coming up. I hear often from college and university leaders about like, what's your AI strategy be? Right? They're talking to their boards about it. They're talking to their faculty, their senior leadership. And it's a goal they seem to be saying out loud.

But is that really what we should be aiming for here, like a single coherent strategy? Is that actually achievable, or is it the wrong thing?

Or is it the wrong word in some ways? Andrea, and then we'll do…

Andrea Goldsmith

Yeah. So I think to some extent it depends on what you mean by coherent. And when faculty and sometimes when university leaders hear coherent, they think, oh, this is going to be some rigid top-down strategy imposed by the president or the provost or the CIO where I'm going to have to do something.

And in any organization, top-down leadership without a shared vision and shared commitment to the goals usually fails. That is especially true in universities. I mean, we talk about getting faculty or students to do something through a top-down approach as herding cats. And if you've ever tried to herd cats, it's very hard to do.

But catnip helps, and so resources or promise of, you know, great jobs at the end is something that you can do.[1]

I think every university, every organization — I sit on two corporate boards as well — it's not just about universities, companies, nonprofits. Every organization needs to have an AI strategy because this truly is a transformational tool. It's like saying, you wouldn't have an internet strategy today or you wouldn't have a wireless strategy or even a quantum strategy. And AI is more transformational, I would say, than any of those technologies today.

So you need a university-wide strategy and that has to be owned by the leader of the university, which is the president.

But what does that mean in terms of a university-wide strategy?

It means shared goals, agreement on outcomes. So when we're talking about a university-wide strategy, it's not just about education, which is teaching and learning and those are two different things. It's about research.

We manage four hospitals and have a huge medical enterprise at Stony Brook. How are we going to use AI in medicine and healthcare and wellness and in our hospitals?

Public universities are about service to our communities and the great state of New York and the country and the world. How are we going to use AI to increase our impact?

And so it also has to be owned by everyone from top to bottom of the university. That doesn't mean the same programs or rules or constraints either across units or even individual faculty within individual disciplines.

The beauty of universities going all the way back to the Middle Ages is that disciplines owned what they taught, what it meant to master a discipline and how do you convey that to a broad range of students who learn differently?

So I believe a university-wide strategy is essential. It has to be owned by the leader of the university and their leadership team who figure out how in an organization that's so diffuse, that's so full of cats that don't like to be herded, where everybody owns the strategy and the outcomes.

And we take the best ideas from throughout the university to figure out how to achieve those outcomes at a time when the tool is changing dramatically.[2]

Jeff Selingo

So Klara, I'm kind of curious about how you think about this given where you sit in the CIO's chair, in terms of a coherent strategy.

Because often, when I talk to some CIOs, they're very interested in standardization across a university. Is that the goal here? Or do you think there's something in between?

Like how are you thinking about this?

Klara Jelinkova

Yeah. When I was listening to Andrea, I was thinking, I would add that there's also an element of having agreement on the level of values and principles actually. And I think values are quite important in this moment.

But I agree that the disciplinary approaches are always going to vary. Right? So disciplines have always been defined by methods. Right? And so, like, you will have different methods as they apply to different disciplines. And I think it's really important that we allow the disciplinary excellence to continue. And we allow disciplines to kind of approach AI differently.

Which sort of speaks against this idea of standardization. Right? What did Fitzgerald say? Like, a first-rate mind can keep two opposing ideas at once and still function. I would say institutions hold more than two opposing ideas and still continue to function, right?

And so I think where we want to have coherence, and I actually really like the word, is we want coherence on capability, but we don't want conformity of use. And I think kind of having a distinction between the two is going to be really important.

And yes, that may mean that you will have a different way of using tools. And you might actually have different tools that are being used within different disciplines.

We all use AI differently. I actually use pretty much all of the AI tools.

Very funny, Andrea. I have to share this. I have to share this. When I knew that Andrea and I were going to be asked this question, so I asked Gemini, because we are Google. I asked Gemini, ’Can you profile Andrea and see what she's going to answer?’

And Gemini said, ‘She is going to say that top down is not going to work.’

I kid you not. Kid you not.

So there you go.[3]

Andrea Goldsmith

And I didn't consult Gemini before I answered that.

Jeff Selingo

Okay. So fair enough. Not coherence on conformity of use.

So how does leadership then at the university see that they're making progress, right?

What are the Klara, what are the concrete checkable signs that an institution is maturing rather than just keeping busy doing this work, right?

I think every institutional leader wants to make sure that they're progressing year after year, right? We have all these KPIs in so many other areas of the university to make sure that we're progressing? How do we understand that when it comes to AI?

Klara Jelinkova

And it's so funny because in technology we absolutely love these maturity curves. You know what I mean? The Gartner Magic Quadrants. You know, are you on the top right quadrant because then you're really sort of cooking with gas.

And so the milestones to me, right? Is like you sort of the baseline is tool access. Right? I mean like you need to have some level of tool access.

But I think the second piece is you start working on pilots and there's just a lot of questions about having pilots and how many should you have. I think it is not about what pilots do you start, but what pilots do you stop.

And so ... I had a coach once that said, when you choose to do something, you choose not to do something else. So it's this idea of discernment. Right?

And so, I think to me the next step is what do you actually stop doing? What things do you choose not to do anymore? When do you decide that this AI thing that you tried to do actually didn't go anywhere, and you just need to sort of let go of it. Right?

So, I think that's another maturity step. [4]

And then, the third is — and it flows from that — is kind of what is the value that is being created and do you have a way of measuring it?

So, it's about learning from the success and kind of defining a little bit what success looks like. I don't know that it's always efficiency. Right? Because efficiency is not… it's more efficacy and individual agency.

But, I think that's kind of how I think about this.

Michael Horn

Andrea, I want to bring you in here as an engineer running a university. A public flagship no less. How do you make a multiyear strategic bet around budget, hires, infrastructure, the tech stack on something as you noted that's moving extremely fast like roughly a four month half-life?

Is there a framework to help others think through what feel like really challenging decisions at this point given this rate of change?

Andrea Goldsmith

I think that's an excellent question that builds on the previous discussion of, ‘What does success look like?’

And so when I approach our strategy for AI, I think of it not just as an engineer, but actually as a successful entrepreneur who built a company.

There's a notion in Silicon Valley of the minimum viable product, which is you don't go out and build your first product that has every bell and whistle on it. You build a minimum viable product, which has all the features that will captivate the imagination of its users and then you learn from that and move on.

The iPhone is a great example of that. The first iPhone was nowhere near as capable as what we have today, but they sold 300,000 phones in the first weekend because it had enough captivating features.

So how do we think about the notion of a minimum viable product for introducing AI into universities? And I think that's what those early adopters did is that they took some of the basic tools or the early versions of AI. They said, okay, how do I use this in my course? Whether it's a history course, and I let my students use AI to take all the history books on this topic and synthesize them and then explain how they use that synthesis and their reasoning. In computer science courses, you allow the students to use AI to code and then they have to explain the code or do oral code reviews or say where the code fails.

So, these early experiments were taking the early AI tools and trying to understand how do we get our students to master domain knowledge, which is what universities are all about, using this new and powerful tool and coupling it with their uniquely human skills of creativity and innovation and perspective and collaboration.

And that is really what the role of the university in this era of AI is going to be. It's we still need to ensure that the students can master domain knowledge so that they can interrogate the AI answers, but they need to be able to do it with AI, and they need to use their uniquely human skills.

So we know kind of what the North Star of using AI in education is. And we're using these early versions of AI and the tools like the ones that Google have developed. So now we have some success stories and we can look at, okay, how do we port those success stories to other disciplines or to other classes?

And the other thing about the minimum viable product is when do you know how to scale and when to scale and how to invest in scaling?

So I think we're at the point now having done those early experiments, seeing what's working, seeing what's not working to scale it up across the university.

And going back to the notion of shutting things down, the second hardest thing to do in a university after a top down approach is shutting things down. And so once we've started any program, it's very difficult to shut things down. And I think we have to be a lot more systematic about that and understanding, ‘This was a great experiment. We learned a lot. Now it's time to move to the next experiment.’

Sponsor

Bring Google's secure, research-backed AI to your institution to accelerate research, streamline operations, and transform education.

With Gemini for Education and Gemini Notebook integrated across disciplines, you can boost student engagement, improve academic outcomes, and personalize instruction,

Empower your entire community to collaborate, solve complex problems, and prepare students for the future, all on a safe, scalable foundation.

Discover more at edu.google.com.

Jeff Selingo

We want to move on to teaching, learning, and assessment.

But before we do that, just a very quick I just want to get a quick readout, a very quick readout from Andrea and Klara on just the capital cost of all this, because it is expensive.

And I think there's been a lot of questions. We had learning management systems. We had online learning. We had all these technologies over the years. Some of it was built internally. Much of it was outsourced.

How should universities think about where they build, where they buy, where they partner on this?

Let's just make this very quick because we want to move on to teaching learning assessment. Klara, build, buy, partner?

Klara Jelinkova

Yeah, so we buy commodity services, right? So LLMs are one of those things that we are ... It would be purchasable, right?

But I think it's a mistake to think about generative AI or AI as LLM only. I mean we somehow equate it in our minds.

And so there are things such as small language models, right? And so there's going to be an area where institutions will be able to build. And that means building not just the connective tissue that connects the tools that we choose to purchase. But also, are there disciplinary specialized, perhaps small language models? Perhaps disciplinary approaches that we can actually build ourselves?

Or things that capture some of the elements that truly are remaining at the university and the intellectual property that should stay there.

I think partnering is actually the toughest piece and the reason why partnering is tough is because you need to have alignment on values.

And this is kind of like, I think there's a huge opportunity between academia and private industry. But we need to be thoughtful about this kind of alignment — especially when we are thinking about how let's say an LLM is used to solve disciplinary problems. Like what is it exactly that we are giving up?

Jeff Selingo

Andrea, any other thoughts on buy, build, partner?

Andrea Goldsmith

I think you do need to use all three.

And the value proposition to companies to partner or give universities, especially public universities that don't have the resources of private universities, is that you tap the creativity and brilliance of the faculty and the students.

And we don't necessarily need the most up-to-date models or the highest compute power because we can use our ingenuity to take a model that's a year old or a computer that's or set of GPUs that aren't as capable and use our ingenuity to use it better and that helps inform industry.

The other thing is to partner with industry, first of all, to bring their problems to our students so that they have experience applying AI to practical things.

And then also we are kind of the zero-th user to help companies improve their products.

So when I look at the suite of products that Microsoft offers for helping faculty and students use AI in teaching and learning, why would we ever try to develop those on their own? I mean, we don't have the resources, people, compute, anything, but we can help evolve those tools.

And so we want to find the cheapest way that Stony Brook can build, buy and partner to take advantage of the full suite of resources out there.

Michael Horn

So I want to shift the conversation to the teaching, learning — two different things as you noted — and assessment part of this. And we've talked strategy, I would say, at some altitude so far. You can have the most elegant strategy written on paper and if you have a thousand faculty doing their own things that depart from it, that's your actual strategy as I teach my students in the classroom.

And so Sandra, I want to start with you on this. One of the big questions I think underneath a lot of the classroom or teaching and learning discussions or food fights if you will sometimes about AI is this, Should undergraduates be learning with AI from day one, or should they build the fundamentals in whatever domain they're learning first and then layer in AI?

What do we know about this question?

Sandra Loughlin

So the answer is both, and it depends on what we're talking about as being learned.

So in higher ed and in general in schools, we have often thought about knowledge as declarative and procedural knowledge. What do you know about something and what are the steps to do something? And that has been the currency that we have used to evaluate what learning looks like.

But in the real world, the thing that matters most is actually conditional knowledge. Under what conditions does this thing apply? Where does it not apply? What are the anti-patterns? What are the exceptions?

And so I think it's like, I used to study expertise and higher-order cognition. And it is fascinating to see what AI can do and where it actually shortcuts and short-circuits learning.

So I think the answer is we need people to have basic foundational knowledge. We're trying to figure out what that is still. Like, we just actually don't know. I mean, it will take some trial and error to figure that out.

But if you have some base understanding, then the real question becomes, can you recognize in the wild when to use that thing?

And the thing that in higher education we have not done as well because we were so focused on declarative and procedural is actually helping students understand under what conditions.

And so I think that … And again, like I've missed academia so much sometimes. I wanna go back in the classroom and actually practice this.

But in enterprise — and we're teaching people because we're teaching all the time as well, how to actually function in the world — we are giving them and helping them learn the foundations on their own of, like, what are the basics. But then we're really spending a lot of time on experimentation, on application, on feedback, on practice, and helping people understand the edge cases where this actually makes sense.

And so I think from a learning perspective. ...

Of course, the other big factor that, you know, is hard to overstate is the importance of friction. People do not learn when there is not friction. That is just the nature of how the mind has evolved.

And so the challenge with AI is that it's so good at seeming ... at removing some of the friction from learning because if we are evaluating students on the things that AI is really good at and we're not evaluating the students on the things they need to be good at, we have removed the friction.

And it isn't as though once we remove the friction it's all neutral. Actually, if it's not learned, it's not used, like the brain will actually prune information out. And so we need as a society again — I say higher ed and enterprise and K-12 — we need to figure out how this whole learning thing is working and what we are looking for when we are assessing and looking for evidence of learning because right now there is a bit of a mismatch in how we define learning and how learning actually needs to look in the wild.

Michael Horn

Well, and there have been a number of early studies starting to produce some evidence around where AI genuinely helps learning and where it feels like learning, but is not actually.

Give us a summary of what the evidence so far is suggesting and we're finding so far.

Sandra Loughlin

The evidence is suggesting that like any technology, AI amplifies. And if you're using it to try to offload cognition, if you're using it to try to not actually think that hard, it's really good at getting you to the bottom level where it looks like you've learned.

But if you know how you learn, and to be clear, most people do not know how they learn. Most faculty don't know how students learn. It's a problem.

If you know how you learn, you actually can use AI to learn certain things faster, faster, better, deeper.

But I think a huge new focus of higher education and again enterprise is teaching people how they learn. Metacognition, when to recognize when you're offloading, How to ask the right questions. When to follow-up. When patterns look off and how to respond. How to ask AI, you know, 'I have a concept. What are related concepts?'

We actually as a society need to get better at understanding how we learn because what the research is showing is that if people who are using AI to learn, don't know how they learn, they are actually going to short-circuit their learning as opposed to get deeper with it.

But if you do know how you learn, AI can be transformational in the process.[5]

Michael Horn

Lightning quick follow-up on this. How do we think about environments and conditions in which people are learning to get more of the actual metacognition but actually using it right for real learning, not stripping away the friction, not stripping away the usage.

Sandra Loughlin

I think it really goes .... Again I'm all about incentives. Right? It goes back to what are we incentivizing students to do?

And if we are giving them resources and assessing them on their ability to ask those follow-up questions to like watch their learning, which we can do if we can observe their interactions with AI, we actually can give them the right conditions to prompt the learning, and we can evaluate them on their individual learning as opposed to what AI produced on their behalf.

But again, this goes back to really understanding teaching students and faculty what they can and should be doing with AI to deepen learning as opposed to offload it.

Michael Horn

Andrea, I want to bring you in here because some faculty we know are starting to experiment with approaches to make sure people aren't doing the cognitive automation. Some people call that AI resistance, if you will, but really trying to figure out are people doing the learning itself, not just the outcome that they got through using the AI.

In general, a lot of these approaches seem like it's a lot of work right now, a lot of redesign, labor intensive in some cases.

At the same time, as you know, AI is also often being sold as the thing that makes higher ed far more efficient.

So, I'm tempted to ask you the binary here of which one is it, but maybe help us make sense of the seeming paradox at the very least between, ‘Is it efficiency or is it a lot more work?’

Andrea Goldsmith

Right. I think it's neither and both at the same time. So in the same way that AI is transforming teaching and learning, it is transforming how we need to assess our students.

So at the end of the day, educators are about teaching students mastery of their disciplinary knowledge, but also there's a much broader set of knowledge that students obtain by coming to our universities — lessons they learn inside as well as outside the classroom.

Now, we can't necessarily assess all of the outside the classroom lessons, but certainly those are as important to employers as the inside the classroom lessons.

So the way that we have actually shortcut assessing mastery of domain knowledge is Scantron exams where you write the answer and it's correct or not correct or it's multiple choice. And this is so ingrained. I mean, standardized test to get into college uses this kind of thing.

And so we've really trained students that get the right answer, not master the subject.

And so when we say, 'Why are students short cutting their mastery of domain knowledge just to get to the right answer, whether it's with AI or other ways?' It's because we haven't really as educators assessed mastery of the discipline.

And that's also an issue for employers when you're trying to look at 1,000 applications for a single job. I know Google and other companies use coding challenges. Well, boy, that's out the window now with AI being able to do coding. What they really wanted was to understand do these students have mastery of computer science?

And so we need to completely rethink how we assess students. And it's not just about domain knowledge because as I said earlier, what we need to do as educators in an era of AI is educate students to use their domain knowledge, which means they need to obtain domain knowledge to be able to collaborate in interdisciplinary ways and use other human skills like creativity and innovation and perspective and do all that with the power of AI that maximizes their domain knowledge and their human capabilities.

How do we assess that? I think we're in the very baby stages of figuring that out. [6]

And there are some faculty that are resistant. I mean, I hear faculty are going back to blue books, if anyone remembers blue books, you know the ancient era where you had to handwrite out your answer. They don't even teach cursive writing in elementary school anymore. So how are you gonna use blue books? You know, it's really that category of adopters that are laggards and resistant.

That is gonna go out the window because that is not the way to train today's student to use AI and their domain knowledge and their human skills and to ensure that they have mastered all of that.

So we need new ways of assessing students. I think we're hearing about those in early experiments of, 'Give me your exchange with the AI tool.' 'How did you interrogate the answer?' 'Can you point out if you do a coding challenge, where did the AI fail? Where could it have been better?' When you look at a history essay. If you're writing a history essay or any essay only using AI, you are not finding your voice, you are not finding your perspective. If you're using AI to solve a homework set, you are not learning anything. And so you will be completely replaceable by AI.

And that's the incentive for our students, that if they're using AI to shortcut finding their voice, finding their ability to communicate, finding their domain knowledge, finding how to work with their human skills, then they will not have a job in the AI era.

And if we can convince them of that and then develop the right educational tools and assessments to help them learn what they need to learn to be AI-proof in whatever job they aspire to, that's the solution.

But we're a long way from that.

Jeff Selingo

So I want to dig a little bit deeper on this connection to the job market.

And Klara, let me start with you because highly selective elite universities never really had to chase the latest labor trends when it came to majors, right?

There was this belief that the graduates of these institutions were highly qualified to do a lot of different things.

So does the current concern around AI and jobs, does that insulate a place like Harvard and other highly-selective institutions? Or does it mean that you do have to shift how you're thinking about preparing students for the labor market?

Klara Jelinkova

So let me just say, I do not speak for Harvard. I don't have a HPAC sign off.

But you know, one of the things about being old is that you remember, still. And so if you happen to have come of age and come to computer science in the late '80s and early '90s, you're actually going to remember what was going on. Right?

And that was the time when there was a lot of discussion about, 'Was there any future in computer science and teaching computer science?' People were dropping out by the boatloads. Right?

But the main question there was, should computer science or, you know, and I will say more broadly, teach durable skills or should it teach just kind of these temporary skills?

And this is a place where I actually completely agree with what Alan Garber said a few years ago, which was, you know, 'The temporary skills are going to erode very, very quickly. But what we are teaching our students, which is kind of these capabilities that they learn in the classroom are largely in, like, working on problem sets together. Being part of a community,' right? Which is critical reasoning, curiosity, empathy, deep listening are going to be durable.

And so one of the things that was really interesting yesterday at dinner, Steven mentioned that the most successful teams, there was a Google research on this, were those where the teams had psychological safety to collaborate.

And guess what? How to create a team where you can disagree with each other and yet have that psychological safety? That is exactly what we do on campuses when we do our jobs well.

Jeff Selingo

I love this because this idea of discernment, critical thinking, communication, all these things that I think universities really excel at actually can differentiate them in this day and age in AI.

And so, Sandra and Andrea, let me just kind of double down on this with each of you.

So then, how can we assess that? So let me start with you, Sandra.

Colleges do talk about, 'Well, we do a great job at teaching these durable skills.' Well, we could debate that, whether they do a great job.

But one way or the other, in this world where you're going to have to differentiate kind of AI from the human capabilities, how are we going to rigorously assess that they do that, that colleges do that?

Like, how are they going to be able to show that?

Or how should they be able to show that, I should say?

Sandra Loughlin

I think what AI is doing is it's shifting the demands around assessment for new hires for, you know, end of courses, but it's also creating opportunities that we didn't have before.

So if I'm looking at durable skills, I'm going to be putting someone in an environment and saying like, 'Go. Like, let me see how you're doing this. Reflect in front of me. I actually wanna hear your thought process. I want you to explain, like, the conditions under which you're gonna make this decision or that decision.'

I think what AI allows us to do is it gives us both the demand and the capability to take the assessment level again at the new hire level and courses to a different level.

But it requires us to actually as educators and as employers think about, ‘How do these skills show up in practice in the job? What does that look like?’

And then to, as best we can, both simulate those environments and seek out signals from students' digital portfolios or experiences or their experiences with chat that allow us to see that.

And so I think it's going to be identifying digital trace and putting people in realistic or real environments and watching how they perform and then asking for the reflection.

Like as AI expands our capabilities, we have to be thinking about how we can expand the use of it to get at these things that have heretofore been too expensive or complex to do. And I think that's as an employer what we're doing.

And I think we're gonna see a lot more employers doing the same.

Jeff Selingo

But Andrea, is higher ed really ready to truly assess learning finally?

Andrea Goldsmith

So, I don't think we've ever been fully ready. We're also learning as an institution or as institutions how we assess students.

But coming back to this notion of, ‘What is it we're trying to assess that universities are trying to provide to our students?’

This notion of the human skills, creativity, one of the stories I like to tell about why students should take classes outside their major. They shouldn't be focused on efficiency. They shouldn't be focused on how soon can I graduate?

Steve Jobs took a class in calligraphy before he dropped out of college. And why did that matter? Well, why does the Mac have the very best fonts of any computer? Because he took this calligraphy class.

And often you, as a student, ideally take classes in a broad range of fields. It opens your mind. It creates a love of lifelong learning. It allows you to connect with people in fields completely different from your own. And it also leads to interdisciplinary collaboration, understanding differences, understanding different perspectives.

How do we measure that?

I don't think we've ever measured that in universities, but we know that that is something that we do for our students. And we know that that's something that employers really value because so much work today is teamwork.

And one of the things I worry about with AI is that it will degrade that aspect of our education in the same way that social media degraded social interaction in many ways. If you're going to AI because you don't understand a concept, rather than going to your professor, rather than going to the study group that you do homework problems with. You are losing the human connection, you are losing … I found that I learn a lot more from explaining something to somebody else than I learned from reading it in a book or hearing it from a chatbot.

And so I think we need to recognize that AI can be a powerful tool to improve teaching and learning, but it can also be severely detrimental to teaching and learning.

And where is that tension and how do we mitigate the dangers while embracing the advantages?

And that gets back to the assessment.

If what we're truly trying to assess is, ‘Can our students master their domain and a broad set of knowledge to be citizens of the world, not just employees, use AI in a powerful way for success in their personal and professional lives. How do we measure that?’

We may never be able to fully measure it, but we can certainly improve on how we're looking at the education of our students to that desired outcome.

Michael Horn

I want to turn a little bit to the cost side of all this.

And Klara, I want to ask you a question about this because it strikes me this is very different from the digital revolution we first had where there was a lot of marginal cost, right?

This actually has real compute costs with the adoption of AI at the same time that we hear everyone saying AI is going to be this huge cost saver.

And so I'm just sort of curious, A, your general perspective on that as the CIO.

But also upfront you mentioned that it's a good thing that we have a diverse array of institutions with different missions, differentiated from each other, different outcomes that they're seeking to produce. But I'm kind of curious, right, like you sit at Harvard with certain budget. There's public institutions that don't maybe have that same budget. Is there going to be haves and have nots with the latest AI models or not?

Andrea suggested maybe that doesn't matter, but I'm curious for your perspective on that as well.

Klara Jelinkova

It's a very compound question.

Michael Horn

Yes, I'm combining a couple of things here, but I'd love you to reflect on cost and its implications.

Klara Jelinkova

Is the cost of technology and the amount of money that institutions spend on technology going to increase? Yes. It has. You know, so I just think that the past trends are going to continue.

Michael Horn

It's going continue. Yeah.

Klara Jelinkova

Exactly. And the thing is, like we have learned this lesson with ERPs. You remember all of the like spend $200-million on your ERP because it's going to save you money yet. I've yet to see that money.

So, I just … I think it's a similar lesson because the only way you save money at institutions is when you start changing workflows and when you start changing roles and service models. And I think that's distinctly harder.

So, to me that's just yet another kind of technology that we are implementing at institutions.

But really, the saving money comes from changing how we do things, which is a different question.

Michael Horn

So less the technology and more the actual organization.

Klara Jelinkova

It's organizational operating models and stuff like that. As far as the tiering, right? I just ... I think it's interesting because you have to realize that leapfrogging is also a possibility, right?

So why does Kenya have better digital … or had digital currencies sooner than the United States, right, or Europe with M-Pesa was because they completely leapfrogged. Right?

And so you can have this possibility where you have institutions that are more agile. And I think agility actually is going to really, really matter.

And so, the fact that you have a ton of resources and you are big … it also implies that you have complexity that you have to overcome. Right?

And so, if you just kind of think about using technology to leapfrog, this kind of have and have nots could look really different because I don't think that AI determines the winners. It's the institutional ability to adopt it in a way that kind of changes itself that is going to be differentiated.

Andrea Goldsmith

Can I add a comment to that? I think it's a beautiful point.

And one of the things I say being president of the number one public university in New York, but hopefully soon to be number one in the country is ...

If this were normal times, that would be a really big stretch goal, but these are not normal times.

There's times of huge disruption in higher education across all institutions. It's also a time of huge disruption in technology with respect to AI.

And I think the … as an entrepreneur who disrupted the WiFi business, this is a time when institutions that are agile and creative and innovative and not afraid to take risks and fail and learn from those failures are going to disrupt higher education and are going to leapfrog and that is certainly the opportunity for Stony Brook.

It's the opportunity for every university whether you are already at the top, you're still gonna have to innovate and take risks and do bold things to stay at the top.

And for universities that not at the top, this is our opportunity to really embrace this moment of profound change in higher education and technology together that rarely happens at the same time to embrace the opportunities and seize them for the benefit of our universities and our students and our regions.

Jeff Selingo

Okay. So that brings me to our last question before we turn it to audience questions, and it's a question to all three of you.

Let's finish this statement. The institutions that get AI right will be the ones that... Fill in the blank. Sandra, let's start with you.

Sandra Loughlin

The ones that are able to capture the best ideas, the best insights, the best questions, and turn those into an infrastructure that raises the floor across the organization. That it creates the cross-pollination that rises all ships.

Jeff Selingo

Which requires very good shared governance.

Sandra Loughlin

And data architecture.

Jeff Selingo

Klara, what does the AI institution that gets this right look like? How do you fill in that blank?

Klara Jelinkova

So the institutions that will be successful are those that understand what is non-negotiable. So they kind of understand their values and the principles and have the agility to innovate around it. So I think.

Andrea Goldsmith

They will be the institutions that understand the transformative power of AI with all of its opportunities and pitfalls and are able to seize those opportunities across all dimensions of their mission — teaching, learning, research, healthcare service — and will do so in a way that adapts as AI tools adapt and get better.

Jeff Selingo

Terrific. We're going to leave it there then. Michael, any closing reactions.

Michael Horn

Gosh. I have several, but I'll try to limit it to a couple.

I'll say one, we talked about the early adopter prototypes upfront. And a question that I've become obsessed with is what do we stop doing in organizations, de-implementation?

And it strikes me that that's an important number one takeaway.

But it connects to the second one which is I think you can only decide what to stop if you know what your north star is around outcomes and values.

Because for testing and learning to work and have that reflection of, 'Do we continue? Do we stop? Do we scale?' You've got to know what you're trying to go toward, and we don't ask both of those questions, I think, nearly enough. You, Jeff?

Jeff Selingo

Well, nobody asked Gemini before we came on, what we would think, but we actually agree on this because I wrote values, values, values, and the value of higher education.

So I think there were so many comments today around values, right?

I think this is going to lead to a conversation on campuses about, 'what we do, why we do it, how we do it, and what we stop doing.' It's going to go back to mission.

Klara, I think you said very early on that higher education is not similar throughout higher ed. And I think that is one of the great benefits of American higher education is its diversity.

But in diversifying higher education, we've always tried to be the same in some ways. Everybody wants to be somebody else in higher ed.

And I thought your comment, Andrea, that this might be the way to leapfrog institutions could be fascinating for the future.

But I think it's going to come down to this idea of values and what we value.

And then second is the value of higher education.

So much around the durable skills, and that friction of learning, which I think is the real benefit of colleges and universities, especially the residential model. Because we've talked a lot about the classroom here, but we don't talk about all the other learning that goes on on our campuses, the research that goes on. But that, to me, is the real value of higher education.

And if we could differentiate what we do that's so much better than anything else around those durable models, I think that will bring back this idea of the value of higher education.

So thank you for all joining us here today in New York City.

A big thank you again to Google for Education for having us here today.

Thank you to our panel of great experts up here who have illuminated us in so many ways today.

Please join me in thanking them.