Lecture 4: Enterprise GenAI Trends and AI Agents

What this lecture covers

This guest lecture, by Burak Gokturk (Google, leading Vertex AI), steps back from algorithms to describe how the AI industry and enterprise customers are actually using large language models today. He opens with a short history of why neural networks won out (parallelizable architectures, huge training sets from next-token prediction, better hardware) and why ChatGPT's late-2022 launch reset expectations. The bulk of the talk covers enterprise adoption trends: falling barriers to entry, the shift from choosing a single model to choosing a flexible platform, dropping inference costs, and the rise of search-grounded LLM applications. He closes with a practical tour of four customization techniques: fine-tuning approaches, distillation, grounding, and function calling.

After watching, you should be able to name the main reasons enterprises now adopt AI faster than before, explain the difference between prompt design, parameter-efficient fine-tuning (like LoRA), and full fine-tuning, describe why grounding with search is needed even for strong models, and explain what function calling adds that a language model alone cannot do.

Key ideas

• Why neural networks won: parallelizable architectures combined with enormous self-supervised training sets (via next-token or masked-word prediction) and improving hardware let neural networks overtake older methods like support vector machines.
• Lower barrier to entry: because base models already carry vast pretraining, enterprises need far less custom data and fewer specialists than before, letting any developer, not just AI experts, build applications in days rather than years.
• Platform over model: with new models released every few weeks, enterprises increasingly choose a platform that gives them access to many models and the flexibility to switch, rather than committing to one "best" model.
• Falling inference cost and latency: increased efficiency (including sparser, less "dense" model execution) has driven API costs down roughly an order of magnitude, enabling more latency-sensitive applications.
• Grounding with search: because LLMs are trained on a past snapshot of data, hallucinate a meaningful fraction of the time, and cannot cite exact sources, combining them with retrieval or search (RAG) is now a dominant pattern for factual applications.
• Fine-tuning spectrum: prompt design (a few examples in the prompt, no training) requires the least data and is what most people use in practice; full fine-tuning updates all model weights and needs heavy compute; parameter-efficient fine-tuning (like LoRA) updates a small added set of parameters, giving most of the benefit at much lower cost.
• Distillation: a large "teacher" model generates labels (hard or "soft," via a temperature-adjusted softmax) for a smaller "student" model to learn from, avoiding the cost of manually labeling large datasets.
• Function calling: LLMs can retrieve and reason about information but cannot take real-world actions (booking a flight, editing a record); function calling lets a model recognize it needs an external tool and select the right one from a library of available functions.

Walkthrough

Why neural networks and LLMs took off (0:03)

Gokturk recalls skepticism about neural networks during his PhD and traces the shift to parallelizable architectures, massive self-supervised training sets built by masking words in web text, and rapid hardware improvement. He cites accuracy jumps on benchmarks like ImageNet and speech recognition over less than a decade, and credits ChatGPT's late-2022 launch with convincing enterprises that AI now genuinely works.

Google's Gemini and the search question (11:12)

He briefly describes Gemini as multimodal from the start with very large context windows (citing internal tests up to 10 million tokens), and poses an open research question: as context windows grow, could an LLM with enough input context eventually replace traditional search algorithms? He introduces the "needle in a haystack" test as a way to evaluate whether a model can retrieve a small piece of information buried in a large context.

Enterprise adoption trends (15:16)

Gokturk lists the trends he sees across enterprise customers: massively increased AI adoption interest, far less data needed to get started, a shift from requiring AI experts to any developer being able to build applications, a move away from per-domain models toward a strong general reasoning base model, a shift from dense to sparse model execution for efficiency, a shift from choosing a single "best" model to choosing a flexible multi-model platform, and steadily falling API costs and latency.

The search and grounding trend (27:26)

He explains why combining LLMs with search has become one of the biggest enterprise patterns: models are trained on stale data, they hallucinate confidently, and they cannot reliably cite which source an answer came from, while search provides freshness, citations, and authority. He also notes growing enterprise interest in employee-productivity tools built on the same underlying models.

Customization techniques: tuning and distillation (34:35)

Gokturk walks through the customization spectrum: prompt design (a handful of examples, no training, and what he says most practitioners actually use), prompt tuning (learning an embedding-based instruction from many examples, which he notes is less used in practice than expected), full fine-tuning (updating all weights, compute-intensive and rare among enterprise customers), and parameter-efficient fine-tuning methods like LoRA and its quantized variant QLoRA, which update a small added set of parameters and are far cheaper to store and run. He then explains distillation: a large teacher model generates labels, sometimes "soft" labels via temperature-adjusted softmax, for a smaller student model, avoiding costly manual labeling.

Grounding and function calling (48:51)

He details a typical retrieval-augmented generation flow: convert a prompt into a search query, retrieve results, and feed them back into the model's prompt, alongside training better, more factuality-focused models. He then turns to function calling: since LLMs cannot take real-world actions like booking a flight or editing a record, function calling lets a model recognize when it needs an external tool and select the correct one from a library of available functions, a task that gets harder as the library of available functions grows.

Closing: evaluation and growth (59:04)

Gokturk closes by noting the growing use of large models themselves to evaluate and rate other models' outputs (side-by-side comparison tools), citing figures on enterprise API usage growth, and encouraging students to pair AI study with other disciplines to build creative and reasoning skills that remain valuable as the field moves quickly.

Before you watch

• No specific prior lecture is required; general familiarity with what an LLM is and what RAG (retrieval-augmented generation) means is helpful, since the lecture assumes both.
• Awareness of basic fine-tuning concepts (updating model weights vs. prompting) makes the tuning section easier to follow.

Check your understanding

1. What factors does the lecture cite for why enterprises can now build AI applications with far less data and fewer specialists than a few years ago?
2. Why are enterprises shifting from choosing a single "best" model to choosing a flexible multi-model platform?
3. What are the three main reasons given for why LLMs need to be combined with search or retrieval?
4. How does parameter-efficient fine-tuning (like LoRA) differ from full fine-tuning, and why do enterprises prefer it?
5. Why is function calling necessary even for a highly capable language model, and what two things must the model get right to use it successfully?