Understanding Generative AI

What is Generative AI?

Generative AI refers to artificial intelligence systems that generate new content.

Examples include generating:

• Text
• Images
• Audio
• Video

Unlike traditional machine learning models that focus on classification or prediction, generative AI creates new outputs.

Examples:

• Writing a social media caption
• Generating an image from a text prompt
• Creating synthetic speech
• Producing chatbot responses

Formally we can describe a generative model as learning a probability distribution:

$$P(x)$$

where (x) represents data such as text, images, or audio.

The model learns this distribution and generates new samples from it.

---

Why Generative AI Matters?

Generative AI is expected to have a significant economic impact.

• $2.6–$4.4 trillion added annually to the global economy
• Up to 7% increase in global GDP over the next decade
• Impact on 10% of tasks performed by 80% of workers

---

AI as a Set of Tools

Artificial Intelligence is not a single technique. It includes multiple learning approaches.

1. Supervised Learning

Supervised learning maps inputs to outputs.

$$A \rightarrow B$$

Examples:

Input	Output	Application
Email	Spam / Not Spam	Spam filtering
Image	Defect / No defect	Quality inspection
Audio	Text transcript	Speech recognition
X-ray image	Diagnosis	Healthcare

2. Unsupervised Learning

Unsupervised learning discovers patterns in data without labels.

Examples:

• clustering
• anomaly detection
• dimensionality reduction

3. Reinforcement Learning

Reinforcement learning trains systems through reward-based feedback.

Example applications:

• robotics
• game playing
• recommendation optimization

4. Generative AI

Generative AI focuses on creating new data rather than predicting labels.

---

Large Language Models (LLMs)

They are trained to predict the next word in a sequence.

For a sequence of tokens:

$$w_1, w_2, ..., w_n$$

the model learns:

$$P(w_{n+1} \mid w_1, w_2, ..., w_n)$$

By repeatedly predicting the next token, the model generates entire sentences and documents.

Example:

Large Language Models power systems like ChatGPT.

Input:

My favorite food is a bagel

Predictions:

My favorite food is a bagel with
My favorite food is a bagel with cream
My favorite food is a bagel with cream cheese

When trained on hundreds of billions of words, the model learns grammar, facts, reasoning patterns, and writing styles.

---

AI as a General Purpose Technology

Electricity powers many systems:

• lighting
• transportation
• manufacturing

Similarly, AI can power many applications:

• spam filtering
• fraud detection
• recommendation systems
• autonomous vehicles
• content generation

This is why AI is often called a general-purpose technology.

Applications of Generative AI

Generative AI is widely used for three main tasks:

1. Writing

Examples include:

• brainstorming product names
• writing marketing content
• drafting emails
• generating press releases
• translation

Example prompt:

Brainstorm five creative names for peanut butter cookies

Possible output:

• Nutty Nirvana Nibbles
• ButterBite Bliss
• Peanut Paddle Pucks

2. Reading

LLMs can analyze and process text.

Examples:

• proofreading documents
• summarizing articles
• analyzing customer emails
• sentiment classification

Example task:

Classify the sentiment of this review:

"The food was amazing and the servers were so friendly!"

Output:

Positive

This corresponds to estimating:

$$P(\text{sentiment} \mid \text{text})$$

---

3. Chatting

Chatbots can handle many conversational tasks.

Examples:

• customer service support
• IT help desks
• travel planning
• career coaching
• product assistance

A typical chatbot pipeline looks like this:

graph TD
A[User Message] --> B[Chatbot Interface]
B --> C[LLM Processing]
C --> D[Generated Response]
D --> E[User]

---

Limitations of Generative AI

Despite its capabilities, generative AI has several limitations.

1. Knowledge Cutoff

An LLM only knows information available during its training.

If trained in 2022, it may not know events after that date.

2. Hallucinations

LLMs sometimes generate confident but incorrect information.

Example:

Asking for Shakespeare quotes about Beyoncé may produce fake quotes that sound plausible.

3. Context Length Limits

LLMs cannot process unlimited text.

Let:

$$L = \text{maximum context length}$$

Then:

$$\text{Input tokens} + \text{Output tokens} \leq L$$

Modern models may support:

• a few thousand tokens
• up to 100k tokens in advanced systems

4. Weak Performance on Structured Data

LLMs work best with unstructured data, such as:

• text
• images
• audio
• video

For structured tables like:

Size	Price
523	100
645	150

traditional machine learning methods may perform better.

5. Bias and Safety

Because LLMs learn from internet data, they may reflect biases present in the data.

Examples include:

• gender stereotypes
• cultural bias
• toxic language

Modern AI systems include safeguards to reduce these issues.

---

Prompt Engineering Tips

Three key principles for effective prompting.

1. Be Specific

Bad prompt:

Write a press release

Better prompt:

Write a press release announcing the hire of our new COO.
Include the following details:
- company background
- executive biography
- industry context

2. Guide the Model Step by Step

Example prompt structure:

Step 1: Generate five fun cat-related words.
Step 2: Create rhyming toy names.
Step 3: Add an emoji to each name.

Breaking tasks into steps often improves results.

3. Experiment and Iterate

There is rarely a perfect prompt on the first try.

Typical workflow:

1. Write an initial prompt
2. Evaluate the response
3. Refine the prompt
4. Repeat

This iterative process is often called prompt engineering.

---

Final Thoughts

Generative AI represents a major shift in how software systems interact with information.

Instead of writing complex rules to process text, developers can now rely on models that understand language at scale.

Key takeaways:

• Generative AI creates new content rather than just predictions
• LLMs generate text by predicting the next token
• AI is a general-purpose technology with many applications
• Prompt engineering plays a critical role in using these models effectively
• Despite their power, LLMs have limitations such as hallucinations and knowledge cutoffs

Understanding these fundamentals is essential before building real-world AI applications.

In the next post, we will explore how companies use LLMs in production systems with real engineering examples.