Exploring Conditional Imitation Learning in the CARLA Simulator

At some point in autonomous driving, rules stop scaling.

You can detect objects.
You can estimate lanes.
You can predict motion.

But deciding what to do next—in complex, real-world traffic—quickly becomes brittle if everything is hand-coded.

That’s where imitation learning enters the picture.

In this project, I explored conditional imitation learning by training a car to drive in the CARLA simulator, learning directly from expert demonstrations rather than explicit rules.

🎥 Project demo:
https://www.youtube.com/watch?v=qaMf3aUWhWo

Why Imitation Learning?

Traditional autonomous driving pipelines look like this:

• perception
• planning
• control

Each module is engineered separately.

Imitation learning flips that idea.

Instead of asking:

How should a car behave in every possible scenario?

It asks:

Can a model learn driving behavior by watching humans drive?

That question fascinated me—because it mirrors how humans learn.

The Key Insight: Driving Depends on Intent

One of the biggest challenges in end-to-end driving is ambiguity.

At an intersection:

• Should the car go left?
• Right?
• Straight?

From camera input alone, the answer is unclear.

That’s where conditional imitation learning becomes powerful.

The model doesn’t just learn how to drive—it learns how to drive given an instruction.

Reference Paper That Guided the Work

This project was inspired by the paper:

End-to-End Driving via Conditional Imitation Learning
Felipe Codevilla, Matthias Müller, Antonio López, Vladlen Koltun, Alexey Dosovitskiy
ICRA 2018

The core idea:

• Learn driving from demonstrations
• Condition behavior on high-level commands like:
  • “turn left”
  • “turn right”
  • “go straight”
  • “follow lane”

This avoids a single ambiguous policy and instead learns multiple behaviors under one model.

CARLA as the Learning Environment

CARLA provided the perfect sandbox:

• realistic urban layouts
• traffic rules
• intersections
• dynamic actors

I used CARLA to:

• record expert driving trajectories
• collect camera images, speed, and control signals
• associate each trajectory with a navigation command
• train a neural network to predict steering and throttle

This wasn’t about beating benchmarks—it was about understanding behavioral learning.

What the Model Actually Learned

The trained model learned to:

• follow lanes smoothly
• slow down at intersections
• execute turns based on commands
• reach the destination without explicit planning logic

There was no hard-coded rule saying “turn left here”.

The behavior emerged from data.

Watching that happen was a turning point.

What Made This Hard

Imitation learning is deceptively simple.

The real challenges were:

• distribution shift (what happens when the car deviates?)
• compounding errors
• sensitivity to data quality
• balancing generalization vs memorization

This project taught me that:

Data quality matters more than model complexity.

A bad demonstration teaches bad behavior—perfectly.

Why This Project Mattered to Me

This work helped me understand something fundamental:

Learning behavior is different from programming behavior.

Imitation learning doesn’t replace classical autonomy—but it complements it.

It’s powerful when:

• rules are hard to specify
• environments are complex
• behavior is contextual

That insight shaped how I think about learning-based decision systems even today.

Closing Thought

In this project, I didn’t teach a car how to drive.

I let it learn by watching.

That shift—from rules to behavior—felt like crossing an invisible boundary in autonomy.

And once you cross it, you don’t think about control systems the same way again.