Explainers / How They Work / Perception & Manipulation

Perception & Manipulation 101

Pillar B: How They Work · Last updated Nov 7, 2025

Bottom Line

  • Robots use cameras, depth sensors, and tactile feedback to "see" and understand objects
  • Grasping requires detecting shape, predicting weight, and adjusting grip force in real time
  • Reflective surfaces, transparent objects, and cluttered scenes remain major challenges

Perception: How Robots "See"

Humanoid robots don't see like humans—they combine multiple sensor types to build a 3D understanding of the world.

RGB Cameras (Vision)

Standard color cameras (like your phone) mounted in the robot's head and hands. Used for object recognition, color detection, and reading text/labels.

Typical Setup

  • Head: 2–4 cameras for stereo vision (depth perception)
  • Hands: 1–2 wrist-mounted cameras for close-up grasping
  • Body: Optional torso camera for wide-angle scene understanding

Depth Sensors (3D Mapping)

Specialized sensors that measure distance to objects. Common types:

  • LiDAR: Laser-based distance measurement; accurate but expensive
  • Structured light: Projects patterns onto objects and reads distortion; fails in bright sunlight
  • Time-of-flight (ToF): Measures light travel time; good range but lower resolution
  • Stereo depth: Two cameras compare images to infer depth; software-based, cheaper but less accurate

Tactile Sensors (Touch)

Force sensors and pressure-sensitive "skin" in fingers and palms. Detect when the robot makes contact, how hard it's squeezing, and whether objects are slipping.

Why tactile matters: Vision alone can't tell if a glass is empty or full, fragile or durable. Touch feedback adjusts grip force in real time—critical for not crushing dishes or dropping delicate items.

Object Recognition: What the Robot Knows

Seeing objects is one thing—understanding what they are and how to manipulate them is another.

The AI Model Pipeline

  1. Detection: Where are objects in the scene? Bounding boxes around each item
  2. Classification: What is each object? "Mug," "plate," "remote control"
  3. Segmentation: Which pixels belong to which object? Precise boundaries for grasping
  4. Pose estimation: How is the object oriented? Is the mug handle on the left or right?

These models are trained on millions of images—but only work reliably on objects similar to the training data. Your custom coffee mug from a local artist? Might confuse the robot.

Common Perception Failures

  • Reflective surfaces: Mirrors, stainless steel, and glossy counters distort depth sensors
  • Transparent objects: Glass cups, plastic wrap, and clear containers are nearly invisible to depth sensors
  • Cluttered scenes: Overlapping objects confuse segmentation; robot can't tell where one item ends and another begins
  • Poor lighting: Too dark = cameras fail; too bright = glare blinds sensors
  • Novel objects: Anything not in training data gets misclassified or ignored

Manipulation: Grasping and Placing

Once the robot knows what an object is and where it is, it must plan and execute a grasp.

Grasp Planning

The robot's AI model predicts:

  • Grasp points: Where should fingers contact the object? Top, sides, handle?
  • Approach angle: Should the hand come from above, below, or the side?
  • Pre-grasp shape: How wide should fingers open before contact?
  • Force estimate: How much grip force is needed? Delicate = light; heavy = firm

Execution: The Grasp Sequence

Step-by-Step

  1. Move hand to pre-grasp position (fingers open, hovering above object)
  2. Lower hand slowly while monitoring force sensors
  3. Detect contact—fingers report "I'm touching something"
  4. Close fingers gradually, applying controlled force
  5. Lift slightly to test grip—if object slips, adjust force and retry
  6. Carry object to destination, maintaining stable grip
  7. Release by opening fingers, monitoring for controlled placement

Why Grasping Is Hard

Infinite variety: Every object has unique shape, weight, surface texture, and center of mass. Training data can't cover everything.

Dynamic forces: Carrying a mug of coffee is different from an empty mug—sloshing liquid changes weight distribution.

Delicate vs. durable: The robot must know to squeeze a tennis ball firmly but barely touch a raw egg.

Recovery from errors: If the object slips mid-grasp, can the robot adjust on the fly or must it start over?

The Role of Imitation Learning

Instead of programming every possible grasp, modern robots learn by watching humans (or teleoperators):

  • Data collection: Teleoperators complete thousands of grasps while the robot records camera feeds, joint angles, and force data
  • Model training: AI models learn patterns—"when I see this object from this angle, humans grasp here with this force"
  • Autonomous execution: Robot applies learned patterns to new objects that resemble training examples
  • Continuous improvement: Every teleop session adds more data, improving the model over time
1X's approach: NEO uses "Expert Mode" (teleoperation) to gather grasp data in real homes, then trains models to autonomously replicate those actions. The more homes NEO enters, the smarter it gets.

What This Means for You

Standardize when possible: The more your dishware, furniture, and clutter match common training data, the better the robot will perform. White ceramic plates > hand-painted artisan plates.

Good lighting matters: Install consistent lighting in areas where the robot works—avoid backlighting and heavy shadows.

Clear workspaces: Robots struggle with cluttered counters. Keep grasping zones tidy for higher success rates.

Expect failures early: Your home is unique. The robot will need time (and teleop assists) to learn your specific objects and layouts.

→ Learn how teleop assists handle perception/manipulation failures

What to Read Next